How to use an ArcArray across threads?

[multimeric]

I have a large matrix as an Array2, and I want to do some processing on it from multiple threads. Normally I would wrap it in a Arc::new(shared_object);. However I suppose that the ArcArray2 already has an Arc somewhere, so instead I matrix.into_shared(), and then I clone() the ArcArray2, and move the entire copy into each thread. Approximately the code looks like this:

use ndarray::{Array2, ArcArray2};
use std::thread;

fn process_matrix(mat: Array2<f64>){
    let matrix: ArcArray2<f64> = data.into_shared();
    for _ in 0..10 {
        matrix = matrix.clone();
        let handle = thread::spawn(move || {
            matrix.sum();
            // Placeholder for processing
        });
    }
}

I just want to check. Is this the way that ArcArray is intended to be used? I don't need any other code to handle the multithreaded access? Is it correct to move the entire ArcArray into each thread, or should I be borrowing them? Although there's documentation on the ArcArray class, it never really explains when or how to use it, and I would appreciate some guidance here.

[jturner314]

Yes, that's correct. The data of an ArcArray is wrapped in an Arc. It's similar to Arc<Array<A, D>>, except that:

• The shape/strides/pointer are uniquely owned by the ArcArray, not wrapped in the Arc. Only the data is wrapped in the Arc. As a result, for example, calling .slice_collapse() on an ArcArray modifies its shape/strides/pointer in-place without cloning the data. In contrast, if you had an Arc<Array<A, D>> and wanted to call .slice_collapse(), you'd have to call .make_mut() (which would clone the wrapped array, including its data) to get a mutable reference in order to modify the shape/strides/pointer.

• When modifying the data in the ArcArray (or getting a mutable reference to data in the ArcArray, such as with .view_mut()), the data is automatically cloned if it's not uniquely held. In contrast, if you had an Arc<Array<A, D>>, you'd need to explicitly call .make_mut().

Somewhat related -- If you're not already familiar with it, you may be interested in the rayon crate, which is simpler in many cases than using std::thread. For example, unlike std::thread::spawn, the ParallelIterator::for_each method does not have a 'static bound on the closure, so you can borrow things from the local stack frame instead of creating Arcs:

use ndarray::Array2;
use rayon::prelude::*;

fn process_matrix2(mat: Array2<f64>) {
    (0..10).into_par_iter().for_each(|_| {
        mat.sum();
        // Placeholder for processing
    });
}

[multimeric]

Very helpful answer, thank you! Do you mind if I (or you) copy this into a PR that describes the ArcArray? It's exactly what I would have wanted as a new user.

Regarding your rayon example, does this replace the need for ArcArray? What is happening to the array in this case? How is it being transferred to/accessed from each thread?