Skip to content

parallelize along user specified dimension #11

@valiantljk

Description

@valiantljk

Currently, H5Spark parallelize the IO along the slowest dimension, i.e., the dimension that changes slowest on disks. For example, for a 2D C array x[10][200], the h5spark will choose the first dimension to partition, and then the maximum partition it can have is only 10, which is also the maximum degree of parallelism.

If we want to parallelize along any user-specified dimension, the current code needs a little bit modification.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions