Skip to content

[OpenViSUS and idx2 integration] idx2 chunks being too small for cloud #8

@hoang-dt

Description

@hoang-dt

I have added an option to group multiple bit planes into a chunk (default is 4). To change it, provide, say --bit_planes_per_chunk 8, when encoding. With this change, the chunks are about 150KB on average.

The chunk size can also be changed with two options when encoding:

  • --bricks_per_chunk (default is 512). For example, --bricks_per_chunk 4096 should create chunks of size 1 MB ish.
  • --brick_size (default is 32 32 32). For example, --brick_size 64 64 64 should also create chunks of size 1 MB ish.

The first option (increasing the bricks-per-chunk) will result in more metadata to be stored, but is better in that the bricks are kept small so fewer samples need to be decoded when we ask for a small region of interest (e.g., a slice).

TODO

  • Can we plot the impact of the chunk size on data retrieval time?
  • Can we allow fetching parts of an object? (so that each chunk does not have to be stored as a file)

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions