MHD_64 file

[1230lw]

Hello, @mikemccabe210 @LTMeyer @MilesCranmer @kazewong

When downloading the MHD_64 dataset, I noticed something unusual in the train/ folder. Most of the files are about 5.87 GB in size, but there is one file, MHD_Ma_0.7_Ms_0.5.hdf5, which is 7.97 GB. Additionally, while the other files each contain 8 trajectories, this MHD_Ma_0.7_Ms_0.5.hdf5 file seems to contain only 5 trajectories.

Is this normal? Because altogether, the number of trajectories doesn’t add up to the 100 trajectories described in the paper.

Thank you for your help!

============================================================
/data/train/MHD_Ma_2_Ms_0.5.hdf5
============================================================

Group: boundary_conditions

Group: boundary_conditions/x_periodic
  Attribute: associated_dims = ['x']
  Attribute: associated_fields = []
  Attribute: bc_type = PERIODIC
  Attribute: sample_varying = False
  Attribute: time_varying = False

  Dataset: boundary_conditions/x_periodic/mask
    Shape: (64,)
    Dtype: bool

Group: boundary_conditions/y_periodic
  Attribute: associated_dims = ['y']
  Attribute: associated_fields = []
  Attribute: bc_type = PERIODIC
  Attribute: sample_varying = False
  Attribute: time_varying = False

  Dataset: boundary_conditions/y_periodic/mask
    Shape: (64,)
    Dtype: bool

Group: boundary_conditions/z_periodic
  Attribute: associated_dims = ['z']
  Attribute: associated_fields = []
  Attribute: bc_type = PERIODIC
  Attribute: sample_varying = False
  Attribute: time_varying = False

  Dataset: boundary_conditions/z_periodic/mask
    Shape: (64,)
    Dtype: bool

Group: dimensions
  Attribute: spatial_dims = ['x' 'y' 'z']

  Dataset: dimensions/time
    Shape: (100,)
    Dtype: float32

  Dataset: dimensions/x
    Shape: (64,)
    Dtype: float64

  Dataset: dimensions/y
    Shape: (64,)
    Dtype: float64

  Dataset: dimensions/z
    Shape: (64,)
    Dtype: float64

Group: scalars
  Attribute: field_names = ['Ma' 'Ms']

  Dataset: scalars/Ma
    Shape: ()
    Dtype: float32

  Dataset: scalars/Ms
    Shape: ()
    Dtype: float32

Group: t0_fields
  Attribute: field_names = ['density']

  Dataset: t0_fields/density
    Shape: (8, 100, 64, 64, 64)
    Dtype: float32

Group: t1_fields
  Attribute: field_names = ['magnetic_field' 'velocity']

  Dataset: t1_fields/magnetic_field
    Shape: (8, 100, 64, 64, 64, 3)
    Dtype: float32

  Dataset: t1_fields/velocity
    Shape: (8, 100, 64, 64, 64, 3)
    Dtype: float32

Group: t2_fields
  Attribute: field_names = []



============================================================
/data/train/MHD_Ma_0.7_Ms_0.5.hdf5
============================================================

Group: boundary_conditions

Group: boundary_conditions/x_periodic
  Attribute: associated_dims = ['x']
  Attribute: associated_fields = []
  Attribute: bc_type = PERIODIC
  Attribute: sample_varying = False
  Attribute: time_varying = False

  Dataset: boundary_conditions/x_periodic/mask
    Shape: (64,)
    Dtype: bool

Group: boundary_conditions/y_periodic
  Attribute: associated_dims = ['y']
  Attribute: associated_fields = []
  Attribute: bc_type = PERIODIC
  Attribute: sample_varying = False
  Attribute: time_varying = False

  Dataset: boundary_conditions/y_periodic/mask
    Shape: (64,)
    Dtype: bool

Group: boundary_conditions/z_periodic
  Attribute: associated_dims = ['z']
  Attribute: associated_fields = []
  Attribute: bc_type = PERIODIC
  Attribute: sample_varying = False
  Attribute: time_varying = False

  Dataset: boundary_conditions/z_periodic/mask
    Shape: (64,)
    Dtype: bool

Group: dimensions
  Attribute: spatial_dims = ['x' 'y' 'z']

  Dataset: dimensions/time
    Shape: (100,)
    Dtype: float32

  Dataset: dimensions/x
    Shape: (64,)
    Dtype: float64

  Dataset: dimensions/y
    Shape: (64,)
    Dtype: float64

  Dataset: dimensions/z
    Shape: (64,)
    Dtype: float64

Group: scalars
  Attribute: field_names = ['Ma' 'Ms']

  Dataset: scalars/Ma
    Shape: ()
    Dtype: float32

  Dataset: scalars/Ms
    Shape: ()
    Dtype: float32

Group: t0_fields
  Attribute: field_names = ['density']

  Dataset: t0_fields/density
    Shape: (5, 100, 64, 64, 64)
    Dtype: float32

Group: t1_fields
  Attribute: field_names = ['magnetic_field' 'velocity']

  Dataset: t1_fields/magnetic_field
    Shape: (5, 100, 64, 64, 64, 3)
    Dtype: float32

  Dataset: t1_fields/velocity
    Shape: (5, 100, 64, 64, 64, 3)
    Dtype: float32

Group: t2_fields
  Attribute: field_names = []

[mikemccabe210]

If I remember correctly, there were a few MHD instances that we found issues with during development so the files not uniformly split. In general, while it's usually the case, there's no actual specification requirement for files to contain the same number of trajectories.

[1230lw]

Thank you @mikemccabe210 very much for your explanation. I understand that the files may not be uniformly split and that there is no strict requirement for each file to contain the same number of trajectories.

I appreciate your clarification. I will proceed accordingly, and if I encounter any further questions or issues, I will be sure to reach out.

Thanks again for your support!