THz imaging data format

[hacknus]

Hello
I am currently working with THz imaging data and I want to try to implement the dotTHz file format in our lab.
However, I am unsure on how to structure it. I'm mostly working with Python and Rust.
So far, I am using a .npz file for the data, where I have one "time" array and one 3-dimensional data structure with dimensions: x * y * len(pulse). Additionally, there is a meta.json file that contains all the meta data (coordinates, ambient conditions, sample content etc).

I'd be glad for a tip on how to include the metadata alongside the 3D structure in a .thz file.
Thanks!

[hacknus]

I created a snipped, but my usage of ds1 and ds2 is not the same as for normal measurements, since I use ds1 as time and ds2 as the dataframe
The coordinates part is still a mystery to me...

import h5py
import numpy as np

if __name__ == "__main__":
    # Sample data
    time = np.linspace(0, 1, 100)  # your time array
    data = np.random.rand(50, 50, 100)  # example 3D data array
    x_coords, y_coords = np.linspace(0, 1, 50), np.linspace(0, 1, 50)

    # Create and save to HDF5 file
    with h5py.File('imaging_data.thz', 'w') as file:

        f = file.create_group("Measurement 1")

        # Save main datasets
        # Better to only save the time array once, than for every pixel, right?
        f.create_dataset('ds1', data=time)
        f.create_dataset('ds2', data=data)

        # Save metadata
        f.attrs['dsDescription'] = ["ds1:time", "ds2:image"]

        f.attrs['date'] = '2023-01-26'
        f.attrs['description'] = 'sample material name or description'
        f.attrs['instrument'] = 'TeraFlash Pro / Toptica'
        f.attrs['md1'] = 1  # thickness in mm
        f.attrs['md2'] = 298.15  # temperature

        # how do we save coordinates?
        # f.attrs['coordinates'] = ???

        # might seem better to just save x_min, x_max and dx (vice versa for y) instead of a full array?
        f.attrs['md3'] = x_coords[0]
        f.attrs['md4'] = x_coords[-1]
        f.attrs['md5'] = x_coords[1] - x_coords[0]
        f.attrs['md6'] = y_coords[0]
        f.attrs['md7'] = y_coords[-1]
        f.attrs['md8'] = y_coords[1] - y_coords[0]

        f.attrs['mdDescription'] = ["Thickness (mm)", "Temperature (K)", "x_min (mm)", "x_max (mm)", "dx (mm)",
                                    "y_min (mm)", "y_max (mm)", "dy (mm)"]
        f.attrs['mode'] = 'THz-TDS/Transmission'
        f.attrs["thzVer"] = "1.00"
        f.attrs["time"] = "12: 29:12"
        f.attrs["user"] = "0000-0000-1111-2222 / J Doe / J.Doe @ thz.ac.uk / University  of Terawick, UK"

    # Open the HDF5 file in read mode
    with h5py.File('imaging_data.thz', 'r') as f:
        # Load the datasets
        print(f.keys())
        print(f.attrs.keys())

[hacknus]

Okay, so after doing some investigations I want to clarify my issue:

in thzVer 1.00, for a single measurement (dataset) all the time-trace points are saved as tuples of length 2 (time, intensity), shape (2,(len(t)) and the coordinates are saved as attributes.

For imaging data, this will blow up the file size incredibly, since all time values will be duplicated for every single pixel, even though they are the same (no one will change the time window throughout an imaging scan).

Proposition:

I propose for imaging datasets to save the time trace as dataset with shape (len(time)) "ds1" and the pulse data structure as "ds2" with shape (nx,ny,len(t)).
All the metadata attributes will remain the same, the coordinates could be saved as md entries or maybe still in the coordinates tag but in a shape like this:

[[x_min,x_max,dx],[y_min,y_max,dy]]

possible breaking changes:

This will however lead to incompatibilities with single-pixel measurements, because the shapes are different. Not sure if it would make sense to change (for example for thzVer 1.10) the shape for single pixel measurements from (2,(len(t)) to (len(t),2)?

Maybe there is a better option for the coordinates.

Happy to hear about your feedback!

[hacknus]

is this repo still being maintained?

[jl2112tag]

Hi Hacknus,

My apologies for the delayed response. Your analysis regarding the possible redundant time array structure is correct. However, each data group (column) can be managed separately and retrieved individually from its application stage. Given this, you may allocate the first data group fro general information about the measurement and time array. Subsequently, all spatial measurement data can be incorporated starting from the second data group, along with coordinate vectors, which can be assigned to item 08 or any available metadata slot.

1. Column 1: DS1 - Time array
2. Column 2 onward: DS1 - E-field data
3. Set [Attribute inclusion] only for the first data group, except for item [08] or a metadata slot [10-16]

Thank you for your support and participation.

Best regards,
Jongmin

[hacknus]

Thanks for the response!

Are you saying something like this:

File --- group 1 --- DS1 time array, meta data
   | --- group 2 --- DS1 e-field data at (0,0)
   | --- group 3 --- DS1 e-field data at (1,0)
                    ...

or like this:

File --- group 1 --- DS1 time array, meta data
                 | - DS2 e-field data at (0,0)
                 | - DS3 e-field data at (1,0)
                 | - DS4 e-field data at (2,0)
                           ...

However, this way there is only a 1D array stored for each DS, whereas in your example files it's always a 2D array.

What I have now is this:

File --- group 1 --- 3D cube structure (x * y * len(time)), meta data

Do you happen to have a dotTHz image scan as an example for my tests?

Thanks a lot!

[hacknus]

another question: are only 7 other metadata slots available or can these be extended indefinitely?

[jl2112tag]

No worries! The dataset doesn't necessarily have to be a 2D array-it can be 1D or n-D, depending on how you design it for maximum compatibility. This flexibility is essentially the core concept of the dotTHz project. Yes, I've also worked with 3-dimensional datasets and temporal measurement datasets as well to observe changes over time.

Per metadata slots, yes, but each metadata can contain an unlimited number of indices in vector form. Beyond that, the number of metadata is essentially limited by the available space in the CaTx user interface. Theoretically, however, the format supports an unlimited number of metadata.

Cheers!
Jongmin

[jl2112tag]

I just noticed that the latest update within the [Deployment Recipe] tab requires a time column as essential. I'll add an option for your case shortly (by this weekend).

[hacknus]

I see, thanks!
I'm working on a THz image explorer, for that it would be nice if the standard for images would be defined. But in that case I will probably stick to my 3D cube for space efficiency.

thanks! might be worth to extend the metadata for more than just 7 slots, since saving lab conditions and sample condition might need more parameters

[hacknus]

unfortunately I do not have access to matlab, so I'm limited to the documentation and example files to write my python code.

[jl2112tag]

No worries, you can download a compiler version of CaTx along with other tools and install them for trial use.

[jl2112tag]

Would you be able to share your image data sample? with a description regarding the measurement environment and the system used. I will see how it can be implemented into CaTx?

[hacknus]

Yes sure, here:
https://drive.google.com/file/d/14yrhFhqfQ53c8xr8SW3Itii-dc7SDnO1/view?usp=share_link

there is a txt file to describe the metadata. let me know how that looks on your end.

Okay, I'll have to check if I manage to install CaTx on my mac.

[jl2112tag]

Thank you for sharing the data and the text file. They are very useful for analysing the datasets. The good news is that your .thz file is readable using CaTx on my end. I'll get back to you, likely early next week, with the analysis results and a recommended direction for efficient and versatile data management. Cheers.