q2netcdf

Note: The package requires Python 3.10+, but the standalone mergeqfiles script is compatible with Python 3.7+ for deployment on legacy systems (e.g., MicroRider instruments).

Python tools for manipulating Rockland Scientific's Q-files as generated by the ISDP data logger.

Features

• Q-file to NetCDF Conversion: Convert binary Q-files to CF-1.8 compliant NetCDF format
• Header Parsing: Extract metadata, sensor configurations, and timing information
• Data Extraction: Read scalar channels and frequency spectra from binary records
• File Manipulation: Merge, reduce, and inspect Q-files
• Configuration Generation: Create syntactically correct isdp.cfg files
• Format Support: Handles Q-file versions 1.2 and 1.3
• Sensor Mapping: Built-in support for 200+ sensor identifier codes

Command-Line Tools

• q2netcdf - Translate Q-files into CF-1.8 compliant NetCDF files
• QFile - Dump header and data records from Q-files
• QHeader - Display header record information
• QReduce - Reduce Q-file size by removing records
• mergeqfiles - Merge multiple Q-files (designed for MicroRider/Slocum Glider integration)
• mkISDPcfg - Generate syntactically correct isdp.cfg configuration files

Quick Start

# Display Q-file header information
QHeader data.q

# Convert Q-file to NetCDF
q2netcdf data.q --nc output.nc

# Dump all records from Q-file
QFile data.q --logLevel INFO

Installation

From PyPI

pip install q2netcdf

Or using pipx for the command-line tools:

pipx install q2netcdf

From Source

git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e .

For Development

git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest  # Run tests

Installation on the MicroRider

For deployment on Rockland Scientific MicroRider instruments:

1. Transfer src/q2netcdf/mergeqfiles.py and src/q2netcdf/mergeqfiles.cfg to the MicroRider's data directory.
2. Dismount the data directory on your host computer.
3. Connect to the MicroRider via a serial connection using your favorite terminal emulator at 115,200 baud.
4. ls data to verify the files are present. This will cause the MicroRider to unmount data from being read-only to the MicroRider and then remount it as read-write. This will typically cause the serial connection to drop then be reestablished. You might have to restart your terminal emulator's connection.
5. mv data/mergeqfiles.py /usr/local/bin/mergeqfiles to install the script as mergeqfiles.
6. Test with mergeqfiles --help to make sure the installation was successful.
7. You can disconnect the MicroRider's serial connection.
8. You should now run a simulation on the Slocum glider to verify Q-files are being uploaded to the science computer's storage as .mri files.
9. Diagnostic information is written into the data/mergeqfiles.log file on the MicroRider.
10. You can modify data/mergeqfiles.cfg to adjust the information included in the .mri file sent to the glider.

This is designed for integration with TWR's Slocum Glider uRider proglet.

Python API

from q2netcdf import QFile

# Stream records from a Q-file
with QFile("data.q") as qf:
    header = qf.header()
    print(header.channels(), header.spectra())
    for record in qf.data():
        print(record.stime, record.channels)

# Convert a Q-file to an xarray Dataset
from q2netcdf.q2netcdf import loadQfile

ds = loadQfile("data.q")
print(ds)
ds.to_netcdf("output.nc")

Usage Examples

Converting Q-files to NetCDF

q2netcdf data.q --nc output.nc

Inspecting Q-file Headers

# Show full header with channel and spectra names
QHeader data.q

# Show condensed header
QHeader data.q --nothing

# Show only configuration
QHeader data.q --channels --spectra --frequencies

Examining Q-file Data Records

# Dump all records
QFile data.q

# Limit number of records displayed
QFile data.q --n 10

Merging Q-files

mergeqfiles -o merged.q file1.q file2.q file3.q

Generating ISDP Configuration Files

mkISDPcfg --help

File Format

Q-files are binary files containing oceanographic data from Rockland Scientific instruments. The format is documented in Rockland Technical Note TN-054.

Q-file Structure

• Header Record (0x1729): File version, timestamp, channel/spectra identifiers, frequencies, configuration
• Data Records (0x1657): Timestamped measurements with scalar channels and frequency spectra

Supported Versions

• v1.2: Original format with full metadata in each data record
• v1.3: Optimized format with reduced redundancy

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Development Setup

git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest                    # Run tests
ruff check src/ tests/    # Lint
mypy src/                 # Type check

To run a module directly: python3 -m q2netcdf.QHeader --help

Additional Documentation

• CHANGELOG.md - Version history and changes
• CONTRIBUTING.md - Contributor guidelines and development setup
• SECURITY.md - Security policy and vulnerability reporting
• QFILE_FORMAT.md - Q-file binary format specification

License

This project is licensed under the GNU General Public License v3.0 or later.

Author

Pat Welch Email: pat@mousebrains.com

Acknowledgments

Developed for use with Rockland Scientific oceanographic instruments and the ISDP data logger.