q2netcdf

Note: The package requires Python 3.10+, but the standalone mergeqfiles script is compatible with Python 3.7+ for deployment on legacy systems (e.g., MicroRider instruments).

Python tools for manipulating Rockland Scientific's Q-files as generated by the ISDP data logger.

Features

• Q-file to NetCDF Conversion: Convert binary Q-files to CF-1.8 compliant NetCDF format
• Header Parsing: Extract metadata, sensor configurations, and timing information
• Data Extraction: Read scalar channels and frequency spectra from binary records
• File Manipulation: Merge, reduce, and inspect Q-files
• Configuration Generation: Create syntactically correct isdp.cfg files
• Format Support: Handles Q-file versions 1.2 and 1.3
• Sensor Mapping: Built-in support for 200+ sensor identifier codes

Command-Line Tools

• q2netcdf - Translate Q-files into CF-1.8 compliant NetCDF files
• QFile - Dump header and data records from Q-files
• QHeader - Display header record information
• QReduce - Reduce Q-file size by removing records
• mergeqfiles - Merge multiple Q-files (designed for MicroRider/Slocum Glider integration)
• mkISDPcfg - Generate syntactically correct isdp.cfg configuration files

Quick Start

# Display Q-file header information
QHeader data.q

# Convert Q-file to NetCDF
q2netcdf data.q --nc output.nc

# Dump all records from Q-file
QFile data.q --logLevel INFO

Installation

Using pipx (Recommended)

The command-line scripts in this package may be installed using pipx:

python3 -m pipx install git+https://github.com/mousebrains/q2netcdf

To update to the latest version:

python3 -m pipx install --force git+https://github.com/mousebrains/q2netcdf

From Source

git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e .

For Development

git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest  # Run tests

Installation on the MicroRider

For deployment on Rockland Scientific MicroRider instruments:

1. Transfer src/q2netcdf/mergeqfiles.py and src/q2netcdf/mergeqfiles.cfg to the MicroRider's data directory.
2. Dismount the data directory on your host computer.
3. Connect to the MicroRider via a serial connection using your favorite terminal emulator at 115,200 baud.
4. ls data to verify the files are present. This will cause the MicroRider to unmount data from being read-only to the MicroRider and then remount it as read-write. This will typically cause the serial connection to drop then be reestablished. You might have to restart your terminal emulator's connection.
5. mv data/mergeqfiles.py /usr/local/bin/mergeqfiles to install the script as mergeqfiles.
6. Test with mergeqfiles --help to make sure the installation was successful.
7. You can disconnect the MicroRider's serial connection.
8. You should now run a simulation on the Slocum glider to verify Q-files are being uploaded to the science computer's storage as .mri files.
9. Diagnostic information is written into the data/mergeqfiles.log file on the MicroRider.
10. You can modify data/mergeqfiles.cfg to adjust the information included in the .mri file sent to the glider.

This is designed for integration with TWR's Slocum Glider uRider proglet.

Usage Examples

Converting Q-files to NetCDF

q2netcdf data.q --nc output.nc

Inspecting Q-file Headers

# Show full header with channel and spectra names
QHeader data.q

# Show condensed header
QHeader data.q --nothing

# Show only configuration
QHeader data.q --channels --spectra --frequencies

Examining Q-file Data Records

# Dump all records
QFile data.q

# Limit number of records displayed
QFile data.q --n 10

Merging Q-files

mergeqfiles -o merged.q file1.q file2.q file3.q

Generating ISDP Configuration Files

mkISDPcfg --help

File Format

Q-files are binary files containing oceanographic data from Rockland Scientific instruments. The format is documented in Rockland Technical Note TN-054.

Q-file Structure

• Header Record (0x1729): File version, timestamp, channel/spectra identifiers, frequencies, configuration
• Data Records (0x1657): Timestamped measurements with scalar channels and frequency spectra

Supported Versions

• v1.2: Original format with full metadata in each data record
• v1.3: Optimized format with reduced redundancy

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Development Setup

git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest                    # Run tests
ruff check src/ tests/    # Lint
mypy src/                 # Type check

To run a module for testing python3 -m src.q2netcdf.QHeader --help

Additional Documentation

• CHANGELOG.md - Version history and changes
• CONTRIBUTING.md - Contributor guidelines and development setup
• SECURITY.md - Security policy and vulnerability reporting
• IMPROVEMENTS_APPLIED.md - Recent improvements and updates
• CLAUDE.md - AI assistant guidance for working with this codebase
• Development Documentation - Internal development notes and analysis

License

See LICENSE file for details.

Author

Pat Welch Email: pat@mousebrains.com

Acknowledgments

Developed for use with Rockland Scientific oceanographic instruments and the ISDP data logger.