v2.2.1

Contributors to this version: Ludwig Lierhammer (@ludwiglierhammer)

Bug fixes

• cdm_reader_mapper.cdm_mapper: set indexes to input data indexed when setting default values (PR/356).

v2.2.0

Contributors to this version: Ludwig Lierhammer (@ludwiglierhammer) and Joseph Siddons (@jtsiddons)

Announcements

This release adds support for Python 3.14 (PR/339).

New features and enhancements

• new parameter in function map_model (PR/327).

  • drop_duplicates: If True remove duplicated rows (default: True).
  • drop_missing_obs: If True remove observation rows without a valid observation_value (default: True).

• new Pub47 testdata (test_data["test_pub47"]) (PR/327)

Breaking changes

• cdm_reader_mapper.cdm_mapper: rename map_and_convert to helper function _map_and_convert (PR/343)
• replace logging.error with raise error statements (PR/345)

Internal changes

• implement map_model test for Pub47 data (GH/310, PR/327)

• rename test data class from test_data to TestData (PR/327)

• update .gitignore (PR/324)

• update and add docstrings for multiple functions (PR/324)

• cdm_reader_mapper.cdm_mapper: update mapping functions for more readability (PR/324)

• cdm_reader_mapper.cdm_mapper: introduce some helper functions (PR/324)

• add more unit tests (GH/311, PR/324)

• cdm_reader_mapper.cdm_mapper: split map_and_convert into multiple helper functions (GH/333, PR/343)

• exclude tests/*.py from pre-commit codespell hook (PR/345)

• replace many os functions with pathlib.Path (PR/345)

• re-work mdf_reader (GH/334, PR/345)

  • remove reader.MDFFileReader class
  • remove utils.configurator module
  • remove both utils.decoder and mdf_reader.utils.converter modules
  • introduce utils.parser module: bunch of functions to parse input data into MDF data
  • introduce utils.convert_and_decode: make converter and decoder functions more modular
  • make utils.validator module more modular
  • utils.filereader.FileReader uses utils.parser function for parsing
  • move many helper function to utils.utilities
  • serialize schemas.schemas module

• add type hints and docstrings to mdf_reader (PR/345)

• add unit tests for mdf_reader module to testing suite (PR/345)
  Bug fixes

---

• add Pub47 mapping code tables (observing_frequency and vessel_type) (GH/308, PR/327)
• observation tables are not empty anymore after mapping Pub47 raw data to the CDM (GH/309,PR/327)

v2.1.1

Contributors to this version: Ludwig Lierhammer (@ludwiglierhammer), Joseph Siddons (@jtsiddons) and Jan Marius Willruth (@JanWillruth)

New features and enhancements

• add encoding optional argument to cdm_reader_mapper.read_mdf and cdm_reader_mapper.read_data which overrides default value set by model schema if set (GH/268, PR/273).
• cdm_reader_mapper.mdf_reader: Added preprocessing function to convert air pressure (PPPP) in IMMT format (PR/287)
• cdm_reader_mapper.cdm_mapper: Added mapping functions for IMMT datetime, latitude, and longitude conversions (PR/287)
• cdm_reader_mapper.cdm_mapper: New mapping function datetime_imma_d701 for icoads_r300_d701 (GH/288, PR/295)
• cdm_reader_mapper.cdm_mapper: New mapping function datetime_imma1_to_utc for mapping local midday to UTC (GH/288, PR/295)

License and Legal

Updated copyright statements in LICENSE (GH/271, PR/272).

Breaking changes

• cdm_reader_mapper: Replace "gcc" with "gdac" (PR/287)
• cdm_reader_mapper: Update gdac schemas to adhere to IMMT-5 documentation (PR/287)
• cdm_reader_mapper: combine icoads_r300_d701_type1 and icoads_r300_d701_type1 test and result data to icoads_r300_d701 (GH/288, PR/295)
• cdm_reader_mapper.cdm_mapper: combine icoads_r300_d701_type1 and icoads_r300_d701_type1 mapping tables to icoads_r300_d701 (GH/288, PR/295)
• cdm_reader_mapper.read: Allow strings as input for cdm_subset (PR/281)
• cdm_reader_mapper.cdm_mapper: Remove timestamps and/or previous history information in column history (PR/281)
• cdm_reader_mapper.DataBundle: Set empty pd.DataFrames as defaults for both data and mask (PR/281)
• cdm_reader_mapper.mdf_reader: read drifter numbers as strings not as integers with C-RAID (PR/281)

Internal changes

• tests: create test data result hidden directory (PR/291)
• ```cdm_reader_mapper.mdf_reader``: update and tidy-up ICOADS mapping tables (PR/281)
• timezonefinde is pinned below v7.0.0 (PR/281)

Bug fixes

• cdm_reader_mapper.write_data: fix doubling of output file name (PR/273)
• cdm_reader_mapper.cdm_mapper.mapping_functions: datetime conversion now ignores unformatable dates (GH/277, PR/278)
• README: fixing hyperlink (GH/279, PR/280)
• tests: raise OSError on checksum mismatch (PR/291)

v2.1.0

Contributors to this version: Ludwig Lierhammer (@ludwiglierhammer) and Joseph Siddons (@jtsiddons)

New features and enhancements

• implement both wrapper functions read and write that call the appropriate function based on mode argument (PR/238):

  • mode == "mdf"; calls cdm_reader_mapper.read_mdf
  • mode == "data"; calls cdm_reader_mapper.read_data or cdm_reader_mapper.write_data
  • mode == "tables"; calls cdm_reader_mapper.read_tables or cdm_reader_mapper.write_tables

• optionally, call cdm_reader_mapper.read_tables with either source file or source directory path (PR/238).

• apply attribute to DataBundle.data if attribute is nor defined in DataBundle (PR/248).

• apply pandas functions directly to DataBundle.data by calling DataBundle.<pandas-func> (PR/248).

• make DataBundle support item assignment for DataBundle.data (PR/248).

• optionally, apply selections to DataBundle.mask in DataBundle.select_* functions (PR/248).

• cdm_reader.reader.read_tables: optionally, set null_label (PR/242)

• new method function: DataBundle.select_where_all_false (PR/242)

• new method functions: DataBundle.split_* which split a DataBundle into two new DataBundles containing data selected and rejected after user-defined selection criteria (PR/242)

  • DataBundle.split_by_boolean_true
  • DataBundle.split_by_boolean_false
  • DataBundle.split_by_column_entries
  • DataBundle.split_by_index

• implement pandas indexer like iloc for not chunked data (PR/242)

Internal changes

• cdm_reader_mapper.common.select: restructure, simplify and summarize functions (PR/242)
• split DataBundle class into main class (cdm_reader_mapper.core._utilities) and method function class (cdm_reader_mapper.core.databundle) (PR/242)

Breaking changes

• remove property tables from DataBundle object. Instead, DataBundle.map_model overwrites .DataBundle.data (PR/238).

• set default overwrite values from True to False that is consistent with pandas inplace argument and rename overwrite to inplace (PR/238, PR/248).

• inplace returns None that is consistent with pandas (PR/242)

• DataBundle method functions return a DataBundle instead of a pandas.DataFrame (PR/248).

• DataBundle.select_* functions write only selected entries to DataBundle.data and do not take other list entries from common.select_* function returns into account (PR/248).

• select functions do not reset indexes by default (PR/242)

• rename DataBundle.select_* functions:

  • DataBundle.select_true -> DataBundle.select_where_all_boolean
  • DataBundle.select_from_list -> DataBundle.select_where_entry_isin
  • DataBundle.select_from_index -> DataBundle.select_where_index_isin

• rename cdm_reader_mapper.common.select_* functions and make them returning a tuple of selected and rejected data after user-defined selection criteria (PR/242):

  • select_true -> split_by_boolean_true
  • select_from_list -> split_by_column_entries
  • select_from_index -> spit_by_index

Bug fixes

• cdm_reder_mapper.metmetpy: set deck keys from ??? to d??? in icoads json files which makes values accessible again (PR/238).
• cdm_reder_mapper.metmetpy: set imma1 to icoads and immt to gcc in icoads/gcc json files which makes properties accessible again (PR/238).
• DataBundle.copy function now makes a real deepcopy of DataBundle object (PR/248).
• correct key index->section for self.df.attrs in open_netcdf (PR/252)
• cdm_reader_mapper.map_model: return null_label if conversion fails (PR/242)
• keep indexes during duplicate check (PR/242)

v2.0.1

Contributors to this version: Ludwig Lierhammer (@ludwiglierhammer) and Joseph Siddons (@jtsiddons)

Announcements

This release drops support for Python 3.9 and adds support for Python 3.13 (PR/228, PR/229)

New features and enhancements

• add environment.yml file (PR/229)

• cdm_reader_mapper now separates the optional dependencies into dev and docs recipes (PR/232).

  • $ python -m pip install cdm_reader_mapper # Install minimum dependency version
  • $ python -m pip install cdm_reader_mapper[dev] # Install optional development dependencies in addition
  • $ python -m pip install cdm_reader_mapper[docs] # Install optional dependencies for the documentation in addition
  • $ python -m pip install cdm_reader_mapper[all] # Install all the above for complete dependency version

Internal changes

• GitHub workflow for testing_suite now uses uv for environment management, replacing micromamba (PR/228)
• rename ci/requirements to CI and tidy up requirements/dependencies (PR/229)

v2.0.0

Contributors to this version: Ludwig Lierhammer (@ludwiglierhammer) and Joseph Siddons (@jtsiddons)

New features and enhancements

• New core DataBundle object including callable cdm_mapper, metmemtpy and operations methods (#84, #188, #197)
• Update readthedocs documentation (#191, #197)
• new function: write_data to write MDF data and validation mask according to write_tables for writing CDM tables (#201)
• new function: read_data to read MDF data and validation mask according to read_tables for reading CDM tables (#201)
• new property: DataBundle.encoding (#222)
• add overwrite option to some DataBundel method functions (#224)

Breaking changes

• cdm_mapper: map_model returns pandas.DataFrame instead of CDM dictionary (#189)
• cdm_mapper: rename function cdm_to_ascii to write_tables (#182, #185)
• cdm_mapper: update parameter names and list of functions read_tables and write_tables (#185)
• main cdm_mapper, mdf_reader and duplicates modules are directly callable from cdm_reader_mapper (#188)
• new list of imported submodules: [map_model, cdm_tables, read_tables, write_tables, duplicate_check and read_mdf] (#188)
• removed list of imported submodules: [cdm_mapper, common, mdf_reader, metmetpy, operations] (#188)
• remove imported submodules from cdm_mapper, mdf_reader (#188)
• read_tables: returning DataBundle object (#188)
• read_tables: resulting dataframe always includes multi-indexed columns (#188)
• duplicates is now a direct submodule of cdm_reader_mapper (#188)
• import read function from mdf_reader.read as read_mdf (#188)
• read_mdf: returning DataBundle object (#188)
• read_mdf: remove parameter out_path to dump attribute information on disk (#201)
• move function open_code_table from common.json_dict to cdm_mapper.codes.codes (#221)
• operations to common (#224)
• cdm_mapper: rename table_writer to writer and table_reader to reader (#224)
• mdf_reader: rename write to writer and read to reader (#224)
• metmetpy: gather correction functions to correct module and validation functions to validate module (#224)
• DataBundle: remove properties selected, deselected, tables_dup_flagged and tables_dups_removed (#224)

Internal changes

• cdm_mapper: dtype conversion from write_tables to new submodule _conversions of map_model (#189)

• cdm_mapper: rename mappings to _mapping_functions (#189)

• cdm_mapper: mapping functions from mapper to new submodule _mappings (#189)

• cdm_mapper: save utility functions from table_reader.py and table_writer.py to _utilities.py (#185)

• reduce complexity of several functions (#25, #200):

  • mdf_reader.read.read
  • mdf_reader.validate.validate
  • mfd_reader.utils.decoders.signed_overpunch
  • cdm_mapper._mappings._mapping
  • metmetmpy.station_id.validate.validate

• split mdf_reader.utils.auxiliary into mdf_reader.utils.filereader, mdf_reader.utils.configurator and mdf_reader.utils.utilities (#25, #200)

• simplify cdm_mapper.read_tables function (#192)

• mdf_reader: Refactored Configurator class, Configurator.open_pandas method, to handle looping through rows (#208, #210)

• mdf_reader: Refactored Configurator class, Configurator.open_data method, to avoid creating a pre-validation missing_value mask (#216)

• mdf_reader: move validate to utils.validators (#216)

• mdf_reader: no need for multi-column key codes (e.g. ("core", "VS")) (#221)

• mdf_reader.utils.validator: simplify function code_validation (#221)

• cdm_mapper.codes.common: convert range-key properties to list (#221)

• testing_suite: new chunksize test with icoads_r300_d721 (#222)

• mdf_reader, cdm_nmapper: use model-depending encoding while writing data on disk (#222)

• code restructuring (#224)

• remove unused functions and methods (#224)

Bug fixes

• Solve SettingWithCopyWarning (#151, #184)
• mdf_reader: utils.converters.decode returns values not only None (#214)
• mdf_reader: solving misleading reading due to German "umlauts"(#212, #214, #222)

v1.0.2

Contributors to this version: Ludwig Lierhammer (@ludwiglierhammer)

Announcements

• New PyPi Classifiers:

  • Development Status :: 5 - Production/Stable
  • Development Status :: Intended Audience :: Science/Research
  • License :: OSI Approved :: Apache Software License
  • Operating System :: OS Independent

v1.0.1

Contributors to this version: Ludwig Lierhammer (@ludwiglierhammer)

Announcements

• set package version to v1.0.1

v1.0.0

Contributors to this version: Ludwig Lierhammer (@ludwiglierhammer)

Announcements

• Final version used for GLAMOD marine processing release 7.0

Bug fixes

• cdm_mapper: Two reports that describe each other as best duplicates are not flagged as duplicates (DupDetect) (:pull:149)
• cdm_mapper: Reindex only if null values available (DupDetect) (:pull:153)

v0.4.3

Contributors to this version: Ludwig Lierhammer (:user:ludwiglierhammer)

Announcements
^^^^^^^^^^^^^

• First release on pypi (:issue:17)
• First release on zenodo (:issue:18)