Skip to content

IRI Data Library dataset entries and update scripts

Notifications You must be signed in to change notification settings

DavidMichaelKelly/dlentries

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

entries is the main folder in the Data Catalog. Its content controls the representation of the Catalog in the Data Library website under SOURCES. Sub-/folders in the Catalog are represented as sub-/datasets in the DL website. The sub-/folders / sub-/datasets organization eventually ends up in index.tex files that define datasets' variables in Ingrid representation. index.tex files define the independent variables (or grids), the variables reading from one or a set of data files, and all the desired metadata. Variables can also be Ingrid calculations derived from other existing DL variables, as opposed to reading from data file(s). They can also include documentation, links to sources, references, and other information of interest. Sub-datasets can also be created within an index.tex file in place of creating sub-folders.
entries can also contain .html files for documentation, readme, etc.

updatescripts contains the scripts used or having been used to fetch new data files for a dataset that increments, typically in time (daily, weekly, monthly, etc.). Older ones are written in Perl while more recent ones were written in Python. Most of the scripts no longer used operationally by the scheduled crontabs are moved to the unused folder. This practice facilitated retrieving information or processes used in older scripts without having to travel back into git history, for those that have been git-removed.

one-time-scripts contains scripts typically used to get an initial whole set of files when setting up a new dataset for the first time. They often resemble their updatescritps counterpart (for those who have one -- not all datasets update). Some update scripts are actually written to cover both cases (get all files or get new increments).

dlhomes.tex defines home Catalog entries in the DL website, listing their name in the DL website Catalog and the path to the Catalog entries in the file system. Then the content under those paths must be identical in nature to the entries. home Catalog entries are typically used as sandbox for DL developers, or "private" DL spaces for personel at the Institution hosting the DL. Their access remains public (unless restrictions are set against its entries). They are just not ostensibly shown to the public.

descriptions and documentation are another mean to keep respectively .html and .pdf files that Catalog's index.tex files can refer to, instead of keeping them where they are desired at the dataset level in entries, as explained above.

About

IRI Data Library dataset entries and update scripts

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • TeX 91.2%
  • HTML 4.9%
  • Python 3.5%
  • Shell 0.3%
  • CSS 0.1%
  • Dockerfile 0.0%