v0.4.4

What's Changed

• Completely local reprocessing and no-per chunk reprocessing by @dachengx in #308
• Fix bug in resubmit mode by @dachengx in #309
• Set PER_CHUNK_DATA_TYPES as empty when no per chunk mode by @dachengx in #310
• Fix of mismatched arguments of no_remote by @dachengx in #311
• Do not make tarballs of pacakges in debug + (relay / resubmit) mode by @dachengx in #312
• Be automatically compatible with SR1 straxen by @dachengx in #313
• Fix bug of missing overwritten storage by @dachengx in #315
• Install only the user_install_package by @dachengx in #314
• Direct all EU RSE based jobs to SURFsara by @dachengx in #316
• Be compatible with AxFoundation/strax#996 by @dachengx in #317
• Minor change of order of codes for saltax compatibility by @dachengx in #318

Full Changelog: v0.4.3...v0.4.4

v0.4.3

What's Changed

• Print total size of a folder in davs by @dachengx in #305
• Use numbered version of docformatter by @dachengx in #306

Full Changelog: v0.4.2...v0.4.3

v0.4.2

What's Changed

• Be compatible with XENONnT/straxen#1564 by @dachengx in #264
• Use more EU sites by @dachengx in #248
• Downsample run_ids from full list by @dachengx in #265
• Increase 20 percent disk/memory in next job retry by @dachengx in #266
• Add eu_only_countries option to constraint the countries of nodes by @dachengx in #267
• Assign GLIDEIN_Site if site_only by @dachengx in #268
• Add scripts of outputs management by @dachengx in #269
• Update memory usage after optimization by @dachengx in #270
• Ask for XENON_DESIRED_Sites in combine and upper job by @dachengx in #271
• Install latest admix before running the jobs by @dachengx in #272
• Increase the num_threads to be consistent with admix by @dachengx in #274
• Increase the overhead(redundancy) of disk and memory from 5% to 25% by @dachengx in #275
• Print more information during job running by @dachengx in #276
• Set periodic_hold of a job by @dachengx in #273
• Set resources increment for walltime by @dachengx in #277
• Increase memory usage of combine and upper jobs by @dachengx in #279
• Be compatible with XENONnT/straxen#1570 by @dachengx in #280
• Only have to consider the largest chunk size of raw_records* by @dachengx in #281
• OSG-RCC relay 1: Add RCC jobs submitters by @dachengx in #278
• Debug get_disk by @dachengx in #282
• Have to set job_id as None before submit muon veto jobs by @dachengx in #283
• Separate TPC, NV and MV folders by @dachengx in #284
• OSG-RCC relay 2: Add OSG-RCC jobs relay submitter by @dachengx in #285
• Do not skip combine job even no lower-level job is submitted by @dachengx in #286
• XENON_CONFIG file name consistency by @dachengx in #287
• Directly stream data to public folder, prevent mv by script by @dachengx in #288
• Add storage and memory usage of afterpulses by @dachengx in #289
• Add special input path for combine-only job by @dachengx in #290
• Save the finished list of run in a file by @dachengx in #291
• Add relay mode for gfal-ls by @dachengx in #292
• Further simply storage of slurm jobs by @dachengx in #293
• Larger memory usage of combine and upper jobs by @dachengx in #294
• Use smaller peaklets and hitlets_nv chunk in per-chunk processing by @dachengx in #295
• Add upper-level jobs only mode of slurm and relay submitter by @dachengx in #296
• When the number of jobs is too high, pause the job submission by @dachengx in #297
• Fix bug in resubmission, should resubmit immediately by @dachengx in #298
• Enable resubmitting a slurm workflow by @dachengx in #299
• Unset CUTAX_LOCATION and PYTHONPATH by @dachengx in #300
• Fix a corner case that the livetime of run is very short by @dachengx in #301
• Set NV as upper-level only workflow by @dachengx in #302
• Submit job to download and decompress files by @dachengx in #303
• Bump version: 0.4.1 → 0.4.2 by @dachengx in #304

Full Changelog: v0.4.1...v0.4.2

v0.4.1

What's Changed

• add timestamp to workflow by @rsriya in #192
• Transfer data to local after processing by @dachengx in #194
• Download log of jobs by @dachengx in #196
• Untar outpus if transferred to local by @dachengx in #197
• Check if there is no crucial user installed package by @dachengx in #198
• Upload miscellaneous information to DB by @dachengx in #199
• Ignore processed data while processing by @dachengx in #200
• Download raw_records* from only raw_records_rse by @dachengx in #201
• Use dependency tree for instruction of processing by @dachengx in #202
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #204
• Limit the number CPU usage of JAX and TensorFlow by @dachengx in #209
• Optimize the order of data_types in processing by @dachengx in #210
• Clearly list the data_type to process when submitting job by @dachengx in #211
• If self-installed package, do not upload to rucio by @dachengx in #212
• Set environmental variable of CPUs usage for all jobs by @dachengx in #213
• Still need local sites even if do not transfer to local by @dachengx in #214
• Set defualt dagman_maxjobs to be 5000 by @dachengx in #215
• Fix $lte for single run by @shenyangshi in #217
• Ask reporter to include the Outsource section of XENON_CONFIG by @dachengx in #222
• Forbid using development container by @dachengx in #221
• No per-chunk processing of MV by @dachengx in #223
• Force renew .dbtoken before workflow submission by @dachengx in #224
• Limit the filename in 255 characters by @dachengx in #226
• Allow staging-out lower data_types and keeping downloaded raw_records* by @dachengx in #228
• Decide whether the dependency exists based on specified RSE by @dachengx in #229
• Add script to test the resources usage: memory, time, storage by @dachengx in #230
• Automatically assign resources by @dachengx in #231
• Fix bug when multiple json files exists by @dachengx in #232
• Keep the folder being tarred by @dachengx in #233
• Stage out lower, combine or upper level outputs by @dachengx in #234
• List a directory in DAV by @dachengx in #235
• FIx bug that non-LED data_types is not filtered out in LED mode by @dachengx in #236
• Assign the resources usage of LED calibration by @dachengx in #237
• Allow multiple raw_records* RSEs by @dachengx in #238
• Set same requirements for lower level and combine jobs by @dachengx in #239
• Only extract input and output folders by @dachengx in #240
• Prioritize the first assign RSE by @dachengx in #241
• Allow multiple per-chunk processing by @dachengx in #242
• Small updates and fixes by @cfuselli in #244
• Change logic for EU sites by @cfuselli in #245
• Print locations of raw_records* by @dachengx in #243
• Lower memory usage of lower level processing by @dachengx in #246
• Revert part of #244 and #245 by @dachengx in #247
• Set special memory usage for LED calibration runs by @dachengx in #249
• Add constraint for SURFSARA2_USERDISK by @dachengx in #250
• Set lowest disk requirement as 200MB by @dachengx in #251
• Update memory usage of upper level jobs by @dachengx in #252
• Save lineage of data_types in summary.json by @dachengx in #253
• Copy .xenon_config to avoid its change after the job submission by @dachengx in #254
• Remove "download" jobs because they are not used by @dachengx in #255
• Add option to first stage the rucio files then download by @dachengx in #256
• Piecewise function for NV disk usage by @dachengx in #257
• Add suffix of combine and untar jobs by @dachengx in #258
• Piecewise function for NV disk usage again by @dachengx in #259
• Update memory usage of NV data by @dachengx in #260
• Copy local files to workflow folder to avoid overwritten by @dachengx in #261
• First dagman_static_retry retries will use the same resources by @dachengx in #262
• Bump version: 0.4.0 → 0.4.1 by @dachengx in #263

New Contributors

• @pre-commit-ci made their first contribution in #204
• @shenyangshi made their first contribution in #217
• @cfuselli made their first contribution in #244

Full Changelog: v0.4.0...v0.4.1

v0.4.0

What's Changed

• Testing single thread processor by @FaroutYLq in #163
• Remove CNAF from glidein site specificiation by @FaroutYLq in #164
• Reduce transfer thread and one more core in event level jobs by @FaroutYLq in #165
• fixed typo by @rsriya in #167
• No longer need production if you don't even try to upload to rucio by @FaroutYLq in #168
• Proposal to use pre-commit for continuous integration by @dachengx in #170
• Use environmental variable to pass the proxy by @dachengx in #171
• First restructure, make it more compatible with strax(en) by @dachengx in #172
• Move Shell to utilix, move all RSE related codes to config.py by @dachengx in #173
• Configuration of disk should be in MB by @dachengx in #174
• Tarball strax(en) and cutax for later user installation by @dachengx in #175
• Move around pegasus.conf by @dachengx in #176
• Remove unused python files by @dachengx in #177
• Apply per-chunk storage by @dachengx in #178
• Add more options for job submission by @dachengx in #179
• Move extraction of run_ids to utils by @dachengx in #180
• Use pyproject.toml to install outsource by @dachengx in #181
• Bump version of utilix by @dachengx in #182
• Upload outousrce to pypi by @dachengx in #190
• Bump to v0.4.0 by @dachengx in #191

New Contributors

• @rsriya made their first contribution in #167
• @dachengx made their first contribution in #170

Full Changelog: v0.3.1...v0.4.0

v0.3.1

Final status for SR1 processing.

What's Changed

• Given up afterpulses computation but doesn't hurt to merge by @FaroutYLq in #135
• Removed attach process by @FaroutYLq in #136
• Enable ref_mon_nv computation by @FaroutYLq in #138
• Debug when cuts_basic fail by @FaroutYLq in #139
• Try requiring AVX2 only by @FaroutYLq in #140
• echo Rundb api accessibility by @FaroutYLq in #141
• Avoid regeneration of .dbtoken on job node by @FaroutYLq in #144
• Debug combine wrapper by @FaroutYLq in #146
• Changed default to development by @FaroutYLq in #149
• Remove event_info re-registering by @FaroutYLq in #153
• Removed outdated cuts removal by @FaroutYLq in #154
• Retire grid proxy info by @FaroutYLq in #155
• Removed outdated plugin unregister by @FaroutYLq in #156
• Put back x509 proxy check by @FaroutYLq in #157
• Improved logging when using gfal2 by @rynge in #159
• Temporary bypassing integrity check in pegasus by @FaroutYLq in #158
• Adding more types for tar balling outputs by @FaroutYLq in #160
• No more local output by @FaroutYLq in #161

Full Changelog: v0.3.0...v0.3.1

v0.3.0

Stable version for SR1 reprocessing for v14 correction.

What's Changed

• Update README.md by @FaroutYLq in #116
• Update README.md by @FaroutYLq in #117
• Debug info for mismatch between chunks and metadata by @FaroutYLq in #115
• Not to updatedb if upload failed by @FaroutYLq in #119
• Added peak_basics into to-check in RunDB by @FaroutYLq in #120
• Debug missing comma in RunConfig by @FaroutYLq in #122
• Trying to add cuts_basic into to-process list by @FaroutYLq in #123
• Revisit check chunk n by @FaroutYLq in #124
• No records or records_nv to upload in rucio by @FaroutYLq in #125
• Save events and changing order by @FaroutYLq in #127
• Added output for events jobs by @FaroutYLq in #129
• Preupload to avoid rucio orphan files by @FaroutYLq in #133

Full Changelog: v0.2.2...v0.3.0

v0.2.2

Stable version for SR1 reprocessing for v13 correction.

What's Changed

• first commit of log_digest executable by @cecilia-ferrari in #91
• Improved usage of raw_records_rses by @FaroutYLq in #94
• Debug check_chunk_n by @FaroutYLq in #95
• Moveing to davs for staging by @rynge in #96
• More debug info before rucio attach by @FaroutYLq in #97
• Bug fix for staging by @FaroutYLq in #98
• Added event_ambience and event_shadow by @FaroutYLq in #99
• Debug dids_to_attach by @FaroutYLq in #101
• Adjustments: set output site to enable proper cleanup, exclude the Da… by @rynge in #100
• Removed hardcode Dakota by @FaroutYLq in #102
• removed s1_cnn temporarily by @FaroutYLq in #103
• Shadow and ambience as a new type by @FaroutYLq in #104
• put s1_cnn back by @FaroutYLq in #105
• imposed check_n_chunk also in runstrax by @FaroutYLq in #107
• More maxidle by @FaroutYLq in #108
• Added requirements for open firewall to the ports we need by @rynge in #109
• Timing info for admix.upload by @FaroutYLq in #111
• Prioritize heavy low level types in runstrax by @FaroutYLq in #113
• Forbid multi-threading by @FaroutYLq in #114

New Contributors

• @cecilia-ferrari made their first contribution in #91

Full Changelog: v0.2.0...v0.2.2

v0.1.0

This release is the first working version of outsource. It converts the xenon1t workflow into a Pegasus framework, apart from a few minor things related to the runDB control.