Skip to content

Enabling Distributed Processing #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
markmac99 wants to merge 135 commits into
base: master
Choose a base branch
from
Draft

Enabling Distributed Processing #61

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
debef9c
split into candidate and simple calc phases
markmac99 Jan 16, 2026
240a720
print candmode in logs
markmac99 Jan 16, 2026
b1040c6
tidy up logging
markmac99 Jan 17, 2026
aa20b49
bug in conversion step
markmac99 Jan 17, 2026
02a9865
avoid committing to sqlite after every update
markmac99 Jan 19, 2026
f569ebe
support for excluding already-failed candidates
markmac99 Jan 19, 2026
12dab86
start adding some standalone management to the sqlite db scripts
markmac99 Jan 19, 2026
9b18baa
add obs-date to database
markmac99 Jan 19, 2026
8208b77
got params in wrong order arg
markmac99 Jan 20, 2026
1dc6ae5
improve archiving process
markmac99 Jan 20, 2026
332c5f3
log how many obs got moved to sqlite
markmac99 Jan 20, 2026
db77938
make logging provisional
markmac99 Jan 20, 2026
1089a1b
remove unnecessary logging
markmac99 Jan 20, 2026
e4320a5
make obs database a class
markmac99 Jan 21, 2026
75f11ea
improve documentation
markmac99 Jan 21, 2026
405d6c0
some tidying up
markmac99 Jan 21, 2026
81a2969
simplify migration to sqlite
markmac99 Jan 21, 2026
9a71d85
keep track of which traj we're doing
markmac99 Jan 21, 2026
297da7c
improve archiving and add merging support
markmac99 Jan 21, 2026
f1f5f5e
updates to improve handling of operation modes and remote data
markmac99 Jan 24, 2026
6eb44fa
improve documentation
markmac99 Jan 25, 2026
9024741
Merge branch 'wmpg:master' into distrib_processing
markmac99 Jan 25, 2026
1afaff1
bugfix to create phase1 trajs properly
markmac99 Jan 26, 2026
625ee4d
bugfix in when to save phase1
markmac99 Jan 26, 2026
3684014
be a bit less verbose
markmac99 Jan 27, 2026
7f5677a
remove unnecessary message
markmac99 Jan 27, 2026
21fa47e
reinstate prior behaviour to skip ignored stations when retrying solu…
markmac99 Jan 27, 2026
2c1ab8b
simplify calls to addTrajectory
markmac99 Jan 27, 2026
610cfa5
create trajectories db in sqlite
markmac99 Jan 27, 2026
22e9932
add trajectories db as json file
markmac99 Jan 27, 2026
558ce81
bugfixes
markmac99 Jan 27, 2026
fd228a9
updates to handle trajectory sqlite db better
markmac99 Jan 28, 2026
b82689b
dont load the traj db in MCMODE2 as we don't need it
markmac99 Jan 28, 2026
8331148
some support to analyse the traj db
markmac99 Jan 28, 2026
62b45cd
improve logging
markmac99 Jan 29, 2026
02b6c0e
handle verbose flags better
markmac99 Jan 29, 2026
13e6836
more debug
markmac99 Jan 29, 2026
9ec2eaa
remove commitnow as its causing issues
markmac99 Jan 29, 2026
7a8d4e3
debug
markmac99 Jan 29, 2026
36a9f0b
debug
markmac99 Jan 29, 2026
ebf5473
typo
markmac99 Jan 29, 2026
7e88468
more doc and debug to track down issue
markmac99 Jan 29, 2026
2eaeb91
fix bug in fails table
markmac99 Jan 29, 2026
ad4eec4
bugfix
markmac99 Jan 29, 2026
d27a13b
simplify code
markmac99 Jan 29, 2026
6c28d9d
bugfixes in correlateDB
markmac99 Jan 29, 2026
863d7e2
avoid removing trajectories incorrectly
markmac99 Jan 29, 2026
50eb9df
ting bug in reporting of removed traj
markmac99 Jan 29, 2026
36a880b
Client-mode remote data processing
markmac99 Jan 30, 2026
bb7b502
reduce amount of legacy data thats loaded
markmac99 Jan 30, 2026
23b258b
make dates tz aware
markmac99 Jan 30, 2026
be29951
improve logging
markmac99 Jan 30, 2026
5eebd9d
more tweaks to copying legacy data
markmac99 Jan 30, 2026
a87fb16
bugfix
markmac99 Jan 30, 2026
71e20aa
bugfix
markmac99 Jan 30, 2026
6083b46
remove unncessary debug
markmac99 Jan 31, 2026
5f986c7
don't try to save an unopened database
markmac99 Jan 31, 2026
6e8bb88
Updates for remote client datahandling
markmac99 Feb 1, 2026
9cd2d00
support to signal when a client is ready for data
markmac99 Feb 3, 2026
458170d
more work on master mode
markmac99 Feb 3, 2026
138d672
fix bug in unpairing if db is empty
markmac99 Feb 3, 2026
69c99ce
fix bug in unpairing if db is empty
markmac99 Feb 3, 2026
fd38bd5
make uploaded database names unique
markmac99 Feb 3, 2026
6aac11f
oops, make sure phase1 dir exists even in mc mode
markmac99 Feb 3, 2026
ce2877b
small bugfixes
markmac99 Feb 4, 2026
3622d0c
remove some excess verbosity
markmac99 Feb 4, 2026
67fefbe
indicate how many are being solved
markmac99 Feb 4, 2026
c4eecac
bugfix to handle nans in the database
markmac99 Feb 4, 2026
92f865a
add log suffix
markmac99 Feb 4, 2026
b271037
add log suffix properly
markmac99 Feb 4, 2026
1d366bf
catch SFTP fails
markmac99 Feb 4, 2026
333b48e
more small changes to protect against sftp fails
markmac99 Feb 4, 2026
c224ac1
defend against sftp failures
markmac99 Feb 11, 2026
47a2d27
bugfix to prevent phase2 crash merging data
markmac99 Feb 11, 2026
6da8bbb
add function to copy json data over
markmac99 Feb 11, 2026
83a2b72
bugfix: only move phase1 if client was doing phase1
markmac99 Feb 11, 2026
6ece9ab
make sure folders exist
markmac99 Feb 12, 2026
c9bb9bd
initialise remote data handling in the right place for phase2
markmac99 Feb 12, 2026
c9ede51
improve logging
markmac99 Feb 12, 2026
cda60ba
prevent crash in mcmode 2
markmac99 Feb 12, 2026
fbd7899
print operation mode more frequently
markmac99 Feb 12, 2026
7677acc
update remote data handling to actually move trajectories!
markmac99 Feb 12, 2026
21c94b4
slight improvement in shutdown messaging
markmac99 Feb 12, 2026
1b73810
bugfix
markmac99 Feb 12, 2026
8d8a325
chmod upload folders as needed
markmac99 Feb 12, 2026
cb20be5
fix perms on upload folders
markmac99 Feb 12, 2026
db2f17c
add some verbosity for debug
markmac99 Feb 13, 2026
e753a1e
bugfix in pairing routine
markmac99 Feb 13, 2026
b2cf365
Merge branch 'wmpg:master' into distrib_processing
markmac99 Feb 14, 2026
b12504a
rename archiveoldrecords to archivemonths
markmac99 Feb 14, 2026
379a5e9
Merge branch 'distrib_processing' of github.com:markmac99/WesternMete…
markmac99 Feb 14, 2026
4d212af
remove unnecessasry flag
markmac99 Feb 14, 2026
22dadc5
whoops changed param name incorrectly
markmac99 Feb 14, 2026
86721cd
initialise obs and traj db
markmac99 Feb 14, 2026
508f40e
a bit of documentation
markmac99 Feb 16, 2026
bd31fa0
check status from merge functions
markmac99 Feb 16, 2026
8de1c47
if connection fails, close gracefully and retry later
markmac99 Feb 16, 2026
e04f9f4
write pidfile
markmac99 Feb 16, 2026
17ef66d
remove unnecessary saving of temp pickles in ph2
markmac99 Feb 16, 2026
0a102b9
tidying up the date formats when reporting candidates, fails etc
markmac99 Feb 18, 2026
b73f07d
bugfix failed to define ref_dt
markmac99 Feb 19, 2026
7b45db1
add some comments
markmac99 Feb 26, 2026
91ab59e
create function to merge broken so we can call it elsewhere if needed
markmac99 Feb 26, 2026
615ea33
commit the write-ahead-log after every candidate save op
markmac99 Feb 27, 2026
43a4e8d
commit the obs database after saving candidates
markmac99 Feb 27, 2026
1a5d592
don't load stations or scan folders if in phase1 or phase2 only modes
markmac99 Feb 27, 2026
4ba68fc
support fractional bins
markmac99 Feb 27, 2026
3c0ae48
small correction in correlateDB
markmac99 Feb 27, 2026
f51e56c
remove unnecessary cursors
markmac99 Feb 27, 2026
d898e1f
support to purge local processed data
markmac99 Feb 27, 2026
704aa0a
make sure dbs are closed in the maintenance section
markmac99 Feb 27, 2026
51b637a
add verbosity
markmac99 Feb 27, 2026
3e7ac46
fix verbosity
markmac99 Feb 27, 2026
7c16e37
error handling in db merge process
markmac99 Feb 27, 2026
739ae25
remote remote WAL dbs after merger
markmac99 Feb 27, 2026
dcc904a
avoid trying to merge empty databases
markmac99 Feb 27, 2026
6415260
more fixes for database or table not existing
markmac99 Feb 27, 2026
d46b06f
documentation
markmac99 Feb 27, 2026
70b59ce
add more debug
markmac99 Feb 27, 2026
8071f49
delete rather than update when unpairing
markmac99 Feb 27, 2026
f663308
handle unpairing more efficiently with multi-insert
markmac99 Feb 27, 2026
6e20cf7
rename function more sensibly
markmac99 Mar 3, 2026
821bf68
switch logic round to mark paired after succeessful phase1. This redu…
markmac99 Mar 3, 2026
5fe13e0
remove unnecessay commit
markmac99 Mar 3, 2026
cfc0b81
pass new obs to solveTrajectory so they can be marked paired if needed
markmac99 Mar 4, 2026
7db6bdf
bug in phase1 processing
markmac99 Mar 5, 2026
ac19e59
working on ref_dt bug
markmac99 Mar 5, 2026
ced94c2
add debug messages to help testing
markmac99 Mar 5, 2026
9db92f7
tidying up
markmac99 Mar 6, 2026
e9d3c6e
avoid reprocessing already processed candidates
markmac99 Mar 7, 2026
c2572de
bugfixes and performance improvements
markmac99 Mar 8, 2026
e9794a8
remove incorrect location of Obs DB
markmac99 Mar 8, 2026
9612432
various improvements
markmac99 Mar 9, 2026
b9d9747
tidying up
markmac99 Mar 9, 2026
d7fabbd
fixes for issues #86, #87, #88, #94
markmac99 Mar 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion wmpl/Rebound/REBOUND.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
REBOUND_FOUND = True

except ImportError:
print("REBOUND package not found. Install REBOUND and reboundx packages to use the REBOUND functions.")
# don't print a message here as its already printed whenever REBOUND_FOUND is False
REBOUND_FOUND = False

from wmpl.Utils.TrajConversions import (
Expand Down
Loading