Skip to content

Support loading a checkpoint in Trainer #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Dec 1, 2025
Merged

Support loading a checkpoint in Trainer #108

merged 4 commits into from
Dec 1, 2025
+79 −26

Conversation

@ATATC
Copy link
Contributor

@ATATC ATATC commented Nov 13, 2025

See #107.

ATATC added 2 commits November 12, 2025 20:48
@ATATC
Added WithNetwork abstraction in layer.py to streamline network c…
bb49be5 
…reation and model loading. Updated imports in `__init__.py`. (#107)
@ATATC
Integrated WithNetwork into Predictor and Trainer for unified m…
8620d9d 
…odel handling and removed redundant `build_network()` abstraction. (#107)
@ATATC ATATC added this to the 1.1.0 milestone Nov 13, 2025
@ATATC ATATC self-assigned this Nov 13, 2025
@ATATC ATATC added enhancement New feature or request todo New task or assignment labels Nov 13, 2025
@ATATC ATATC linked an issue Nov 13, 2025 that may be closed by this pull request
Support loading a checkpoint in Trainer #107
Closed
@ATATC
Copy link
Contributor Author

ATATC commented Nov 13, 2025

Tomorrow's todo: add method recover_from(experiment_id: str) -> Self

@ATATC
Enhanced training recovery system in Trainer by adding `recover_fro…
245dad4 
…m()` and `continue_training()` methods, tracking training arguments, and updating epoch handling for checkpoint recovery (#107).
@ATATC
Copy link
Contributor Author

ATATC commented Nov 30, 2025

Note that now epoch starts counting at 0.

@ATATC
Copy link
Contributor Author

ATATC commented Dec 1, 2025

@perctrix Can you quickly test the new features?

@perctrix
Copy link
Contributor

perctrix commented Dec 1, 2025

LGTM

@ATATC ATATC merged commit e9ce7f9 into main Dec 1, 2025
3 checks passed
@ATATC ATATC deleted the 107 branch December 1, 2025 19:48
@perctrix
Copy link
Contributor

perctrix commented Dec 1, 2025

@ATATC Seems a problem in Trainer:

Traceback (most recent call last):
  File "D:\workstation\ML\MIPCandy\mipcandy-bundles\lsnet-seg\test_lsnet_BRaTS_random_patch_sliding.py", line 31, in <module>
    trainer.train(num_epochs=100, note="BRaTS", early_stop_tolerance=10)
  File "D:\workstation\ML\MIPCandy\mipcandy\training.py", line 441, in train
    raise e
  File "D:\workstation\ML\MIPCandy\mipcandy\training.py", line 435, in train
    self.save_everything_for_recovery(toolbox, self._tracker, **training_arguments)
TypeError: Trainer.save_everything_for_recovery() got multiple values for argument 'self'

@ATATC
Copy link
Contributor Author

ATATC commented Dec 1, 2025

Can you create a new issue for that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@ATATC ATATC

Labels

enhancement New feature or request todo New task or assignment

Projects

None yet

Milestone

1.1.0

Development

Successfully merging this pull request may close these issues.

Support loading a checkpoint in Trainer

3 participants

@ATATC @perctrix