-
Notifications
You must be signed in to change notification settings - Fork 52
CodeRabbit: "Fix" Catch-up-math #486
Copy link
Copy link
Open
Description
From #484 --
- for _ in range(
- self.start_window,
- (loaded_checkpoint_window + 1) * self.hparams.inner_steps,
- ):
- self.inner_scheduler.step()
+ # Advance scheduler by the exact number of inner steps since the start window
+ steps_to_apply = max(
+ 0,
+ (loaded_checkpoint_window - self.start_window + 1)
+ * self.hparams.inner_steps,
+ )
+ for _ in range(steps_to_apply):
+ self.inner_scheduler.step()
self.sync_window = loaded_checkpoint_window
Code rabbit explains that one over-steps the LR scheduler by start_window * (inner_steps-1)
This is likely negligible but we should still modify the math to achieve the desired state
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels