feat: Add simulation's controllers

[filimarc]

Technical proposal and requirements

Impact

• This feature will control the start-stop behavior of simulations.
• It may impact simulation performance.
• It impacts the simulation progress interfaces and progress listeners (by default there are no progress listeners, or configuration for it).

Current Behavior

• In Arbor: no start-stop: run simulation from start to finish.
• In NEST: start-stop in 1ms blocks.
• In NEURON start-stop in 1ms blocks.

N.B.: resolution config attr only affects simulation resolution, not start-stop behavior.

Context

Sometimes devices or other simulation components need to execute a callback at certain checkpoints. The current use case is LFP where the recorded data of the entire simulation exceeds memory capacity, but one can calculate less memory intense intermediates and flush the recorded data in between.

Desired Behavior

Technical proposal

We introduce a concept of "simulation controllers" which are simulation components that implement get_next_checkpoint. Without any controllers present, simulation should run from start to finish without steps (breaking change), no progress listeners triggered.

Uniformizing implement

The implement interface should be lifted from Device to SimulationComponent and be made optional for all components, and called the same way it's done for Devices now to give each component the opportunity to store the context they need to operate. The implement function should remain required for devices.

Finding the "controllers":

def prepare():
  # ...
  controllers = [*(d for d in simulation.devices if hasattr(d, "get_next_checkpoint")), *(<same for cell_models>), *(<same for connection_models>)]

And honestly let's just add a util function get_simulation_components that returns an iterator with all of the components, and then:

def prepare():
  # ...
  controllers = [*(comp for comp in get_simulation_components(simulation) if hasattr(comp, "get_next_checkpoint"))]

run_checkpoint

A required callback to be ran at requested checkpoints.

Note

• We have to request the arbor.clear_samples(handle) feature to the arbor dev team
• We need to ask the NEST devs if it's possible to clear recorded data during a simulation

As a workaround, run_checkpoints in Arbor may clear all the samples, but should take care so that all devices in the simulation first get flushed, so that no data is lost.

get_next_checkpoint

At the start of the simulation, and at each checkpoint, call all get_next_checkpoint and find the closest one (min). Thenh store all controllers, that requested this checkpoint. At the checkpoint, run all run_checkpoint functions of those controllers:

Controller A requests: 3, 10, 16
Controller B requests: 4,  10, 12

Then checkpoints are: 3, 4, 10, 12, 16
And run_checkpoint is called for A at requested checkpoints, and B requested checkpoints

FixedStepController

Provide component to do simple fixed step, which also adds a progress listener that prints progress or shows progress bar in TTY. Add option to control addition of this controller to simulations, with configuration of time step. Its goal is that the user has a simple way of start-stopping simulation in certain intervals with progress reporting.

Describe the work done

• Now the implement method is raised to SimulationComponent level, it still need to be implemented for Devices.
• The method create_device (or prepare_samples for Arbor) is substitute with implement_components() that is general for all the adapters.
• Now every adapter has a _controllers list where selected controllers (can be any simulation components) can be registered, if it has get_next_checkpoint and progress methods.
• In the Adapter is implemented a get_next_checkpoint method that creates an iterator that span all the registered controllers for the next checkpoint
• before simulation starts the adapter will look for controllers with on_start() method an run them.
• Is created a BasicSimulationListener class, can be added with a new BsbOption ( --sr). In the presence of a TTY terminal the simulation execution is now rendered as a progress bar.
• Interface for add custom listeners is removed

Tasks

• Added tests
• Updated documentation

---

📚 Documentation preview 📚: https://bsb-arbor--169.org.readthedocs.build/en/169/

---

📚 Documentation preview 📚: https://bsb-nest--169.org.readthedocs.build/en/169/

---

📚 Documentation preview 📚: https://bsb-core--169.org.readthedocs.build/en/169/

---

📚 Documentation preview 📚: https://bsb-neuron--169.org.readthedocs.build/en/169/

---

📚 Documentation preview 📚: https://bsb--169.org.readthedocs.build/en/169/

---

📚 Documentation preview 📚: https://bsb-test--169.org.readthedocs.build/en/169/

[codecov]

Codecov Report

❌ Patch coverage is 81.25000% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.85%. Comparing base (812a974) to head (48ac68a).

Files with missing lines	Patch %	Lines
packages/bsb-core/bsb/simulation/adapter.py	76.92%	13 Missing and 2 partials ⚠️
packages/bsb-arbor/bsb_arbor/devices/probe.py	33.33%	2 Missing ⚠️
packages/bsb-neuron/bsb_neuron/adapter.py	75.00%	2 Missing ⚠️
...ages/bsb-arbor/bsb_arbor/devices/spike_recorder.py	66.66%	0 Missing and 1 partial ⚠️
packages/bsb-core/bsb/_options.py	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             next     #169      +/-   ##
==========================================
+ Coverage   78.78%   78.85%   +0.07%     
==========================================
  Files         121      143      +22     
  Lines       12545    13596    +1051     
  Branches     1385     1505     +120     
==========================================
+ Hits         9883    10721     +838     
- Misses       2318     2499     +181     
- Partials      344      376      +32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Helveg]

I made a mess of the review, but most parts are still not as discussed in the technical proposal. Can we have a call?