Add `agent_data` table by Grufoony · Pull Request #433 · physycom/DynamicalSystemFramework

Grufoony · 2026-04-08T13:54:14Z

Add agent_data table with columns:

simulation id
agent id
edge id
timestep in
timestep out

Solved issues

Closes Returning aggregate metrics #432

test/mobility/Test_dynamics.cpp

+
+          SQLite::Statement schema(db, "PRAGMA table_info(agent_data)");
+          std::set<std::string> columns;
+          while (schema.executeStep()) {


test/mobility/Test_dynamics.cpp

+          while (schema.executeStep()) {
+            columns.insert(schema.getColumn(1).getString());
+          }
+          CHECK(columns.count("id") == 1);


test/mobility/Test_dynamics.cpp

+            columns.insert(schema.getColumn(1).getString());
+          }
+          CHECK(columns.count("id") == 1);
+          CHECK(columns.count("simulation_id") == 1);


test/mobility/Test_dynamics.cpp

+          }
+          CHECK(columns.count("id") == 1);
+          CHECK(columns.count("simulation_id") == 1);
+          CHECK(columns.count("agent_id") == 1);


test/mobility/Test_dynamics.cpp

+          CHECK(columns.count("id") == 1);
+          CHECK(columns.count("simulation_id") == 1);
+          CHECK(columns.count("agent_id") == 1);
+          CHECK(columns.count("edge_id") == 1);


test/mobility/Test_dynamics.cpp

+          }
+          CHECK(agentColumns.count("id") == 1);
+          CHECK(agentColumns.count("simulation_id") == 1);
+          CHECK(agentColumns.count("agent_id") == 1);


test/mobility/Test_dynamics.cpp

+          CHECK(agentColumns.count("id") == 1);
+          CHECK(agentColumns.count("simulation_id") == 1);
+          CHECK(agentColumns.count("agent_id") == 1);
+          CHECK(agentColumns.count("edge_id") == 1);


test/mobility/Test_dynamics.cpp

+          CHECK(agentColumns.count("simulation_id") == 1);
+          CHECK(agentColumns.count("agent_id") == 1);
+          CHECK(agentColumns.count("edge_id") == 1);
+          CHECK(agentColumns.count("time_step_in") == 1);


test/mobility/Test_dynamics.cpp

+          CHECK(agentColumns.count("agent_id") == 1);
+          CHECK(agentColumns.count("edge_id") == 1);
+          CHECK(agentColumns.count("time_step_in") == 1);
+          CHECK(agentColumns.count("time_step_out") == 1);


test/mobility/Test_dynamics.cpp

          CHECK(simColumns.count("save_avg_stats") == 1);
          CHECK(simColumns.count("save_road_data") == 1);
          CHECK(simColumns.count("save_travel_data") == 1);
+          CHECK(simColumns.count("save_agent_data") == 1);


Copilot

Pull request overview

This PR extends the mobility simulation’s SQLite persistence to record per-agent per-edge traversal intervals by introducing a new agent_data table, and updates tests and metadata accordingly.

Changes:

Add agent_data table creation and periodic dumping to SQLite when enabled via FirstOrderDynamics::saveData(...).
Track edge entry/exit times in Street and flush them to the database during FirstOrderDynamics::evolve(...).
Update dynamics DB tests to validate agent_data + add save_agent_data to simulations metadata; bump DSF patch version and citation.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
test/mobility/Test_dynamics.cpp	Adds coverage for `agent_data` table creation/content and updates “all options enabled” expectations.
src/dsf/mobility/Street.hpp	Introduces static storage + accessors for collecting agent traversal data.
src/dsf/mobility/Street.cpp	Records `(agent_id, time_in, time_out)` when agents leave a street queue.
src/dsf/mobility/FirstOrderDynamics.hpp	Extends `saveData` API to support enabling agent-data persistence.
src/dsf/mobility/FirstOrderDynamics.cpp	Creates `agent_data`, writes rows during save intervals, and adds `save_agent_data` to `simulations`.
src/dsf/dsf.hpp	Bumps patch version.
CITATION.cff	Updates release version/date.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-08T14:03:01Z

src/dsf/mobility/Street.hpp

+        throw std::runtime_error(
+            "agentData not initialized. Please call acquireAgentData() first.");
+      }
+      return std::move(m_agentData.value());


Street::agentData() moves the underlying map out of the std::optional but leaves m_agentData engaged, so subsequent calls to Street::dequeue() will keep writing into a moved-from tbb::concurrent_unordered_map. This is brittle (moved-from state is unspecified) and makes the lifetime/clearing semantics unclear. Consider extracting via swap/exchange and immediately re-initialize (or reset()) m_agentData so writers always see a well-defined, empty container after a flush.

Suggested change

return std::move(m_agentData.value());

decltype(m_agentData.value()) extractedData;

extractedData.swap(m_agentData.value());

return extractedData;

src/dsf/mobility/FirstOrderDynamics.hpp

Copilot · 2026-04-08T14:03:02Z

src/dsf/mobility/FirstOrderDynamics.cpp

    // Create simulations table if it doesn't exist
    SQLite::Statement createTableStmt(*this->database(),
                                      "CREATE TABLE IF NOT EXISTS simulations ("
                                      "id INTEGER PRIMARY KEY, "
                                      "name TEXT, "
                                      "speed_function TEXT, "
                                      "weight_function TEXT, "
                                      "weight_threshold REAL NOT NULL, "
                                      "error_probability REAL, "
                                      "passage_probability REAL, "
                                      "mean_travel_distance_m REAL, "
                                      "mean_travel_time_s REAL, "
                                      "stagnant_tolerance_factor REAL, "
                                      "force_priorities BOOLEAN, "
                                      "save_avg_stats BOOLEAN, "
                                      "save_road_data BOOLEAN, "
-                                      "save_travel_data BOOLEAN)");
+                                      "save_travel_data BOOLEAN, "
+                                      "save_agent_data BOOLEAN)");
    createTableStmt.exec();
    // Insert simulation parameters into the simulations table
    SQLite::Statement insertSimStmt(
        *this->database(),
        "INSERT INTO simulations (id, name, speed_function, weight_function, "
        "weight_threshold, error_probability, passage_probability, "
        "mean_travel_distance_m, mean_travel_time_s, stagnant_tolerance_factor, "
-        "force_priorities, save_avg_stats, save_road_data, save_travel_data) "
-        "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
+        "force_priorities, save_avg_stats, save_road_data, save_travel_data, "
+        "save_agent_data) "
+        "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
    insertSimStmt.bind(1, static_cast<std::int64_t>(this->id()));


CREATE TABLE IF NOT EXISTS simulations (...) will not add the new save_agent_data column when opening an existing DB created with an older schema; the subsequent INSERT INTO simulations (..., save_agent_data) will fail with "no such column". Add a lightweight migration step (e.g., PRAGMA table_info(simulations) + ALTER TABLE simulations ADD COLUMN save_agent_data BOOLEAN DEFAULT 0) before the insert to keep backward compatibility.

Copilot · 2026-04-08T14:03:02Z

src/dsf/mobility/FirstOrderDynamics.cpp

+      if (m_bSaveAgentData) {
+        auto agentData = Street::agentData();
+        SQLite::Statement insertStmt(*this->database(),
+                                     "INSERT INTO agent_data (simulation_id, "
+                                     "agent_id, edge_id, time_step_in, time_step_out)"
+                                     "VALUES (?, ?, ?, ?, ?)");
+        for (auto const& [edge_id, data] : agentData) {
+          for (auto const& [agent_id, ts_in, ts_out] : data) {
+            insertStmt.bind(1, simulationId);
+            insertStmt.bind(2, static_cast<std::int64_t>(agent_id));
+            insertStmt.bind(3, static_cast<std::int64_t>(edge_id));
+            insertStmt.bind(4, static_cast<std::int64_t>(ts_in));
+            insertStmt.bind(5, static_cast<std::int64_t>(ts_out));
+            insertStmt.exec();
+            insertStmt.reset();
+          }


Agent-data inserts are not covered by the hasWritePayload/transaction logic (so when only m_bSaveAgentData is enabled, inserts run in autocommit mode). This can drastically slow down writes and makes this block behave differently from the other tables. Update the transaction condition to include agent-data payload and ensure these inserts run inside the same transaction.

Suggested change

if (m_bSaveAgentData) {

auto agentData = Street::agentData();

SQLite::Statement insertStmt(*this->database(),

"INSERT INTO agent_data (simulation_id, "

"agent_id, edge_id, time_step_in, time_step_out)"

"VALUES (?, ?, ?, ?, ?)");

for (auto const& [edge_id, data] : agentData) {

for (auto const& [agent_id, ts_in, ts_out] : data) {

insertStmt.bind(1, simulationId);

insertStmt.bind(2, static_cast<std::int64_t>(agent_id));

insertStmt.bind(3, static_cast<std::int64_t>(edge_id));

insertStmt.bind(4, static_cast<std::int64_t>(ts_in));

insertStmt.bind(5, static_cast<std::int64_t>(ts_out));

insertStmt.exec();

insertStmt.reset();

}

auto agentData = Street::agentData();

if (m_bSaveAgentData && !agentData.empty()) {

this->database()->exec("SAVEPOINT agent_data_insert");

try {

SQLite::Statement insertStmt(*this->database(),

"INSERT INTO agent_data (simulation_id, "

"agent_id, edge_id, time_step_in, time_step_out)"

"VALUES (?, ?, ?, ?, ?)");

for (auto const& [edge_id, data] : agentData) {

for (auto const& [agent_id, ts_in, ts_out] : data) {

insertStmt.bind(1, simulationId);

insertStmt.bind(2, static_cast<std::int64_t>(agent_id));

insertStmt.bind(3, static_cast<std::int64_t>(edge_id));

insertStmt.bind(4, static_cast<std::int64_t>(ts_in));

insertStmt.bind(5, static_cast<std::int64_t>(ts_out));

insertStmt.exec();

insertStmt.reset();

}

}

this->database()->exec("RELEASE SAVEPOINT agent_data_insert");

} catch (...) {

this->database()->exec("ROLLBACK TO SAVEPOINT agent_data_insert");

this->database()->exec("RELEASE SAVEPOINT agent_data_insert");

throw;

Copilot · 2026-04-08T14:03:02Z

src/dsf/mobility/FirstOrderDynamics.cpp

+          for (auto const& [agent_id, ts_in, ts_out] : data) {
+            insertStmt.bind(1, simulationId);
+            insertStmt.bind(2, static_cast<std::int64_t>(agent_id));
+            insertStmt.bind(3, static_cast<std::int64_t>(edge_id));
+            insertStmt.bind(4, static_cast<std::int64_t>(ts_in));
+            insertStmt.bind(5, static_cast<std::int64_t>(ts_out));
+            insertStmt.exec();
+            insertStmt.reset();


ts_in/ts_out are std::time_t, but they’re bound without an explicit cast. On some platforms time_t is not an int64_t, which can lead to narrowing/overload surprises. Cast ts_in/ts_out to std::int64_t when binding to match the INTEGER schema.

Copilot · 2026-04-08T14:03:02Z

src/dsf/mobility/FirstOrderDynamics.cpp

@@ -1652,6 +1697,7 @@
        m_bSaveStreetData = false;
        m_bSaveTravelData = false;
        m_bSaveAverageStats = false;


When saveAgentData is enabled, Street::acquireAgentData() makes Street start collecting agent events based on m_agentData.has_value(). However, in the one-shot savingInterval == 0 reset path only m_bSaveAgentData is cleared; the static Street::m_agentData remains engaged, so streets will keep accumulating agent events even though saving is disabled. Add an explicit teardown/clear (e.g., Street::releaseAgentData()/resetAgentData()), or gate collection on a flag that is reset alongside m_bSaveAgentData.

Suggested change

m_bSaveAverageStats = false;

m_bSaveAverageStats = false;

Street::releaseAgentData();

Copilot · 2026-04-08T14:03:03Z

src/dsf/mobility/FirstOrderDynamics.cpp

    spdlog::info(
        "Data saving configured: interval={}s, avg_stats={}, street_data={}, "
-        "travel_data={}",
+        "travel_data={}, agent_data={}",
        savingInterval,
        saveAverageStats,
        saveStreetData,
-        saveTravelData);
+        saveTravelData,
+        saveAgentData);


The log message uses interval={}s, but savingInterval is documented as "in time steps" (not seconds). Consider adjusting the unit in the message to avoid misleading logs (e.g., interval={} steps).

codecov · 2026-04-08T14:03:23Z

Codecov Report

❌ Patch coverage is 91.66667% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.32%. Comparing base (d93e300) to head (956ece0).

Files with missing lines	Patch %	Lines
test/mobility/Test_dynamics.cpp	85.71%	0 Missing and 3 partials ⚠️
src/dsf/mobility/FirstOrderDynamics.cpp	96.15%	1 Missing ⚠️
src/dsf/mobility/Street.hpp	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #433      +/-   ##
==========================================
+ Coverage   87.27%   87.32%   +0.04%     
==========================================
  Files          52       52              
  Lines        6461     6516      +55     
  Branches      717      730      +13     
==========================================
+ Hits         5639     5690      +51     
- Misses        803      804       +1     
- Partials       19       22       +3

Flag	Coverage Δ
unittests	`87.32% <91.66%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Grufoony added 3 commits April 8, 2026 15:34

Add agent_data table if enabled

2bba407

Add tests

4284091

Bump version

687a52d

Grufoony self-assigned this Apr 8, 2026

Copilot AI review requested due to automatic review settings April 8, 2026 13:54

Copilot started reviewing on behalf of Grufoony April 8, 2026 13:55 View session

github-advanced-security bot found potential problems Apr 8, 2026

View reviewed changes

Fix macos

dc413a4

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Copilot fixes

956ece0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `agent_data` table#433

Add `agent_data` table#433
Grufoony wants to merge 5 commits intomainfrom
agentdata

Grufoony commented Apr 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

codecov bot commented Apr 8, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-      return std::move(m_agentData.value());
+      decltype(m_agentData.value()) extractedData;
+      extractedData.swap(m_agentData.value());
+      return extractedData;

	m_bSaveAverageStats = false;
	m_bSaveAverageStats = false;
	Street::releaseAgentData();

Conversation

Grufoony commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Grufoony commented Apr 8, 2026 •

edited

Loading

codecov bot commented Apr 8, 2026 •

edited

Loading