Implement SQLite by drekdrek · Pull Request #68 · drekdrek/grinding-stats

drekdrek · 2025-05-01T00:20:03Z

closes #67

drekdrek · 2025-05-01T00:42:49Z

this is going to require something similar to when we moved the data to the PluginStorage folder. a whole migration process that happens when someone loads the plugin with this update

drekdrek · 2025-05-01T01:25:40Z

Recap also will need to be moved over

edit: this will require a large re-write of recap

drekdrek · 2025-05-02T01:59:19Z

all thats left is migrating the old data into sqlite

drekdrek · 2025-05-02T21:47:42Z

currently in my tests with ~17,000 files it takes approx. 2 minutes and 30 seconds to migrate all files into the SQL db, i believe that the slowest part of this is definitely the medals, because its usually about 7 medals per map. I'll be optimizing this more.

unnecessarilycomplex

this looks almost complete I think. I haven't tested the branch but let me know if it would be useful

unnecessarilycomplex · 2025-07-10T12:07:14Z

src/Data/SQLite.as

+        string grinds_query_string = """
+        SELECT * FROM grinds WHERE map_id = ?;
+        """;
+
+        auto grinds_query = db.Prepare(grinds_query_string);
+        grinds_query.Bind(1, mapUid);
+        grinds_query.Execute();
+
+        grinds_query.NextRow(); // im not sure why i have to do this twice, but its blank otherwise
+        grinds_query.NextRow();
+
+        finishes = uint64(grinds_query.GetColumnInt64("finishes"));
+        resets = uint64(grinds_query.GetColumnInt64("resets"));
+        time = uint64(grinds_query.GetColumnInt64("time"));
+        respawns = uint64(grinds_query.GetColumnInt64("respawns"));
+
+
+        debug_print("Loaded finishes " + finishes + " resets " + resets + " time " + time +
+					" respawns " + respawns + " with map_id " + mapUid);


For readability this block could be put this in a separate load_grinds() method. etc for load_medals(), save_grinds()...

unnecessarilycomplex · 2025-07-10T12:12:01Z

src/Data/SQLite.as

@@ -0,0 +1,182 @@
+class SQLite : AbstractData {


The class name SQLite conflicts with the namespace name SQLite

unnecessarilycomplex · 2025-07-10T12:13:35Z

src/Data/SQLite.as

+        @this.db = db;
+        initialize();
+
+        try {migrate();} catch {}


Shouldn't all these issues be handled by migrateToSQLite() in main ?

this migrate() function is in case more columns need to be added later, which then a db migration would be necessary

unnecessarilycomplex · 2025-07-10T12:19:03Z

src/Main.as

 	migrateOldData();
+	startnew(CoroutineFunc(migrateToSQLite));


I think this should be

migrateOldData(); migrateToSQLite(); data.start()

p.s. Also I don't understand either why new coroutines are used at startnew(CoroutineFuncUserdata(insert_medals),strings); and startnew(CoroutineFunc(mergeData)); and startnew(CoroutineFunc(TurboSTM::LoadSuperTimes));. Can't this cause concurrency issues ?

unnecessarilycomplex · 2025-07-10T12:34:10Z

src/Data/SQLite.as

+            return;
+        }
+        @this.db = db;
+        initialize();


I think this would be better named check_tables_exist() (or init_tables_if_not_exist()) as 99.9% of the time this will not initialize tables

unnecessarilycomplex · 2025-07-10T12:48:59Z

src/Main.as

+
+
+	auto old_path = IO::FromStorageFolder("data");
+	if (IO::FolderExists(old_path)) {


This also needs to check if the database tables are empty.

unnecessarilycomplex · 2025-07-10T13:14:35Z

src/Main.as

+}
+
+void insert_medals(ref@ medals_ref) {
+				array<string> strings = cast<array<string>>(medals_ref);


unnecessarilycomplex · 2025-07-10T13:27:29Z

src/Main.as

+		int BATCH_AMOUNT = 50;
+		uint batches = uint(Math::Ceil(old.Length / float(BATCH_AMOUNT)));
+		for (uint i = 0, idx = 0; i < batches; i++) {
+			uint amt = Math::Min(BATCH_AMOUNT, Math::Max(0, old.Length - (i * BATCH_AMOUNT)));


Math::Max confuses me, is it necessary ? (as the last value of i is batches - 1)

unnecessarilycomplex · 2025-07-10T13:37:02Z

src/Main.as

+				strings.InsertAt(1, medals_string);
+
+
+				startnew(CoroutineFuncUserdata(insert_medals),strings);


This creates one coroutine and one insert per map. This kills any performance gains from batching on grinds

I think the structure would be clearer as

for each batch batch_map_ids = (parse filenames) batch_jsons = array of Json::Values for each map insert_into_grinds(batch_maps, batch_jsons) insert_into_medals(batch_maps, batch_jsons)

and then

void insert_into_grinds(batch_maps, batch_jsons) { statement = "INSERT INTO ..." for each map statement += "..." for each map parse json statement.Bind execute statement }

and same for medals

unnecessarilycomplex · 2025-07-10T14:14:39Z

src/Data/DataManager.as


+	SQLite::Database@ db = SQLite::Database(IO::FromStorageFolder("data.db"));
 	// Cloud @cloudData = Cloud();
-	Files @localData = Files();


as far as I understand the Files class isn't used anymore so the source file may be removed from the codebase (?)

create sqlite class

2be3847

drekdrek added 5 commits April 30, 2025 21:22

instantiate db outside instead of every creation

b8c6f8d

wrap migrate with try catch

b4a819b

add save and load functions

0cdd26b

create db outside

04644b4

replace files with SQLite

08a5fc2

drekdrek added 6 commits April 30, 2025 23:21

save and load medals

c7a7c98

create medals when track is first played

8f892ee

add timestamp to table

c33fdd0

convert recap from files to SQLite

8f98554

invert list instead of re-sorting

40e0a18

add total track amount to initial screen

e2a9d4e

drekdrek added 2 commits May 1, 2025 22:05

parameterize and increase batch amount

a341d2c

create migration to sqlite

39115ec

drekdrek added 2 commits May 2, 2025 17:55

fix off by 1 in progress logs

00a0977

fix error with files that contain medals data

86fb20a

drekdrek marked this pull request as ready for review June 8, 2025 12:02

drekdrek added 2 commits June 8, 2025 08:14

Merge branch 'origin' into sqlite

b7b9532

remove some prints

ddf5de7

unnecessarilycomplex reviewed Jul 10, 2025

View reviewed changes



		auto old_path = IO::FromStorageFolder("data");
		if (IO::FolderExists(old_path)) {

		strings.InsertAt(1, medals_string);


		startnew(CoroutineFuncUserdata(insert_medals),strings);

Conversation

drekdrek commented May 1, 2025

Uh oh!

drekdrek commented May 1, 2025

Uh oh!

drekdrek commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drekdrek commented May 2, 2025

Uh oh!

drekdrek commented May 2, 2025

Uh oh!

unnecessarilycomplex left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

unnecessarilycomplex Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

drekdrek commented May 1, 2025 •

edited

Loading

unnecessarilycomplex Jul 10, 2025 •

edited

Loading