Possible serverChunkSize un-awareness of lazyFile's speedup system incompatible with rangemapper db chunking.

[nelsonjchen]

I have a hacked up copy/paste/extraction of the lazyFile in this project put into my test lab project. I think it's possible there's a bug upstream here where the speed stuff and using the rangemapper functionality with split db chunks can sometimes result in the rangeMapper being called with absolute bytes that span server DB chunks. The reasoning for this is that the head/speed stuff can sometimes increase the next read to span the DB chunks and the rangemapper would not be able to properly map a URL.

I'm sorry, my stuff is so hacked up, that by all means, if you want to just close this, I'm fine with that since this is one of those crazy downstream people hacking it too far sort of issues. I did try to fix it and this commit of mine seems to do it. It will not try to call rangemapper with something itself could not resolve as lazyFile is now aware of the serverChunkSize and will not speedup if the next wantedChunk spans to another DB chunk.

Here's the commit:

nelsonjchen/datasette-lite@ccdc574

Without this, it seems to try to get something from the 11MB mark of a 10MB chunk for my project which I've fixed:

https://datasette-lite-lab.mindflakes.com/index.html?url=https://datasette-lite-lab.mindflakes.com/sdb/2022-10-02_93eff57de3573985_ca_unclaimed_property.sqlite#/2022-10-02_93eff57de3573985_ca_unclaimed_property?sql=SELECT+*+FROM+records+WHERE+records.Owner_name+MATCH+%0A%22charles+munger%22%0AORDER+BY+cast%28current_cash_balance+AS+float%29+DESC%3B

This commit itself may be buggy, but I just wanted to note this here. I'm 80% certain this isn't handled at the higher level SQLite worker and that this issue could still happen with this project's lazyFile. I'm only using this project's lazyFile, but I'm pretty sure my use of it would be similar to this project's sqlite worker.

My guess on the best way to reproduce this is to probably try to do many reads of a contiguous block near the end of a server chunk boundary until the speedup results in rangeMapper being called with something that it could not return a reasonable from, to and url structure where to is beyond the file size of the server chunk.

[phiresky]

Hey, thank you for reporting this! Good find. I'd say it's fairly likely that this bug applies here too, though I'm not sure why I haven't encountered it then (maybe because it's just somewhat unlikely to hit with large server-chunk sizes and small page sizes).

I'm not really actively working on this project anymore so if someone else hits this with the code in this repo itself I'm happy for a PR, I'll leave it open in the meanwhile. I had since continued on reusing / repurposing the code in here but the goal there was somewhat different. I think the concept of the "lazyFile" thing with the chunk size heuristics is pretty useful for other applications so I'm happy to see different uses ;)

[nelsonjchen]

No kidding, I think it was extracted out to this:

https://github.com/bmaranville/lazyFileLRU/blob/main/src/lazyFileLRU.ts

They added a LRU, so it can be configured to not grow until the page dies from being closed or memory is exhausted.

I think also might have this bug. Which I'll report too.

[nelsonjchen]

Other stuff too like speed-up but going down in opposite direction might be neat as well.