Large Lua string allocation/concatenations crashes Lua environment.

[tlubke]

Doing iterative string building like this can lead to issues.

local n = 9

for i=1,n do
  local s = ""
  for i=1,(2^i) do
    s = s.."!"
  end
end

For n = 9, this is fine, but n = 10 appears to hang something in the firmware. LEDs will hold their state, diii continues to show as connected, but all code execution seems to stop. Soft reboot via ^^r doesn't work, but physically reconnecting seems to fix things.

Interesting to note that building one constant string of size 2^10 does not cause the same behavior. Maybe the strings spawned by concatenation are not garbage collected fast enough and cause an out-of-memory failure because of it?

EDIT: Quick calc using the summation formula would mean n=9 => 131,238B and n=10 => 524,800B assuming nothing gets free'd before the end of the loop. n=10 would be more than available SRAM of the Pico 2.

EDIT 2: Yeah, I think garbage collection is the issue. Adding collectgarbage("collect") to the inner for-loop fixes the crash. I don't think this allocation/collection problem is realistically something to fix, but would be nice to keep the Lua environment running if possible.

[tehn]

thanks for taking a look.

i need to do some real benchmarking of the memory limits within this system. will see if i can put in some introspection utilities for better understanding what's going on.

[tehn]

heap has about 220k so i think your assessment makes sense!

i'll update the README to indicate the memory limitations. let me know if you'd like this issue reopened, if there is a proposal for a structural change.

[catfact]

i think this can be fixed, and is worth fixing. i would try using a custom allocator that errors gracefully rather than panicking...

[robbielyman]

apologies in advance—without the source in the repo, it's quite possible this is worse than useless as a basically drive-by suggestion. as a first try, i might see what happens with something like this as an allocator function?

static void *iii_alloc (void *ud, void *ptr, size_t osize,
                                                size_t nsize) {
  lua_State *l = (lua_State *)ud;
  (void)osize;  /* not used */
  if (nsize == 0) {
    free(ptr);
    return NULL;
  } else {
    void *ret = realloc(ptr, nsize);
    if (ret == NULL) {
      lua_gc(l, LUA_GCOLLECT);
      return realloc(ptr, nsize);
    }
    return ret;
  }
}

you'd set it passing the lua_State pointer to lua_setallocf as the userdata. unclear to me whether lua_gc has bad interactions with in-progress allocation, but it might be worth a shot?

[catfact]

thanks for the suggestion! but, calling GC here doesn't realy help (and might deadlock.) we need to properly raise a memory error, and then (the tricky part) handle / communicate it in a useful way.

[beels]

Just to add a data point to this issue: I'm working on a port of the Descartes norns script to iii, and have bumped into similar problems. In my case, there were no string allocations, just insertions and deletions from tables. After not long (800 sequencer steps, driven by two separate per-track metros) the grid would run out of memory.

Three things fixed the issue for me, independently:

• adding print calls to debug the script. Something about print triggers garbage collection?
• calling collectgarbage("step") once every 100 sequencer steps. "collect" was not necessary.
• for some variants of the script, turning on a very aggressive collectgarbage("generational", 5, 20) did the trick. For other variants, it did not, no doubt because of differing patterns of memory allocation. No level of aggressiveness with "incremental" did the trick for me without also explicitly triggering GC steps as described above.

[tehn]

@beels thank you, this is helpful feedback. definitely memory is more tricky to deal with on the rp2040. i'm hoping to develop some guidance around the limitations.

[tehn]

Spent awhile on this without success. Here's a custom allocator:

https://codeberg.org/tehn/iii/src/branch/main/vm.c#L348

I've been actively working on this branch: https://codeberg.org/tehn/iii-grid-one

Basically continue to hit lockup without getting a serial warning, so it's difficult to ascertain at what point the lockup is occurring.

[Dewb]

A perhaps naive question: does the hardware/platform have a flag set that causes it to automatically panic on allocation error? (I have seen this with other MCUs but no direct experience with rp2040)

[tehn]

Lua itself has a lua_atpanic function that I couldn't get working meaningfully.

Might be worth implementing a watchdog timer, though I believe that simply resets the device (which is probably better than lockup)?

But really I don't think we should be running out of memory so easily. I need to find/commission someone more familiar with this little corner of lua.

[Dewb]

Aha, it appears that panic on malloc fail is a Pico SDK default behavior?

https://github.com/raspberrypi/pico-sdk/blob/a1438dff1d38bd9c65dbd693f0e5db4b9ae91779/src/rp2_common/pico_malloc/include/pico/malloc.h#L23

If so, then Lua code (or user C code) is never going to get a chance to react to the error.

So perhaps you want to #define PICO_MALLOC_PANIC 0 (and maybe also #define PICO_DEBUG_MALLOC 1?)

[tehn]

oh ho good find @Dewb