Fatal JavaScript invalid size error 169220804

[RezaAb]

I got this error in my app with a large object that includes arrays of arrays.
The error closed the app even by error handling.

nodejs version: 16.20

#
# Fatal error in , line 0
# Fatal JavaScript invalid size error 169220804
#
#
#
#FailureMessage Object: 0x7ffc7697ca00
 1: 0xb72001  [node]
 2: 0x1bfb174 V8_Fatal(char const*, ...) [node]
 3: 0xe67af8  [node]
 4: 0x1015422  [node]
 5: 0x1015d66  [node]
 6: 0x11d6543 v8::internal::Runtime_GrowArrayElements(int, unsigned long*, v8::internal::Isolate*) [node]
 7: 0x15d9e19  [node]

[miktam]

Try increasing the heap memory available to your app by using the --max-old-space-size flag, like node --max-old-space-size=4096 yourScript.js. The value (4096) is in megabytes.

[RezaAb]

@miktam Thanks for your response.
My current heap size is big enough.
heap_size_limit: 4345298944,
My object size was not around this value. (Maybe around 100Mb)
I didn’t have any problems like this until I used the package.

[miktam]

@RezaAb, thank you for the update!

Heavily nested arrays will cause the module to throw, as the engine can not allocate memory for a huge data structure with the nested arrays.

This code crashes tests:

describe('Testing with Large Objects - Arrays of Arrays with Huge Strings', () => {
  it.only('should not crash the app when handling large nested arrays with huge strings', function () {
    // Optional: Increase the timeout for this test as it will likely take a while.timeout(30000)

    // Create large nested arrays with huge strings
    const largeNestedArray = []
    const hugeString = 'x'.repeat(1000) // A string with 10,000 characters

    for (let i = 0; i < 500; i++) {
      let currentArray = largeNestedArray
      for (let j = 0; j < 500; j++) {
        const newArray = [hugeString]
        currentArray.push(newArray)
        currentArray = newArray
      }
    }

    // Test that this doesn't crash the app
    // Note: Depending on the machine this test is run on, this might still crash
    sizeof(largeNestedArray).should.be.equal(-1)
  })
})

Regarding remediation, the only way I see at the moment is to look at the object first and nicely return -1 when the object's depth exceeds a certain threshold. At least it will not crash the invoking app.

This does not feel like an ideal solution. Are there any better ways?

[bjwktcsnzh]

big-json can be used to calculate the size of JSON objects @miktam

https://stackoverflow.com/a/72005609/21185704