Cause minetest crash (serialization issue)

[jimy-byerley]

I guy ! Your mod is a pure dream when it works !

Sometimes, when I use a ship (aroud 10m radius) it brings minetest to crash after few seconds during a flight. For now it always has happend during the first fly. After the first it seems to work always.

Here is the minetest output, (no problem before)

2017-07-01 17:22:56: ERROR[Server]: An unhandled exception occurred: String too long for serializeString
2017-07-01 17:22:56: ERROR[Server]: In thread 7f31f7d1b700:
2017-07-01 17:22:56: ERROR[Server]: /home/jimy/minetest-master/src/server.cpp:73: virtual void* ServerThread::run(): A fatal error occured: String too long for serializeString
2017-07-01 17:22:56: ERROR[Server]: Debug stacks:
2017-07-01 17:22:56: ERROR[Server]: DEBUG STACK FOR THREAD 7f31dc954700: 
2017-07-01 17:22:56: ERROR[Server]: #0  virtual void* UpdateThread::run()
2017-07-01 17:22:56: ERROR[Server]: DEBUG STACK FOR THREAD 7f31dd555700: 
2017-07-01 17:22:56: ERROR[Server]: #0  virtual void* EmergeThread::run()
2017-07-01 17:22:56: ERROR[Server]: DEBUG STACK FOR THREAD 7f31dffff700: 
2017-07-01 17:22:56: ERROR[Server]: #0  virtual void* UpdateThread::run()
2017-07-01 17:22:56: ERROR[Server]: DEBUG STACK FOR THREAD 7f31f7d1b700: 
2017-07-01 17:22:56: ERROR[Server]: #0  virtual void* ServerThread::run()
2017-07-01 17:22:56: ERROR[Server]: DEBUG STACK FOR THREAD 7f320a4087c0: 
2017-07-01 17:22:56: ERROR[Server]: #0  int main(int, char**)
2017-07-01 17:22:56: ERROR[Server]: #1  void ClientMap::renderMap(irr::video::IVideoDriver*, irr::s32)

I think it can be a error during serialization of node's metadata.

When I restart the server, The ship has disappeared ( :-<)

[stujones11]

I was not aware there was a size limit on string serialization, it would be interesting to know what that limit was then I could at least prevent a crash, though it could still result in loss of inventory items.

It may also be possible to compress the strings using zlib but I think a better long-term solution would be to move away from metadata storage to the new world storage api.

Thank you for the detailed report and sorry about your loss.

[stujones11]

Please see if this helps aa542be

Sorry I realized that was overly simplistic, I will try to post a better solution asap.

Update: 2692918 should hopefully be better, it just depends whether the compression level is sufficient or not, I have not yet been able to reproduce this error locally.

[jimy-byerley]

I just tried it but the problem is still here, and it crashed after the first use.
It seems that itsn't due to your mod: after rebooting the server I get

2017-07-03 10:28:29: WARNING[Server]: meshnode: stray object removed
2017-07-03 10:28:29: ERROR[Server]: suspiciously large amount of objects detected: 197 in (15,3,61); removing all of them.

I think it crashes because of the too large number of "meshed" blocks (that are using one entity each ?)

[jimy-byerley]

Found: minetest/util/serialize.cpp
This function is used to serialize. It stores the length of strings on 2 bytes, and raise an error for each string over 2^(2*8) = 65536. This size is used by minetest from a while to store its maps, so it's not possible to modify it without lost all existing worlds ...

[stujones11]

They are indeed using one entity each as there is currently no way to create proper voxel area entities so you need to increase max_objects_per_block to a more sensible level to prevent that error.

Thanks for the info on serialization, at least by knowing the limit I can include a check to prevent crashing, however, a better solution would be to use tables rather than strings for inventory items but I keep getting invalid key to 'next' error when I do meta:from_table()

[HybridDog]

You could save an id number to meta and use modstorage to save the ships, nodemeta is sent to clients and causes unneeded information sending. To fix the serialization problem you could serialize manually.
https://github.com/minetest/minetest/blob/master/doc/lua_api.txt#L3114

[stujones11]

Yes, mod storage is definitely the way to go in future versions but for now I think f35f481 should at least prevent crashes and does compress strings where possible, books for example could be problematic, I'm not sure.

[jimy-byerley]

I got a crash with this one too. Same error with string length, but I didn't seen any warning.
(I probably put more blocks than the last time to get the crash)

[stujones11]

Strange, I felt sure it must be a string formatted inventory item (like a book) that could possibly exceed that limit. It must be some specific node that causes this, are you using anything in particular as building materials that could possibly contain very long strings in their meta?

cdc3b1f might help to identify the culprit.

[jimy-byerley]

I tested it on a terrain piece (it's easier than building an entire ship), so dirt, stone, grass, including "plantlike" grass, leaves, tree, ...
For what I seen, that is more likely the large amount of blocks than the block type, that generates a too long string. It is strange with f35f481

How should I use cdc3b1f ?

[stujones11]

Sorry, the last commit merely compresses and checks all string data, if the problem persists then we know the issue is not with node-meta serialization. It's beginning to look that way as the building materials you mention should not contain much (if any) metadata.

Yes, it looks like the controller entity is attempting to serialize the already serialized metadata when it statically saves. I will work on a fix asap, the nodemeta compression is likely not needed after all.

[stujones11]

Hopefully e99b25a will fix the problem, or at least fail a little more gracefully if a string is too long.

[jimy-byerley]

Let see that: no difference ...

[stujones11]

As far as I can see, the only serialization (besides meta positions) is done by the controller node when statically saving. I have tried activating large volumes of landscape as you suggested but have still been unable to reproduce the serialization error, however, I have noticed some other problems with large volumes being removed along with the controller but that's another issue.

I have moved the string length check back to the serialized inventory itemstacks which is the only place I can think they could possibly exceed the limit. If none of these attempts at fixing has actually produced the log message then I wonder if the limit is actually less than 65535? I am running out of ideas now :-/