Limit size of HeronTupleSet.

[objmagic]

Fix #2234. We limit the size of HeronTupleSet. If it is larger than the maximum size, we release it back to allocator instead of memory pool.

Tested on local machine.

[srkukarni]

ByteSizeLong?

[objmagic]

I could not find ByteSizeLong API in generated code. Let me try again...

[objmagic]

seems this is introduced in 3.1.0 and then deprecated in 3.4.0

[objmagic]

also, ByteSize calculates size of serialized message, which seems will be slightly larger than actual size in memory.

[objmagic]

so if the message is 60MB but only little part of it is used to hold the incoming message (because incoming message is small), we will not delete it because I believe serialized size is much smaller than 60MB. i'm not sure about

[objmagic]

Did some experiment:

  LOG(INFO) << _message->ByteSize();
  LOG(INFO) << _message->SpaceUsed();
  _message->Clear();
  LOG(INFO) << _message->ByteSize();
  LOG(INFO) << _message->SpaceUsed();

gives us

I0828 14:10:23.306396 2502742976 stmgr-server.cpp:437] 97
I0828 14:10:23.306404 2502742976 stmgr-server.cpp:438] 832
I0828 14:10:23.306416 2502742976 stmgr-server.cpp:440] 0
I0828 14:10:23.306422 2502742976 stmgr-server.cpp:441] 832

[srkukarni]

Thats interesting obs. So ByteSize is probably more related to wire format size and SpaceUsed is related to actual in memory repr. In that case, shouldn't you use SpaceUsed? But isn;t that very slow?

[objmagic]

@srkukarni It seems that we have no choice but to use SpaceUsed here. We can do some benchmark here to measure its performance.

[huijunw]

If the performance is a concern, I would suggest the old method: put it in mempool and run a garage collection against mempool to remove large tuples every 1 min.

[objmagic]

Some benchmarking for SpaceUsed. Size is in bytes. It looks not bad @srkukarni

I0828 15:22:59.759600 2502742976 stmgr-server.cpp:439] size: 1105717
I0828 15:22:59.759625 2502742976 stmgr-server.cpp:440] time: 5 microseconds
I0828 15:22:59.967779 2502742976 stmgr-server.cpp:439] size: 1250892
I0828 15:22:59.967838 2502742976 stmgr-server.cpp:440] time: 10 microseconds
I0828 15:23:00.128224 2502742976 stmgr-server.cpp:439] size: 1250892
I0828 15:23:00.128247 2502742976 stmgr-server.cpp:440] time: 5 microseconds
I0828 15:23:00.290628 2502742976 stmgr-server.cpp:439] size: 1250892
I0828 15:23:00.290647 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:00.485448 2502742976 stmgr-server.cpp:439] size: 2150892
I0828 15:23:00.485477 2502742976 stmgr-server.cpp:440] time: 6 microseconds
I0828 15:23:00.642523 2502742976 stmgr-server.cpp:439] size: 2150892
I0828 15:23:00.642571 2502742976 stmgr-server.cpp:440] time: 5 microseconds
I0828 15:23:00.818881 2502742976 stmgr-server.cpp:439] size: 2150892
I0828 15:23:00.818902 2502742976 stmgr-server.cpp:440] time: 5 microseconds
I0828 15:23:00.970676 2502742976 stmgr-server.cpp:439] size: 3100892
I0828 15:23:00.970695 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:01.183621 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.183640 2502742976 stmgr-server.cpp:440] time: 3 microseconds
I0828 15:23:01.350108 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.350129 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:01.512719 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.512740 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:01.721442 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.721462 2502742976 stmgr-server.cpp:440] time: 3 microseconds
I0828 15:23:01.896715 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.896742 2502742976 stmgr-server.cpp:440] time: 7 microseconds
I0828 15:23:02.010160 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.010188 2502742976 stmgr-server.cpp:440] time: 6 microseconds
I0828 15:23:02.057590 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.057623 2502742976 stmgr-server.cpp:440] time: 6 microseconds
I0828 15:23:02.272822 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.272848 2502742976 stmgr-server.cpp:440] time: 6 microseconds
I0828 15:23:02.437445 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.437464 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:02.597956 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.598012 2502742976 stmgr-server.cpp:440] time: 8 microseconds

[srkukarni]

Could you also share the thruput figures? Particularly in exclamation or other topologies that are used for Heron benchmarking?

[objmagic]

word count, parallelism=20

Stmgr CPU user time doubled, Data Tuples from Instances dropped ~25%.

We need to fix #1908 first to see more metrics.