After some time, seeing: Query error: MySQL server has gone away #7
Description
Every few days, we will see a whole string of messages with the "queue size" increasing by 2 bytes each second.
We attempted the usrsigkill -
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5902
Query error: MySQL server has gone away in flush peers with a qlength: 1982 queue size: 1000
Torrent flush queue size: 5902, next query length: 520
Peer flush queue size: 1000, next query length: 1982
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5904
Query error: MySQL server has gone away in flush peers with a qlength: 4824 queue size: 1000
Reloading from database
Loaded 12269 torrents
Loaded 0 tokens
Loaded 2898 users
Assuming no whitelist desired, disabling
Torrent flush queue size: 5904, next query length: 520
Peer flush queue size: 1000, next query length: 4824
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5906
Query error: MySQL server has gone away in flush peers with a qlength: 1741 queue size: 1000
Torrent flush queue size: 5906, next query length: 520
Peer flush queue size: 1000, next query length: 1741
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5908
Query error: MySQL server has gone away in flush peers with a qlength: 2751 queue size: 1000
Torrent flush queue size: 5908, next query length: 520
Peer flush queue size: 1000, next query length: 2751
But as you can see from the above, it didn't fix the problem.
Next is bouncing the tracker:
Peer flush queue size: 1000, next query length: 822
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5984
Query error: MySQL server has gone away in flush peers with a qlength: 570 queue size: 1000
Torrent flush queue size: 5984, next query length: 520
Peer flush queue size: 1000, next query length: 570
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5986
Query error: MySQL server has gone away in flush peers with a qlength: 573 queue size: 1000
Torrent flush queue size: 5986, next query length: 520
Peer flush queue size: 1000, next query length: 573
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5988
Query error: MySQL server has gone away in flush peers with a qlength: 577 queue size: 1000
Torrent flush queue size: 5988, next query length: 520
Peer flush queue size: 1000, next query length: 577
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5990
Query error: MySQL server has gone away in flush peers with a qlength: 580 queue size: 1000
Torrent flush queue size: 5990, next query length: 520
Peer flush queue size: 1000, next query length: 580
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5992
Query error: MySQL server has gone away in flush peers with a qlength: 701 queue size: 1000
^C
Session terminated, terminating shell...Caught SIGINT/SIGTERM
closing tracker... press Ctrl-C again to terminate
Torrent flush queue size: 5992, next query length: 520
Peer flush queue size: 1000, next query length: 701
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5994
Query error: MySQL server has gone away in flush peers with a qlength: 854 queue size: 1000
...killed.
And then we'll restart the process and everything will be fine until it hits a busy time with the database:
Clearing xbt_files_users and resetting peer counts...done
Loaded 2898 users
Loaded 12269 torrents
Loaded 0 tokens
Assuming no whitelist desired, disabling
Sockets up on port 34000, starting event loop!
0 open, 8 connections (2/s), 8 requests (2/s)
0 open, 277 connections (1/s), 277 requests (1/s)
0 open, 418 connections (1/s), 418 requests (1/s)
0 open, 534 connections (1/s), 534 requests (1/s)
Torrent flush queue size: 2, next query length: 374
Peer flush queue size: 1, next query length: 1076
Torrent flush queue size: 4, next query length: 374
Peer flush queue size: 2, next query length: 1076
Torrent flush queue size: 6, next query length: 374
Snatch flush queue size: 1, next query length: 97
Peer flush queue size: 3, next query length: 1076
MySQL error in flush_snatches: Too many connections
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
3 open, 708 connections (4/s), 705 requests (3/s)
Is there a way to have Ocelot reset its connections to the database?
I just attempted various mysql commands from the CLI, and inadvertently made the flush problem stop.
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
3 open, 1971 connections (15/s), 1968 requests (16/s)
0 open, 2090 connections (2/s), 2090 requests (2/s)
0 open, 2239 connections (1/s), 2239 requests (1/s)
0 open, 2351 connections (2/s), 2351 requests (2/s)
0 open, 2546 connections (3/s), 2546 requests (3/s)
0 open, 2652 connections (1/s), 2652 requests (1/s)
Reloading from database
Loaded 12271 torrents
Loaded 0 tokens
Loaded 2898 users
Assuming no whitelist desired, disabling
1 open, 2756 connections (3/s), 2755 requests (3/s)
0 open, 2924 connections (2/s), 2924 requests (2/s)
(just above is an automatic USRSIG when new torrents have been added to the database. Might that be contributing to the problem? We made it not reload anytime sooner than 5 minutes even if 5 or 10 new torrents are added quickly.)
Thanks for any advice.