Next steps after conditional DBN <= 0.11

[km-git-acc]

@rudolph-git-acc
Opening the new thread..
Agree dbn < 0.1 is a good place to stop. I think the tloop part could potentially also be passed through Boinc. Will try out on a smaller X to see how it goes (essentially apart from t1,t2,y parameters (appropriately chosen for each job) we would also pass an additional parameter to the boinc job - the web address of the relevant storedsums file).

I will check if there are readymade scripts for RH verification to help in detailed understanding of the process involved. The one at https://www.wstein.org/simuw/misc/zeta_grid.html seems to be a dead link. Is Gourdon-Demichel's code available?

[rudolph-git-acc]

@km-git-acc

Great. Have done extensive testing on the tloop script today and found that at heights > 10^19 and with a larger number of threads (>12) it started to hit memory issues. The root cause was that for sarr, zarr, bexpo, afac, bsums etc., I used matrices that were initialised for each thread with range 1..num (i.e. the number of mesh points on one side of the rectangle). Then realised that matrices weren't actually required (except for ests, that is of size '4xnum-4, to which the argument principle is applied at the end) and replaced these by simple variables. All memory issue are gone and tloop now runs another factor 1.5 faster than before :) Will upload the new version to Github.

Even in case ests becomes too large for memory (no issues yet), I do have a routine to apply the argument principle directly to each two subsequent mesh points (except for 4 'gaps' that I need to evaluate at the end of the rectangle process). So, I might be able to do without matrices all together if push comes to shove.

You are correct that the tloop process could potentially be 'Boinc-ed', however there are many connections between the process steps to be handled with caution. Only after a rectangle has been completely processed, we know the 'minmodabb' and that in turn is required to establish the next t-step hence determines the t for the next rectangle. Just splitting the range 0..t_0 into smaller steps might therefore cause problems, right?

Will also search for routines to numerically verify the RH. I'd thought that these all boil down to counting the zeros on the critical line (using the arg-function to count sign changes) and then applying the argument principle (+Rouhé) to verify the rectangle 0..1/2, 0..iT and comparing the two outcomes. But there are probably different and smarter ways to do this for larger heights

[km-git-acc]

@rudolph-git-acc
Indeed, i checked the new script even with a single thread and it was slightly faster.

In the tloop step, for a given t range [tstart,tend], tstart does not rely on minmodabb values, which are used only for determining subsequent t's. For eg. this is the output generated by splitting the 6x10e10 tloop into 4 parts.

docker run dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/SingleStoredSums_x60000155019_dig_20.txt -O sss.txt; ./tloopsingle 0.00 0.02 0.2 0 sss.txt";
docker run dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/SingleStoredSums_x60000155019_dig_20.txt -O sss.txt; ./tloopsingle 0.02 0.05 0.2 0 sss.txt";
docker run dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/SingleStoredSums_x60000155019_dig_20.txt -O sss.txt; ./tloopsingle 0.05 0.10 0.2 0 sss.txt";
docker run dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/SingleStoredSums_x60000155019_dig_20.txt -O sss.txt; ./tloopsingle 0.10 0.20 0.2 0 sss.txt";

Rectangle(1) : 0, 12880, 2770, 0.000000, 4.6295791526613631141, 11076
Rectangle(2) : 0.00032061949943023005544, 12758, 2744, 0.000000, 4.614940433284710646, 10972
Rectangle(3) : 0.0006431575487549447949, 12637, 2717, 0.000000, 4.6002924173972816219, 10864
Rectangle(4) : 0.00096762478127827166219, 12517, 2691, 0.000000, 4.5856355203733802906, 10760
Rectangle(5) : 0.0012940317094857798743, 12397, 2666, 0.000000, 4.5709701672512488026, 10660
Rectangle(6) : 0.0016224152028512109304, 12277, 2640, 0.000000, 4.5562956124780297, 10556
Rectangle(7) : 0.0019528131512488674996, 12158, 2614, 0.000000, 4.5416110947562938455, 10452
Rectangle(8) : 0.0022852371597006107012, 12039, 2589, 0.000000, 4.5269170418175539788, 10352
Rectangle(9) : 0.0026197264895301276028, 11921, 2563, 0.000000, 4.5122126737156098699, 10248
Rectangle(10) : 0.0029562932770240970576, 11803, 2538, 0.000000, 4.4974984245255042342, 10148
Rectangle(11) : 0.0032949782236076354999, 11686, 2512, 0.000000, 4.4827734949074217703, 10044
Rectangle(12) : 0.003635794028408886721, 11569, 2487, 0.000000, 4.468038325386630336, 9944
Rectangle(13) : 0.0039787829060462478011, 11452, 2462, 0.000000, 4.4532920964433414579, 9844
Rectangle(14) : 0.0043239882934408811803, 11336, 2437, 0.000000, 4.4385339702129262533, 9744
Rectangle(15) : 0.0046714242470588175117, 11221, 2412, 0.000000, 4.4237643890564660553, 9644
Rectangle(16) : 0.0050211047023708633236, 11105, 2387, 0.000000, 4.408983806720157697, 9544
Rectangle(17) : 0.0053731068461547586588, 10991, 2363, 0.000000, 4.394190032321037396, 9448
Rectangle(18) : 0.0057274140094994076804, 10877, 2338, 0.000000, 4.3793848524889713264, 9348
Rectangle(19) : 0.0060840734608636598939, 10763, 2313, 0.000000, 4.3645673815044667182, 9248
Rectangle(20) : 0.0064431338883935741109, 10649, 2289, 0.000000, 4.349736712950571079, 9152
Rectangle(21) : 0.0068046454587711279732, 10536, 2265, 0.000000, 4.3348919188494100758, 9056
Rectangle(22) : 0.0071686253295806771451, 10424, 2240, 0.000000, 4.3200334549891288049, 8956
Rectangle(23) : 0.0075350905497446380819, 10312, 2216, 0.000000, 4.3051617898216623318, 8860
Rectangle(24) : 0.0079040938264922779512, 10200, 2192, 0.000000, 4.2902759656198110106, 8764
Rectangle(25) : 0.0082756895093961809914, 10089, 2168, 0.000000, 4.2753750011789116258, 8668
Rectangle(26) : 0.0086498965667040322775, 9979, 2144, 0.000000, 4.2604593649000756862, 8572
Rectangle(27) : 0.0090267338615131389702, 9869, 2121, 0.000000, 4.2455295386920883992, 8480
Rectangle(28) : 0.0094062585893165727932, 9759, 2097, 0.000000, 4.2305845086894495099, 8384
Rectangle(29) : 0.0097885297757792687159, 9650, 2074, 0.000000, 4.215007805813597432, 8292
Rectangle(30) : 0.010173504677936118191, 9541, 2050, 0.000000, 4.1982387825648442214, 8196
Rectangle(31) : 0.010561120104260910584, 9432, 2027, 0.000000, 4.1814734530114327098, 8104
Rectangle(32) : 0.010951437476293505231, 9325, 2004, 0.000000, 4.164710611339387204, 8012
Rectangle(33) : 0.011344435933273600373, 9217, 1980, 0.000000, 4.1479526075106240624, 7916
Rectangle(34) : 0.011740221178745079603, 9111, 1957, 0.000000, 4.1311964114021889107, 7824
Rectangle(35) : 0.012138771986713709711, 9004, 1935, 0.000000, 4.1144444216005460327, 7736
Rectangle(36) : 0.012540198510658683726, 8899, 1912, 0.000000, 4.0976935323640538815, 7644
Rectangle(37) : 0.012944479163806683937, 8793, 1889, 0.000000, 4.0809461925901332627, 7552
Rectangle(38) : 0.013351728816097157412, 8689, 1867, 0.000000, 4.0641992174345833147, 7464
Rectangle(39) : 0.013761925526585658196, 8584, 1844, 0.000000, 4.0474551078718644905, 7372
Rectangle(40) : 0.014175189169161597676, 8481, 1822, 0.000000, 4.0307105962423514738, 7284
Rectangle(41) : 0.014591497457835380422, 8377, 1800, 0.000000, 4.0139682374478265404, 7196
Rectangle(42) : 0.015010975580963926028, 8275, 1777, 0.000000, 3.9972246770515417588, 7104
Rectangle(43) : 0.015433600919580426543, 8172, 1755, 0.000000, 3.9804825258218744988, 7016
Rectangle(44) : 0.015859504312363328464, 8071, 1733, 0.000000, 3.9637383383124382356, 6928
Rectangle(45) : 0.016288662822871622137, 7970, 1712, 0.000000, 3.94699478319153939, 6844
Rectangle(46) : 0.016721159031553120178, 7869, 1690, 0.000000, 3.9302504125623896801, 6756
Rectangle(47) : 0.017157078514659282293, 7769, 1668, 0.000000, 3.9135037425705781281, 6668
Rectangle(48) : 0.01759645343325505757, 7669, 1647, 0.000000, 3.8967553991424493332, 6584
Rectangle(49) : 0.018039373683501823685, 7570, 1626, 0.000000, 3.8800038505658406127, 6500
Rectangle(50) : 0.018485873531661115708, 7471, 1604, 0.000000, 3.8632497298611131823, 6412
Rectangle(51) : 0.018936047501659926199, 7373, 1583, 0.000000, 3.8464914546481039986, 6328
Rectangle(52) : 0.01938993214219269495, 7275, 1562, 0.000000, 3.8297296647830023536, 6244
Rectangle(53) : 0.019847626941475581871, 7178, 1541, 0.000000, 3.8129627241766841528, 6160
------------------------------------------------------------------------------------------------------------
Rectangle(1) : 0.02, 7146, 1534, 0.000000, 3.8074107470722755211, 6132
Rectangle(2) : 0.02046283385769273377, 7050, 1514, 0.000000, 3.790637706543950258, 6052
Rectangle(3) : 0.020929590979188329551, 6954, 1493, 0.000000, 3.7738602682577860282, 5968
Rectangle(4) : 0.021400379053427297919, 6858, 1473, 0.000000, 3.7570767113976133805, 5888
Rectangle(5) : 0.021875310040799361696, 6763, 1452, 0.000000, 3.7402852664502645934, 5804
Rectangle(6) : 0.022354429553803984584, 6669, 1432, 0.000000, 3.7234865863117914924, 5724
Rectangle(7) : 0.022837783367915814167, 6575, 1412, 0.000000, 3.7066813453853006423, 5644
Rectangle(8) : 0.023325491557328027193, 6482, 1392, 0.000000, 3.6898676942278805828, 5564
Rectangle(9) : 0.023817603204077160266, 6389, 1372, 0.000000, 3.6730463127283116076, 5484
Rectangle(10) : 0.024314245294033070676, 6296, 1352, 0.000000, 3.6562152811298696591, 5404
Rectangle(11) : 0.024815550135381525196, 6205, 1332, 0.000000, 3.6393726210105860333, 5324
Rectangle(12) : 0.025321492540056881527, 6113, 1312, 0.000000, 3.6225217011948328452, 5244
Rectangle(13) : 0.025832292752913880193, 6022, 1293, 0.000000, 3.6056577634379342173, 5168
Rectangle(14) : 0.026348011411737848014, 5932, 1273, 0.000000, 3.5887814744038446653, 5088
Rectangle(15) : 0.026868709569931348463, 5842, 1254, 0.000000, 3.5718935256877936742, 5012
Rectangle(16) : 0.027394538656834428517, 5753, 1235, 0.000000, 3.5549917550062233717, 4936
Rectangle(17) : 0.027925564513779713303, 5664, 1216, 0.000000, 3.5380768558167503903, 4860
Rectangle(18) : 0.028461948139453574601, 5576, 1197, 0.000000, 3.5211465734047317206, 4784
Rectangle(19) : 0.029003760652617981294, 5488, 1178, 0.000000, 3.5042016017305627769, 4708
Rectangle(20) : 0.029551173480921655269, 5401, 1159, 0.000000, 3.4872395866818605919, 4632
Rectangle(21) : 0.030104263572882747769, 5314, 1140, 0.000000, 3.4702612211139438986, 4556
Rectangle(22) : 0.030663213746219959643, 5228, 1122, 0.000000, 3.4532640438139307031, 4484
Rectangle(23) : 0.031228107404179778064, 5142, 1103, 0.000000, 3.4362487446321530652, 4408
Rectangle(24) : 0.031799139832151803164, 5057, 1085, 0.000000, 3.4192127467155946915, 4336
Rectangle(25) : 0.032376401597371418487, 4973, 1067, 0.000000, 3.4021567346652582991, 4264
Rectangle(26) : 0.03295998429084925044, 4889, 1049, 0.000000, 3.3850814239875312095, 4192
Rectangle(27) : 0.033550101170372165394, 4805, 1031, 0.000000, 3.3679840831599296477, 4120
Rectangle(28) : 0.034146976109635418182, 4722, 1013, 0.000000, 3.3508618762058697134, 4048
Rectangle(29) : 0.034750716447671392285, 4639, 995, 0.000000, 3.3337154691257773319, 3976
Rectangle(30) : 0.035361562636316742001, 4557, 978, 0.000000, 3.3165418785168438326, 3908
Rectangle(31) : 0.035979631953524739331, 4476, 960, 0.000000, 3.2993417556940263842, 3836
Rectangle(32) : 0.036605043427093556664, 4395, 943, 0.000000, 3.2821157860383463611, 3768
Rectangle(33) : 0.037238061808444716245, 4314, 926, 0.000000, 3.2648607858067105606, 3700
Rectangle(34) : 0.037878966023977101633, 4234, 908, 0.000000, 3.2475734374977726163, 3628
Rectangle(35) : 0.038527896925606240182, 4155, 891, 0.000000, 3.2302543495994149576, 3560
Rectangle(36) : 0.039184997852104294323, 4076, 874, 0.000000, 3.2129041665502009853, 3492
Rectangle(37) : 0.039850577873338396626, 3998, 858, 0.000000, 3.1955193294807797031, 3428
Rectangle(38) : 0.040524794814179011859, 3920, 841, 0.000000, 3.1781004538383664706, 3360
Rectangle(39) : 0.041207983705464309428, 3843, 824, 0.000000, 3.1606437662229190217, 3292
Rectangle(40) : 0.041900318799459344302, 3766, 808, 0.000000, 3.1431498457654896447, 3228
Rectangle(41) : 0.042602164217878221, 3690, 791, 0.000000, 3.1256146823394191948, 3160
Rectangle(42) : 0.043313712912279147611, 3614, 775, 0.000000, 3.1080388099852412452, 3096
Rectangle(43) : 0.044035361725224704126, 3539, 759, 0.000000, 3.0904179564319201859, 3032
Rectangle(44) : 0.04476732497937331113, 3465, 743, 0.000000, 3.0727526000857139908, 2968
Rectangle(45) : 0.04550982212225519107, 3391, 727, 0.000000, 3.0550432594079790701, 2904
Rectangle(46) : 0.046263299933935515184, 3317, 711, 0.000000, 3.0372852913654193306, 2840
Rectangle(47) : 0.04702823369678309412, 3244, 696, 0.000000, 3.0194738113833471357, 2780
Rectangle(48) : 0.047804890235433940955, 3172, 680, 0.000000, 3.0016091471252901391, 2716
Rectangle(49) : 0.048593543812711775173, 3100, 665, 0.000000, 2.9836916652765423449, 2656
Rectangle(50) : 0.049394734672478401736, 3029, 649, 0.000000, 2.9657160138628231997, 2592
------------------------------------------------------------------------------------------------------------
Rectangle(1) : 0.05, 2976, 638, 0.000000, 2.9522854869097730149, 2548
Rectangle(2) : 0.050824020660923982868, 2906, 623, 0.000000, 2.9342044506715830815, 2488
Rectangle(3) : 0.05166166844160931772, 2836, 608, 0.000000, 2.9160614186774491472, 2428
Rectangle(4) : 0.052513594188674708816, 2767, 593, 0.000000, 2.8978502361952538818, 2368
Rectangle(5) : 0.053380182636884052467, 2698, 579, 0.000000, 2.8795709396590789615, 2312
Rectangle(6) : 0.054262158522599055788, 2630, 564, 0.000000, 2.8612168467459935979, 2252
Rectangle(7) : 0.055159959605011981111, 2563, 549, 0.000000, 2.8427878135224122781, 2192
Rectangle(8) : 0.056074039906815497411, 2496, 535, 0.000000, 2.8242837259674491114, 2136
Rectangle(9) : 0.057005243322667840805, 2430, 521, 0.000000, 2.8056971120346269447, 2080
Rectangle(10) : 0.057954089870830238725, 2364, 507, 0.000000, 2.787027630540792851, 2024
Rectangle(11) : 0.05892152964686272301, 2299, 493, 0.000000, 2.7682670902709444841, 1968
Rectangle(12) : 0.059908161700047126875, 2234, 479, 0.000000, 2.7494148680444493168, 1912
Rectangle(13) : 0.060915061820031213409, 2170, 465, 0.000000, 2.7304619372591331505, 1856
Rectangle(14) : 0.06194292446392943145, 2107, 452, 0.000000, 2.7114073231610684564, 1804
Rectangle(15) : 0.062992477061537911971, 2044, 438, 0.000000, 2.6922500466767215669, 1748
Rectangle(16) : 0.064065006438581317825, 1982, 425, 0.000000, 2.6729797774086914542, 1696
Rectangle(17) : 0.065161363541209315532, 1920, 412, 0.000000, 2.6535950781737507564, 1644
Rectangle(18) : 0.066283027644424810718, 1859, 399, 0.000000, 2.6340844103956137285, 1592
Rectangle(19) : 0.06743100204485279013, 1799, 386, 0.000000, 2.6144457568995753746, 1540
Rectangle(20) : 0.068606347101495133307, 1739, 373, 0.000000, 2.5946770359366546042, 1488
Rectangle(21) : 0.069810876736881363672, 1680, 360, 0.000000, 2.5747647538781468191, 1436
Rectangle(22) : 0.071045855757046927254, 1621, 348, 0.000000, 2.5547060485509722846, 1388
Rectangle(23) : 0.072313410382926614043, 1564, 335, 0.000000, 2.5344855811097456159, 1336
Rectangle(24) : 0.073614232365733356764, 1506, 323, 0.000000, 2.5141123937629420219, 1288
Rectangle(25) : 0.074951624393464393963, 1450, 311, 0.000000, 2.4935562093242084437, 1240
Rectangle(26) : 0.076326490744722468752, 1394, 299, 0.000000, 2.4728260224807039901, 1192
Rectangle(27) : 0.077741717446645498874, 1339, 287, 0.000000, 2.451902447643588012, 1144
Rectangle(28) : 0.079199448923601128457, 1284, 276, 0.000000, 2.430779665076989285, 1100
Rectangle(29) : 0.080703171404190683975, 1230, 264, 0.000000, 2.4094350352041349598, 1052
Rectangle(30) : 0.082255557611673720528, 1177, 253, 0.000000, 2.38786069368305136, 1008
Rectangle(31) : 0.08385951741939084148, 1125, 242, 0.000000, 2.3660481794414819034, 964
Rectangle(32) : 0.085518226912227714283, 1073, 230, 0.000000, 2.343988366430657, 916
Rectangle(33) : 0.087236762202470637861, 1022, 220, 0.000000, 2.3216508269056636167, 876
Rectangle(34) : 0.089019199410793205001, 971, 209, 0.000000, 2.2990230407708591685, 832
Rectangle(35) : 0.090871952284913554299, 922, 198, 0.000000, 2.2760679883587006346, 788
Rectangle(36) : 0.092798273313502166701, 873, 188, 0.000000, 2.2527932834293598402, 748
Rectangle(37) : 0.094806054852367412795, 825, 178, 0.000000, 2.2291555917080474073, 708
Rectangle(38) : 0.096902001024134742986, 778, 167, 0.000000, 2.205133652159038079, 664
Rectangle(39) : 0.099093689523053815067, 731, 158, 0.000000, 2.1807036973877882761, 628
-------------------------------------------------------------------------------------------------------
Rectangle(1) : 0.1, 713, 154, 0.000000, 2.1708014191620375438, 612
Rectangle(2) : 0.10234333999882473709, 668, 144, 0.000000, 2.1457246061758069186, 572
Rectangle(3) : 0.10480699958890828038, 623, 134, 0.000000, 2.1201533731715991913, 532
Rectangle(4) : 0.10740756680106173012, 580, 125, 0.000000, 2.0940080337280439738, 496
Rectangle(5) : 0.11015585651438594387, 537, 116, 0.000000, 2.0672835444069747591, 460
Rectangle(6) : 0.11307444784475274976, 495, 107, 0.000000, 2.0398780066455815685, 424
Rectangle(7) : 0.11618531250464281353, 454, 99, 0.000000, 2.0117221742296555833, 392
Rectangle(8) : 0.11951509702937773772, 415, 90, 0.000000, 1.9827332721171080344, 356
Rectangle(9) : 0.12308794828749125106, 376, 82, 0.000000, 1.952881624089661062, 324
Rectangle(10) : 0.12695199516007013686, 338, 74, 0.000000, 1.9219786885750103792, 292
Rectangle(11) : 0.13115903270023288946, 302, 66, 0.000000, 1.8898714959262929079, 260
Rectangle(12) : 0.13576125619667756797, 267, 59, 0.000000, 1.8564726636159636002, 232
Rectangle(13) : 0.14084167815778604587, 233, 51, 0.000000, 1.8215572003906827267, 200
Rectangle(14) : 0.1465135974727675168, 200, 45, 0.000000, 1.7848250115559063391, 176
Rectangle(15) : 0.15293772253054704849, 169, 38, 0.000000, 1.7458613952661560903, 148
Rectangle(16) : 0.16030968344928169992, 140, 32, 0.000000, 1.7043045866414617366, 124
Rectangle(17) : 0.16891185906814928375, 113, 26, 0.000000, 1.6596632676841007873, 100
Rectangle(18) : 0.17917436586181389249, 88, 21, 0.000000, 1.6112351926148756764, 80
Rectangle(19) : 0.19180203850516475245, 65, 16, 0.000000, 1.557945213295379675, 60

I have now submitted 4 Boinc jobs to test this out in the grid as well

bin/boinc2docker_create_work.py --min_quorum 1 --target_nresults 1 dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/SingleStoredSums_x60000155019_dig_20.txt -O sss.txt; ./tloopsingle 0.00 0.02 0.2 0 sss.txt > /root/shared/results/tloopout_X_6x10e10_t_0.00_0.02_y_0.2.txt";  
bin/boinc2docker_create_work.py --min_quorum 1 --target_nresults 1 dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/SingleStoredSums_x60000155019_dig_20.txt -O sss.txt; ./tloopsingle 0.02 0.05 0.2 0 sss.txt > /root/shared/results/tloopout_X_6x10e10_t_0.02_0.05_y_0.2.txt";  
bin/boinc2docker_create_work.py --min_quorum 1 --target_nresults 1 dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/SingleStoredSums_x60000155019_dig_20.txt -O sss.txt; ./tloopsingle 0.05 0.10 0.2 0 sss.txt > /root/shared/results/tloopout_X_6x10e10_t_0.05_0.10_y_0.2.txt";  
bin/boinc2docker_create_work.py --min_quorum 1 --target_nresults 1 dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/SingleStoredSums_x60000155019_dig_20.txt -O sss.txt; ./tloopsingle 0.10 0.20 0.2 0 sss.txt > /root/shared/results/tloopout_X_6x10e10_t_0.10_0.20_y_0.2.txt";  

One thing I have noticed is that it is now quite difficult for a non-aggressive client to get assigned jobs, since the aggressive clients likely search for new jobs very frequently (maybe every few seconds), and maintain a healthy queue on their machines.

5000 subtotal matrices for the 9x10e21 storedsums are already complete, so this shouldn't take long now. 2x10e20 storedsums has about 300 subtotals pending, which should also likely complete by today or tomorrow.

I have the same impression about RH verification. While the algorithm used by Gourdon-Demichel has been described in detail in
http://www.dtc.umn.edu/~odlyzko/unpublished/zeta.10to20.1992.pdf and
zetazeros1e13-1e24.pdf
what is missing are the links to the codes used.
It also seems that Platt (dave.platt@bris.ac.uk) (in https://research-information.bristol.ac.uk/files/78836669/platt_zeta_submitted.pdf
used interval arithmetic and the method used is more rigorous. It's possible the second method is slower than the first. Once we have decided which method to use, we could contact them and request for the code.

[rudolph-git-acc]

@km-git-acc

Great. You could indeed split up the rectangle processes this way, however the trick will be to allocate the work to client in an exponential way. The first t-steps are very small and the process greatly speeds up at the end. We therefore need some smart way to 'cut the cake' efficiently. The table below lists all data from the last t-loop runs:

X	Barrier offset	t0	y0	Λ	Starting 'num'	Number of rectangles	Time (secs)
6.00E+11	183290	0.2000	0.20000	0.22	19388	242	25
2.00E+12	129093	0.1980	0.15492	0.21	34980	384	74
5.00E+12	194858	0.1860	0.16733	0.20	40344	435	99
2.00E+13	131252	0.1800	0.14142	0.19	68376	689	280
6.00E+13	123375	0.1680	0.15492	0.18	81236	803	401
3.00E+14	188911	0.1610	0.13416	0.17	142012	1374	1241
2.00E+15	122014	0.1530	0.11832	0.16	259748	2239	3915
7.00E+15	68886	0.1390	0.14832	0.15	273548	2174	3993
6.00E+16	156984	0.1320	0.12649	0.14	562360	3949	16213
6.00E+17	88525	0.1220	0.12649	0.13	988908	6066	44170
9.00E+18	35785	0.1130	0.11832	0.12	2083348	12947	241003
2.00E+20	66447	0.1020	0.12649	0.11	?	?	?
9.00E+21	70686	0.0930	0.11832	0.10	?	?	?

Have not been yet been able to spot a predictive pattern to be able to predict what will happen in the next round, but a safe bet is to expect the number of triangles will double at least. We could also calculate the 'starting num' as an additional data point once the stored sums are ready and use that as an indicator. What is it that drives the number of rectangles needed? I guess this has something to do with the quality of the Barrier location choice (i.e. the height of the 'peak' of ABB_eff at that point), however went back to my selection data (attached) and this doen't explain the drop in rectangles 2x10^15 and 7x10^15 (EDIT: but the higher y might!)

BarrierlocationselectionNoTriangle.txt

I also noticed that I got kicked out of Boinc, since there were not tasks left 'ready to send'. Guess some of the high performing clients indeed grabbed a substantial chunk from the workload to create a solid buffer. The speed of progress is impressive though.

Also read the Gourdon paper and there are quite a few nuances to consider that could substantially impact speed (hence having the (pseudo) code would be great). Surprised to see that Odlysko actually already calculated a couple of billion zeros in the 10^23 domain in the early 2000's (though not published).

[km-git-acc]

@rudolph-git-acc
Yeah, number of rectangles required is dependent on the ddtbound (dependent on X,y0,t) and minmodabb (dependent on X,t) values, and their values at t=0 make a significant impact on the overall proceedings.

It seems there are a few ways to simulate the minmodabb values, and hence the incremental t-steps. The tloop code on a single core processes anywhere between 6000 and 10000 mesh points per second (exact value dependent on number of taylor terms used), so if we assume an average of 8000 points, we can simulate the amount of time required at each t-step (better to slightly overestimate). This can help in identifying how to split the t range between jobs.

To simulate minmodabb values, we can either use approx. abbeff (with say 100k terms), or eulerprod(x,t) or just assume it to be a constant like 3.5. The first method is the most accurate (but slow atleast in gp), the second one not so much but is fast and the trend is alright. A good thing about the 3rd one apart from being fast is that it predicts smaller than actual tsteps (since minmodabb generally starts above 4) where most of the computation resides, so the critical jobs will take lesser time than estimated.

On trying this out with X=3x10^14+188911,

eulerprodt(x,t)=abs(prod(v=1,30,1/(1-1/prime(v)^(1-I*X/2+(t/4)*log((1-I*X/2)/prime(v))))));

X=3*10^14+188911;
y0=0.13416;
tstart=0;tend=0.161;
filename="tsplit.txt";

\\minmodabbstart=vecmin([abs(abbeff_approx(X,1,tstart)),abs(abbeff_approx(X+0.5,1,tstart)),abs(abbeff_approx(X+1,1,tstart))]);
\\minmodabbstart=vecmin([eulerprodt(X,tstart),eulerprodt(X+0.5,tstart),eulerprodt(X+1,tstart)]);
minmodabbstart=3.5;

t=tstart;
minmodabb=minmodabbstart;
tottimest=0;
counter=1;
while(t<=tend,{
Dtabb = ceil(ddtboundint(X+1,y0,t));
Dzabb = ceil(ddzboundint(X+1,y0,t));
timest=Dzabb/2000.0;
print([counter,t,Dtabb,4*Dzabb,minmodabb,timest]);
t = t + (minmodabb-0.5)/Dtabb;
\\minmodabb = vecmin([abs(abbeff_approx(X,1,t)),abs(abbeff_approx(X+0.5,1,t)),abs(abbeff_approx(X+1,1,t))]);
\\minmodabb = vecmin([eulerprodt(X,t),eulerprodt(X+0.5,t),eulerprodt(X+1,t)]);
tottimest=tottimest+timest;
if(tottimest>3600,tottimest=0;write(filename,t));
counter=counter+1;
});

we get in the tsplit file (excluding tstart and tend),

0.0033938467544100506907145464893504019839
0.0090426139257163597884666889062147590431
0.030102294503472653902564881426563175383

Have submitted these 4 jobs in Boinc to see whether they actually take less than an hour (the last one ofcourse will be much faster),

bin/boinc2docker_create_work.py --min_quorum 1 --target_nresults 1 dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/storedsum_nolemma_300000000188911_dig20.txt  -O sss.txt; ./tloopsingle 0 0.0033938467544100506907145464893504019839 0.13416 0 sss.txt > /root/shared/results/tloopout_X_300000000188911_part1.txt";  

bin/boinc2docker_create_work.py --min_quorum 1 --target_nresults 1 dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/storedsum_nolemma_300000000188911_dig20.txt  -O sss.txt; ./tloopsingle 0.0033938467544100506907145464893504019839 0.0090426139257163597884666889062147590431 0.13416 0 sss.txt > /root/shared/results/tloopout_X_300000000188911_part2.txt";  

bin/boinc2docker_create_work.py --min_quorum 1 --target_nresults 1 dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/storedsum_nolemma_300000000188911_dig20.txt  -O sss.txt; ./tloopsingle 0.0090426139257163597884666889062147590431 0.030102294503472653902564881426563175383 0.13416 0 sss.txt > /root/shared/results/tloopout_X_300000000188911_part3.txt";  

bin/boinc2docker_create_work.py --min_quorum 1 --target_nresults 1 dbnupperbound/arbcalc:v9 bash -c "wget http://anthgrid.com/dbnupperbound/storedsums/storedsum_nolemma_300000000188911_dig20.txt  -O sss.txt; ./tloopsingle 0.030102294503472653902564881426563175383 0.161 0.13416 0 sss.txt > /root/shared/results/tloopout_X_300000000188911_part4.txt";  

EDIT:
The 4 boinc jobs completed, and attaching the concatenated output
tloop_boincout_X_3x10e14+188911.txt

These were the timings of the 4 jobs:
part1 - 2.27 hours
part2 - 0.97 hours
part3 - 1.08 hours
part4 - 0.4 hours

As expected part2 took under an hour, part3 slightly more than an hour, and part4 was anyways the residual part. Part1 unexpectedly took 2.27 hours, which I am assuming was due to the peculiarities of the host machine that job was assigned to (that host has been taking more time on the other jobs too).

[rudolph-git-acc]

@km-git-acc

Great work and nice results! Will try the same run once again on 32 cores and see how fast it is (EDIT: using 64 virtual threads, the task completed in 332 secs). Did a quick check on the output from my earlier run and that was 1241 secs (on a 6-core machine).

Did some further work on the contour integral for Ht to isolate the zeros. Using the derivative as an integral instead of the H'-function, gives the correct results (after a while). It won't work for t=0, but hey, we won't need the complex integral to evaluate that level. Will now try this in ARB to see how fast a rectangle could be processed. Splitting the area of the contour into smaller parts is easily done, hence a root finding approach like this would also be very suitable for grid computing.

Xi(s)=(s/2)*(s-1)*Pi^(-s/2)*gamma(s/2)*zeta(s);
Ht(x, y, t)=1/(8*sqrt(Pi))*intnum(v=-8,8,Xi((1+I*(x+y*I))/2+sqrt(t)*v)*exp(-v^2));
Htd(x, y, t)=I/(8*sqrt(t*Pi))*intnum(v=-8,8,Xi((1+I*(x+y*I))/2+sqrt(t)*v)*exp(-v^2)*v);
f(z) = Ht(real(z),imag(z),0.1);
fd(z) = Htd(real(z),imag(z),0.1);

contint(func,vertices) = round(abs((1/2/Pi/I)*(intnum(v=vertices[1],vertices[2],fd(v)/func(v))+intnum(v=vertices[2],vertices[3],fd(v)/func(v))+intnum(v=vertices[3],vertices[4],fd(v)/func(v))+intnum(v=vertices[4],vertices[1],fd(v)/func(v)))));

contint(f,[25-I,25+I,30+I,30-I]) --> 1

[km-git-acc]

@rudolph-git-acc
Interesting. Having a direct expression for the derivative is a great development, and should result in more accurate results too.

332 seconds is quite impressive. I was analyzing the Boinc users, and these are some of the stats,
+--------+------+
| userid | #open tasks
+--------+------+
| 40 | 6009 |
| 68 | 2149 |
| 36 | 357 |
| 43 | 352 |
| 89 | 296 |
| 7 | 219 |
| 114 | 191 |
| 31 | 100 |
| 92 | 31 |
| 51 | 20 |
| 84 | 17 |
| 62 | 17 |
| 77 | 10 |

+--------+------+
| userid | #completed tasks
+--------+------+
| 40 | 8810 |
| 2 | 1476 |
| 36 | 1061 |
| 29 | 757 |
| 7 | 682 |
| 1 | 379 |
| 86 | 268 |
| 8 | 252 |
| 48 | 170 |
| 9 | 138 |
| 89 | 116 |
| 122 | 110 |
| 68 | 106 |
| 106 | 79 |
| 114 | 73 |
| 92 | 54 |
| 34 | 46 |
| 58 | 31 |
| 55 | 14 |
| 6 | 11 |

+--------+-----+
| userid | #host machines
+--------+-----+
| 36 | 47 |
| 40 | 28 |
| 84 | 12 |
| 8 | 6 |
| 9 | 5 |

(not showing the tail in any of the three tables). Assuming all the hosts are low cost single core (may not be always true but messy to find out), your single machine is quite up there in terms of config.
Also, I have tried to now restrict the number of open tasks to each host as 3*number of cores, so we should hopefully see a more equitable distribution of future jobs to users.

The 9x10e21 storedsums jobs are halfway complete, while the 2x10e20 ones are going through the tail effect (maybe they will both complete at roughly the same time).

[rudolph-git-acc]

@km-git-acc

Great, a bit of load balancing is smart (we otherwise could end up with only one client spending ages on the last 'cell').

Could you do a quick sanity check on whether the returned outputs still have decent accuracy?

[km-git-acc]

@rudolph-git-acc
The 9x10e21 outputs seem to have 10 to 12 digits of accuracy.

Subtotalindex: 0, start: 0, end: 1338093
9000000000000000070686.00000000, 114, 114, 10
50.2343729045, 6.5382939665, 3494.06987472, 448.99437777, 140382.1897443, 17649.7162562, 4201525.330523, 516012.659033, 102538664.80792, 12284069.86469, 2132637430.2851, 248305233.4740, 38794458594.213, 4366006626.541, 628004007388.80, 67896079323.83, 9161342100985.4, 945794755479.0, 1.21613597610789e+14, 1.1924458391587e+13, 1.4806....

Subtotalindex: 100, start: 133809300, end: 135147393
9000000000000000070686.00000000, 114, 114, 10
0.03304019628, 0.06834751427, 0.1746325774, 0.3612212935, 0.4615068731, 0.9545398146, 0.813092741, 1.68160.....

Subtotalindex: 997, start: 1334078721, end: 1335416814
9000000000000000070686.00000000, 114, 114, 10
0.01920383224, -0.00799040265, 0.02551016990, -0.01059822934, 0.01694372127, -0.00702857.....

It seems some of the allocated tasks have come back to the pool, and my boinc client has started receiving tasks again on updating the project. You may see similar behavior on your side.

EDIT:
Links to the partial output generated so far,
https://storage.googleapis.com/dbnupperbound/subt2x10e20.zip
https://storage.googleapis.com/dbnupperbound/subt9x10e21.zip

[rudolph-git-acc]

@km-git-acc

Great. Accuracy is in line of the expectation! I asked, since I did a run to calculate a single cell (to serve as checksum). It ran 2 days and came back with 0... Have now started a new one with higher accuracy. Will check Boinc!

The contour integration runs pretty fast in pari/gp (e.g. 3 mins 20 for the rectangle 25-4i, 25+4i, 2000+4i, 2000-4i at t=-12 on single core machine), however evaluation is still way too slow in ARB. When running along the imaginary side of the rectangle all is fast, but when I try to integrate along the real axis side of the rectangle, it becomes unworkably slow. Very strange and seem to be related to the integral evaluation itself. Need to do some further digging. All in all this seems to be a promising route for root finding, esp. when we could parallelise the work and reduce time by a significant factor. What is actually the best approach to properly isolate/certify the individual roots and then to find them up to a certain accuracy? Seems some recursive method is needed here and did read about triangulation.

This is our formula to evaluate. Do you maybe see any opportunity to further simplify this? I have considered writing xi(s) as a polynomial (which is easily done with readily available polynomials for zeta, Gamma and exp), since I'd thought integration and differentiation would become much easier (i.e. just another polynomial). This obviously doesn't work since the integration variable v is not the polynomial variable...

EDIT: this is the pari/gp code:

default(realprecision, 20)
Xi(s)=(s/2)*(s-1)*Pi^(-s/2)*gamma(s/2)*zeta(s);
Ht(x, y, t) =1/(8*sqrt(Pi))  *intnum(v=-8,8,Xi((1+I*(x+y*I))/2+sqrt(t)*v)*exp(-v^2));
Htd(x, y, t)=I/(8*sqrt(t*Pi))*intnum(v=-8,8,Xi((1+I*(x+y*I))/2+sqrt(t)*v)*exp(-v^2)*v);
f(z) = Ht(real(z),imag(z),-12);
fd(z) = Htd(real(z),imag(z),-12);

contint(vertices) = round(abs((1/2/Pi/I)*(intnum(v=vertices[1],vertices[2],fd(v)/f(v))+intnum(v=vertices[2],vertices[3],fd(v)/f(v))+intnum(v=vertices[3],vertices[4],fd(v)/f(v))+intnum(v=vertices[4],vertices[1],fd(v)/f(v)))));
#
contint([25-4*I,25+4*I,2000+4*I,2000-4*I])

[km-git-acc]

@rudolph-git-acc
Well, I don't know whether something like a vectorized integration can be made to work, but that can be one way to exploit the similarity of the numerator and denominator integrands.
So instead of calculating the two integrals separately, we could calculate something like

Const = I/sqrt(t);
h(v,z,t) = Xi((1+I*z)/2+sqrt(t)*v)*exp(-v^2);
g(z,t) = {
[D,F] = Integral{[Const*v,1]*h(v,z,t)};
return(D/F);
}

If this function can then be used within the contour integral, it should provide a 2x speedup.

I think the recursive approach you mentioned is generally the way to zero in on the roots. At each step, the contour area should decrease by half, so the number of steps required should ideally be logarithmic in terms of the digits accuracy.

On the Boinc front, I have enabled the forums (http://anthgrid.com/dbnupperbound/forum_index.php), after a suggestion by one of the users.

I have been thinking of mailing Prof. Platt about the work done so far, and whether he can share the RH verification scripts used in his paper. What do you think?

EDIT:
Well, it does work. parigp can be neat sometimes.

t=-12;
const = I/sqrt(t);
h(v,z) = Xi((1+I*z)/2+sqrt(t)*v)*exp(-v^2);
g(z) = {[D,F] = intnum(v=-8,8,[const*v,1]*h(v,z)); return(D/F);}

contint(vertices) = round(abs((1/2/Pi/I)*(intnum(v=vertices[1],vertices[2],g(v))+intnum(v=vertices[2],vertices[3],g(v))+intnum(v=vertices[3],vertices[4],g(v))+intnum(v=vertices[4],vertices[1],g(v)))));

contint([25-4*I,25+4*I,2000+4*I,2000-4*I])

This took around 3 minutes on my machine, compared to 9 minutes earlier.
A more compact version,

m4(j)=1+(j-1)%4;
h(v,z,t) = Xi((1+I*z)/2+sqrt(t)*v)*exp(-v^2);
g(z,t) = {[D,F] = intnum(v=-8,8,[v,1]*h(v,z,t)); return(D/F);}

contint(p,t) = round(abs((1/2/Pi/sqrt(t))*sum(j=1,4,intnum(v=p[m4(j)],p[m4(j+1)],g(v,t)))));

contint([25-4*I,25+4*I,2000+4*I,2000-4*I],-12)

[km-git-acc]

@rudolph-git-acc
I have been quite intrigued by the formula for the derivative, since it seems to be quite elegant and a deeper result, not just a standard application of the Leibniz rule. In fact, it seems there are two different integrands that on integrating give the same answer, which is amazing. How did you hit upon this formula..

h(z,t,v) = Xi((1+I*z)/2+sqrt(t)*v)*exp(-v^2);
k(z,t,v) = (I/sqrt(t))*v*h(z,t,v);

r(z,t) = intnum(v=-8,8,h(z,t,v));
f(z,t) = intnum(v=-8,8,h'(z,t,v));
g(z,t) = intnum(v=-8,8,k(z,t,v));

r'(30+8*I,-8) -> 0.0038989093364183130987 + 0.0021132950015132515725*I
f(30+8*I,-8)  -> 0.0038989093364357291900 + 0.0021132950016088488043*I
g(30+8*I,-8)  -> 0.0038989093365395572872 + 0.0021132950011405614916*I

h'(2+3*I,6,-5) -> -4.7516483118896075523 E-11 - 7.3920052300069603964 E-11*I
k(2+3*I,6,-5)  -> -3.4209137327030026867 E-10 - 5.9845755513689015352 E-10*I

[rudolph-git-acc]

@km-git-acc

I actually found the derivative formula with a bit of luck. Was playing with the derivatives listed on the integral 3rd approach page, changed a few parameters and there it suddenly was... The only intuitive guess I made was: when t->0 in the Ht-integral, you'll end up with xi(s). This obviously can't happen for Ht', so I assumed there should at least be a division by t somewhere (i.e. to prevent it from being valid at t=0 since xi(s) is in it). Did try to derive the result through a variable change, but didn't completely get there. Also got intrigued by it's surprising beauty and that it still works for -t. It makes t=0 even more special and maybe the fact that this derivative integral breaks down is a sign that all complex zeros have to have collided before t=0 (i.e. a collision without a derivative can't occur).

And yes, also tried your proposed contour integral routine and also got a 100% speed boost. Nice idea, KM :)

On the RH-zeros front, yes, it would a good idea to contact prof. Platt and explain what we have been doing here. He actually might be interested. BTW, I stumbled upon this arxiv-paper by Fredrik (ARB): https://arxiv.org/pdf/1802.07942.pdf in which he explains a routine that does the contour for the logarithmic integral of zeta for a rectangle (i.e. a bit more than the critical strip) all the way up to T=10^9 (x=2*10^9) in only 1600 secs(!). I mean, if 10^9 is feasible this way (and still unthreaded/gridded), then why not just counting the zeros in the rectangle and comparing it with the number of sign changes on the critical line. Wouldn't that be sufficient?

[km-git-acc]

@rudolph-git-acc
If this is a new formula, i.e not yet published in literature, you should definitely go ahead and do it or atleast post about it. The xi function is a pretty important, and anything new regarding it is always looked forward to. Also, this may have an interesting proof.

Yeah we should use Fredrick's routine to establish the number of roots in the rectangle. The Odlyzko/Platt papers should help us on efficient ways to count the sign changes. On the latter it seems we have to decide whether we count the sign changes as in Odlyzko's work, which is fast, or we find the exact locations of zeros as done by Platt which would be comparatively slower but more rigorous.

[rudolph-git-acc]

@km-git-acc

Okay, will put a question on the blog. The formula actually does look quite close to this one on: http://michaelnielsen.org/polymath1/index.php?title=Bounding_the_derivative_of_H_t_-_third_approach, so still wondering whether just a smart change of variables will do the trick:

[km-git-acc]

@rudolph-git-acc Actually, I was now going through that wiki page myself, and it makes sense. I guess the key trick was that although s and z are related, s becomes a formal variable under integration, and then differentiation wrt z leaves the xi function untouched. Didn't know earlier that this is allowed, but this pdf is quite illuminating http://www.math.uconn.edu/~kconrad/blurbs/analysis/diffunderint.pdf

…

On Tue, Oct 2, 2018, 11:42 PM rudolph-git-acc ***@***.***> wrote: @km-git-acc <https://github.com/km-git-acc> Okay, will put a question on the blog. The formula actually does look quite close to this one on: http://michaelnielsen.org/polymath1/index.php?title=Bounding_the_derivative_of_H_t_-_third_approach, so still wondering whether just a smart change of variables will do the trick: [image: image] <https://user-images.githubusercontent.com/36714969/46367894-1fb67e80-c67f-11e8-8a53-990b28734ba3.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#113 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AiK-4j6SRIgcGJiEXGQXwC-AlebH1vQAks5ug6ySgaJpZM4W-3lD> .