Potential divide and conquer strategy?

[km-git-acc]

@rudolph-git-acc and @mariotrevi

Few days back, I had noticed in the fourth thread at Prof. Tao's blog you were focused on the Ceff term, and improving Ht precision at smaller values of x, while I have been working more on the medium values of x.

This could be a good divide and conquer strategy in light of the fifth thread. Do let me know if there are any thoughts on this, and if we can help each other more.

[rudolph-git-acc]

@km-git-acc

Happy to follow your suggestion and divide up the work. If there are any specific requirements you would like me to work on, just let me know. I do have the Pari/gp code working for A_eff, B-eff and E1,2,3 as well as their bounds. Today I have attempted to find a faster formulae to produce the exact Ht, say for x<10^3, however have not succeeded yet.

P.S.
Did you see my comment from last Saturday on your E1/E2 bound calculation? It is just a small thing, but probably good to get the Wiki-displays for Sn+ and Sn- fixed.

Regards,

Rudolph

[km-git-acc]

@rudolph-git-acc
I had seen your comment and your strategy of calculating N from x and then calculating the bound seems correct.
Didn't completely follow you on the sn+ and sn- part. Is there a typo there?

EDIT: I replied on the latter part in the other thread. Although it would be good to ask Prof. Tao about this.

[rudolph-git-acc]

@km-git-acc

It indeed just seems a typo and as you say, we do need the extra divide by 2 in both sn+ and sn-. Will ask him directly on the blog.

Just saw that you published the bounds for the normalised derivative of A_eff + B_eff and I do fully match your numbers in Pari/gp code.

I also produced the code for the normalised derivatives of A_eff + B_eff and I guess you have coded this as well. So, would like to check some numbers that I got (way below the bound).

t=y=0.4

x= 10^2 0.19243854524130085276
x= 10^3 0.29955042110470048266
x= 10^4 0.028704733253812567905
x= 10^5 0.22145785748176529207
x= 10^6 0.48873623232165474190

Could you confirm?

In the mean time I have started to produce the exact values for H_t, in steps of 0.1 for x <= 300 at a precision of 40 digits. Have already produced up to the first 200 and expect the last csv-file to complete tonight. Maybe these could also be posted in the research folder.

[km-git-acc]

@rudolph-git-acc

I have been trying to produce the normalised derivatives for A_eff + B_eff, but going a bit slow. Since this is supposed to be exact, and we cannot use inequalities, I wanted to be a bit careful. For example, in the A_eff section, differentiation gives (−log(n)ddx(sA)+ddxlogλ), which turns into (log(n)ddx(sA)+ddxlogλ) due to the inequality. Just checking if there are any such other nuances.

[rudolph-git-acc]

@km-git-acc

Yep. I spotted that one as well. I believe he assumed that -log(n)d/dxSa +d/dxlog(lambda) = I/2log(n) +d/dx log(lambda), so that the entire LHS can be replaced by the O<=(...) inequality.

[km-git-acc]

@rudolph-git-acc
Also, I noticed while calculating the bounds that =(−i/4)*log(s/2πn)−(i/4)*log[(1−s)/2πn] in the next step becomes (1/4)*log(|s||1−s|/4πn^2], whereas I think it should be (1/4)*log(|s||1−s|/4π^2n^2]. Please check on your side as well.

[rudolph-git-acc]

@km-git-acc

Good catch. This should change to π^2 in both the bound as well as in the derivative, right?

[rudolph-git-acc]

@km-git-acc

After the correction of the π^2 in both the bound and the derivative itself, I now get:

Derivative:
x= 10^1 0
x= 10^2 0.19257287640109451000
x= 10^3 0.29955745401936522916
x= 10^4 0.028704710631748847242
x= 10^5 0.22145782200606050720
x= 10^6 0.48873623602067662246

Bound:
N= 10^1 5.2342874398389429723
N= 10^2 6.1615758838756614645
N= 10^3 2.5190679826910524273
N= 10^4 0.73829280418351811203
N= 10^5 0.28681345856569107338
N= 10^6 0.13681914060945870938

And a few variables for x = 1000, y=t=0.4 in the derivative calculation:

Sa=500.15696362404976313
Sb=500.15763352746266180
ddxsa=0.50000001570603390126
ddxsb=0.50012532371741988033
ABsum1=0.29971404441875070455
Absum2=0.00086679552518089579374
lambda=0.41669127076771460641

[km-git-acc]

@rudolph-git-acc

For the bound, our values match.

I am getting different values for the exact derivative at x.

10^2, 0.5534567762349945175223569395
10^3, 1.0368922891066215403945020717
10^4, 0.654362756931538124392228692915
10^5, 0.862586325880511659690526817054
10^6, 1.41149560304156500014976609879

I have noticed one more nuance. In the triangle inequality, it did not matter whether s is chosen as 1+y-ix or 1-y+ix to derive ddx log_lambda. But in the exact derivative, if we use the former, then log lambda is actually -log lambda.
and then the expression in the numerator would be −logn * ddx(sA) - ddxlogλ

EDIT: Have added the function ddx_abbeff_x for the exact derivative calculation in another branch
https://github.com/km-git-acc/dbn_upper_bound/blob/Derivatives-for-eff-model/dbn_upper_bound/python/research/ddx_abbeff_bounds.py

May not be the final version, so havent merged it into the main branch yet. Once our values match, then that can be done.

[km-git-acc]

@rudolph-git-acc
Also, as Prof. Tao commented just a while back, for the bound formula, there is an additional factor of 4 giving 16pi^2n^2, if we are dealing in x and y, and correcting that leads to further improvement. For the exact derivative, I was computing using s, so no change there.

[rudolph-git-acc]

@rudolph-git-acc

Yes, saw it. Not only impacts the bound, it Is also induced a change in the derivative calculation itself (same factor).

Currently running through the code in detail to spot the difference. Could you maybe post a few variables for say x=1000? Let's just use the corrected version (i.e. with 16 Pi^2).

[rudolph-git-acc]

@km-git-acc

Please ignore my comment on the 16*Pi^2. That just sits in the O<-() part. Symbols dancing in front of my eyes now :-)

[rudolph-git-acc]

@km-git-acc

Our data is very close now. Here is one difference I found:

• (I/4.0)log(sb/(2pi)) - (I/4.0)log((1-sb)/(2pi))

should be:

• (I/4.0)log(sb/(2pin)) - (I/4.0)log((1-sb)/(2pin)) (the n is missing)

Will keep digging further.

EDIT: ignore my comment. This is not an error in your code, I now understand how the n's got removed.

[rudolph-git-acc]

@km-git-acc

The differences are small, but I would really like to get it fully matched. These are my variables for x=1000. Could we compare, so we can zoom in?

alfap1 = 0.0010000302193660523824
ddxsa=0.50000066999756590754
ddxsb=0.50000074999804750530
ddxloglambda=1.0892570119404423856 (n=9)
sumB=0.29963939014413037597
sumA=2.3543846706393712008
lambda=0.41669127076771460641
overall=1.0571656625195835368

Thanks.

[rudolph-git-acc]

@km-git-acc

Never mind, I have got an exact match :-) Although, I see again at 16 digits, the numbers start to diverge, so maybe there still is somewhere a non-mpmath variable in your code?

I had made two errors in the ddxloglambda. Mistook an "i" for a "1" and had a 1-s where it should have been an s.

Just saw prof Tao's comment on the blog that we should use the +-sign. However, the definition of s=(1+y-ix)/2 puzzled me as well, since usually we have s=(1-y+ix)/2. I'll now try to check how well it works against the Newton quotient.