Add scoreboard 🏆

[mutatrum]

This PR adds a scoreboard screen that shows the top 20 shares:

And an api/system/scoreboard api call:

[
  {
    "difficulty": 956725.8468403114,
    "job_id": "684934fc00047bfd",
    "extranonce2": "0600000000000000",
    "ntime": 1758023248,
    "nonce": "F298047E",
    "version_bits": "040BC000"
  },
  {
    "difficulty": 942190.512488009,
    "job_id": "684934fc00047bc7",
    "extranonce2": "0e00000000000000",
    "ntime": 1758021685,
    "nonce": "5B7D04AA",
    "version_bits": "0122A000"
  },
  {
    "difficulty": 699177.2079349027,
    "job_id": "684934fc00047b90",
    "extranonce2": "1f00000000000000",
    "ntime": 1758020167,
    "nonce": "8B1F05EA",
    "version_bits": "04CA2000"
  },

Currently it captures all the fields from mining.submit, but others can be added. Not sure if adding the full header and resulting hash should be added.

[github-actions]

Test Results

42 tests  ±0   42 ✅ ±0   0s ⏱️ ±0s
 1 suites ±0    0 💤 ±0 
 1 files   ±0    0 ❌ ±0 

Results for commit 0e2d660. ± Comparison against base commit 5156662.

♻️ This comment has been updated with latest results.

[mutatrum]

Thanks to @duckaxe for combing over my NG stuff 🤝

[skot]

Next step would be to show it somewhere on AxeOS and to persist this table in NVS. Adding the current all time best to the table would be a nice-to-have, even though we don't have the other fields.

As with all flash we need to be careful to not erase from NVS at high frequency as this can wear out the flash. I think keeping the high score list in NVS is a great idea. Just need to be aware that we only get something like 100,000 erase cycles on a ESP32 flash sector before it will wear out and be unreliable. An erase every minute would be bad (~ 2 months MTBF), an erase every hour (~ 11 years MTBF) would probably be fine. The esp-idf NVS library does some wear leveling afaik, but I wouldn't want to rely on that.

generally speaking, just changing a value in flash involves a erase cycle on the whole sector.

[mutatrum]

Next step would be to show it somewhere on AxeOS and to persist this table in NVS. Adding the current all time best to the table would be a nice-to-have, even though we don't have the other fields.

As with all flash we need to be careful to not erase from NVS at high frequency as this can wear out the flash. I think keeping the high score list in NVS is a great idea. Just need to be aware that we only get something like 100,000 erase cycles on a ESP32 flash sector before it will wear out and be unreliable. An erase every minute would be bad (~ 2 months MTBF), an erase every hour (~ 11 years MTBF) would probably be fine. The esp-idf NVS library does some wear leveling afaik, but I wouldn't want to rely on that.

generally speaking, just changing a value in flash involves a erase cycle on the whole sector.

Apologies for the long reply, it was a deeper rabbit hole than I expected.

I did some reading, and initially bad news: the current NVS partition is only 24kb, which means 6 pages of 4kb. However, the write pattern is more subtle, and that's good news: each page is an append log, and writes are append onto the same page without an erase. Updates are also appended, and the old value is marked as deleted. Only when a page is full, it starts to use a next page. If all but the last empty page is used, a full page is recycled, where still valid key/values are moved to a new page and it's actually flash erased. link

We are currently at ~33% usage of NVS, Added some statistics:

I (1104) nvs_device: Used entries: 249
I (1104) nvs_device: Free entries: 507
I (1104) nvs_device: Available entries: 381
I (1107) nvs_device: Total entries: 756

The difference between free and available is old values that are marked as deleted, and will at some time be garbage collected page. Each entry is 32 bytes: small values (8, 16 or 32 bit) are a single entry, blobs and strings can be more than one entry. Blobs have 2 entries overhead and string have 1 entry overhead.

I also looked at repartitioning, but that's a tricky avenue. It has been done, but makes f.e. downgrades very challenging. This PR doesn't warrant that.

So, not sure here. It's possible to add 20 NVS string entries with all values concatenated per line. This would be around 80 NVS entries. IMO this is a reasonable trade-off for write granularity. Having the scoreboard persistent is making it a lot more fun. More ways of celebrating high difficulties and to have an idea when they happened and how rare a high difficulty is gives a lot of insight in the process.

[mutatrum]

Besides the NVS storage aspect, are there other fields that should be added or replaced?

[skot]

This does feel a little bit outside of the practical use of the ESP32's limited NVS. I could image the high scoreboard being a good thing to punt to the client dashboard to keep track of.. Or maybe just ESP32 RAM?

[mutatrum]

lol I added it, as I wanted to test if it would work. With 20 entries as strings, I get the following NVS statistics:

I (1116) nvs_device: Used entries: 329
I (1116) nvs_device: Free entries: 427
I (1116) nvs_device: Available entries: 301
I (1119) nvs_device: Total entries: 756

Exactly 80 entries are added, the math works out. It uses about 10% of NVS storage, it's now at 43%.

The first few minutes have quite a few updates, but as the difficulty rises quickly, the number of updates will drop really fast. A minimum threshold before saving an entry would reduce that.

This does feel a little bit outside of the practical use of the ESP32's limited NVS. I could image the high scoreboard being a good thing to punt to the client dashboard to keep track of.. Or maybe just ESP32 RAM?

I agree, it's pushing it a bit. But as I mentioned above, at the same time it's also a really informative addition, IMO. And having the scoreboard survive a reboot is chefs kiss 😆

To not paint ourselves into the corner, I'll experiment a bit with adding a second NVS partition at the end of the partition table. I can't find a conclusive answer from the documentation if this is allowed/possible or not.

[duckaxe]

Some improvements mutatrum#4

[Travetown]

After a little over a month with a total of 5 Bitaxe Gamma with scoreboard:
The lowest difficulty is now between 50 and 75M, and new entries have become correspondingly rare.
There have been no problems with the scoreboard and I think it's great.

[Travetown]

Another quick update.
After about 10 weeks, # 20 is now around 150M for most Bitaxe. This means that on average, a new share can be expected in 5 days (at 1.5 Th/s).
Unfortunately, this PR seems to have fallen out of focus somewhat.
Perhaps it will be tested again with the current master. 😇

[mutatrum]

Another quick update. After about 10 weeks, # 20 is now around 150M for most Bitaxe. This means that on average, a new share can be expected in 5 days (at 1.5 Th/s). Unfortunately, this PR seems to have fallen out of focus somewhat. Perhaps it will be tested again with the current master. 😇

It won't be forgotten, it just takes time to mature 😉

[0xf0xx0]

only nits are SystemScoreboardEntry should be moved up to components.schemas, and there should be some dupe protection in case the axe is looping and happens to get a high score (somehow i did this)

[mutatrum]

only nits are SystemScoreboardEntry should be moved up to components.schemas, and there should be some dupe protection in case the axe is looping and happens to get a high score (somehow i did this)

How did you do this? Not even sure how to do the duplicate protection either. Is it also submitting the share multiple times to the pool?

[0xf0xx0]

my pool fell over while i was gone for a day x3 so it was never submitted, just looping old work.

[mutatrum]

my pool fell over while i was gone for a day x3 so it was never submitted, just looping old work.

I think that would be covered by #911

[Travetown]

I would be very curious to know why this PR is not being accepted.
Four months of maturation does not seem to be enough.

I found it quite exciting to observe the development of the difficulties over several months. But now I will probably delete it with the new firmware.

[mutatrum]

I would be very curious to know why this PR is not being accepted. Four months of maturation does not seem to be enough.

I found it quite exciting to observe the development of the difficulties over several months. But now I will probably delete it with the new firmware.

It'll happen, just need patience. There were several big PRs going in each version already, and this is not a high-prio item.

Edit: Oops, wrong button. Didn't mean to close the PR.