Source-built binaries fail to sync past block 450,947 (Fork #1)

[melvincarvalho]

Summary

Binaries compiled from source (master branch) fail to sync past block 450,947 with PoW validation errors, while the official v0.9.7.2 release binary syncs successfully.

Environment

• OS: Ubuntu 22.04 / 24.04
• Compiler: GCC 11+
• OpenSSL: 3.0.x
• Boost: 1.74+

Reproduction Steps

1. Build from master branch (with PR Fix build on Ubuntu 22.04 and 24.04 (GCC 11+, Boost 1.74+) #138 fixes for Blake2 alignment and Boost placeholders)
2. Start bitmarkd and let it sync
3. Observe failure at block 450,947

Error Details

UpdateTip: new best=d4b12e844971df57d6e1017e1c0c191f2e2c31175ee0d63313142754393a8956  height=450946  log2_work=72.842976  tx=1090252  date=1528347329 progress=0.031011 nbits=453792464 algo=0
ProcessBlock: ACCEPTED
ERROR: CheckProofOfWork() : hash doesn't match nBits (hash is dd64d857462cf74a732c555b70f36efb82691956d9cb600b8f81df73760fa267, nbits is 0000003e7fc10000...
ERROR: CheckBlock() : proof of work failed

• Block 450,946 (algo=0, Scrypt) validates successfully
• Block 450,947 (should be algo=5, Lyra2REv2) fails
• The computed hash has no leading zeros - suggesting wrong hash algorithm

Investigation Findings

Official Binary Works

The official v0.9.7.2 binary (Ubuntu 18.04, OpenSSL 1.0.2n) syncs past Fork #1:

height=450947  nbits=490635201 algo=5  ✓
height=450948  nbits=490635201 algo=5  ✓
...continuing to 668,000+

Code Comparison

Compared v0.9.7.2 tag with current master:

• BLOCK_VERSION_ALGO mask: identical (7 << 9)
• GetAlgo() function: identical
• GetPoWHash() function: identical
• CheckProofOfWork(): identical

The only consensus-related changes are:

• Fork2 functions commented out (March 2021)
• VARIANT flags added to block version (bits 13-14)
• Neither should affect blocks before 511,111

Related Issues

• bitmarkd compiled from source doesn't sync after block 450627 #54 - "bitmarkd compiled from source doesn't sync after block 450627" (2018)
• Bitmark doesn't sync after block 518192 #87 - "Bitmark doesn't sync after block 518192" (2018)

Potential Causes

1. OpenSSL 1.1+ incompatibility - BIGNUM became opaque structure
2. Compiler optimization differences - GCC 11+ vs GCC 7.x
3. Undefined behavior - code that worked on older compilers
4. Hash function library differences - sphlib, Lyra2, etc.

Proposed Solutions

1. Add debug logging to trace algo detection and hash computation
2. Build in Ubuntu 16.04/18.04 Docker with OpenSSL 1.0
3. Identify specific code causing different behavior
4. Create compatibility layer for OpenSSL 1.1+

Questions

1. Was the official v0.9.7.2 binary built from exactly commit 79262c2?
2. Are there any unpublished patches in the release binaries?
3. Has anyone successfully built from source on modern systems?

/cc @dbkeys @piratelinux

[melvincarvalho]

Detailed Error Analysis

Summary

Master branch cannot sync past block 450,946. Block 450,947 is the first post-Fork1 block requiring multi-algorithm PoW validation, and it fails with "hash doesn't match nBits" errors.

Test Environment

• Ubuntu 24.04 LTS, GCC 13
• Official v0.9.7.2 binary: Successfully syncs to 668,181+ blocks
• Master branch build: Fails at block 450,946

Error Output

ERROR: CheckProofOfWork() : hash doesn't match nBits 
  (hash is dd64d857462cf74a732c555b70f36efb82691956d9cb600b8f81df73760fa267, 
   nbits is 0000003e7fc10000000000000000000000000000000000000000000000000000)
ERROR: CheckBlock() : proof of work failed
ERROR: ProcessBlock() : CheckBlock FAILED

Root Cause Analysis

The computed hash dd64d857... has no leading zeros, indicating a completely wrong hash algorithm is being used. Valid PoW hashes should have many leading zeros to satisfy the difficulty target.

Key Code Differences Found

1. BLOCK_VERSION_ALGO Mask (src/pureheader.h)

// Master branch:
BLOCK_VERSION_ALGO = (7 << 9),  // 3-bit mask, supports algos 0-7

// 0.9.9 branch:  
BLOCK_VERSION_ALGO = (31 << 9), // 5-bit mask, supports algos 0-31

2. Fork2/Fork3 Functions Disabled (src/chainparams.h, src/core.h)
Master branch has Fork2 code commented out with note "Fork2 in suspension (3/21/21)":

// Fork2 in suspension (3/21/21)
//int64_t GetFork2Height() const { return nForkHeight2; }
//bool    OnFork2(int64_t blockHeight) const { return blockHeight >= nForkHeight2; }

3. GetAlgo() and GetPoWHash() Functions
The GetAlgo() function extracts the algorithm from block version, and GetPoWHash() computes the appropriate hash. With the 3-bit mask limitation, some algorithm selections may be incorrect.

Debug Build Output

Added tracing to main.cpp:

LogPrintf("CheckBlock PoW: nVersion=%d (0x%x) algo=%d hash=%s\n",
          block.nVersion, block.nVersion, block.GetAlgo(),
          block.GetPoWHash().ToString().substr(0,16).c_str());

Pre-fork blocks show: algo=0 (SHA256D) - all pass
Post-fork block 450,947 fails before debug line is reached (or algo detection is wrong)

Suggested Next Steps

1. Compare algorithm detection: Check what algorithm the official binary detects for block 450,947 vs what master detects

2. Review BLOCK_VERSION_ALGO mask: The 3-bit mask may be truncating algorithm IDs incorrectly

3. Check AuxPow serialization: Fork1 enables merge mining; the block deserialization may not correctly handle auxpow data

4. Test with 0.9.9 branch: The 0.9.9 branch has different mask values and may work correctly

5. Bisect commits: Find the exact commit that broke post-fork sync (likely around March 2021 "Fork2 suspension" changes)

Files to Investigate

• src/pureheader.h - GetAlgo(), GetPoWHash(), BLOCK_VERSION_ALGO
• src/core.h - onFork2(), onFork3() always return false
• src/chainparams.h - Fork parameters
• src/pow.cpp - CheckProofOfWork()
• src/main.cpp - CheckBlock()

Related Issues

• bitmarkd compiled from source doesn't sync after block 450627 #54 "bitmarkd compiled from source doesn't sync after block 450627"
• Bitmark doesn't sync after block 518192 #87 "Bitmark doesn't sync after block 518192"

[melvincarvalho]

Root Cause Identified: AuxPow Algorithm Mismatch

Discovery

After adding detailed debug logging, the exact failure point is now identified:

The CheckAuxPowProofOfWork() function uses the CHILD block's algorithm to compute the PARENT block's PoW hash, which is incorrect for merge-mined blocks.

Debug Output Analysis

# Passing blocks (algo 0,1,2,4):
AuxPoW debug: childAlgo=1 parentVersion=0x20000000 parentAlgo=1 parentPoWHash=0000000000000000d9efca65...
AuxPoW debug: childAlgo=4 parentVersion=0x20000000 parentAlgo=4 parentPoWHash=00000000000be1f47452fc06...

# Failing blocks (algo 5 - LYRA2REv2):
AuxPoW debug: childAlgo=5 parentVersion=0x20000000 parentAlgo=5 parentPoWHash=50802358b88dc659d51768ef...
ERROR: CheckProofOfWork() : hash doesn't match nBits

For LYRA2REv2 auxpow blocks, the parent PoW hash has no leading zeros, indicating the wrong hash algorithm is being used.

Root Cause in Code

In src/core.cpp line ~674:

// PROBLEM: Uses child block's algo for parent block's hash
if (!CheckProofOfWork(block.auxpow->getParentBlockPoWHash(algo), block.nBits, block.GetAlgo()))

The parent block (from the merge mining pool) uses its own algorithm (typically SCRYPT), not the child block's algorithm (LYRA2REv2).

Proposed Fix

// FIX: Use parent block's algorithm, not child block's
int parentAlgo = block.auxpow->parentBlock.GetAlgo();
if (!CheckProofOfWork(block.auxpow->getParentBlockPoWHash(parentAlgo), block.nBits, parentAlgo))

Testing in Progress

Currently syncing with the proposed fix to verify it resolves the issue. Will update with results.