Skip to content

Investigage mIndex errors related to S3 archive #338

@jayz22

Description

@jayz22

We have encountered a few bucketlist assertion failures (https://github.com/stellar/stellar-core/blob/3c6146baa94cd2c41a8b49d1b94915e3c5ba8ee9/src/bucket/LiveBucket.cpp#L610) since switching to S3 archive.
This is likely due to S3 issues, but it causes the entire catch-up mission to fail. Restarting the mission fixes the problem.
We need to

  • investigate and understand the issue
  • if is S3 stability related, make this more tolerant (e.g. allowing a few retries)

Sample error message:

2025-11-04T20:32:17.878 GAJSL [Ledger INFO] Ledger close complete: 10853701
2025-11-04T20:32:17.887 GAJSL [Ledger INFO] Ledger close complete: 10853702
2025-11-04T20:32:17.910 GAJSL [Ledger INFO] Ledger close complete: 10853703
mIndex at bucket/LiveBucket.cpp:610
Completed 832 Bytes/832 Bytes (2.7 KiB/s) with 1 file(s) remaining
download: s3://ssc-history-archive/prd/core-live/core_live_003/results/00/a5/a0/results-00a5a0ff.xdr.gz to ../../data/buckets/tmp/catchup-5fe1314fee098152/results/00/a5/a0/results-00a5a0ff.xdr.gz.tmp
Cannot provide readable stack trace. Use ParseDump.py to translate stack.
/usr/bin/stellar-core(___interceptor_backtrace+0x46) [0x55d5f60a2b56]
/usr/bin/stellar-core(+0x18034ec) [0x55d5f6ff44ec]
/usr/bin/stellar-core(+0x18222a0) [0x55d5f70132a0]
/usr/bin/stellar-core(+0xb66643) [0x55d5f6357643]
.
.
.
/usr/bin/stellar-core(+0x13345b0) [0x55d5f6b255b0]
/usr/bin/stellar-core(+0x14f2df4) [0x55d5f6ce3df4]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fd6b0b5cd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fd6b0b5ce40]
/usr/bin/stellar-core(+0x86b1b5) [0x55d5f605c1b5]
Aborted
Finish processing job: 10864575/16320, duration: 346s
Error processing job: 10864575/16320 (exit code: 134)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions