Skip to content

feat!: migrate additional indexes from synchronous to async operation #7101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
UdjinM6 wants to merge 14 commits into
base: develop
Choose a base branch
from
+1,924 −1,108
Open

feat!: migrate additional indexes from synchronous to async operation #7101

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a77ec64
feat: migrate TimestampIndex from synchronous to async operation
UdjinM6 Jan 19, 2026
931d03a
test: update timestampindex functional test for async behavior
UdjinM6 Jan 19, 2026
f9e86c2
feat: migrate SpentIndex from synchronous to async operation
UdjinM6 Jan 19, 2026
36cc2a9
test: update feature_spentindex for async behavior
UdjinM6 Jan 19, 2026
aa5bdca
feat: migrate AddressIndex from synchronous to async operation
UdjinM6 Jan 19, 2026
87e49e7
test: update feature_addressindex for async behavior
UdjinM6 Jan 19, 2026
e65ddb1
feat: add CompactRange template method to CDBWrapper
UdjinM6 Jan 19, 2026
3d890ec
feat: migrate old synchronous index data to new async index databases
UdjinM6 Jan 19, 2026
51f07f1
fix: add missing include in `src/zmq/zmqpublishnotifier.cpp`
UdjinM6 Jan 19, 2026
50679b7
docs: add release notes
UdjinM6 Jan 19, 2026
9b73190
refactor: address code review feedback
UdjinM6 Jan 19, 2026
0521e9f
refactor: apply clang-format
UdjinM6 Jan 19, 2026
60db74a
feat: add async indexes to getindexinfo RPC
UdjinM6 Jan 19, 2026
88a169d
fix: check index sync status before querying and show sync progress
UdjinM6 Jan 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions doc/release-notes-7101.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
## Indexes

### Async Index Migration (#7101)

`TimestampIndex`, `SpentIndex`, and `AddressIndex` have been migrated from synchronous to asynchronous operation, following the same pattern as `TxIndex` and other indexes using `BaseIndex` framework.

When enabling an index for the first time, the node will build the index in the background while remaining fully operational. Progress can be monitored via the `getindexinfo` RPC.

Existing nodes with indexes enabled will automatically migrate data from the old location (block index database) to new separate databases on first startup. This migration may take 20-40 minutes or longer depending on hardware specifications and index sizes. The node will log progress during migration.

Breaking changes:
- `SpentIndex` and `AddressIndex` are incompatible with pruned nodes as they require access to undo data now
17 changes: 11 additions & 6 deletions src/Makefile.am
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,8 +157,6 @@ BITCOIN_CORE_H = \
active/masternode.h \
active/quorums.h \
addrdb.h \
addressindex.h \
spentindex.h \
addrman.h \
addrman_impl.h \
attributes.h \
Expand Down Expand Up @@ -247,10 +245,17 @@ BITCOIN_CORE_H = \
httprpc.h \
httpserver.h \
i2p.h \
index/addressindex.h \
index/addressindex_types.h \
index/addressindex_util.h \
index/base.h \
index/blockfilterindex.h \
index/coinstatsindex.h \
index/disktxpos.h \
index/spentindex.h \
index/spentindex_types.h \
index/timestampindex.h \
index/timestampindex_types.h \
index/txindex.h \
indirectmap.h \
init.h \
Expand Down Expand Up @@ -344,7 +349,6 @@ BITCOIN_CORE_H = \
rpc/blockchain.h \
rpc/client.h \
rpc/evo_util.h \
rpc/index_util.h \
rpc/mempool.h \
rpc/mining.h \
rpc/protocol.h \
Expand Down Expand Up @@ -379,7 +383,6 @@ BITCOIN_CORE_H = \
support/events.h \
support/lockedpool.h \
sync.h \
timestampindex.h \
threadsafety.h \
timedata.h \
torcontrol.h \
Expand Down Expand Up @@ -489,7 +492,6 @@ libbitcoin_node_a_SOURCES = \
active/masternode.cpp \
active/quorums.cpp \
addrdb.cpp \
addressindex.cpp \
addrman.cpp \
banman.cpp \
batchedlogger.cpp \
Expand Down Expand Up @@ -536,9 +538,13 @@ libbitcoin_node_a_SOURCES = \
httprpc.cpp \
httpserver.cpp \
i2p.cpp \
index/addressindex.cpp \
index/addressindex_util.cpp \
index/base.cpp \
index/blockfilterindex.cpp \
index/coinstatsindex.cpp \
index/spentindex.cpp \
index/timestampindex.cpp \
index/txindex.cpp \
init.cpp \
instantsend/db.cpp \
Expand Down Expand Up @@ -601,7 +607,6 @@ libbitcoin_node_a_SOURCES = \
rpc/coinjoin.cpp \
rpc/evo.cpp \
rpc/fees.cpp \
rpc/index_util.cpp \
rpc/masternode.cpp \
rpc/mempool.cpp \
rpc/governance.cpp \
Expand Down
43 changes: 0 additions & 43 deletions src/addressindex.cpp
View file
Open in desktop

This file was deleted.

3 changes: 1 addition & 2 deletions src/core_write.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,7 @@
#include <util/strencodings.h>
#include <util/system.h>

#include <addressindex.h>
#include <spentindex.h>
#include <index/spentindex.h>

#include <evo/assetlocktx.h>
#include <evo/cbtx.h>
Expand Down
16 changes: 16 additions & 0 deletions src/dbwrapper.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,22 @@ class CDBWrapper
{
pdb->CompactRange(nullptr, nullptr);
}

/**
* Compact a specific range of keys in the database.
*/
template<typename K>
void CompactRange(const K& key_begin, const K& key_end) const
{
CDataStream ssKey1(SER_DISK, CLIENT_VERSION), ssKey2(SER_DISK, CLIENT_VERSION);
ssKey1.reserve(DBWRAPPER_PREALLOC_KEY_SIZE);
ssKey2.reserve(DBWRAPPER_PREALLOC_KEY_SIZE);
ssKey1 << key_begin;
ssKey2 << key_end;
leveldb::Slice slKey1(CharCast(ssKey1.data()), ssKey1.size());
leveldb::Slice slKey2(CharCast(ssKey2.data()), ssKey2.size());
pdb->CompactRange(&slKey1, &slKey2);
}
};

template<typename CDBTransaction>
Expand Down
Loading
Loading