Base layer actions (#485)

Everything starts when a scheduled Intent arrives...

# Architecture

## Schedulers
We can't directly spawn bunch of **IntentExecutor**s. The reason is that
one message can block execution of another message. To handle this the
messages have to go through Scheduling.

Details: Once message make it to
`CommittorProcessor::schedule_base_intents` it outsources intent to
tokio task `IntentExecutionEngine` which figures out a scheduling.

## IntentExecutionEngine
Accepts new messages, schedules them, and spawns up to 50 parallel
**IntentExecutor**s for each Intent. Once a particular
**IntentExecutor** finishes execution we broadcast result to
subscribers, like: `RemoteScheduledCommitsProcessor` or
`ExternalAccountsManager`
 
details: For scheduling logic see **IntentScheduler**. Number of
parallel **IntentExecutor** is controller by Semaphore.

## IntentExecutor
IntentExecutor - responsible for execution of Intent. Calls
**TransactionPreparator** and then executes a transaction returning as
result necessary signatures

## TransactionPreparator
TransactionPreparator - is an entity that handles all of the above
"Transaction preparation" calling **TaskBuilderV1**, **TaskStrategist**,
**DeliveryPreparator** and then assempling it all and passing to
**MessageExecutor**

## DeliveryPreparator
After our **L1Task**s are ready we need to prepare eveything for their
successful execution. **DeliveryPreparator** - handles ALTs and commit
buffers

## TaskBuilder
First, lets build atomic tasks from scheduled message/intent.

High level: TaskBuilder responsible for creating L1Tasks(to be
renamed...) from ScheduledL1Message(to be renamed...).
Details: To do that is requires additional information from
DelegationMetadata, it is provided **CommitIdFetcher**

### L1Task
High level: L1Task - is an atomic operation that is to be performed on
the Base layer, like: Commit, Undelegate, Finalize, Action.

Details: There's to implementation of L1Tasks: ArgsTask, BufferTask.
ArgsTask - gives instruction using args. BufferTask - gives instruction
using buffer. BufferTask at the moment supports only commits

### CommitIdFetcher
High level: for account to be accepted by dlp it needs to have
incremental commit ids. CommitIdFetcher provides a user with the correct
ids/nonces for set of committees

Details: CommitIdTracker - implementation of CommitIdFetcher, that
caches and locally increments commit ids using LruCache

## TaskStrategist
After our tasks were built with **TaskBuilder**, they need to be
optimized to fit into transaction. That what TaskStrategist does.

Details: Initially **TaskBuilder** builds ArgsTasks, **TaskStrategist**
if needed optimzes them to BufferTask.


### Changes
- Persister changed from reqid & bundle_id format to message_id. Meaning
row created per message. A particular Row tracking lifespan of Intent
- Persister will be passed along into Executors & Scheduler for them to
update Intent statuses during execution
- No notion of bundles anymore, we represent things by Intent id
- AccountsManager doesn't use custom `AccountCommitter` for periodic
commits of accounts but instead uses CommittorService
- RemoteScheduledCommitsProcessor extracted from AccountsManager since
has nothing to do with it

# Benchmarks

Using the same config as in this
[PR](#366)
Here's best result on master I got on my machine:
```
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Metric                        | Observations | Median | Min  | Max   | Avg  | 95th Perc | Stddev |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| sendTransaction Response (μs) | 2000         | 3401   | 1499 | 23918 | 3543 | 5079      | 1244   |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Account Update (μs)           | 0            | 0      | 0    | 0     | 0    | 0         | 0      |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Signature Confirmation (μs)   | 2000         | 3460   | 1520 | 13509 | 3609 | 5430      | 1123   |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Transactions Per Second (TPS) | 51           | 40     | 40   | 40    | 40   | 40        | 0      |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Requests Per Second (TPS)     | ---          | ---    | ---  | ---   | ---  | ---       | ---    |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| getX Request Response (TPS)   | ---          | ---    | ---  | ---   | ---  | ---       | ---    |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
```

Here's the best result on feat/base-layer-ix/main which is current PR on
review
```
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Metric                        | Observations | Median | Min  | Max   | Avg  | 95th Perc | Stddev |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| sendTransaction Response (μs) | 2000         | 3241   | 1450 | 17898 | 3397 | 4907      | 954    |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Account Update (μs)           | 0            | 0      | 0    | 0     | 0    | 0         | 0      |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Signature Confirmation (μs)   | 2000         | 3332   | 1504 | 17947 | 3517 | 5187      | 988    |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Transactions Per Second (TPS) | 51           | 40     | 40   | 40    | 40   | 40        | 0      |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Requests Per Second (TPS)     | ---          | ---    | ---  | ---   | ---  | ---       | ---    |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| getX Request Response (TPS)   | ---          | ---    | ---  | ---   | ---  | ---       | ---    |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
```

So results are almost the same, but current `feat/base-layer-ix/main`
sends 2 txs for commit & finalize, while master sends 1.
Here's results on optimized branch feat/base-layer-ix/single-tx-intent
which sends just 1 tx as well. We get ~13% increase in performance on
our trait fuel

```
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Metric                        | Observations | Median | Min  | Max   | Avg  | 95th Perc | Stddev |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| sendTransaction Response (μs) | 2000         | 2896   | 1360 | 14462 | 3050 | 4619      | 1060   |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Account Update (μs)           | 0            | 0      | 0    | 0     | 0    | 0         | 0      |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Signature Confirmation (μs)   | 2000         | 2977   | 1376 | 15452 | 3149 | 4916      | 1149   |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Transactions Per Second (TPS) | 51           | 40     | 40   | 40    | 40   | 40        | 0      |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| Requests Per Second (TPS)     | ---          | ---    | ---  | ---   | ---  | ---       | ---    |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
| getX Request Response (TPS)   | ---          | ---    | ---  | ---   | ---  | ---       | ---    |
+-------------------------------+--------------+--------+------+-------+------+-----------+--------+
```

---------

Co-authored-by: Thorsten Lorenz <thlorenz@gmx.de>
Co-authored-by: Dodecahedr0x <90185028+Dodecahedr0x@users.noreply.github.com>
Co-authored-by: Arthur Bretas <158767751+BretasArthur1@users.noreply.github.com>
Co-authored-by: Dodecahedr0x <hexadecifish@gmail.com>
Co-authored-by: Gabriele Picco <piccogabriele@gmail.com>

Author: 5 people

Cargo.lock

@@ -66,14 +66,15 @@ borsh = { version = "1.5.1", features = ["derive", "unstable__schema"] }
 borsh-derive = "1.5.1"
 bs58 = "0.4.0"
 byteorder = "1.5.0"
-cargo-lock = "10.0.0"
 cargo-expand = "1"
+cargo-lock = "10.0.0"
 clap = "4.5.40"
-convert_case = "0.8.0"
 conjunto-transwise = { git = "https://github.com/magicblock-labs/conjunto.git", rev = "bf82b45" }
 console-subscriber = "0.2.0"
-crossbeam-channel = "0.5.11"
 const_format = "0.2.34"
+convert_case = "0.8.0"
+crossbeam-channel = "0.5.11"
+dyn-clone = "1.0.20"
 ed25519-dalek = "1.0.1"
 enum-iterator = "1.5.0"
 env_logger = "0.11.2"
@@ -100,7 +101,9 @@ lazy_static = "1.4.0"
 libc = "0.2.153"
 libloading = "0.7.4"
 log = "0.4.20"
+lru = "0.16.0"
 macrotest = "1"
+magic-domain-program = { git = "https://github.com/magicblock-labs/magic-domain-program.git", rev = "ea04d46", default-features = false }
 magicblock-account-cloner = { path = "./magicblock-account-cloner" }
 magicblock-account-dumper = { path = "./magicblock-account-dumper" }
 magicblock-account-fetcher = { path = "./magicblock-account-fetcher" }
@@ -118,8 +121,7 @@ magicblock-config = { path = "./magicblock-config" }
 magicblock-config-helpers = { path = "./magicblock-config-helpers" }
 magicblock-config-macro = { path = "./magicblock-config-macro" }
 magicblock-core = { path = "./magicblock-core" }
-magicblock-delegation-program = { git = "https://github.com/magicblock-labs/delegation-program.git", rev = "4af7f1c" }
-magic-domain-program = { git = "https://github.com/magicblock-labs/magic-domain-program.git", rev = "ea04d46", default-features = false }
+magicblock-delegation-program = { git = "https://github.com/magicblock-labs/delegation-program.git", rev = "5fb8d20", features = ["no-entrypoint"] }
 magicblock-geyser-plugin = { path = "./magicblock-geyser-plugin" }
 magicblock-ledger = { path = "./magicblock-ledger" }
 magicblock-metrics = { path = "./magicblock-metrics" }
@@ -148,20 +150,20 @@ protobuf-src = "1.1"
 quote = "1.0"
 rand = "0.8.5"
 rayon = "1.10.0"
-rustc_version = "0.4"
 rusqlite = { version = "0.34.0", features = ["bundled"] } # bundled sqlite 3.44
+rustc_version = "0.4"
 semver = "1.0.22"
 serde = "1.0.217"
 serde_derive = "1.0"
 serde_json = "1.0"
 sha3 = "0.10.8"
 solana-account = { git = "https://github.com/magicblock-labs/solana-account.git", rev = "176540a" }
-solana-accounts-db = { version = "2.2" }
 solana-account-decoder = { version = "2.2" }
+solana-accounts-db = { version = "2.2" }
 solana-address-lookup-table-program = { version = "2.2" }
 solana-bpf-loader-program = { version = "2.2" }
-solana-compute-budget-program = { version = "2.2" }
 solana-compute-budget-instruction = { version = "2.2" }
+solana-compute-budget-program = { version = "2.2" }
 solana-cost-model = { version = "2.2" }
 solana-frozen-abi-macro = { version = "2.2" }
 solana-geyser-plugin-interface = { version = "2.2", package = "agave-geyser-plugin-interface" }
@@ -175,17 +177,17 @@ solana-program = "2.2"
 solana-program-runtime = { version = "2.2" }
 solana-program-test = "2.2"
 solana-pubkey = { version = "2.2" }
-solana-rayon-threadlimit = { version = "2.2" }
 solana-pubsub-client = { version = "2.2" }
+solana-rayon-threadlimit = { version = "2.2" }
 solana-rpc = "2.2"
 solana-rpc-client = { version = "2.2" }
 solana-rpc-client-api = { version = "2.2" }
 solana-sdk = { version = "2.2" }
+solana-storage-proto = { path = "storage-proto" }
 solana-svm = { git = "https://github.com/magicblock-labs/magicblock-svm.git", rev = "e93eb57", features = [
   "dev-context-only-utils",
 ] }
 solana-svm-transaction = { version = "2.2" }
-solana-storage-proto = { path = "storage-proto" }
 solana-system-program = { version = "2.2" }
 solana-timings = "2.2"
 solana-transaction-status = { version = "2.2" }

Cargo.toml

@@ -16,6 +16,7 @@ magicblock-account-fetcher = { workspace = true }
 magicblock-account-updates = { workspace = true }
 magicblock-account-dumper = { workspace = true }
 magicblock-accounts-api = { workspace = true }
+magicblock-rpc-client = { workspace = true }
 magicblock-config = { workspace = true }
 magicblock-core = { workspace = true }
 magicblock-committor-service = { workspace = true }

magicblock-account-cloner/Cargo.toml

@@ -10,9 +10,10 @@ use magicblock_account_fetcher::AccountFetcherError;
 use magicblock_account_updates::AccountUpdatesError;
 use magicblock_committor_service::{
     error::{CommittorServiceError, CommittorServiceResult},
-    ChangesetCommittor,
+    BaseIntentCommittor,
 };
 use magicblock_core::magic_program;
+use magicblock_rpc_client::MagicblockRpcClient;
 use solana_sdk::{clock::Slot, pubkey::Pubkey, signature::Signature};
 use thiserror::Error;
 use tokio::sync::oneshot::{self, Sender};
@@ -76,9 +77,9 @@ pub enum AccountClonerUnclonableReason {
     DelegatedAccountsNotClonedWhileHydrating,
 }
 
-pub async fn map_committor_request_result<T, CC: ChangesetCommittor>(
+pub async fn map_committor_request_result<T, CC: BaseIntentCommittor>(
     res: oneshot::Receiver<CommittorServiceResult<T>>,
-    changeset_committor: Arc<CC>,
+    intent_committor: Arc<CC>,
 ) -> AccountClonerResult<T> {
     match res.await.map_err(|err| {
         // Send request error
@@ -96,10 +97,21 @@ pub async fn map_committor_request_result<T, CC: ChangesetCommittor>(
                             format!("{:?}", table_mania_err),
                         ));
                     };
-                    let cus =
-                        changeset_committor.get_transaction_cus(&sig).await;
-                    let logs =
-                        changeset_committor.get_transaction_logs(&sig).await;
+                    let (logs, cus) = if let Ok(Ok(transaction)) =
+                        intent_committor.get_transaction(&sig).await
+                    {
+                        let cus = MagicblockRpcClient::get_cus_from_transaction(
+                            &transaction,
+                        );
+                        let logs =
+                            MagicblockRpcClient::get_logs_from_transaction(
+                                &transaction,
+                            );
+                        (logs, cus)
+                    } else {
+                        (None, None)
+                    };
+
                     let cus_str = cus
                         .map(|cus| format!("{:?}", cus))
                         .unwrap_or("N/A".to_string());

magicblock-account-cloner/src/account_cloner.rs

@@ -11,7 +11,7 @@ use magicblock_account_dumper::AccountDumper;
 use magicblock_account_fetcher::AccountFetcher;
 use magicblock_account_updates::AccountUpdates;
 use magicblock_accounts_api::InternalAccountProvider;
-use magicblock_committor_service::ChangesetCommittor;
+use magicblock_committor_service::BaseIntentCommittor;
 use solana_sdk::pubkey::Pubkey;
 use tokio::sync::oneshot::channel;
 
@@ -34,7 +34,7 @@ impl RemoteAccountClonerClient {
         AFE: AccountFetcher,
         AUP: AccountUpdates,
         ADU: AccountDumper,
-        CC: ChangesetCommittor,
+        CC: BaseIntentCommittor,
     {
         Self {
             clone_request_sender: worker.get_clone_request_sender(),

magicblock-account-cloner/src/remote_account_cloner_client.rs

@@ -17,7 +17,7 @@ use magicblock_account_dumper::AccountDumper;
 use magicblock_account_fetcher::AccountFetcher;
 use magicblock_account_updates::{AccountUpdates, AccountUpdatesResult};
 use magicblock_accounts_api::InternalAccountProvider;
-use magicblock_committor_service::ChangesetCommittor;
+use magicblock_committor_service::BaseIntentCommittor;
 use magicblock_config::{
     AccountsCloneConfig, LedgerResumeStrategyConfig, PrepareLookupTables,
 };
@@ -137,7 +137,7 @@ where
     AFE: AccountFetcher,
     AUP: AccountUpdates,
     ADU: AccountDumper,
-    CC: ChangesetCommittor,
+    CC: BaseIntentCommittor,
 {
     #[allow(clippy::too_many_arguments)]
     pub fn new(
@@ -737,11 +737,10 @@ where
 
                 // Allow the committer service to reserve pubkeys in lookup tables
                 // that could be needed when we commit this account
-                if let Some(committor) = self.changeset_committor.as_ref() {
+                if let Some(committor) = self.changeset_committor.clone() {
                     if self.clone_config.prepare_lookup_tables
                         == PrepareLookupTables::Always
                     {
-                        let committor = Arc::clone(committor);
                         let pubkey = *pubkey;
                         let owner = delegation_record.owner;
                         tokio::spawn(async move {

magicblock-account-cloner/src/remote_account_cloner_worker.rs

@@ -181,7 +181,6 @@ impl AccountDumper for AccountDumperBank {
 
         let mut program_id_modification =
             AccountModification::from((program_pubkey, program_account));
-
         // point program account to the derived program data account address
         let program_id_state =
             bincode::serialize(&UpgradeableLoaderState::Program {

magicblock-account-dumper/src/account_dumper_bank.rs

@@ -1,7 +1,9 @@
 use std::sync::Arc;
 
 use magicblock_bank::bank::Bank;
-use solana_sdk::{account::AccountSharedData, clock::Slot, pubkey::Pubkey};
+use solana_sdk::{
+    account::AccountSharedData, clock::Slot, hash::Hash, pubkey::Pubkey,
+};
 
 use crate::InternalAccountProvider;
 
@@ -33,4 +35,7 @@ impl InternalAccountProvider for BankAccountProvider {
     fn get_slot(&self) -> Slot {
         self.bank.slot()
     }
+    fn get_blockhash(&self) -> Hash {
+        self.bank.last_blockhash()
+    }
 }

magicblock-accounts-api/src/bank_account_provider.rs

@@ -1,9 +1,12 @@
-use solana_sdk::{account::AccountSharedData, clock::Slot, pubkey::Pubkey};
+use solana_sdk::{
+    account::AccountSharedData, clock::Slot, hash::Hash, pubkey::Pubkey,
+};
 
 pub trait InternalAccountProvider: Send + Sync {
     fn has_account(&self, pubkey: &Pubkey) -> bool;
     fn remove_account(&self, _pubkey: &Pubkey) {}
     fn get_account(&self, pubkey: &Pubkey) -> Option<AccountSharedData>;
     fn get_all_accounts(&self) -> Vec<(Pubkey, AccountSharedData)>;
     fn get_slot(&self) -> Slot;
+    fn get_blockhash(&self) -> Hash;
 }