Skip to content

[aave-v2-revm] OutOfGas when executing repayETH #261

@sekisamu

Description

@sekisamu

Problem Overview

In pallet-revive's Ethereum compatibility mode, the gas estimate returned by eth_estimateGas (DRY_RUN) is insufficient when the actual transaction executes (REAL_TX), causing OutOfGas errors.

Environment Information

revive Version

polkadot-sdk commit: 1a788e01360730c36db448194bf8086a20e14b22

Test Project

Reproduction Steps

1. Environment Setup

# Install nvm (if not already installed)
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash

# Reload shell configuration
source ~/.nvm/nvm.sh

# Install and use Node.js 18
nvm install 18
nvm use 18

# Verify Node.js version
node --version  # Should display v18.x.x

2. Clone and Compile Project

# Clone project
git clone https://github.com/papermoonio/aave-protocol-v2-revm.git
cd aave-protocol-v2-revm

# Install dependencies
npm install

# Compile contracts
npm run compile

3. Start Local Nodes

In terminal 1, start revive-dev-node:

revive-dev-node --dev

In terminal 2, start eth-rpc:

eth-rpc --dev

4. Run Tests

# Ensure Node.js 18 is being used
source ~/.nvm/nvm.sh && nvm use 18

# Run tests (will fail before fix)
TS_NODE_TRANSPILE_ONLY=1 npx hardhat --network local test \
    ./test-suites/test-aave/__setup.spec.ts \
    ./test-suites/test-aave/weth-gateway.spec.ts \
    --grep "stable WETH"

Expected Result

Tests should pass, repayETH transaction executes successfully.

Actual Result (Before Fix)

Tests fail, repayETH transaction reverts due to OutOfGas:

Error: transaction failed [ See: https://links.ethers.org/v5-errors-CALL_EXCEPTION ]
(transactionHash="0x...", transaction={...}, receipt={"status":0,...}, code=CALL_EXCEPTION)

Hypothesis A: Timestamp/BlockNumber Difference (Primary Cause)

Investigation Process

Step 1: Observing the Phenomenon

When running Aave tests, repayETH transaction reports OutOfGas error:

Error: transaction failed [ See: https://links.ethers.org/v5-errors-CALL_EXCEPTION ]
(transactionHash="0x...", transaction={...}, receipt={"status":0,...}, code=CALL_EXCEPTION)

Initial observations:

  • Gas value returned by eth_estimateGas is insufficient to complete the actual transaction
  • Same transaction succeeds during estimation but fails during execution

Step 2: Log Comparison Analysis

Comparing DRY_RUN and REAL_TX execution through node logs (node.log):

[DRY_RUN] dry_run_eth_transact:
  input: 0x02c5fcf8... (repayETH)
  gas_limit: 18446744073709551615 (unlimited)
  Result: Success, returns estimate 987,968

[REAL_TX] bare_call:
  input: 0x02c5fcf8... (repayETH)
  gas_limit: 987,968 (using estimated value)
  Result: OutOfGas ❌

Key Finding: Same transaction input, DRY_RUN succeeds but REAL_TX fails.

Step 3: Deep Call Stack Analysis

Further analysis of output.log (30,710 lines containing complete opcode execution trace) reveals the call stack:

Frame#0: WETHGateway.repayETH()
  │
  ├─> Frame#1: Helpers.getUserCurrentDebtMemory() [delegate_call]
  │     │
  │     └─> Frame#2: StableDebtToken.balanceOf(address)
  │           │
  │           └─> Internal call to calculateCompoundedInterest()
  │                 │
  │                 └─> Multiple calls to rayMul()  ← Divergence happens here!

Step 4: Locating Opcode Execution Divergence

Using a Python script to compare DRY_RUN and REAL_TX opcode execution sequences line by line:

# Comparison results
First 1835 opcodes: IdenticalOpcode #1836 (PC 6747 JUMPI): Divergence point ❌
  - DRY_RUN: Jump not takenPC 6748
  - REAL_TX: Jump takenPC 6763

Precise Divergence Point Location:

PC 6740: JUMPDEST      ; Code block entry
PC 6741: SWAP1
PC 6742: POP
PC 6743: DUP1          ; Duplicate top stack element (condition value)
PC 6744: PUSH2 0x1a6b  ; Push jump target (6763)
PC 6747: JUMPI         ; ← Divergence point! Conditional jump
Execution Mode Condition Value Jump? Next PC
DRY_RUN 0 ❌ No jump 6748
REAL_TX Non-zero ✅ Jump 6763

Step 5: Bytecode Reverse Engineering

Reverse engineering confirmed PC 6747 corresponds to Solidity code:

// WadRayMath.sol - rayMul function
function rayMul(uint256 a, uint256 b) internal pure returns (uint256) {
    if (a == 0 || b == 0) {           // ← PC 6747 JUMPI
        return 0;                      // ← DRY_RUN takes this path
    }
    // ← REAL_TX takes this path (full overflow check)
    require(a <= (type(uint256).max - halfRAY) / b, ...);
    return (a * b + halfRAY) / RAY;
}

Question: Why does the same function call have different values for a?

Step 6: Tracing the Source of Value a

The parameter a for rayMul comes from the result of calculateCompoundedInterest:

// MathUtils.sol
function calculateCompoundedInterest(uint256 rate, uint40 lastUpdateTimestamp, uint256 currentTimestamp) {
    uint256 exp = currentTimestamp.sub(uint256(lastUpdateTimestamp));
    //            ↑ exp value depends on block.timestamp!

    if (exp == 0) {
        return WadRayMath.ray();  // Returns 1e27
    }

    // ... calculate compound interest using exp
    uint256 basePowerTwo = ratePerSecond.rayMul(ratePerSecond);
    //                                          ↑ If exp=0, this might return 0
}

Key Finding: The value of exp depends on block.timestamp!

Step 7: Verifying Timestamp Difference

Checking TIMESTAMP opcode (0x42) execution:

DRY_RUN:  TIMESTAMP @ PC 5321 returns value = T1 (old block time)
REAL_TX:  TIMESTAMP @ PC 5321 returns value = T2 (new block time, T2 > T1)

Causal chain confirmed:

DRY_RUN: timestamp = T1 (old) → exp = T1 - lastUpdate ≈ 0 → rayMul(0, x) → simplified path
REAL_TX: timestamp = T2 (new) → exp = T2 - lastUpdate > 0 → rayMul(non-zero, x) → full path

Step 8: Locating polkadot-sdk Code Issue

Tracing the timestamp source, located in eth_estimateGas implementation:

File: substrate/frame/revive/rpc/src/lib.rs

async fn estimate_gas(&self, transaction: GenericTransaction, block: Option<BlockNumberOrTag>) {
    let block = block.unwrap_or_default();  // ← Problem is here!
    // ...
}

Checking BlockTag default value:

File: substrate/frame/revive/src/evm/api/rpc_types_gen.rs

pub enum BlockTag {
    Earliest,
    Finalized,
    Safe,
    #[default]
    Latest,    // ← Default is Latest, not Pending!
    Pending,
}

Continuing to trace dry_run function:

File: substrate/frame/revive/rpc/src/client/runtime_api.rs

pub async fn dry_run(&self, tx: GenericTransaction, block: BlockNumberOrTagOrHash) {
    let timestamp_override = match block {
        BlockNumberOrTagOrHash::BlockTag(BlockTag::Pending) =>
            Some(Timestamp::current().as_millis()),  // ← Only Pending sets this!
        _ => None,  // ← Latest goes here, timestamp_override = None
    };
}

Finally confirmed the issue in Stack::new():

File: substrate/frame/revive/src/exec.rs

let mut timestamp = T::Time::now();
// Only updates when timestamp_override exists
if let Some(timestamp_override) = exec_config.is_dry_run.as_ref().and_then(|cfg| cfg.timestamp_override) {
    block_number = block_number.saturating_add(1u32.into());
    timestamp = cmp::max(timestamp.saturating_add(delta), timestamp_override);
}
// If timestamp_override = None, uses old block's timestamp!

Root Cause Summary

eth_estimateGas called without block parameter
         ↓
block.unwrap_or_default() → BlockTag::Latest
         ↓
In dry_run: Latest → timestamp_override = None
         ↓
Stack::new(): Uses confirmed block's timestamp (old)
         ↓
TIMESTAMP opcode returns old value T1
         ↓
calculateCompoundedInterest: exp = T1 - lastUpdate ≈ 0
         ↓
rayMul(0, x) → simplified path → low gas estimate
         ↓
Actual transaction executes in new block, timestamp = T2 > T1
         ↓
exp > 0 → rayMul(non-zero, x) → full path → needs more gas
         ↓
Estimated gas < Actual gas → OutOfGas ❌

Opcode Execution Statistics

Frame#2 Local Statistics (balanceOf function)

Metric DRY_RUN REAL_TX Difference
Total opcodes 342 752 +410 (2.2x)
Unique PC count 328 472 +144
child_weight_consumed 270,718,988 292,477,708 +21,758,720
child_gas_consumed 5,415 5,850 +435

Complete Transaction Statistics (output.log analysis)

Metric DRY_RUN REAL_TX Difference
Total opcode executions 11,684 13,918 +2,234 (+19.1%)
PC 5326-5332 loop 18 times 43 times +25 times
PC 7040-7047 execution 0 times 30 times +30 times
JUMPI@6747 taken 0 times 5 times +5 times
JUMPI@6747 not taken 5 times 0 times -5 times

POC Verification: Simplified Contract Reproduces Issue

To verify the correctness of Hypothesis A, we created a simplified POC contract to reproduce this issue.

POC Repository

  • repo url: https://github.com/papermoonio/time-sensitive-poc
  • Contract: TimeSensitiveContract.sol
  • Test: test/test-standard.ts

POC Design Principle

The POC contract simulates Aave's time-sensitive behavior:

// Core logic: Choose different execution paths based on timeDelta
if (timeDelta < 1) {
    // Simple path: Only calculate base interest (what eth_estimateGas sees)
    emit ZeroDeltaPath(msg.sender, timeDelta, gasleft());
    newBalance = balance + baseInterest;
} else {
    // Complex path: Execute loop calculation (what actual transaction executes)
    uint256 iterations = timeDelta * 100;
    emit NonZeroDeltaPath(msg.sender, timeDelta, iterations, gasleft());
    // ... execute iterations loop calculations
}

Running POC

cd time-sensitive-poc
pnpm install
npx hardhat compile

# Ensure revive-dev-node and eth-rpc are running
npx hardhat test --network local

POC Results

======================================================================
  POC: eth_estimateGas Timestamp Mismatch Issue
======================================================================

Contract deployed: 0x5CC307268a1393AB9A764A20DACE848AB8275c46
Account initialized with 1000 ETH balance

──────────────────────────────────────────────────────────────────────
Phase 1: Gas Estimation (uses latest block timestamp)
──────────────────────────────────────────────────────────────────────
  timeDelta seen by estimator: 0 seconds
  Expected path: Simple (no loop)
  Estimated gas: 307892

──────────────────────────────────────────────────────────────────────
Phase 2: Actual Transaction (executes in new block)
──────────────────────────────────────────────────────────────────────
  Using estimated gas limit: 307892
  Result: ❌ Failed - OutOfGas

──────────────────────────────────────────────────────────────────────
Phase 3: Retry with 150% gas limit
──────────────────────────────────────────────────────────────────────
  Using gas limit: 461838
  Result: ✅ Succeeded
  Actual gas used: 401586

──────────────────────────────────────────────────────────────────────
Analysis: Timestamp Difference Impact
──────────────────────────────────────────────────────────────────────
  Estimation phase:
    - timeDelta: 0 seconds
    - Path taken: Simple (no complex calculation)
  Execution phase:
    - timeDelta: 2 seconds
    - Path taken: Complex (200 loop iterations)

  Gas comparison:
    - Estimated: 307892
    - Actual: 401586
    - Difference: +93694 (30.43%)

======================================================================
  Conclusion
======================================================================

  ✅ Successfully reproduced the timestamp mismatch issue!

  Root Cause:
  ┌─────────────────────────────────────────────────────────────────┐
  │ eth_estimateGas uses BlockTag::Latest (old timestamp)          │
  │ → timeDelta = 0 → Simple path → Low gas estimate               │
  │                                                                 │
  │ Actual transaction runs in new block (new timestamp)           │
  │ → timeDelta ≥ 1 → Complex path → Higher gas needed             │
  │                                                                 │
  │ Result: Estimated gas < Actual gas → OutOfGas error            │
  └─────────────────────────────────────────────────────────────────┘

  This is exactly what happens in Aave's calculateCompoundedInterest!

POC Verification Conclusion

Metric Estimation Phase Execution Phase Description
timeDelta 0 seconds 2 seconds Caused by BlockTag::Latest
Execution Path Simple path Complex path (200 loops) Path divergence
Gas Consumption 307,892 401,586 +30.43%
Transaction Result - OutOfGas Fails when using estimated value

POC successfully reproduced the issue found in Aave: eth_estimateGas uses BlockTag::Latest (old timestamp) for estimation, but the actual transaction executes in a new block (new timestamp), leading to different code paths and OutOfGas error.

Hypothesis B: Weight Limit Metering Difference (Secondary Cause)

Investigation Process

Step 1: Observing Abnormal Gas Allocation in Logs

While analyzing node.log, found significant differences in gas allocation between DRY_RUN and REAL_TX:

[DRY_RUN] delegate_call:
  gas_left=18446744073708943984  ← Near uint64::MAX!

[REAL_TX] delegate_call:
  gas_left=396339  ← Normal value

Anomaly: DRY_RUN gas value is abnormally large (~1.8×10¹⁹), while REAL_TX is normal.

Step 2: Tracing Gas Decrement Pattern in Nested Calls

Extracting gas values from multi-level nested calls reveals completely different decrement patterns:

Call Level DRY_RUN gas REAL_TX gas Notes
1 18,446,744,073,708,977,836 430,191
5 18,446,744,073,708,937,093 389,013
10 18,446,744,073,708,840,927 292,376
15 18,446,744,073,708,802,627 253,640
20 18,446,744,073,708,744,410 194,970
25 18,446,744,073,708,925,474 309,832 With refund
30 18,446,744,073,708,901,100 267,480
31 18,446,744,073,708,893,387 238,971
32 (Completed successfully) OutOfGas

Key Observations:

  • DRY_RUN: Gas at each level is near uint64::MAX, barely decrements
  • REAL_TX: Gas decreases from 430,191 down to 238,971, eventually exhausted at level 32

Step 3: Searching Gas Allocation Related Code

Searching for maybe_weight_limit keyword, found the entry point difference:

File: substrate/frame/revive/src/lib.rs

// DRY_RUN - dry_run_eth_transact function
let transaction_limits = TransactionLimits::EthereumGas {
    eth_gas_limit: call_info.eth_gas_limit.saturated_into(),
    maybe_weight_limit: None,  // ← No weight limit!
    eth_tx_info: EthTxInfo::new(call_info.encoded_len, base_weight),
};

// REAL_TX - eth_call function
TransactionLimits::EthereumGas {
    eth_gas_limit: eth_gas_limit.saturated_into(),
    maybe_weight_limit: Some(weight_limit),  // ← Constrained by weight_limit!
    eth_tx_info: EthTxInfo::new(encoded_len, extra_weight),
}

Finding: DRY_RUN uses maybe_weight_limit: None, REAL_TX uses Some(weight_limit).

Step 4: Analyzing weight_limit Impact on Gas Calculation

Tracing weight_left calculation logic:

File: substrate/frame/revive/src/metering/math.rs

pub fn weight_left<T: Config, S: State>(...) -> Option<Weight> {
    let unbounded_weight_left = eth_tx_info.weight_remaining(...)?;

    // Critical branch!
    Some(match meter.weight.weight_limit {
        Some(weight_limit) => unbounded_weight_left.min(
            weight_limit.checked_sub(&self_consumed_weight)?
        ),
        None => unbounded_weight_left,  // ← DRY_RUN takes this, returns unlimited
    })
}

Impact:

  • weight_limit = None → Returns unbounded_weight_left (nearly unlimited)
  • weight_limit = Some(X) → Returns min(unbounded, X - consumed) (constrained)

Step 5: Analyzing Nested Call Propagation Rules

Continuing to trace weight_limit propagation in nested calls:

File: substrate/frame/revive/src/metering/math.rs

pub fn new_nested_meter<T: Config, S: State>(...) {
    let nested_weight_limit =
        if meter.weight.weight_limit.is_none() {
            None           // Parent None → Child None
        } else {
            Some(weight_left)  // Parent has limit → Child inherits remaining
        };
}

Propagation Rules:

  • DRY_RUN: Parent None → All child calls are None → Never constrained
  • REAL_TX: Parent Some(X) → Child calls inherit remaining value → Decreases at each level

Root Cause Summary

DRY_RUN (maybe_weight_limit = None):
  Root:    weight_limit = None
    ↓
  Call 1:  nested_weight_limit = None  (unconstrained)
    ↓
  Call 2:  nested_weight_limit = None  (unconstrained)
    ↓
  Call N:  nested_weight_limit = None  (always unconstrained) ✅ Success

REAL_TX (maybe_weight_limit = Some(X)):
  Root:    weight_limit = X
    ↓      consumed_1
  Call 1:  nested_weight_limit = min(unbounded, X - consumed_1)
    ↓      consumed_2
  Call 2:  nested_weight_limit = min(unbounded, prev - consumed_2)
    ↓      consumed_3
  Call N:  nested_weight_limit keeps decreasing... may < needed → OutOfGas! ❌

Problem Summary

Comparison Summary

Dimension Hypothesis A: Timestamp Difference (Primary) Hypothesis B: Weight Limit Difference (Secondary)
Problem Level eth-rpc → pallet-revive → EVM pallet-revive metering layer
Direct Cause BlockTag::Latest vs new block maybe_weight_limit set differently
Impact Mechanism Different timestamp → different calculation results → different code branches Different weight decrement rules, differences accumulate
Quantified Impact 2.2x opcode difference, +21.7M ref_time Gas exhausted after 32 nested levels
Verification Status ✅ Verified and fixed 🟡 Pending independent verification

Compound Effect Model

┌─────────────────────────────────────────────────────────────────────────┐
│                        Problem Compound Model                           │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  DRY_RUN Phase:                                                         │
│    • BlockTag::Latest → timestamp = old value (Hypothesis A)           │
│    • maybe_weight_limit = None → unconstrained (Hypothesis B)          │
│    • Takes simplified path (342 opcodes)                               │
│    • Returns estimate: 987,968 gas                                     │
│                                                                         │
│  REAL_TX Phase:                                                         │
│    • New block → timestamp = new value → different code path (A)       │
│    • maybe_weight_limit = Some(X) → constrained, differences           │
│      accumulate (Hypothesis B)                                         │
│    • Takes full path (752 opcodes, 2.2x)                               │
│    • Actual need > estimate → OutOfGas                                 │
│                                                                         │
│  Compound Effect:                                                       │
│    Hypothesis A causes: More opcodes executed (+410)                   │
│    Hypothesis B causes: Insufficient gas in deep calls                 │
│    Combined: OutOfGas triggered earlier and more severely              │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

Verification Status

Hypothesis Status Verification Method Conclusion
A: Timestamp Difference Verified and Fixed Modified dryrun timestamp logic All 44 identical tests pass
B: Weight Limit Difference 🟡 Pending Independent Verification Requires separate modification Secondary factor

Impact

  1. Gas value returned by eth_estimateGas is unreliable for time-sensitive contracts
  2. DeFi protocols with interest calculations (like AAVE) cannot execute normally
  3. Problem is more severe for deeply nested calls
  4. Users submitting transactions with estimated values will get OutOfGas

Suggested Fix Solutions

Solution 1: Change estimate_gas Default Block Tag

For Hypothesis A - Alternative Solution

// substrate/frame/revive/rpc/src/lib.rs
async fn estimate_gas(
    &self,
    transaction: GenericTransaction,
    block: Option<BlockNumberOrTag>,
) -> RpcResult<U256> {
    // Use Pending instead of Latest as default
    let block = block.unwrap_or(BlockNumberOrTag::Tag(BlockTag::Pending));
    // ...
}

Fix Verification Results

Using Solution 1, running test suite after fix:

Test File Test Count Status
weth-gateway.spec.ts 11 ✅ Pass
flashloan.spec.ts 17 ✅ Pass
liquidation-atoken.spec.ts 6 ✅ Pass
liquidation-underlying.spec.ts 6 ✅ Pass
atoken-transfer.spec.ts 4 ✅ Pass
Total 44 ✅ All Pass

Solution 2: Add Safety Factor

Universal Solution

// Add buffer when returning estimate
let estimated_gas = dry_run_result.gas_used;
let safe_gas = estimated_gas * 150 / 100;  // 1.5x safety factor

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions