Question about file system sync performance

[beckend]

I am testing the creation and opening of the database, inserting 1_000_000 keys and values.
Is this expected comparing to redb?

git repo

Clone

$ cargo bench

I open the database, run the same function twice for redb and fjall to get a grip on creation and reopening of the database.

fn mapper(
  path: impl AsRef<Path> + ToOwned,
) -> impl Future<Output = std::result::Result<DBTest<TransactionalKeyspace>, Report>> {
  let path = path.as_ref().to_owned();
  let fn_create_open = get_db_fs_inner_fjall;

  async move {
    tokio::task::spawn_blocking(move || {
      let db = fn_create_open(path, false)?;
      let partition = db.open_partition(PARTITION, Default::default())?;

      {
        let mut write_txn = db.write_tx();

        for i in 0..BENCH_ITEMS_AMOUNT {
          let v = Bytes::from(format!("test{i}"));
          write_txn.insert(&partition, v.as_ref(), v.as_ref());
        }
        write_txn.commit()?;
      }

      db.persist(fjall::PersistMode::SyncAll)?;

      Ok::<_, Report>(db)
    })
    .await?
  }
}

For fjall, I use PersisMode::SyncAll.
For redb, I use Durability redb::Durability::Immediate.

     Running benches/fjall_benchmark.rs (target/release/deps/fjall_benchmark-49a2e5d08d486de7)
Benchmarking fjall: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 40.8s.
fjall                   time:   [4.0070 s 4.0074 s 4.0078 s]
                        change: [-0.0234% -0.0042% +0.0138%] (p = 0.69 > 0.05)
                        No change in performance detected.

     Running benches/redb_benchmark.rs (target/release/deps/redb_benchmark-3e38810964d04089)
Benchmarking redb: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 23.3s.
redb                    time:   [2.2651 s 2.2927 s 2.3238 s]
                        change: [-2.4912% -1.0028% +0.7741%] (p = 0.27 > 0.05)
                        No change in performance detected.

[marvin-j97]

ReDB uses fsyncdata.

PersistMode::SyncAll uses fsync, which tends to be 2x slower than fsyncdata, because it needs 2 disk hops (data + metadata) instead of 1 (just data).

[beckend]

I actually tried fsyncdata now on fjall:
I get about the same time as with SyncAll.

This is my system.

System:
  Kernel: 6.9.10-arch1-1 arch: x86_64 bits: 64 compiler: gcc v: 14.1.1
    clocksource: tsc avail: hpet,acpi_pm parameters: initrd=\initramfs-linux.img
    root=LABEL=OSRoot rootfstype=bcachefs rootflags=relatime rw
    pcie_aspm.policy=powersupersave transparent_hugepage=madvise nowatchdog
    loglevel=4 resume=LABEL=Swap amd_pstate=guided zswap.enabled=1
    zswap.compressor=zstd zswap.max_pool_percent=20 zswap.zpool=z3fold
  Desktop: Hyprland v: N/A tools: swayidle avail: swaylock dm: SDDM
    Distro: Arch Linux

[marvin-j97]

What output do you get when running:

fn main() {
    {
        let path = std::path::Path::new("./my_db.redb");
        if path.exists() {
            std::fs::remove_file(path).unwrap();
        }

        const TABLE: redb::TableDefinition<&str, &str> = redb::TableDefinition::new("my_data");
        let db = redb::Database::create(path).unwrap();

        for _ in 0..5 {
            let start = std::time::Instant::now();

            {
                let mut write_txn = db.begin_write().unwrap();
                write_txn.set_durability(redb::Durability::Immediate);

                {
                    let mut table = write_txn.open_table(TABLE).unwrap();
                    for i in 0..1_000_000 {
                        let v = format!("test{i}");
                        table.insert(v.as_str(), v.as_str()).unwrap();
                    }
                }
                write_txn.commit().unwrap();
            }
            eprintln!("redb: {:?}", start.elapsed());
        }
    }

    {
        let path = std::path::Path::new("./fjall");
        if path.exists() {
            std::fs::remove_dir_all(path).unwrap();
        }

        let keyspace = fjall::Config::new(path).open_transactional().unwrap();
        let table = keyspace
            .open_partition("table", Default::default())
            .unwrap();

        for _ in 0..5 {
            let start = std::time::Instant::now();

            {
                let mut write_txn = keyspace.write_tx();

                for i in 0..1_000_000 {
                    let v = format!("test{i}");
                    write_txn.insert(&table, &v, &v);
                }
                write_txn.commit().unwrap();
            }
            keyspace.persist(fjall::PersistMode::SyncAll).unwrap();

            eprintln!("fjall: {:?}", start.elapsed());
        }
    }
}

On two different SSDs I get:

redb: 2.585170534s
redb: 1.372190959s
redb: 1.387693487s
redb: 1.363789312s
redb: 1.377867999s
fjall: 1.217479161s
fjall: 1.448499643s
fjall: 1.291164503s
fjall: 1.41874197s
fjall: 1.361954581s

and on an HDD I get:

redb: 4.517093614s
redb: 3.316358699s
redb: 5.091505315s
redb: 2.574284788s
redb: 2.619253589s
fjall: 1.751218769s
fjall: 2.062881187s
fjall: 1.852786121s
fjall: 1.966631933s
fjall: 2.815284892s

[beckend]

I have an NVME, a fast one, however I am running bcacheFS so it affects a lot.
I made some slight modification on the test above you gave me since I wanted to repopen an existing database, I moved the open table/partition into the loop.

redb: 4.00804402s
redb: 3.243745319s
redb: 3.219940113s
redb: 3.209736979s
redb: 3.193510602s
fjall: 1.497419337s
fjall: 1.441717064s
fjall: 1.437103679s
fjall: 1.41079111s
fjall: 1.40825687s

[beckend]

I don't see why fjall would be much slower in my bench, is it because of tokio async?

[marvin-j97]

I don't see why fjall would be much slower in my bench, is it because of tokio async?

I get more-or-less the same performance when wrapping the loop body in a tokio task and awaiting it.

since I wanted to repopen an existing database, I moved the open table/partition into the loop

Dropping a partition handle doesn't cause it to reload when "opening" the second time. In fact, all partitions are opened once after recovery anyway, so partition_open is almost a no-op on previously created partitions. But get_partition_handle would be a bit verbose of a name.

[beckend]

Thank you for the help, it's a great database.

[beckend]

I will however use fjall in a tokio context, so this is quite bothersome using tokio::task::spawn_blocking.

[marvin-j97]

I found that 2.0 degraded the sync write performance (but only on certain disks??), I think 9cf22eb should deal with that...