opt(stream): add option to directly copy over tables from lower levels (#1700) #1872

mangalaman93 · 2023-02-14T10:51:18Z

Also takes a bug fix from PR #1712, commit 58d0674

This PR adds FullCopy option in Stream. This allows sending the table entirely to the writer. If this option is set to true we directly copy over the tables from the last 2 levels. This option increases the stream speed while also lowering the memory consumption on the DB that is streaming the KVs.

For 71GB, compressed and encrypted DB we observed 3x improvement in speed. The DB contained ~65GB in the last 2 levels while remaining in the above levels.

To use this option, the following options should be set in Stream.

stream.KeyToList = nil
stream.ChooseKey = nil
stream.SinceTs = 0
db.managedTxns = true

If we use stream writer for receiving the KVs, the encryption mode has to be the same in sender and receiver. This will restrict db.StreamDB() to use the same encryption mode in both input and output DB. Added TODO for allowing different encryption modes.

CLAassistant · 2023-02-14T10:51:23Z

All committers have signed the CLA.

stream_writer.go

level_handler.go

coveralls · 2023-02-22T09:23:46Z

Coverage: 61.149% (+0.4%) from 60.713% when pulling 081b41b on aman/copy-levels into a89c52c on main.

mangalaman93 · 2023-02-26T09:35:47Z

This code has a data race, I am looking into it:

==================
WARNING: DATA RACE
Write at 0x00c000b100e8 by goroutine 2957:
  github.com/dgraph-io/badger/v3.(*StreamWriter).Write.func1()
      /home/aman/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.6-0.20230214155941-0e7c6a7a614a/stream_writer.go:212 +0x167
  github.com/dgraph-io/ristretto/z.(*Buffer).SliceIterate()
      /home/aman/gocode/pkg/mod/github.com/dgraph-io/ristretto@v0.1.1/z/buffer.go:290 +0x30a
  github.com/dgraph-io/badger/v3.(*StreamWriter).Write()
      /home/aman/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.6-0.20230214155941-0e7c6a7a614a/stream_writer.go:146 +0x194
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*countIndexer).writeIndex()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/count_index.go:188 +0x9c7
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*countIndexer).addCountEntry.func1()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/count_index.go:100 +0x47

Previous write at 0x00c000b100e8 by goroutine 228:
  github.com/dgraph-io/badger/v3.(*StreamWriter).Write.func1()
      /home/aman/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.6-0.20230214155941-0e7c6a7a614a/stream_writer.go:212 +0x167
  github.com/dgraph-io/ristretto/z.(*Buffer).SliceIterate()
      /home/aman/gocode/pkg/mod/github.com/dgraph-io/ristretto@v0.1.1/z/buffer.go:290 +0x30a
  github.com/dgraph-io/badger/v3.(*StreamWriter).Write()
      /home/aman/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.6-0.20230214155941-0e7c6a7a614a/stream_writer.go:146 +0x194
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).startWriting.func2()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:349 +0xd1
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).startWriting()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:389 +0x284
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).reduce.func4()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:475 +0x64

Goroutine 2957 (running) created at:
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*countIndexer).addCountEntry()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/count_index.go:100 +0x4c9
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).startWriting.func1.2()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:338 +0x4e
  github.com/dgraph-io/ristretto/z.(*Buffer).SliceIterate()
      /home/aman/gocode/pkg/mod/github.com/dgraph-io/ristretto@v0.1.1/z/buffer.go:290 +0x30a
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).startWriting.func1()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:336 +0x1cb
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).startWriting()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:392 +0x2bc
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).reduce.func4()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:475 +0x64

Goroutine 228 (running) created at:
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).reduce()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:475 +0x4c5
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).run.func1()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:109 +0xbe4
  github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).run.func2()
      /home/aman/gocode/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:123 +0x66
==================

ghost

Can you write a benchmark for this?

github-actions · 2024-07-19T02:16:25Z

This PR has been stale for 60 days and will be closed automatically in 7 days. Comment to keep it open.

netlify · 2025-01-27T10:40:52Z

✅ Deploy Preview for badger-docs canceled.

Name	Link
🔨 Latest commit	`ffd74f3`
🔍 Latest deploy log	https://app.netlify.com/sites/badger-docs/deploys/67976e13711d860008cabd6a

ryanfoxtyler · 2025-02-15T00:02:47Z

closing PR for now, keeping branch active

#1700) Also takes a bug fix from PR #1712, commit 58d0674 This PR adds FullCopy option in Stream. This allows sending the table entirely to the writer. If this option is set to true we directly copy over the tables from the last 2 levels. This option increases the stream speed while also lowering the memory consumption on the DB that is streaming the KVs. For 71GB, compressed and encrypted DB we observed 3x improvement in speed. The DB contained ~65GB in the last 2 levels while remaining in the above levels. To use this option, the following options should be set in Stream. stream.KeyToList = nil stream.ChooseKey = nil stream.SinceTs = 0 db.managedTxns = true If we use stream writer for receiving the KVs, the encryption mode has to be the same in sender and receiver. This will restrict db.StreamDB() to use the same encryption mode in both input and output DB. Added TODO for allowing different encryption modes.

mangalaman93 requested review from akon-dey, billprovince, joshua-goldstein and skrdgraph as code owners February 14, 2023 10:51

mangalaman93 force-pushed the aman/block-length branch 2 times, most recently from e8583aa to a3c70d1 Compare February 14, 2023 14:39

mangalaman93 force-pushed the aman/copy-levels branch from 16131ef to 0c4052d Compare February 14, 2023 15:36

mangalaman93 force-pushed the aman/block-length branch from a3c70d1 to 5c374e2 Compare February 14, 2023 15:46

mangalaman93 force-pushed the aman/copy-levels branch 2 times, most recently from 5e7a48d to 1f99a5c Compare February 14, 2023 15:52

mangalaman93 force-pushed the aman/block-length branch from 5c374e2 to d1fe28b Compare February 15, 2023 19:36

mangalaman93 force-pushed the aman/copy-levels branch from 1f99a5c to 0e2acc6 Compare February 15, 2023 19:37

mangalaman93 force-pushed the aman/block-length branch 2 times, most recently from 1375846 to 12a0e42 Compare February 18, 2023 11:40

mangalaman93 force-pushed the aman/copy-levels branch 3 times, most recently from f1a05da to 63a4948 Compare February 20, 2023 05:37

joshua-goldstein reviewed Feb 20, 2023

View reviewed changes

stream_writer.go Show resolved Hide resolved

level_handler.go Outdated Show resolved Hide resolved

mangalaman93 force-pushed the aman/block-length branch from 12a0e42 to 72a4c72 Compare February 21, 2023 03:43

Base automatically changed from aman/block-length to release/v4.0 February 21, 2023 06:22

mangalaman93 force-pushed the aman/copy-levels branch from 63a4948 to 1ab2de4 Compare February 22, 2023 09:11

mangalaman93 requested a review from joshua-goldstein February 23, 2023 18:35

mangalaman93 force-pushed the aman/copy-levels branch 2 times, most recently from f7c65e6 to ed0f577 Compare February 24, 2023 13:10

mangalaman93 force-pushed the aman/copy-levels branch from ed0f577 to 741ab83 Compare March 1, 2023 07:18

mangalaman93 changed the base branch from main-deprecated-v4 to main March 1, 2023 07:18

mangalaman93 force-pushed the aman/copy-levels branch from 081b41b to edb2318 Compare March 2, 2023 07:28

mangalaman93 changed the base branch from main to aman/sw March 9, 2023 18:10

mangalaman93 force-pushed the aman/copy-levels branch from 8873caa to 4615899 Compare March 15, 2023 19:44

mangalaman93 force-pushed the aman/sw branch from edcc231 to 663ea22 Compare March 15, 2023 19:45

ghost reviewed Mar 28, 2023

View reviewed changes

mangalaman93 force-pushed the aman/sw branch from 663ea22 to 1bd711c Compare May 17, 2023 04:43

mangalaman93 force-pushed the aman/copy-levels branch from 4615899 to e39ebb5 Compare May 17, 2023 04:43

mangalaman93 force-pushed the aman/copy-levels branch from e39ebb5 to 32230b1 Compare June 7, 2023 14:37

mangalaman93 force-pushed the aman/sw branch 2 times, most recently from 77973c6 to 908259a Compare June 12, 2023 04:07

Base automatically changed from aman/sw to aman/bug June 12, 2023 04:07

Base automatically changed from aman/bug to main June 12, 2023 04:37

mangalaman93 force-pushed the aman/copy-levels branch from 32230b1 to 08a3e2d Compare June 12, 2023 05:03

mangalaman93 force-pushed the aman/copy-levels branch from 08a3e2d to 1218725 Compare July 19, 2023 04:41

github-actions bot added the Stale label Jul 19, 2024

github-actions bot closed this Jul 26, 2024

mangalaman93 reopened this Nov 30, 2024

mangalaman93 requested a review from a team November 30, 2024 02:22

github-actions bot removed the Stale label Nov 30, 2024

mangalaman93 force-pushed the aman/copy-levels branch from 1218725 to f31ff23 Compare January 27, 2025 10:40

mangalaman93 force-pushed the aman/copy-levels branch 2 times, most recently from c024d71 to ffd74f3 Compare January 27, 2025 11:29

ryanfoxtyler closed this Feb 15, 2025

mangalaman93 reopened this Jun 27, 2025

mangalaman93 requested a review from a team June 27, 2025 12:44

mangalaman93 force-pushed the aman/copy-levels branch from ffd74f3 to 88c37d9 Compare June 27, 2025 13:31

mangalaman93 force-pushed the aman/copy-levels branch from 88c37d9 to 48955c8 Compare June 27, 2025 13:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

opt(stream): add option to directly copy over tables from lower levels (#1700) #1872

opt(stream): add option to directly copy over tables from lower levels (#1700) #1872

Uh oh!

mangalaman93 commented Feb 14, 2023 •

edited

Loading

Uh oh!

CLAassistant commented Feb 14, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

coveralls commented Feb 22, 2023 •

edited

Loading

Uh oh!

mangalaman93 commented Feb 26, 2023

Uh oh!

ghost left a comment

Uh oh!

github-actions bot commented Jul 19, 2024

Uh oh!

netlify bot commented Jan 27, 2025 •

edited

Loading

Uh oh!

ryanfoxtyler commented Feb 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

Uh oh!

opt(stream): add option to directly copy over tables from lower levels (#1700) #1872

Are you sure you want to change the base?

opt(stream): add option to directly copy over tables from lower levels (#1700) #1872

Uh oh!

Conversation

mangalaman93 commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mangalaman93 commented Feb 26, 2023

Uh oh!

ghost left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 19, 2024

Uh oh!

netlify bot commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for badger-docs canceled.

Uh oh!

ryanfoxtyler commented Feb 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

mangalaman93 commented Feb 14, 2023 •

edited

Loading

CLAassistant commented Feb 14, 2023 •

edited

Loading

coveralls commented Feb 22, 2023 •

edited

Loading

netlify bot commented Jan 27, 2025 •

edited

Loading