Need more basic explanations and examples for bitfield

[ZhouHansen]

These are some 🔦questions:

I red up bitfield, sparse-bitfield though. I've still struggled to figure out the bitfield module for few days. The source of indexed-field and hypercore/bitfield are both hard to read. And sleep.md/bitfiled is similar to this document. So I decide to ask questions now.

1. Is this module for sparse file, or something similar to it？
2. In Types of Bitfields
   "Data Bitfield: Indicates which data you have, and which data you don't. " What does 'the data you have' means? Does this means data entries I've already downloaded, then when update coming, I don't need to download them again? I don't know. Hypercore is a append-only log. So if there are 3 entries in metadata.data, the Data Bitfield will be 111000000?

I want to figure out this one first.

[yoshuawuyts]

What does 'the data you have' means? Does this means data entries I've already downloaded, then when update coming, I don't need to download them again?

Yeah, that covers the gist of it! -- because Hypercore is append-only, data can be tracked by just flipping a bit, and once it's flipped we don't need to download that piece of data ever again. sparse-bitfield is mostly an efficient way of storing that data on disk.

So if there are 3 entries in metadata.data, the Data Bitfield will be 111000000?

The data is stored in a tree format, so it would be represented differently on disk. But it seems you've got the right intuition about it!

---

I'll be spending some time after this month to write more docs. Questions like these are super helpful to know where we should focus more on writing docs. Please keep the questions coming! 🎉

[ZhouHansen]

From bitfield.md

Example

bits:  00101
index: 01234

The set above contains 2 and 4.

Hypercore is append-only, It seems when the previous bit is 0 then the next rest of bits should all be 0, so why this happens: the 0 and 1 is 0, but 2 is 1.

[yoshuawuyts]

Oh yeah, so while Hypercore is append-only, it also supports sparse replication. So:

• new data can only be appended
• data can be randomly replicated between peers

The bitfield represents the data that you have stored on your machine. Because we can do sparse copies (e.g. only copy a few entries in hypercore, or just one file in Hyperdrive) it must be able to represent that in its data structure, which means interior mutability.

Does that make sense?

[ZhouHansen]

Yeah, that's what I want to make sure. Thanks

[ZhouHansen]

I'm learning hypercore, I think SLEEP files prefixed by content are in charge of hypercore, files prefixed by metadata are in charge of hyperdrive. Right?

[yoshuawuyts]

I'm learning hypercore, I think SLEEP files prefixed by content are in charge of hypercore, files prefixed by metadata are in charge of hyperdrive. Right?

Not quite; they're both hypercore instances. One keeps track of the content, the other keeps track of the metadata. In newer version of hyperdrive, the metadata core is a hypercore that's formatted as a hyperdb.