Does the syncs download data?

[lonnietc]

Hello,

I am new to and am exploring the Autonomi data storage network and came across your ryyn project.

In reading its description, it suggest that it syncs data across devices in vaults. Does this mean that it downloads the data to the physical devices, or does it map the device to the virtual storage on the network that is chunked across nodes? Basically, I am wondering where the data is stored and also if it is mutable since I am thinking about an application in which might need to store a mutable database on the distributed data storage network and am trying to get a feel for how it works.

Thanks for any information that you might be able to provide.

Best and have a great day

[oetyng]

Hi!

Yes, the data is stored to the network.

I'll go through the steps:

1. For a specific path on your device you create a "vault". The path will immediately be a mount of that vault. The data at that path is uploaded to the network, and that then becomes the logical "container" in the network that we call the vault, where the data is stored. Replicas connect to this vault, "clone" it, by mounting it to a path.

2. A replica is created when you mount that vault on any path on any device (could be on the same device as well, no problem). Any changes you do on that path will be stored to the vault, and be reflected on the other replicas as well - replicated. The vault conceptually is a hub, and the replicas are nodes. Changes on any replica goes into the vault, and the local mount on every replica reflects what is in the vault. "Reflect" is a key word.

3. Every replica can choose if they want that reflection of the vault to also contain the actual data (download it to the local device), or if they only want to know about it, i.e. see the folders and the files without the actual content occupying space locally.

4. For a replica to modify a file though, it has to be downloaded (as things work currently). But once the change is made, the file can be "evicted" again. Evicted is not deleted, it is just to not have the file contents physically present on the device.

On your question of mutability:

Yes, absolutely. Everything in your vault is mutable. Additionally, every version of a file that ever existed, can be accessed at any time.

But the data in the Autonomi network is never mutated (even though there exists mutable containers, this app never mutates the data). Instead, when files are modified, the delta is stored in the network. This is done by chunking the data. Additionally, chunks are deduplicated, so you are not necessarily uploading data when something changes locally.

Please, if you'd like to, tell me more about this mutable database, is it custom built or an existing one? Also what are the needs, and what type of application? I'm happy to help with figuring out how you can use the network. Feel free to ask more!

Best, and have a great day you too

[lonnietc]

Hello my friend and Thanks for getting back to me on this.

I truly appreciate you taking the time to clarify how the Ryyn project works and was wondering if you had run any speed and latency tests on it to see how it performs in the real world against some other projects like Syncthing, Rcolne, and some others just to be curious.

The project that I am developing here is to be part of a larger project. The Autonomi DDN runs basically at the network layer 4 based upon what I have read in their documentation which is great and I am working on a type of massively scaling routing VPN system that will run at the network layer 3 and have Autonomi as the data management within it. Think of this as a type of Yggdrasil VPN (https://yggdrasil-network.github.io/) global VPN system although I am not using Yggdrasil as it is much slower than what I have now, but the approach provides the data encryption P2P framework that is needed.

The database that I am seeking to develop is a property-based P2P graph database in which thousands of peers can join to effectively create a massive virtual graph database in which the graph nodes can store many different types of data properties and have many edge connections. The idea, is that there can be public and private graphs with mutable data so that users can change their data as needed.

The problem that I see with many immutable data network approaches like IPFS and many others is just that the data is immutable which means that any updates and changes creates permanent copies previous data and thus greatly consumes network storage resources. This is a problem for database where many thousands of uses might constantly be changing their data so I wanted to try and research some ways to possibly resolve this delima and come up with a low latency and speedy P2P solution if possible.

Maybe you have some ideas on this as well.

On a side note, I am listed as tallylc on the Autonomic Discord channel if you want to discuss things more as well.

Thanks again and have a great day

[oetyng]

Hey, thanks for the patience.

The public Ryyn repo is still a prototype. It hasn’t been tuned or benchmarked yet. I put it together pretty fast just to show what the system might look like. So now I'm cleaning+building it up piece by piece to get a cleaner core and a more modular layout. Performance-wise, it’ll always depend mostly on the Autonomi network itself, latency, routing, chunk locality and all that. What we can try to do is to use those primitives as efficiently as possible. There’s still a lot of headroom to explore on the client side and in how we map data to Autonomi’s structures.
But of course there are plenty of important performance design decisions and solutions tied to the implementation of Ryyn regardless of remote storage. I am always evaluating high capacity use cases / power users, as well as edge cases, like for example handling of TB file sizes. Those things always require very deliberate and careful design to even be feasible.

From what you described, users with public/private graphs and mutable data, I wonder if you expect shared mutable state across different users (not just a single user’s devices)? Ryyn right now treats a workspace as a single user, with any number of vaults under one key and multiple trusted peers writing. Each peer keeps its own event stream, and the vault’s history is basically the union of those. It has a DAG Eventstore that handles ordering and conflict resolution so that peers converge.
That same idea could work across users too (that's the plan worked with). Everyone keeps their own keys and streams, and anyone with read access can merge them into a combined view. There’s no real owner in that setup. It just depends on everyone more or less using the same merge rules and having roughly the same idea of what the shared view should be.

For your graph database, my first thought would still be to build on the DAG Eventstore. It models mutability as a sequence of immutable events connected by causality. Add an adjacency index and a blob store and you end up with something like a distributed, versioned, deduplicated property graph. Ryyn has that delta/dedup blob layer, but it’s wired for the distributed filesystem -> Autonomi object store pipeline. Your setup might want to go straight to the Eventstore and blob layers instead. I’m doing that in another project (not yet public). Both layers are suitable as standalone libraries so that they can be used in different kinds of pipelines.

I'm curious about how cross-user writes should behave (single-writer per entity, CRDT-style merges, LWW-HLC, etc.) and how “public vs private” should gate access to events/blobs, and generally what the flows are.

Happy to go into more details on discord. I sent you a request.