Super Duper WIP Start to Server Structure

[zicklag]

Hey there! Just figured I'd share my totally unready code so that you can see what I'm working on. This doesn't really do much yet but get's some structure. Also, it isn't really gun wire protocol compatible or anything yet.

I know you're more on the no-dependencies side of things, but I've tried to keep it as minimal as possible so far while keeping what I think is useful.

Currently an optimized WASM build still only weighs 374k and the native build on Linux is 392k stripped.

Just in case that's not good enough ( 😉 ), I think the use of extra dependencies are more justified in the Rust implementation than the JavaScript implementation because in Rust, you don't have the same pre-existing utilities such as you do in JavaScript.

In JavaScript if you want to access a HashMap ( equivalent ) concurrently, you just do it. JavaScript comes with it's implementation and you are stuck with it, in Rust, we use a concurrent HashMap implementation from crates.io. The same applies for async. In Rust you have to have an async executor, but JavaScript comes with one. The cool part is that with Rust we get to choose.

I'm probably over-selling this, but I just wanted to address concerns about what might seem a lot of deps.

I also think that while I think it's useful to start with pre-built dependencies for a set of things such as the HashMap and async executor, we could also rewrite some of this ourselves to optimize for our use-case and goals, but start of using these external dependencies to help us get something that works with less initial development.

Also, we might be able to do some more fancy tricks to cut down further on size, but I don't know them all yet.

[zicklag]

Aha! I just got the WASM build to be 156k by compiling std with stripped out panic handling, and getting rid of the async executor that we don't need when running in the browser.

There might still be more to trim out, but not sure. Also I'm not sure if this will be much bigger or not when we actually write more of the implementation, as this will mostly just sum up the weight of our dependencies right now.

[zicklag]

Getting more of the basic structure fleshed out. So far I've got some Rust structures for the graph data, conversions of those structures for serializing/de-serializing to efficient binary representations, and another conversion for serializing/de-serializing from the GUN JSON format used by the official JavaScript implementation.

I've also gotten a basic storage interface and a super simple filesystem implementation of the storage interface that will later be replaced with a sled database ( which is supposed to really beat the performance of LevelDB and RocksDB, BTW ).

I use a convention for storing and serializing binary data that allows you to write code that writes and read binary values transparently, which will be converted automatically to base64-encoded strings, but only for those serialization backends that don't support binary data natively ( i.e. JSON & LocalStorage ).

This should allow supporting clients to use the binary transport that uses Borsh binary serialization when possible, but to also transparently use the official GUN JSON implementation if necessary.

Need to do some work now on the conflict resolution and the basic database API.

Also, for reference, here is the Rust API docs so far: https://katharostech.github.io/rod/rod/index.html.

---

Edit: Also, the WASM build is now only 47k gzipped! I'm pretty sure we can squeeze even more out of it, but it will require replacing or modifying some dependencies a little and creating shims for things we don't need out-of-the-box on WASM ( like maybe tracing ). I think it's plenty tiny enough for now and once we get it working we can squeeze every last bit out of it if we want.

[zicklag]

Got a first pass for the Rust API now:

    let rod = &Rod::new().await?;

    let mut mary = rod.get("users/mary").await?;
    mary.set("name", "Mary".to_string());
    mary.set("age", 32);
    rod.put("users/mary", &mary).await?;

    let mut john = rod.get("users/john").await?;
    john.set("name", "John".to_string());
    john.set("wife", &mary);
    rod.put("users/john", john).await?;

    let john_wife_name = rod
        .get("users/john")
        .await?
        .get("wife")
        .unwrap()
        .follow()
        .await?
        .get("name")
        .unwrap()
        .owned();

More verbose than the JavaScript version, but that's typical of Rust and it's actually still fairly succinct. I might be able to add some more chaining for setting node values, but not sure if I want to yet.

By re-exporting the tap crate we can also do things more chained without any extra effort on our part:

    let mary = rod
        .get("users/mary")
        .await?
        .tap_mut(|x| x.set("name", "Mary".to_string()))
        .tap_mut(|x| x.set("age", 32));
    rod.put("users/mary", &mary).await?;

    rod.get("users/john")
        .await?
        .tap_mut(|x| x.set("name", "John".to_string()))
        .tap_mut(|x| x.set("wife", &mary))
        .pipe(|x| rod.put("users/john", x))
        .await?;

    let wife_name = rod
        .get("users/john")
        .await?
        .get("wife")
        .unwrap()
        .follow()
        .await?
        .get("name")
        .unwrap()
        .owned();

[zicklag]

@amark I'm going to try to get on a call with you as soon as I can. Unfortunately my schedule is as predictable as the stock market so we might just have to try to get lucky on when to do it. When I get time I'll try to ping you on Matrix to see if we can get on a call. 🤞

[zicklag]

With the discovery of wasm-strip now the gzipped WASM binary is only 23k!