Doesn't DAT Need Official HTTPS Gateway?

[ilyaigpetrov]

IPFS has a https-gateway used as https://ipfs.io/ipfs/\<hash\>.
The existence of such gateway gives the following benefits:

1. Search engine crawlers may access these pages.
2. Users may access ipfs/dat pages without ipfs installed.
3. If access to the pages is blocked by censors then:
   1. pages should still appear in search results.
   2. users may still install ipfs and ipfs-companion browser extension to access these pages.

@RangerMauve has implemented a http(s) gateway based on pfrazee/dat-gateway working in a way similar to ipfs (repo, demo).
It even redirects each page to a subdomain to insulate cookies and provide better security.

Concerning datbase.org — I couldn't get it to serve me a html page the right way (without the header and maybe with absolute css paths which is not critical).

I kindly ask the DAT team to take a view on the work @RangerMauve and similar works and provide users with an official DAT https gateway, which we may use to build our future censorship-resistant websites.

[RangerMauve]

One thing I'd also like to note is that I'm working on having dat-gateway automatically inject a DatArchive polyfill so that sites that make use of it can work without any extensions.

[joehand]

Hey! Been really great to see the work @RangerMauve and others are doing on gateway and related work. You are right on the benefits of having the gateway. While an official gateway can be helpful it could also lead to copyright & legal issues beyond the resources of our nonprofit. We'd like to keep the scope of the Dat Project focused on the core technology to ensure sustainability in the long-term, and using resources otherwise may detract from that.

It's really great to see the community efforts around this and we'll continue to support them however we can.

Concerning datbase.org...

Dat Base was intended a registry not necessarily a gateway and I'm not sure we could cover all uses without doing subdomains.

[RangerMauve]

Hi, I'd like to hijack this issue to talk about gateways and the such.

There was a meeting with some people from the Dat community to talk about gateways and getting Dat working in the browser on Wednesday Feb 27. Here are the meeting notes (courtesy of @substack)

Details

# dat in browsers notes

2019-02-27

# on the call

* diana: working on dat gateway
* franz: working on archipel
* mauve: working on webrtc/websockets/discovery-swarm-stream
* substack: working on peermaps

# franz: working on archipel

archipel is an electron app using an rpc layer over websockets

hyperstack module: hyperdrive and hypergraph, but probably extensible

same apis for using dat apis on the browser or the server

goal: allow the same backend code to run in a webworker in the browser

# webrtc/gateway/signaling

mauve: got webrtc working on signalhub

prioritize webrtc connections by setting a delay to start using websockets
establishes a websocket connection too after some seconds

signalhub is great for a single swarm
swarm per key in a browser is very limiting

discovery-swarm-stream: rpc over websockets for the discovery-swarm api
mux several streams through websockets

with dat gateway,
dat key gets leaked to the gateway: privacy issue

applications shouldn't necessarily share data with 3rd party gateways

does discovery-swarm-stream make sense?
act as a signaling server over the same protocol?

franz: many use-cases for browser/server storage/processing splits
easy to use rpc api for easier frontend apps
browser vs server has different trade-offs for storage/network etc

substack: peermaps is browser-first p2p for maps
using iframes to handle cross-domain maps?

mauve: gozala is working on lunet using service workers paired with iframes
to serve ipfs or will proxy to local ipfs daemon

substack: how do custom extensions work with gateway/signaling/rpc work?

mauve: gateway probably wouldn't work with custom extensions, but custom
extensions over discovery-swarm-stream should work.

franz: with hyperstack, the backend has the different implementations
how to make it run in the browser? looking at lunet, different replication
possibilities for gateways etc

mauve: do we really need shared storage across origins? what about iframe post
messages

if you have different apps with different data, do you need to necessarily
coordinate on different origins? seems complicated

substack: running tools like lunet makes sense for seeding from the browser and
managing connections from a place/single interface

We also spoke about this on Thursday the 28th at the dat comm-comm call.

The gist of it is that it'd be nice to find a standard way of doing dat stuff in the browser.

The main pieces that I see (feel free to add more) are:

• A standard and efficient way of working with webrtc-swarm (Something similar to signalhubws??)
• A standard gateway protocol for reaching out to the Dat network (Maybe discovery-swarm-stream)
• An standard HTTP interface for dealing with hyper-* data structures for when you don't want to create them client-side (@Frando could elaborate)
• A way to share resources like connections to the swarms / gateways and storage between origins. (Maybe something based on gozala/lunet)

I looked at some this stuff a while ago when I was working on dat-polyfill and again recently when working on dat-js.

I propose meeting on Wednesday February the 6th at 20:00 GMT to discuss this stuff and maybe start putting parts together. We could use audio-only setups with https://talky.io/dat-in-browsers to talk about it.

Personally, I'd like to work on combining the signalhub / discovery-swarm-stream code so that we could support replicating multiple hyperdrives through both WebRTC and proxying to the discovery-swarm all with a single websocket connection. (Also integrating hyperswarm once that stabilizes)

Does that time work for you all? Is there a better date or time? Any other items that I could add to the list of stuff to talk about?

CC @garbados @substack @Karissa @Frando @dpaez @tinchoz49 @Gozala

[RangerMauve]

Also, I'm making a calendar invite. Email me at ranger@mauve.moe if you'd like to be added to the calendar.

• A standard and efficient way of working with webrtc-swarm (Something similar to signalhubws??)

I think a good way of accomplishing this is to publish a set of capabilities to the peer table when joining a swarm. So for example if I am a browser I can communicate as a websocket-client and webrtc-peer. And if I'm an electron app I can communicate as a websocket-client, websocket-server (if I have a public IP or can hole-punch), tcp client, tcp server (if I have a public IP or can hole-punch), udp etc. Peers could also publish their connection preferences to the table. Then to make a peer connection, clients can consult this table along with their own heuristics to make the best connection possible according to some mutually acceptable preferences.

These kind of hybrid swarms would be very useful for merging what would otherwise be fairly separate networks based on transport protocols.

Apologies if this is already the plan, although if so I guess this comment will help to disambiguate.

[RangerMauve]

Yeah, that's a great idea! One thing I was thinking of is that it'd be cool if these gateway servers published their existence to the discovery swarm under a know key. Then you could connect to one and discover more through it and potentially save them for later.

[okdistribute]

@pvh o/

[RangerMauve]

Re: the capabilities. We should discuss (on the call) how to do this stuff without reimplementing libp2p. 😅

[Gozala]

Re: the capabilities. We should discuss (on the call) how to do this stuff without reimplementing libp2p. 😅

Is there reason to not collaborate on libp2p itself?

Reading this thread I was feeling: oh that’s exactly goal of libp2p, which also happens to have rust implementation so you could in theory wasm it.

[pvh]

I think for the same reason it's healthy to have both KDE and Gnome, or Mozilla and Firefox, or Linux and FreeBSD, it's not wise to create a monoculture.

The dat community and the IPFS community have different ethos, technical goals, funding models, development methodology and values. I think both should inspire and be inspired by each other and drive one another to improve but it doesn't make much sense to me for the dat community to adopt the IPFS codebase.

[pvh]

As for web gateways, if you've been following the work Ink & Switch has been doing, we've been discussing something morally along the lines of the DatArchive injection (though I think we envision a quite different actual implementation) to extend our system to non-Electron computers like iPhones and browsers.

Roughly, because a first-order goal for us to is to support totally offline usage we've discussed bridging hypermerge repositories over a websocket gateway but also wrapping all of that magic in a PWA that kept the data in localStorage (or something) for improved durability.

[RangerMauve]

@pvh Would you be interested in attending the call?

Also ping @sammacbeth. He's using gateway stuff in https://github.com/cliqz-oss/dat-webext

[Gozala]

I think for the same reason it's healthy to have both KDE and Gnome, or Mozilla and Firefox, or Linux and FreeBSD, it's not wise to create a monoculture.

The dat community and the IPFS community have different ethos, technical goals, funding models, development methodology and values. I think both should inspire and be inspired by each other and drive one another to improve but it doesn't make much sense to me for the dat community to adopt the IPFS codebase.

I think there are few caveats here that is worth considering:

• Multiple implementations of the same standard is important (otherwise what's the point of standard) there is no compatibility at any layer here.
• Adopting libp2p doesn't imply implementing Dat over IPFS. libp2p is a modular networking stack used by community that steches way beyond IPFS. It's actually really well structured allowing one to plug different transport implementation and keep the application layer (like IPFS) the same. That is to say this group can implement own even competing transport which would in fact benefit larger community. e.g. Manyverse got Bluetooth transport that PL is also building for libp2p and down the line this group probably will end up doing it too. If that was a libp2p module instead everyone would gain additional transport rather than reinventing a wheel.
• If you consider porting to different languages you face the same reinvention of the wheel. Libp2p has Rust implementation both SSB and Dat have ongoing efforts to have one too.
• If there was one networking stack it would be far more easier to get wider adoption both in browsers and OSs. I can promise you that if there were more interop I would have being far more successful in getting it into browser but now every conversation derails into picking a winner.
• P2P network will be more resilient the more peers it has, sharing DHT, Routing nodes, clients applications each one will be better off.

Please note that does not imply that:

• You have to share code base, if you want to compete on alternative but compatible implementation everyone wins!
• You don't have to buy into ethos, technical goals or funding model. Only thing you need to buy into is an architecture and even there a good argument towards some changes will get you pretty far (in my experience).

I apologize for derailing this conversation, it just saddens me that instead of making greater progress towards decentralization communities across the board choose to keep reinventing the same wheel which slight technical differences. It could be that overhead of coordination across groups would have higher overhead than the value to be gained by that, but that's rarely argument.