A method for transferring HTTP communication over Nostr direct-messages

[oren-z0]

Example use case: https://v.nostr.build/VAREX7f6XB8sTPQr.mp4
(and here is the old video that was implemented with NIP-04 direct-messages not following this NIP exactly but showing the purpose: https://vimeo.com/950881613 ).

Implementation: http://github.com/oren-z0/http2nostr http://github.com/oren-z0/nostr2http
rpm-packages: http://npmjs.com/http2nostr http://npmjs.com/nostr2http

[vitorpamplona]

Very cool!

But don't use NIP-04 or NIP-17 for this. Just create your own event kind. NIP04 and NIP-17 are for plain text, human-readable content. You can just use the same encryption (NIP44) in a different kind that is designed for this, like NIP-46 and NIP-47 do.

[oren-z0]

Very cool!

But don't use NIP-04 or NIP-17 for this. Just create your own event kind. NIP04 and NIP-17 are for plain text, human-readable content. You can just use the same encryption (NIP44) in a different kind that is designed for this, like NIP-46 and NIP-47 do.

HTTP is also plain text, human readable content (over TCP). I don't see a reason for a new event kind - it will just add unnecessary complexity and security concerns. A remote observer should not be able to tell if this is a direct-message with "human" content or an http request/response. If the direct-messages are secure enough, then they're good for any type of content. K.I.S.S.

[vitorpamplona]

A new event kind allows processors to filter only the messages they need to parse. There is no need to download the 1000s of DMs just to process one of these calls. In the same way, there is no need to download these messages if you only want to show DMs to the user.

Don't overload event kinds. Reusing the same kind for everything is not kiss.

[shocknet-justin]

We do this with Lightning.Pub (kind 21000) using protos for http-nostr

An ephemeral kind is necessary because otherwise you're going to fill up a relay with all http events

As for the lnurl piece, there's still an ssl requirement for reverse compatibility so a new bi-directional RPC has replaced it since we no longer have the limitations of url's themselves

[oren-z0]

We do this with Lightning.Pub (kind 21000) using protos for http-nostr

An ephemeral kind is necessary because otherwise you're going to fill up a relay with all http events

As for the lnurl piece, there's still an ssl requirement for reverse compatibility so a new bi-directional RPC has replaced it since we no longer have the limitations of url's themselves

I totally agree with the ephemeral part. I can also imagine a use-case for ephemeral direct-messages (sending encrypted direct-messages that there is no reason to keep for long-term). We can have kind 20004 be the ephemeral equivalent of kind 4 (as in NIP-04) and kinds 20014, 20013 the ephemeral equivalent of kinds 14 and 13 (as in NIP-17).

Give me a js library that support it and I'll upgrade http2nostr and nostr2http to use them (Let's start with a js library that implements NIP-17?).

Managing your node over Nostr instead of TOR or Tailscale is also really cool, but from a quick look at Lightning.Pub it feels like you're trying to reinvent the idea of HTTP JSON apis. It may be more efficient, but does it worth it?

My initial idea was not even to implement "HTTP over Nostr direct-messages", but implement "TCP over Nostr direct-messages", but there are more edge-cases to support.

[vitorpamplona]

We can have kind 20004 be the ephemeral equivalent of kind 4

NIP-04 has been deprecated. There is no benefit on creating something based on that encryption standard.

kinds 20014, 20013 the ephemeral equivalent of kinds 14 and 13 (as in NIP-17).

You are confusing some of the kinds.

The encryption you are looking for is NIP-44. That can be used in any kind you see fit, including directly in the .content of the new kinds for this protocol.

Kind 14 are for human-readable chat messages alone. They must be rendered as chat messages in all supporting clients. HTTP request JSONs should not be rendered as chat messages.

Kind 13 is the companion to kind 1059 if you need to use GiftWraps NIP-59. Kind 13 and 1059 are used for extreme privacy needs when not even the type of event is available for the public to query on relays. I don't actually see much value of that level of privacy for this protocol, but I might be missing something.

Again, NIP-46 and NIP-47 are the model you should be copying. Just replace their use of NIP-04 encryptions for NIP-44 encryptions.

Give me a js library that support it and I'll upgrade http2nostr and nostr2http to use them (Let's start with a js library that implements NIP-17?).

The NDK and nostr-tools both support NIP-44 encryptions.

[oren-z0]

Kind 14 are for human-readable chat messages alone. They must be rendered as chat messages in all supporting clients. HTTP request JSONs should not be rendered as chat messages.

Who cares if the content is a human-readable text or a string containing a JSON object?
The DMs are intended to be used between a web-client and a web-server with their own private-public keys, unrelated to the users' regular Nostr accounts.

It's too much math for me 😨... Please implement in @rust-nostr/nostr-sdk or in a different js library some client.sendDirectMessage(...) and the equivalent handler to decrypt incoming messages, and I'll happily update http2nostr and nostr2http. It would also be great if the events would be ephemeral for a minute or two, because there is no need to store them for a long time.

[vitorpamplona]

Who cares if the content is a human-readable text or a string containing a JSON object?

I do. I don't want to have to deal with custom JSONs polluting everything. We don't even accept Markdown in chat messages.

I don't even understand why you are insisting on this. Just pick a different number to separate things out. It's not hard. This is why event kinds were designed for... So that we don't need to parse every type of payload out there.

[vitorpamplona]

The DMs are intended to be used between a web-client and a web-server

I don't know who told you this, but that is not what DMs were designed for.

[oren-z0]

OK, I actually found out that @rust-nostr/nostr-sdk already implemented NIP17, so implementing something similar with different kinds shouldn't be too difficult.

I just wanted the implementation to be quick, and was frustrated to find out that I need patches for many repos (both the main LNbits repo, the LNbits lnurlp extension, the LNbits lnurl parser, etc.). I'm worried that if we add another kind it could take months for actual wallets to implement this.

Let's see if I understand how NIP17, etc. work.

So suppose I'll use kind:50 instead of kind:14 for the request-message, and we can have the stringified json object in the content field (I dropped the "e" and "subject" tags which are irrelevant to this use case, and we have a single receive-pubkey):

{
  "id": "<usual hash>",
  "pubkey": "<sender-pubkey>",
  "created_at": "<now>",
  "kind": 50,
  "tags": [
    ["p", "<receiver-pubkey>", "<relay-1-url>", "<relay-2-url>"] // can we have more than one relay here?
    ...
  ],
  "content": "{
    \"id\": \"0da8a3dd-80e2-4fca-ba93-5c4261edcc9e\",
    \"url\": \"/api/test\",
    \"method\": \"GET\",
    \"headers\": {
      \"accept\": \"*/*\",
      \"user-agent\": \"curl/8.6.0\"    
    },
    \"bodyBase64\": \"\"
  }"
}

The response message could have kind:51 because the content is slightly different (has status instead of url and method):

{
  "id": "<usual hash>",
  "pubkey": "<sender-pubkey>",
  "created_at": "<now>",
  "kind": 51,
  "tags": [
    ["p", "<receiver-pubkey>", "<relay-1-url>", "<relay-2-url>"] // can we have more than one here?
    ...
  ],
  "content": "{
    \"id\": \"0da8a3dd-80e2-4fca-ba93-5c4261edcc9e\",
    \"status\": 200,
    \"headers\": {
      \"date\": \"Fri, 31 May 2024 00:00:00 GMT\",
      \"content-type\": \"application/json; charset=utf-8\"
    },
    \"bodyBase64\": \"eyJoZWxsbyI6IndvcmxkIn0=\"
  }"
}

I don't actually see much value of that level of privacy for this protocol, but I might be missing something.

I actually do think it's important. Suppose I'm scanning a static lnurl-pay QR code - I would really prefer that nobody could tell that I interacted with this server - by wrapping whatever needs to be wrapped.
However, unlike direct-messages between humans, the web-client doesn't need a fixed "private-public key pair" - it can generate a new key pair for every request (If the server wants to keep a session, it could use cookies etc.).

Suppose the client uses clientOneTimePublicKey, clientOneTimePrivateKey and the server uses serverPublicKey, serverPrivateKey. To obfuscate the response from the request, I suggest that the request from the client to the server will be wrapped and look like:

{
  "id": "<usual hash>",
  "pubkey": randomPublicKey,
  "created_at": randomTimeUpTo2DaysInThePast(),
  "kind": 1059, // gift wrap - is there an equivalent ephemeral gift-wrap?
  "tags": [
    ["p", serverPublicKey, "<relay1-url>", "<relay2-url>"] // we can have more than one, right?
  ],
  "content": nip44Encrypt(
    {
      "id": "<usual hash>",
      "pubkey": clientOneTimePublicKey,
      "created_at": randomTimeUpTo2DaysInThePast(),
      "kind": 13, // seal
      "tags": [], // no tags
      "content": nip44Encrypt(unsignedKind50, clientOneTimePrivateKey, serverPublicKey),
      "sig": "<signed by clientOneTimePrivateKey>"
    },
    randomPrivateKey, serverPublicKey
  ),
  "sig": "<signed by randomPrivateKey>"
}

The server response doesn't need to be wrapped, because it's going to send only one message to clientOneTimePublicKey, so it could be simpler and look like the seal part alone:

{
  "id": "<usual hash>",
  "pubkey": serverPublicKey,
  "created_at": randomTimeUpTo2DaysInThePast(),
  "kind": 28000, // a new kind for "ephemeral-server-response"?
  "tags": [
    ["p", clientPublicKey, "<relay1-url>", "<relay2-url>"]
  ],
  "content": nip44Encrypt(unsignedKind50, serverPrivateKey, clientOneTimePublicKey),
  "sig": "<signed by serverPrivateKey>"
}

WDYT? The only thing remote users/relays might see is requests sent to the server (by unknown users) and the server responds to one-time public-keys that are never seen again. If the load on the server is high enough, it would be even harder to match a request to a response (and they are also encrypted).
By not gift-wrapping the response-message I did expose that server has responded (i.e. that it's alive), but that's not a big deal.

P.S. Will you be at BTC Prague? Maybe we could discuss over there.

[vitorpamplona]

Nice, I think you got it. You can use the expiration tag to set when the relay can delete the outmost event.

On the p tags, you can only use one relay url. But you can use the relays tag to add a list.

[oren-z0]

Thanks @vitorpamplona 🙏
I'm having second thoughts about not gift-wrapping the response.
Maybe in some use cases the server operators wouldn't want random people to know that there is an http-server behind their pub-key.
On the other hand, if people will notice lots of ephemeral gift-wraps being sent to the same pub-key, they can even send an empty GET request to check if it's responding - meaning that if the server has enough load it wouldn't be able to hide anyways...

[vitorpamplona]

You can just gift wrap both ways. Yes, people can send a GET to see if it is a server, but the server might not reply to random GET commands if they are truly private. Their protocol can add a little invite token to register keys that can use the service.

Also, a single key receiving multiple wraps might just be any other type of service, using a different protocol. So randomly sending GET might not be that effective of an attack.

[oren-z0]

@vitorpamplona I've added the changes for sealing and gift-wrapping (in a new commit - could be squashed in the future).

New kinds:

• 80, 81 for the unsigned request and response messages (equivalent of kind 14) - will be easy to remember because the default port of HTTP is 80.
• 21059 - ephemeral gift wrap. We don't need expiration tags here, the relays are not expected to store the communication anyways.

[oren-z0]

@vitorpamplona I've added the partIndex and parts fields in case the http request/response does not fit in a single encrypted message (NIP-44 limits their size to 64kb) - so we can split them over multiple events.

See the implementation using nostr-tools: https://github.com/oren-z0/http2nostr https://github.com/oren-z0/nostr2http

@fiatjaf - Who is in charge of giving numbers to NIPs? I suggest "NIP-80" after the default http port.

[4xvgal]

Hi, is it still proposal state?

[oren-z0]

@4xvgal Closed it. It's not a good idea to transfer large amounts of data over Nostr notes, despite the advantages of bypassing NAT etc.
LNURLs specifically should use something else:

lnurl/luds#203

[4xvgal]

@4xvgal Closed it. It's not a good idea to transfer large amounts of data over Nostr notes, despite the advantages of bypassing NAT etc.
LNURLs specifically should use something else:

lnurl/luds#203

Calling lnurl like small, one time rpc/api are reasonably more than whole http.

Is the reason close the pr is latency issue or care about relays's infrastructure?

Cuz i made nostr tunneling middleware for hide app server behind nat

https://github.com/4xvgal/nostr-tun

[shocknet-justin]

CLINK is a nostr successor for lnurl, also has a bridge and support for debits etc

https://clinkme.dev

[oren-z0]

@4xvgal It's more about the infrastructure. It doesn't make sense to split large traffic into many many small signed notes, and relays will rightfully treat it as spam.

HTTP-based protocols like LNURLs that pass small amount of data, should build an equivalent nostr-based implementation.

For large amounts of p2p traffic that needs to bypass home routers NAT (which needs a static public ip address & port-forwarding), there are TOR and other technologies (my favorite is https://holesail.io ).

[oren-z0]

@shocknet-justin
How is it different from lnurl/luds#203 ?
Maybe you should write about it over there.

[4xvgal]

@4xvgal It's more about the infrastructure. It doesn't make sense to split large traffic into many many small signed notes, and relays will rightfully treat it as spam.

HTTP-based protocols like LNURLs that pass small amount of data, should build an equivalent nostr-based implementation.

For large amounts of p2p traffic that needs to bypass home routers NAT (which needs a static public ip address & port-forwarding), there are TOR and other technologies (my favorite is https://holesail.io ).

What if run holesail connection string exchange server behind nostr-tunneling?

That makes holesail host server can be run on no domain, CA/TLS just pubkey.

This could be self VPN network service with just npub

[shocknet-justin]

@shocknet-justin
How is it different from lnurl/luds#203 ?
Maybe you should write about it over there.

Lots of ways, ground up approach like an rpx, bi directional with sub specs, and sdk.. in use in various projects

https://GitHub.com/shocknet/clink

[oren-z0]

@4xvgal I'm not familiar enough about how holesail works in the background, but it creates an hs:// URL followed by a public-key (I think), so it's not that different from an npub anyways.