Make base32 CIDv1 the default (Kubo 1.0)

[kyledrake]

I understand there's a switch to CIDv1 soon. I think go-ipfs should use lowercased base32 (rfc4648 - no padding - highest letter) as the default multibase.

The reason this encoding is preferable: it's the one encoding that will work with subdomains (RFC1035 + RFC1123). The restrictions are: case-insensitive, a-b0-9 and less than 63 bytes.

For a slight increase in length, you reap enormous benefits:

• The ability to do proper security origins for the HTTP gateway with subdomains (cidv1abcde.dweb.link). This is very important if we want to handle reports with Google's safe browsing system (which is designed for origins). With the current design, all content is on the same browser origin, and a single phishing/malware report on any of the IPFS gateways (hosted by us or someone else) will make web browsers block every single thing on the origin with a giant red warning message until it's cleared up with Google (which from experience can take several days!)
• Root paths are in the right place, which dramatically improves compatibility with existing web sites that tend to do a lot of this:
  <img src="/rootimg.jpg">
• Allows us to register dweb.link (and ipfs.io, etc.) to the Public Suffix List, which will prevent the sandboxed content from reading/manipulating cookies on the parent domain (and on other cidv1 subdomains).
• Opens up the ability for go-ipfs to do HTTP Host Header parsing and automatic Let's Encrypt support (if we wanted to), so anyone can set up a public IPFS gateway without additional software. Once Let's Encrypt gets their wildcard cert domains shipped (Dec 2017), this could be a fully automated process. Otherwise something like nginx would be needed (I could write an example nginx.conf that people could use for it).

It should use lowercase base32 characters by default, so that it's consistent with subdomain usage (all the browsers will force lowercase). IIRC the RFC doesn't care if it's lowercased, I think people just default to upper case for legacy reasons.

Obviously an abstraction layer could be written that converts between base32 and something else for use with web gateways and then have a different default, but I think it would less confusing for end users to use one default: the one that will let origins in browsers work.

This approach shouldn't be a problem for webextension plugins, but @lidel feel free to chime in.

Further reading: https://github.com/neocities/hshca

[lidel]

Good arguments, especially the one about GoogleSafeBrowsing's false-positives for public gateways 👍

CIDv1 format is strongly related to discussion at ipfs/in-web-browsers: Tackle identifying origins with (or without?) fs: paths.

I was unable to find definitive, final decision on which exact encoding will be used apart from @lgierth initially pondering "base16 or base32" and @samholmes suggesting base32 with Crockford's Encoding.

Was the decision made elsewhere?
If not, this ticket provides good opportunity to do so 🔧

[daviddias]

Thank you for creating this issue, @kyledrake. I agree with your proposal, we can take the opportunity that we are bringing CID to the world for the first time to get base32 as the new default.

If not handled internally correctly (i.e using the string format vs the binary format) it will add significant overhead, but that is just something we can change internally to make sure that we use memory efficiently.

I was unable to find definitive, final decision on which exact encoding will be used apart from @lgierth initially pondering "base16 or base32" and @samholmes suggesting base32 with Crockford's Encoding.

I'm strongly in favour of making base32 the general default encoding for CIDs everywhere. We need base32 for the ipfs:// URL scheme, and it'd suck if people had to deal with different CID encodings, or even have to use converter tools.

And, I think we haven't had any decision on it -- we just sticked with base58 as that was the original encoding used from the beginning.

[daviddias]

e just sticked with base58 as that was the original encoding used from the beginning.

Yeah, that was pretty much how the decision got made. Still in time to change though.

Still in time to change though.

Well I'm all for it :):)

[samholmes]

Notes on Base 32 Encoding

What would it take to get a new base added to the multibase table? Specifically, what would it take to add Crockford's Encoding to the table. As of commenting, it appears RFC4648 and z-base-32 are the only base 32 encodings included in the multibase spec.

An added reason to push for Crockford's Base32 is that it meets the same criteria as Base58Check, the base58 encoding Bitcoin uses for bitcoin addressses:

// Why base-58 instead of standard base-64 encoding?
// - Don't want 0OIl characters that look the same in some fonts and
// could be used to create visually identical looking account numbers.
// - A string with non-alphanumeric characters is not as easily accepted as an account number.
// - E-mail usually won't line-break if there's no punctuation to break at.
// - Doubleclicking selects the whole number as one word if it's all alphanumeric.

It seems to be like Crockford's Base32 naturally fits the same goals as Base58Check with the added feature of being case-insensitive.

Let's use whatever base32 variant Javascript and (less important) other programming languages use as their default base32. (I assume it's Crockford's)

[samholmes]

Notes on URLs and URI Schemes

From what I can tell, there is no obvious direction for solving issues surrounding URI Schemes and browser origin policies coupled with them thus far. However, my rough proposal is up for further commenting.

However, my inclination is to specify an alternative format and standard from URI. Then, leave it up to implementations to bridge this new format to a purposed URI scheme. Although a hack at the implementation level, it would open up an opportunity to re-think what a web address could be. Maybe a multiresource standard should be defined and added to the multiformats basket?

[samholmes]

@lgierth I don't know if there is a default base32 encoding in Javascript. If you would consider Javascript's native toString Number method:

var a = []; for (var i = 0; i < 32; i++) a.push((i).toString(32))
console.log(a);
// (32) ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v"]

It appears toString uses Base 32 Encoding with Extended Hex Alphabet from RFC4648.

Other than this, the Javascript community modules include many variants of base32 encodings; among them is Crockford's. So, it's safe to say that it's not an obscure encoding at the least.

[kevina]

@samholmes

What would it take to get a new base added to the multibase table?

I am not sold that Crockford's Base32 is better than than rfc4648 (that we already use the to in the flatfs datastore) but I don't see any problem with adding an entry to the table and implementing it in go-multibase. Step one would be to open an issue here: https://github.com/multiformats/multibase/issues.

[kyledrake]

Support for crockford base32 (base32check?) seems fairly widespread:

• lua
• ruby
• node
• go

Since there seems to be a strong preference for it, I hereby revise the proposal to use crockford base32.

[kyledrake]

Worth noting is that there's several different flavors of base32, including (my personal favorite) one that Nintendo games used that was designed to avoid profanity.

I'm kindof indifferent as to which version gets used. I chose RFC because it's a standard, it's been around a while, nginx-misc-module supports it, and it probably has the widest support across all programming languages. My only strong preference here is that it's a variation most programming languages already support, so we can minimize devs having to re-invent wheels.

Crockford seems to fit the bill more-or-less as well as RFC, which is my rationale for being OK with using it.

@kevina would you have very strong objections to crockford being used by default by go-ipfs with cidv1?

[kevina]

@kyledrake concerning some of the issues (in particular the use of cidv1abcde.dweb.link) have a look at #1678 (comment) it the full issue rather long but contains lots of useful context in why we currently use /ipfs/Qm.../hash