[RFC Discussion]: XOR-URLs

joshuef · January 21, 2019, 5:20pm

This is a thread for further discussion of The XOR-URLs RFC.

Please hold forth and opine here.

joshuef · January 21, 2019, 5:22pm

Currently there’s some discussion around the use of : for distinction of the typeTag: https://github.com/maidsafe/safe_browser/issues/429

Essentially, it’s reserved in url schema for port numbers and so cannot be larger than 65535. Which is problematic.

There have been suggestions for alternative characters. !, eg.

joshuef · January 21, 2019, 5:22pm

@bochaco FYI.

Also, @bzee, you were looking where to comment!

bzee · January 21, 2019, 6:14pm

Thanks, @joshuef. I think I wanted to know what was envisioned with a possible ‘fallback mechanism’ in the ‘Private content’ section:

At the same time, it may be considered to have either a fallback mechanism, or an additional bit encoded in the URL to realise if the XOR-URL is targeting a private or public content to act accordingly.

mav · January 22, 2019, 1:37am

This is a very exciting rfc!!

RFC-0053 may need some modification to the bold part

#somesection?somekey=5 is the <fragment> and <query> parts which will be ignored by the resoler and simply returned back to the application for it to use it.

The URL spec says

No fragmentid part of a WWW URI (the hash sign and following) is sent with the request.

Maybe change to

#somesection?somekey=5 is the <fragment> and <query> parts which will not be sent with the request or received with the response data, but may be added to the response data by the client for the application to use.

My suggestion is to exclude fragments from the rfc completely, leave it to the app to use.

RFC 3986 Section 3.5 has useful claims about how fragments are meant to be used:

[fragments are] the primary form of client-side indirect referencing
…
the identifying information within the fragment itself is dereferenced solely by the user

(also there’s a typo resoler/resolver in the rfc)

In RFC-0053 why do both the querystring and fragment do the same thing, ie “to be used by the client app and not for retrieving the content”? Might be best to leave the fragment alone completely for apps to do with as they please.

Querystring is still useful to keep in the spec (see below about MD)

From RFC-0053

#somesection?somekey=5 is the and parts which will be ignored by the resoler and simply returned back to the application for it to use it.

The order of querystring and fragment is back to front (querystring should be second last, fragment should be last). See RFC 3986 Section 1.2.3, 3.4 and 3.5

The URI syntax is organized hierarchically, with components listed in order of decreasing significance from left to right.
…
The query component is indicated by the first question mark (“?”) character and terminated by a number sign (“#”) character or by the end of the URI.
…
A fragment identifier component is indicated by the presence of a number sign (“#”) character and terminated by the end of the URI.

I’m dubious about the use of ports as a placeholder for MD types (which has been discussed by others in the github issue).

Introducing ! delimiter instead of : will mean xor urls are broken in every existing url parser. I reckon it’s best to put the type value in the querystring as ?t=1234

A missing t parameter means it’s immutable.

Then optional content version can be encoded into the querystring with v, ie ?t=1234&v=1; leaving out v means the latest version is returned.

The querystring seems like the right place to encode metadata into xor-urls. It should not be encoded in the port. Or the host. Or the path. Or the fragment.

It may be encoded in the userinfo part of the url, eg type 1234 and version 1 could be safe://1234:1@<cid>/path/to/file?querysting#fragment but I feel like this is abusing the intended userinfo semantics, depending how RFC 3986 Section 3.2 is interpreted:

[userinfo is a] means for distinguishing an authority based on a registered name or server address

Ultimately I reckon MD metadata belongs in the querystring. It might have collision issues if someone wants to use t or v keys in the querystring for their application data, but probably the fragment should be the canonical way for clients to manage data internally, leaving querystring only for request metadata.

Please do not break standard url parsers by introducing custom delimiters.

bzee · January 22, 2019, 9:55am

I missed the tag encoding problem.

I could argue that both the name and tag are equally important for locating an MD. If the tag is metadata, then so is the name. Both could be encoded into the cid part I assume. (Version number too, or into the port part.)

Another option would be the top-level domain (TLD). TLDs can’t start with a digit, so a SAFE URL could turn into: safe://<cid>.t12345 (for tag 12345).

Semantically I don’t think the query string is suited for the tag. It follows the authority (userinfo, host and port) and path; reading from left to right it should narrow the scope. A query string is for querying, for passing parameters to the page. Web app developers (like with Vue) might have great trouble constantly re-encoding the tag type into the query string for every URI pointing to its own app.

mav · January 23, 2019, 2:00am

I don’t agree about their importance for locating an MD.

The xorname is the only data used to locate the section/group/vault for an MD (ie the route). This means the xorname is like host in the url - the xorname is everything needed to route the request.

There are probably two main options for encoding type+version metadata into urls.

Encode them as subdomains, eg

safe://<cid>/path?querystring#fragment - immutable (no subdomain)
safe://1234.<cid>/path?querystring#fragment - mutable with type 1234 (latest version)
safe://1.1234.<cid>/path?querystring#fragment - mutable with type 1234 version 1

I don’t think it should be encoded as the TLD, since type+version is lower in the hierarchy than the xorname (since they’re not used at all for routing purposes).

There’s a loose hierarchy here, with version being a child of type, and type is a child of the xorname. But I think it sets up a false idea that MD under the same xorname are related when they’re not. MD under the same xorname+type obviously are related by their version numbers, but the xorname has no meaningful relation to the various MD type values that have been created.

Encode them as querystring parameters, eg

safe://<cid>/path#fragment - immutable (no querystring)
safe://<cid>/path?t=1234#fragment - mutable with type 1234 (latest version)
safe://<cid>/path?t=1234&v=1#fragment - mutable with type 1234 version 1

I prefer querystring because it’s extendable and semantic. MD urls are literally querying a specific type and/or version of mutable data at a specific xorname.

Overall, existing urls have quite strong semantics and I think where possible xor-urls should try to match those and not repurpose fields for different purposes.

The difference is certainly not majorly significant in practice. If we need to have a custom xor-url parser in order to have meaningful semantics then so be it (this will be needed to some degree anyhow for the cid component). But I’d like to see the design be maximally consistent with existing url wisdom where possible. The conceptual mapping isn’t always perfect so it’s good to see discussion and ideas happening.

Yes, good point, it’s important to consider the impact on the end users. But I feel it doesn’t make sense to model the design of xor-urls around implementations and libraries, it should be modelled on the existing design/purpose of urls.

Since all MD for the app will (presumably) have the same type it does map reasonably well onto the subdomain concept. But I still feel this is a query not a domain

bzee · January 23, 2019, 10:13am

I agree, but again I would argue that without the port (analogous to MD tag) there is no way to reach the service — in that sense I think host and port are equally important. Analogously, without either the name or tag you won’t get an MD.

Because I view it like this, I feel the query string isn’t suited. It’s too specific. It comes after the path. The path doesn’t make sense without having specifief the MD first.

Touche.

Another reasonable option would be the first component in the path: safe://<cid>/1234 for tag 1234.

mav · January 23, 2019, 10:32pm

Looking at maidsafe/routing/src/messages/request.rs GetMData

There’s no way to specify which version is desired to be fetched. Is version history retained? I’m not clear about accessing MD, can someone please clarify if only the latest version is available or if all versions are.

Krishna · January 24, 2019, 6:11am

Only the latest version is available for now.

This is something we are debating internally with in the teams on and off. We are planning to bring this as a mainstream discussion in the upcoming weeks to explore possible approaches for a versioned datatype.

joshuef · January 24, 2019, 8:52am

Right now the type tag is necessary component of the URL for fetching the data (it needs to be passed to the browser for grabbing MD).

Any webapps dealing with this would effectively be having to run their own fetch implementation instead of relying on baked in browser set fetch to retrieve XOR-URLs (which would negate the point of them, IMO).

On where should type/version live in the URL…

Using the path I was initially against, to make it clear what’s part of the URL and what’s not as, the path could be used to fetching parts of RDF data eg., but, that’s not necessarily precluded by using paths…

safe://dsaiudhasiuhiufhsiduhiusfgdsdsfs/t1234/v1/some/portion/of/the/RDFdata

Though we’re creating limitations (or if not, just confusion) about what paths could be reliably stored at a given XOR-URL…

Subdomains is interesting, in the current plan we first attempt to resolve a XOR, and if fail, hit PNS.

So now it would be, if a XOR is hit with XOR-URL, we check for subdomains, if there’s at least one, we resolve that typetag. if there’s two… typetag and version. If not we move to PNS system.

This would more clearly retain the ability to utilise paths / fragments to further specify the data you might want from an MD (if it was RDF eg).

happybeing · January 24, 2019, 10:29am

Another variation which avoids confusing version and tag with subNames or path, assuming there’s a suitable separator we can use (here I use hyphen):

safe://dsaiudhasiuhiufhsiduhiusfgdsdsfs-t1234-v1

bochaco · January 28, 2019, 10:54pm

@bzee, I think this was just trying to mean that if the resolver supports to fetch and decrypt MD entries it may need to try first to decrypt them and if that fails then simply assume that entry is unencrypted, in priv MDs entries can be both encrypted but some unencrypted. I’ll be looking at applying some of the corrections made above by @mav so I’ll try to also enhance this paragraph as this bit is definitely not clear at all.

I think the confusion here has to do with the fact that the RFC is talking not just about the XOR-URL spec being proposed, but also how we can/want to support it with our resolver, i.e. our webFetch and fetch browser embedded functions.
E.g., if you are fetching a WebID with a fragment, I thought it would be desirable to have the fetch/webFetch to already take care of it and return the corresponding graph (in cases you have more than one foaf:Person graph in the same WebID), but it’s also true an app may not want that and would prefer to obtain the complete WebID profile doc for it to process the fragment (or not ). This can probably be an option which can be passed in when invoking the resolver stating if the fragment shall be processed by the resolver, or as you suggested, simply have the resolver to not support fragment inspection/resolution.

When looking at how to encode the type tag I was having the same type of thoughts and point of view as @bzee, I don’t think MD urls are literally querying a specific type and/or version of a MD in the sense that querying is supposed to be something targetting the client app or website, and nothing to do with the resolver or how to route the request, just like the fragment. So we thought that since the type tag is used by the vault to retrieve the service/content (as bzee says you cannot do it without a type tag), it fits quite well trying to make an analogy with port numbers.
I think along the same lines for the version, this is not defined yet how it will work, so we don’t know if different versions of the “same” content would be store/hold by the same section/group, I presume it all depends how this will be implemented. And even if they are in the same section/group, providing the version is/will still be part of the routing/locating mechanism rather than something dealt by the client app.

Now, I have recently started to think about the need of a type tag number, simply considering not having type tags at all to route and locate any type of content. From a user’s perspective, why do I need a “number” to locate a content on a network which holds immutable versionable data, for which I already have a hash of the content I’m looking for? AFAIK, the idea behind the type tag was to be able to have the network (vaults) to act upon data in specific ways for predefined type tags, e.g. safecoin is typetag X so mutations/requests for such MDs can be treated in a special way. But what if we can do that in some other way and remove the type tag altogether? wouldn’t that be a much cleaner CAS and URLs?

(Note I have no clear understanding yet if it’s possible to get rid of type tags but throwing it for consideration as well)… thinking out loud, what if some bits of the XOR address are actually the type tag, and what if we even expand the XOR name address space if we want to have 256 bits for it and add some more bits which are used for type tag? @ustulation / @nbaksalyar does this sounds too crazy/wrong?

I would be more inclined to this option rather than any of the other where querystring or fragment is used for any of the type tag or versioning. I’m also thinking of the suggestion from @hunterlester Cannot fetch content with a XOR-URL which typeTag is greater than 65535 · Issue #429 · maidsafe/sn_browser · GitHub, what if we propose an addition to the multiformat protocol where you can append some custom data and which we use to encode the type tag number. I’ll have to think about this more thoroughly though.

mav · January 31, 2019, 1:36am

The use of the term website is not really ideal here (page was also used in an earlier post). I reckon this url scheme is closest to an api design, not a website or webpage. Querystrings are a way to refine the data returned from the specified endpoint. Querystrings are consumed by the endpoint, not by the client.

The host field in the url maps exactly in concept to the xor location of the data, which is used to route the request. Host/xorname is not used to refine data returned from the endpoint.

The type tag and / or version number maps almost exactly in concept to the querystring as a way to get the endpoint to refine the result from a larger set of data.

The point I want to make here is that type tag is not used for routing the request. This is my reason for wanting it not to be encoded as subdomain in the host field but instead as querystring. To my mind this is closer to the semantics of the current url rfc 3986. Type tag doesn’t affect routing thus it doesn’t belong in the host field.

In a similar vein, it might be good to also think toward the future when compute is on SAFE and how futureproofing that kind of feature might impact current URL design thinking. It’s an unknown for now but that doesn’t mean we can’t at least consider keeping some options open for future uses.

I disagree (to an extent). Type/version is only used by the endpoint to filter the potential results at that xor address down to the correct data, ie it’s not used for routing (I think I’m using a different definition of routing than you; I consider routing to stop when the request reaches the vault containing the data, operations internal to the vault such as picking the correct chunk off the disk are not routing). Type/version is not part of routing.

Anyway I think I’ve already said more than necessary on this topic considering the relatively small magnitude of the point. Please carefully consider what is ‘routing’, ‘location’, ‘filtering’ etc and apply those semantics consistently within the url encoding. As IPv6 and BIP70 has shown it can be very hard to change or extend once an existing standard is in use.

rob · January 31, 2019, 7:10am

While I agree here, the is perhaps another way to consider it.

For instance when the address is routed only the upper “x” number of bits is used. The rest are contained within the section/group. Not the number of bits will vary as the number of sections vary and how they split. But it will still only be a small portion of the address used for routing.

Now the tag when requesting a MD could be considered the lower 64 bits of a combined address.

bochaco · January 31, 2019, 4:12pm

It sounds like it just has to do with the fact there is no server, so we now have only the routing side of it, and the client side.
From my point of view the client side is everything that a traditional script would do on the client or server side, or even the web server process itself, this includes the query string used by a script and path which would be taken care by the web server process. Then the type tag or version is kinda in the middle/gray-area where traditionally has to be resolved before a server side script or webserver process deals with the request, which in my view/model is still part of the routing the request to the targetted content. So it’s a matter of where we draw the line I guess.
EDIT: however for the version, as I said before, I think it all depends how it’s implemented, if different versions are kept in the same section or not, then it would go on one side or the other.

mav · January 31, 2019, 11:58pm

Agreed, this is a very reasonable perspective. The traditional role of querystring maps onto fairly ambiguous territory when used in SAFE.

riddim · March 20, 2019, 11:39am

Stupid question

Xor urls - how are they defined by now…? I saw a lot of discussion and this rfc

Sorry for not participating in this earlier and complaining now… But since the proposed z-base32 encoding is no standard encoding I wasn’t able to simply just generate the string and try to see my uploaded data with the browser…

Since this was kind of a bummer for me (and the z-base32 implementation for python that can simply installed with pip is python2 only for now) I thought it would ask why you want to use a non standard encoding which might cause trouble with other programming languages too…?

The argument with easier readability doesn’t count imho because we are talking about xor addresses which are hashes of data (at least for immutable data…?) which should result in kind of randomly distributed bits… So the characters should appear pretty much equally distributed (and with creating a new mutable data with the create new random it should be the same)

In addition to this I like the argument that base64 encoding is shorter (and nobody would type those links by hand anyway)

joshuef · March 20, 2019, 12:03pm

hey @riddim, not stupid questions at all!

The RFC is still on going so all comments / concerns / questions are very welcome!

Can’t say I agree here, I’m afraid.

z-base32 is case insensitive, which can be important when copy-pasting /passing around / normalizing urls to avoid issues (we’ve had issues with base64, already). It also strives to remove characters that could be easily confused, which should make them just that bit more useable.

In addition to this I like the argument that base64 encoding is shorter (and nobody would type those links by hand anyway)

If no one would type them, what does it matter if they are longer?

If you do type them out for whatever reason, and one or two or three characters can be confused, that’d be a huge pain (IMO).

That, at least ,is part of the reasoning for the choice of z-base32 (@bochaco maybe you can remember anye more ?)

The availability of parses might be something worth having a look at, certainly

Although as a developer you are able to use the SAFE APIs to decode the strings and retrieve the mutable data info if that’s what you’re after. Or are you imagining other situations to be using the XOR-URLs?

I’m not sure I follow.

You need to upload the data to the network and retrieve the given xor-url based on its location in the network. You cannot simply base32 some data and pass this link for the browser to decode / display data (Is that what you’re imagining?)

bochaco · March 20, 2019, 1:39pm

That was the reasoning about the choice of z-base32 so far @joshuef @riddim, for a URL having a case insensitive encoding seems a must to me.
Also, regarding different languages, you can take a look at the CID implementation for different languages if you don’t want to use the functions exposed by the SAFE API: GitHub - multiformats/cid: Self-describing content-addressed identifiers for distributed systems, we are simply using the JS implementation of CID and multiformats, and whenever this is implemented in lower layers in Rust we will presumably also be using the available one https://github.com/multiformats/rust-cid, once it’s in our SCL layer you shouldn’t need any other package/lib but just use what’s available in our API through the lang binding you are using.

@riddim, please take a look at the snippets and little description in the following post, that’s how you can retrieve a XOR-URL (remember you need the experimental APIs enabled), this may help you in getting a sense of the current state of the proposal, this is part of https://safenetforum.org/t/release-safe-browser-v0-11-0/26792:

XOR-URLs can be obtained for any ImmutableData and MutableData on the Network

As mentioned above, applications can now fetch any MutableData/ImmutableData from the Network using its XOR-URL. The APIs are now capable of generating and returning the XOR-URL for any MutableData and ImmutableData the application has access to. For more details/information about the XOR-URLs you can reference directly from the proposed RFC , from the discussions taking place in our dev forum , or by watching this screencast . The following snippets are examples of how an application can obtain the XOR-URL for a MutableData and an ImmutableData respectively:

Get MutableData XOR-URL
const typeTag = 15015;
const md = await safeApp.mutableData.newRandomPublic(typeTag);
await md.quickSetup({ key1: 'value1', key2: 'value2' });
const info = await md.getNameAndTag();
console.log('MutableData\'s XOR-URL:', info.xorUrl);
Get ImmutableData XOR-URL
const imdWriter = await safeApp.immutableData.create();
const cipherOpt = await safeApp.cipherOpt.newPlainText();
await imdWriter.write('my text');
const getXorUrl = true;
const info = await imdWriter.close(cipherOpt, getXorUrl);
console.log('ImmutableData\'s XOR-URL:', info.xorUrl);