[RFC Discussion]: XOR-URLs

#26

Oooooh - beautiful! :flushed: :dizzy_face: :hugs:

Ps and sorry for not simply adding the other address as text for copying…

3 Likes
#27

TL;DR: the current cid solution looks nice and is a very elegant way of doing it but is imho not very flexible and since we need to provide only 24 byte of data to discover a piece of data (160 for including all the keys) this solution is overy complicated for a super simple task


current implementation/suggestion is this:

def getXorAddresOfMutable(data, ffi):
    xorName_asBytes = ffi.buffer(data.name)[:]
    myHash = multihash.encode(xorName_asBytes,'sha3-256')
    myCid = cid.make_cid(1,'dag-pb',myHash)
    encodedAddress = myCid.encode('base32z').decode()
    return 'safe://' + encodedAddress + ':' + str(data.type_tag)

we sha3-256 the 24 bytes of the xor-name, then we twist it somehow into a cid and in the end we encode it into base32z-ish

if we look at the bytes we can see that the difference between the hashed and unhashed value is not large - and if we look closer than we see that the sha-3’d values are just patched with (hex) 1618 from 24 to 26 byte

then some magic with cid happens and we can somehow revert that to get the xor-name back.

if we want to be case-insensitive and use a base32-encoding we are only slightly shorter than just using the hex-value. - do we plan on integrating additional information into the cid? or is it just a fancy way to do it …? if the second … then why would we do this instead of the simple hex-value that is easy and well-known/understood and not calculation-intense …?

in addition the this i saw somewhere the question/idea (i think by @happybeing) that we could make even xor-urls for private data; while the argument against it was that we cannot encode the additional keys in the cid …

if we just take the hex value of the MdataInfo we can have 160 byte in hex-representation that are describing our piece of data perfectly (the following piece of data is unencrypted so all the keys are 0); yes nobody wants to type those by hand - but as clickable link or qr-code that’s an absolute valid solution imho

if we want to give someone-read-only-access to a file we mask the key that enables modification of the file and only provide the decryption key … simple as that …

and since the MDataInfo is exactly what a program needs to handle a piece of data; and hex representation is super easy to implement and clearly defined (and easy to adapt in case something changes with appendable data/any future data format change) i really really really don’t see why we would go for something unnecessarily complicated like cids …

ps: and in addition to this the currently selected encoding base32z is definitely not a super-widely used encoding that has a different implementation in e.g. python than in Javascript and the python-generated base32z-links don’t work with the browser and i cannot decode a link that i get from the browser/JS API in python (!)… (the good thing is that “standard encoded” links do work in the browser too - but this means again that we have redundantly defined many links to the same location in the network - while the redundancy doesn’t provide any additional benefits because it’s not like check-bits that show you at least that the link is correct or wrong but it’s just randomly spread links that suddenly end up in the same location …)


so i guess my suggestion would be to just use the hex-values of the name or the whole Info*-object as xor-urls where one can mask the properties which should not be shared … simple to integrate for private data too and in the ‘easy case’ only 5 characters longer than the current proposal

(and for e.g. the type-tag 18446744073709551615 (largest possible if i’m not mistaken) the difference is [including separator] exactly 1 character vs. included in the hex string:

)

2 Likes
#28

pps: okay - and if i really missed something about the multiformats-thing that would make it very useful and cool for the future…

…you don’t want bas64 because of upper/lowercase and special characters … you don’t want base32 because of similar looking characters … why don’t simply choose base16 (hex) as default because it’s standardized and well known …?

(and if someone just wants to generate a link in a random language that doesn’t implement cids yet [maybe e.g. rust for exposing this through the client libs?:roll_eyes:] he can just hex the xor-name, put e.g. ‘safe://f01701620’ (description: sha3-encoded + first patch-bytes because the xor-address is shorter than the checksum, bytes then hex-encoded) in front of it and have a working link to the newly generated mutable:

safe://f017016202976d0fbd38b8d2d29bde345379b4d541b4c76c4df33920171ef20fa70f33ac8:777

because of the self describing nature of the multicodec-thing this already works anyway - it’s just not obvious to someone wanting to do it … or if you want to have your address to be keccak-512-encoded you put ‘f01701d20’ in front of it and magically end up still in the same place

safe://f01701d202976d0fbd38b8d2d29bde345379b4d541b4c76c4df33920171ef20fa70f33ac8:777

(the trick with the cids seems to be that they choose a hashing function to patch the data to a working size - and then they put the information about the used hashing function + the used encoding for the following string at the front of the string of the output)

if you personally prefer your base32z encoded strings because they are a bit shorter and easier to identify/type in then you can still generate them on demand and the browser will accept them and show you the location you want it to show …?

but please don’t use a non-standard-encoding as default behaviour in your api …


yes it’s nice that you can choose the representation of your liking for the data you encode:

safe://mAXAWII9/3YMeDvNet/RpZbiwkV5jZSKg/9qMcyYZg7UBiCid:777
safe://bafybmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
safe://zdjCkrCPzAj3i5AoLGmfsjvpoz1tU9LXEinMkPrf2QzD4t9cg:777
safe://f017016208f7fdd831e0ef35eb7f46965b8b0915e636522a0ffda8c73261983b50188289d:777

some of them are implemented in JS, some not … but i don’t see the value in it as of now … would be nice to be able to use the base58btc because of the length … but that again isn’t implemented in JS as of now … and if you choose to do it then i would tend to just append the type-tag in encoded form instead of doing the :777 thing … i don’t know where the real value is there [to encode it base10 and introduce a separator]

other examples for working links for the interested reader - all ends up at the same place
bin::          safe://bafkrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
base1::        safe://baearmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
base8::        safe://baedrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
base10::       safe://baeermiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
cbor::         safe://bafirmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
protobuf::     safe://bafibmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
rlp::          safe://bafqbmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
bencode::      safe://bafrrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
multicodec::   safe://baeybmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
multihash::    safe://baeyrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
multiaddr::    safe://baezbmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
multibase::    safe://baezrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
sha1::         safe://baeirmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
sha2-256::     safe://baejbmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
sha2-512::     safe://baejrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
dbl-sha2-256:: safe://baflbmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
sha3-224::     safe://baelrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
sha3-256::     safe://baelbmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
sha3-384::     safe://baekrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
sha3-512::     safe://baekbmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
shake-128::    safe://baembmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
shake-256::    safe://baemrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
keccak-224::   safe://baenbmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
keccak-256::   safe://baenrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
keccak-384::   safe://baeobmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
keccak-512::   safe://baeormiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
murmur3::      safe://baerbmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
blake2b-8::    safe://baga6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-16::   safe://bagboiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-24::   safe://bagb6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-32::   safe://bagcoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-40::   safe://bagc6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-48::   safe://bagdoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-56::   safe://bagd6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-64::   safe://bageoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-72::   safe://bage6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-80::   safe://bagfoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-88::   safe://bagf6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-96::   safe://baggoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-104::  safe://bagg6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-112::  safe://baghoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-120::  safe://bagh6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-128::  safe://bagioiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-136::  safe://bagi6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-144::  safe://bagjoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-152::  safe://bagj6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-160::  safe://bagkoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-368::  safe://bagxoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-376::  safe://bagx6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-384::  safe://bagyoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-392::  safe://bagy6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-400::  safe://bagzoiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-408::  safe://bagz6iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-416::  safe://bag2oiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-424::  safe://bag26iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-432::  safe://bag3oiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-440::  safe://bag36iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-448::  safe://bag4oiaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
blake2b-456::  safe://bag46iaqwechx7xmddyhpgxvx6ruwlofqsfpggzjcud75vddteymyhnibrauj2:777
ipfs::         safe://bagsqgfrar5753ay6b3zv5n7unfs3rmerlzrwkiva77niy4zgdgb3kamifcoq:777
http::         safe://bahqagfrar5753ay6b3zv5n7unfs3rmerlzrwkiva77niy4zgdgb3kamifcoq:777
https::        safe://bag5qgfrar5753ay6b3zv5n7unfs3rmerlzrwkiva77niy4zgdgb3kamifcoq:777
quic::         safe://bahgagfrar5753ay6b3zv5n7unfs3rmerlzrwkiva77niy4zgdgb3kamifcoq:777
ws::           safe://bahoqgfrar5753ay6b3zv5n7unfs3rmerlzrwkiva77niy4zgdgb3kamifcoq:777
onion::        safe://bag6agfrar5753ay6b3zv5n7unfs3rmerlzrwkiva77niy4zgdgb3kamifcoq:777
p2p-circuit::  safe://bagraefrar5753ay6b3zv5n7unfs3rmerlzrwkiva77niy4zgdgb3kamifcoq:777
dag-pb::       safe://bafybmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
dag-cbor::     safe://bafyrmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
git-raw::      safe://baf4bmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777
eth-block::    safe://bagiacfrar5753ay6b3zv5n7unfs3rmerlzrwkiva77niy4zgdgb3kamifcoq:777
eth-block-list::safe://bagiqcfrar5753ay6b3zv5n7unfs3rmerlzrwkiva77niy4zgdgb3kamifcoq:777

ps:

if we would append a checksum instead of the encoding-information-stuff we could not only check for ‘incorrect characters in the encoded string’ but we could do offline-checks for typos …

as of now e.g. both of the strings are valid cids … while the last one has the last character mis-typed if you want to end up at the address of the mutable …

3 Likes
#29

The key and main reason is simply to allow us to evolve without breaking things, as an example, your next statement:

Let’s imagine in the future we decide to change the encoding, or even the hash function we use for our immutable data XOR addrs, we will have to make sure that whatever new format we adopt we don’t break backward compatibility with older URLs.

Remember we are after the perptual web, we don’t want to break old URLs just because we are moving away from one encoding to another, or even from one hash function to another. So using the hex encoded XOR addr wouldn’t be enough if we want to accomplish that, we need some other ways to make sure that if I give you a URL to an immutable data on SAFE, it’s immutable and perpetual regardless what’s the most used encoding at any moment.

I get your point, although just FYI Rust seems to be covered already: https://github.com/multiformats/cid#implementations

This is just another type of CID you are creating, what they are trying to achieve with CID is to have something standard that can be use to encode additional information to the content address. Where is the Rust implementation for that CID you are creating, or the golang one :slight_smile: just kidding ofc, I hope you understand what I mean

2 Likes
#30

So we patch the data we want to encode up a couple of bytes for the possibility that we might at some point randomly decide to change the format…? Hex is the 1:1 representation of the bytes (the information we want to encode) altering it (except from changing the base) always will always be less efficient… Why would we want to become less efficienct?
(and hex is around since the beginning of computing - hard to belief it won’t be understood/be hard to handle at any point in time)

1 Like
#31

And imho the multiformat thing just create the impression of a future proof format…

… It can handle different encodings and different patching algorithms (hash functions)… But can it handle if we decide to add 2 additional leading bytes as check sum for offline validation? How does it handle if we decide to move from cid:typeTag to [cidWithTypeTagCodedInForNotHavingASeparator] how does it handle if we expand the address space from 32 byte to 64 byte?

All those cases cannot be simply coded into the cid but we would need to extract the bytes from the cid and then do a case decision on the length of the bytes… Just as we would do without cid… Only that with using cid we need to extract the bytes from it before we can use them… (so one additional step with cid) wasted resources in my opinion…

#32

Let me say it differently.

CIDs are an elegant way of encoding random data bytes into one data encoding of your choosing (taking care of the issues that arise if the data you want to encode doesn’t fit the alignment of the encoding you want to use… you will always get the exact byte string length back you wanted to encode initially) [the chosen hashing function is just a random property of the cid to identify the length of the encoded byte string and to patch/unpatch it in the process …] they don’t take care of any data format changes

So… Unless you plan on changing the base for encoding the xor URL cids can only solve a problem they create themselves… That’s why I’m against cids for xor urls! We know the length of our xor URL, we don’t need to solve issues we cannot have - cids are answering the wrong question and are not the proper tool for this… Why would you add complexity you don’t need? but we could add some additional bits as checksum to verify the validity of a xor URL [and maybe add a byte to describe the encoding if you want… So we’d have self description again if we at some point randomly would think that it makes way more sense to move from base32 to base16 or base58] (!)

Ps:

oh well - you know what… I don’t care anymore… If you love your cids that much then go with them… Imho it’s a bad decision because it makes it overly complicated - but I don’t want to waste more of my life time for this issue that doesn’t matter anyway in the long run… just please don’t use the non standardised base32z encoding in your official api that possibly creates incompatibilities (or at least tell me which characters I need to replace by which others to get back to standard base 32 encoding to be able to decode urls you created)! (and why not append a checksum after the address too for typo-recognition…? Even the Iban comes with check bits nowadays… But please (!) not a complex solution this time but just e.g. Counting one’s in the bytes and take the last 1 or 2 digits or so… )

2 Likes
#33

Hey @riddim, I don’t think everyone is really in love with CID and/or completely sold on it, this was just a proposal made and one way of achieving the goal. I think there are valid points in your critic which are not being ignored, if that’s what you feel. I’m personally waiting for others to also chime in here with their opinions and perspectives, I’m aware some other people are trying to catch up with these discussions. I’m trying to explain what were the decisions and the reasoning behind what has been done, but it’s good we are looking at them and reviewing them from different angles.

7 Likes
#34

Aye, @riddim don’t be disheartened please.

It’s awesome you’re certainly raising your concerns. This is how we progress on this front :+1:

I’d been of the opinion that flexibility and future proofing are worthwhile additions. But you raise some good points. I need to digest and re read some of your posts above before I can opine something more though :slight_smile:

But aye, please don’t mistake a lack of immediate response as a lack of interest in your posts/points. :bowing_man:

8 Likes
#35

All good just mega busy

tbh I’m not 100% against cid… Probably the point where I got a bit upset was when I realised that cid is just a way of representing bytes and that you chose a non standard encoding for doing so as default behaviour … I don’t care that much about a couple of bytes more or less… But I would rather discuss if it wouldn’t make sense to include a check byte for offline typo recognition and to include the type tag just as bytes instead of the :typeTag thing that looks a bit pointless to me tbh…

Ps: and since I found out that indeed I can just hex the name and set something in front of it I had the impression you don’t know what cid precisely is and think it’s a careless package for all data (but it’s not - it’s just one way to represent it - actually a smart way because you first say how the following data will be structured - but it is really no more then just a representation of bytes… (that can even be base 8 or base 2 as pure zeros and one’s… If you exclude the type tag from the cid you make it complicated to transfer an xor address in such an environment instead of simple… Same goes with an environment where someone in the future wants to use pure base base64 or base 128 data… )

1 Like
#36

Side note:

Note: base64 or base128 would for example be possible if someone used 6 or 7 parallel data transfer channels (just going away from the visual data representation layer and looking at the technical level)

So indeed there might be future use cases with different encodings where cids then could be natively at home and one wouldn’t need to decode and re-encode the data by hand (if the type tag is included in the xor address cid - otherwise you need to split it up again and treat the type tag different from the rest and need to re-encode it)

#37

Finally had the chance to scroll shortly through the new primer…

That’s something I didn’t pay attention to earlier…

Plus while making Screenshots for this here I got a bit confused…

Immutable data Name: 32byte array

I thought for mutable data it would be 32byte too (24 byte name +8byte type tag) did this change…?

Anyway - but to sum it up the current proposal is to do:

  • cid(name):typeTag for mutable public
  • cid(name+mime type) for immutable public
  • David said 32byte array for safecoin (which is just data)
  • format unclear for private data

… Looks like a bunch of different formats arising…

I know I sound a bit like a broken record now - but I would vote for cid(relevantBytes+checksum) for simply everything…

You can just return all three of base58, base32 and base16 through the api and people themselves can decide which one they want to use (the browser should be able to decode at least base32 and base16)

So it would be

  • cid(name+mimeType+checksum) for immutables
  • cid(name+checksum) for safecoin
  • cid(name+typeTag+checksum) for public mutable
  • cid(name+keys+typeTag+checksum) for private mutable

… Still many different lengths of data but a bit more homogeneous…


[i cannot post more than 3 times in a row - so here an EDIT]


… Okay more on encoding and the differences between base32 and base32z… Just that everyone knows what we are talking about

I had a look at the JS implementation of those 2 and can make a base32 string from a base32z-js-string now…

So this here

safe://hyfabcerxx9qag8oq6pxmx7djcshmbrk6cp11fe895kg8gjo3oq4odnbeuw:777

is as base32

safe://bafybmiepp7oyghqo6nplp5djmw4lbek6mnssfih73kghgjqzqo2qdcbitu:777

As we can see the base32z leaves out l, v and 2

IMG_20190415_122813_546

And re-sorts all other characters :thinking:

1 Like
#38

Hey @riddim, thanks for your valuable comments! I’m still reading through this discussion and will chime in soon. For now I’ve got just a quick remark:

we never had 24 byte XorNames – for Mutable Data it’s 32 byte XorName + an 8 byte (64 bit) type tag:
https://docs.rs/safe_core/0.32.0/safe_core/client/mdata_info/struct.MDataInfo.html

2 Likes
#39

Then I need to check this in pySafe! Thx!

ps: all good - don’t know why i thought it would be 24 bytes …

2 Likes
#40

Okay – playing with immutables now …

and here again there is the question how xor-links are supposed to work for me

i uploaded a jpg to this xor-name (hex)

30a3fcb0130310087d0890da0143a594b40dd96b7536de15892d78ee264a6813

and the same file as png to this xor-name (hex)

2d50dd18645fe99a98b89271b44d10c612ca2cb168eba51e4ea3ce7909689583

i know that it‘s really there because i downloaded it on a different pc and both downloads succeeded without error

IMG_20190505_160452_162

since the prefix you used for your png link at safe://hygjdkftyx3k7kr51q9mxapy418zk3stdsss8suyqcim3b56jcten8d4j9emo is not in the python implementation of multicodec i „added it manually“ to my local version of it (just smuggled it into the source code)

then i used the safe://toolbox.dapp to analyse the picture link you provided to extract the xor-name – i can download the lamp and get the data …

IMG_20190505_160851_510

when i convert it to a cid i get as base32:

safe://bagkdefraipltiobuvllajcbkfl34dbiphitkor5jz2r6lqwdptkzpqbgvz2a

which seems to be fine (page loads – toolbox analyses)

for base32z then suddenly i only get ‚roughly‘ what your link is (pay attention to the 2 additional y‘s) and safe://hygkdrftyexmueqbwimmyjnbkfm5hdbex8eukqt7j34t6mosdxuk3xobgi34y then analyses fine again with the toolbox + loads the picture (so it‘s definitely not ‚just base32z encoded‘ but somehow there are additional characters that were not there before [and imo are not supposed to be there – since it‘s 2 additional characters that obviously don‘t contain any information … otherwise the base32 encoded data wouldn‘t analyze and work…])

if i try the same with my uploaded png i get:

safe://bagkdefrafvin2gdel7uzvgfysjy3itiqyyjmulfrndv2khsouphhscliswbq

which doesn‘t let me view the png in the browser and doesn‘t analyse with the toolbox,

base32z
‘safe://hgkdrftyfiep4gdrm9w3igfa1ja5eueoaajcwmftpdi4k81qwx881nme1sbo’
fails too – and with the 2 additional y‘s (as in the example link)
‘safe://hygkdrftyexmueqbwimmyjnbkfm5hdbex8eukqt7j34t6mosdxuk3xobgi34y’
it fails as well …

so what am i doing wrong with my png?

If i messed up something how is the precise specification of the xor-url of an immutable? why are there those 2 additional y’s? (:face_with_raised_eyebrow:) and why don’t we just append the mime type to the bytes and encode it just the same way we did before …?

As it is now for me in python - I need to manually patch the multicodecs implementation to have the required mime types (not sure how standardised that is - and how widely used… The last update of the hash constants on github for python was 2015… where do those codec-numbers come from anyway? I didn’t see them in the iana link from the github issue and the used codec for the png is not the in the issue mentioned x1910 but x1914…? May we run into collisions with the definitions suddenly? ) then I can generate a link (which only works in some cases as it seems)…

Ps: oh sorry! My mistake with the y! I think I made a copy&paste error with the base32z declaration dict… On second view the lengths of the links looked fishy :roll_eyes: :thinking:

Then your base32z link is perfect - it’s just that I fail with generating the right link to my uploaded png

2 Likes
#41

It’s in the RFC as well: https://github.com/maidsafe/rfcs/blob/master/text/0053-xor-urls/0053-xor-urls.md#xor-urls-specification

It’s just the CID and as you already know we use the multicodec-content-type part for the mime types as suggested

As you can see I worked on a PR against the multicode repo, which wasn’t merged yet, they were suggesting some minors changes that I/we will need to work on to presumably get it there.

Now, the python implementation is perhaps not using the master list of codec as it should, which is the one from the multicodec repo from my understanding: https://github.com/multiformats/multicodec/blob/master/table.csv and this is why you and me had to patch the table to have them in there untill they are effectively approved and make it part of the master list (our SAFE experimental api uses my patch to the table: https://github.com/bochaco/js-multicodec/tree/mime-types-as-codecs, which is used from https://github.com/bochaco/js-cid/tree/temp-use-bochaco-multicodec that in turn is the safe_app_nodejs’s dependency).

Therefore the CID implementation (in any lang) shall follow the spec from https://github.com/multiformats/cid, the issues you had so far seem to be all due to some tiny difference in the CID implementation and/or encodings used within them. Do I know if the js one is the correct one and the python is wrong, no I don’t know, since the browser and tools all use the same implementation. In any case, if we use CID and multiformats, we should be able to work on PRs to be sent to those implementations, in fact these problems you are seeing could be a good issue to be reported in the python implementation repo

3 Likes
#42

Ah - sorry for my impatience =D… Thx I’ll dig a bit further why my link doesn’t generate correctly as soon as I have some time again - you don’t happen to be able to generate a working png link to

@bochaco - that would be super nice and might speed up finding my mistake

1 Like
#43

These are the XOR-URLs I’m manually generating with JS:

  • For 2d50dd18645fe99a98b89271b44d10c612ca2cb168eba51e4ea3ce7909689583:

    • image/png: safe://hygkdrftyfiep4gdrm9w3igfa1ja5eueoaajcwmftpdi4k81qwx881nme1sbo
    • imag/jpeg: safe://hygjurftyfiep4gdrm9w3igfa1ja5eueoaajcwmftpdi4k81qwx881nme1sbo
  • For 30a3fcb0130310087d0890da0143a594b40dd96b7536de15892d78ee264a6813:

    • image/png: safe://hygkdrftygnt93cyuyceyo9ee1dpyno7f114y5smmqw5phfcjfihqhj1kpyjo
    • image/jpeg: safe://hygjurftygnt93cyuyceyo9ee1dpyno7f114y5smmqw5phfcjfihqhj1kpyjo

I cannot fetch the files with any of them, and the safe-URL Analyser is decoding them correctly to the correct xorname and mime type (you can try them). Perhaps because they are private, i.e. encrypted, are they?

I just pushed a commit you can use to generate ImmD XOR-URLs with different mime types, as long as you didn’t encrypt the targeted Immd files (we are missing an API which allow you to generate the XOR-URL of an ImmD without trying to fetch it, as it fails to decrypt it if not owned).

Just clone the repo https://github.com/bochaco/safe-tools, then npm i && npm start, then open it with the browser at localhost:1357, you’ll see a new tab for obtaining ImmD XOR-URLs.

2 Likes
#44

Yapp - it’s encrypted - that’s correct

I’ll test it asap (sorry - not possible before tomorrow) =)

2 Likes
#45

oh you mean owned by the app and not owned by my account - aren’t you? and the browser doesn’t own my png …?

ps: oh nice - and when i don’t accidentally damage the link before re-encoding it to base32z i get exactly your link! @bochaco

1 Like