SAFE communication protocols

For a laugh I looked into compiling the safe-api into wasm and creating a simple SAFE:GET function in a normal browser so I could look at text and pictures on the SAFE network in a browser without needing to proxy it. It’s not that simple of course!

That exercise took me down the path of trying to do a simple SAFE:GET from golang, nodejs, python etc without depending on any maidsafe code, just using quic libraries.

I managed to connect to the vault but quic is still very experimental so I never got past the handshake. Honestly even being able to connect was above my expectations, so please consider this topic very experimental.

It made me wonder, what’s the full spec for doing something like a GET on SAFE, without using rust or maidsafe.

It’s not something I have a clear answer on (or expect to have one any time soon) but here’s some brain dump. If anyone has anything to add that would be cool.

  • Uses quic protocol
  • Vault offers quic versions of 0xa1a2a3a 0xff00001b
  • Looks like messages are serialised with serde using bincode, not sure how cross-language bincode is…?

Anyway, just taking notes on my experiments, if you have any other info or experiments would be cool to hear it.

6 Likes

You’re right on serde + bincode. The comms are all serialised and sent through quic-p2p right now.

A quick google makes it look like bincode is designed for rust only. Which, probably means we should look to remove that down the line and use something a bit more universal.

qp2p shepherds bytes with a WireMsg format, everything coming from SCL is a Message type (or will be… this is in the midst of an overhaul clarify things a bit more)

I’m not suuupper au fait with our network layer comms (yet! digging into qp2p at the moment actually), but definitely an interesting exercise! And happy to help answer any more Qs that may come up :+1:

3 Likes

Might be of relevance: Writing a core library in Rust --- possibilities for SAFE Browser (WASM)

2 Likes

This is the list of formats implemented for serde

JSON (inefficient)
Bincode (rust only)
CBOR
YAML (inefficient)
MessagePack
TOML (inefficient)
Pickle
RON (rust only)
BSON
Avro
JSON5 (inefficient)
Postcard (rust only)
URL (inefficient)
Envy (impractial)
Envy Store (impractical)
S-expressions (impractial)
D-Bus’s binary wire format
FlexBuffers (rust only)

I’m surprised not to see protobuf or capnproto in the list.

From the remaining options there are some that are fairly specific to certain use cases even though they are portable

CBOR
MessagePack
Pickle (vendor specific, python)
BSON (vendor specific, mongodb)
Avro (vendor specific, hadoop)
D-Bus’s binary wire format (seems to have a fairly complex interface)

So that leaves CBOR and MessagePack.

2 Likes

(Ignore my earlier post, I misread the source I linked and mentally swapped CBOR and MessagePack in my head, but can’t edit or delete the post because of permissions right now) :upside_down_face:
EDIT: Seems to display properly now

Seems CBOR is claimed to be more “flexible”, but the design and stability seems to come into question. Don’t know all the requirements for datatypes SAFE would need, but MessagePack is extensible at any rate, and seems like less complexity and fewer potential bugs has good security implications in the long term.

1 Like

I can’t delete the reply but have made it a wiki and edited it. You may not be able to edit though - but I can remove the content of you want.

1 Like

I’m not familiar with what it means to make the reply a wiki, but, at any rate, it seems to display as intended in the timeline, so I suppose it’s fine now, thanks!

2 Likes

You need more trust to do things like make a post into a wiki, also to edit your own posts it seems. You can always ask the mods to do stuff.

2 Likes

I tried modifying vault and dependencies replacing bincode with cbor or messagepack to hopefully do some benchmarking of performance, but their interfaces are different enough that this is not a trivial task. Just making a note of it here, might go back to this one day cause I’m fairly curious about it.

Interesting that all the serde options seem to be schemaless, whereas protobuf and capnproto (which seem like good options to me) require schema definitions.

I feel there’s some strength in having schemas, especially for versioning and upgrades and deprecation purposes.

Also interesting that serde does not have xml as an option (not that I would want it just an observation!).

I also note that safe-vault uses pickledb quite a bit (which is not the same as serde-pickle used for serialization despite having similar origins in python pickle).
https://github.com/maidsafe/safe-vault/search?q=pickle

3 Likes