Getting a safe:// url for a file

It’s currently not possible to get the safe:// url for a file without being connected to a network - see sn_api/…/subcommands/xorurl.rs#L74 which calls connect(safe).await

Most of the logic for creating the safeurl is sn_api/…/safeurl.rs

Is it possible to use the safeurl logic elsewhere? I haven’t had any luck importing it to my test app.

In theory safe xorurl --dry-run /tmp/data.bin should give the url without being connected to the network. But following through the chain of logic we get to safe_client.rs#L203 - “TODO: allow this operation to work without a connection when it’s a dry run”, ie dryrun is currently ignored.

I appreciate the format of safeurls is specific to sn_api (especially nrs urls); safeurls are not required to fetch data, just convenient. However I wonder if it’s worth pulling this safeurl logic out to a library since I imagine it will be the standard way to publicize and access data on the network so it perhaps should be more easily available than within sn_api?

To give my specific use case that led to this, I’m making a gui for self_encryption, and I want to show the safeurl for each chunk so they can be fetched individually from a network. But right now it’s almost impossible to easily generate safeurls. I have the content and the xorname, but it’s not easy to convert into a safeurl.

Would it be helpful to refactor into a safeurl library? sn_url? I’m happy to do this if it’s of value.

I really appreciated the example in self_encryption that builds a standalone self_encryption binary for encrypting/decrypting any file. Maybe we could also have a sn_url example binary that easily allows us to get the safeurl for any file (like what safe xorurl --dry-run /tmp/data.bin is meant to do)?

Any thoughts on the best way to generate safeurls, without cherrypicking a bunch of code from safeurl.rs?

1 Like

This makes complete sense.

self encrypt is also similar, you should not need a network (here we are fine), but same situation.

Oh yes, it would be really appreciated.

Again, yes that is a valid example.

We can lean on @bochaco (again, sorry Gab) for help here. He is all over the codebase right now but let’s see if we can grab a wee while as this would be really helpful Ian and I think making this more modular would be a wise move.

2 Likes

I started looking into this issue earlier today, this was disabled after the big refactoring, it needs some minor changes in sn_client plus https://github.com/maidsafe/sn_api/pull/772 then dry-run and xorurl commands should be working again without connecting.

+1 …It makes sense, SafeUrl is all about encoding, so it sounds like it could be a very useful standalone crate/lib for any Safe app.

Unless I missunderstand, this is an example of one of the APIs it exposes already:

    use sn_api::safeurl::{SafeUrl, XorUrlBase};
    ...
    let xor_name = XorName::default();
    let url = SafeUrl::encode_safekey(xor_name, XorUrlBase::Base32z).unwrap();
    println!("XorUrl: {}", url);

There are other ways, but there are a few helpers like that one above, otherwise you can go for SafeUrl::new(...)

1 Like

Thanks for this example code, it’s really helpful.

I get a compile error with it though (which is what eventually led to the original post to pull this into a library, not understanding how to correctly import sn_api to my test project, and such a large lib just to get to the safeurl logic):

error[E0432]: unresolved import `sn_api::safeurl`
 --> src/main.rs:2:13
  |
2 | use sn_api::safeurl::{SafeUrl, XorUrlBase};
  |             ^^^^^^^ could not find `safeurl` in `sn_api`


Cargo.toml includes this dependency:

 sn_api = "0.23.2"

The new name of the module SafeUrl was not released yet, it’s a refactoring made last Friday, so if you change it to use sn_api::xorurl::{SafeUrl, XorUrlBase}; I think you should be fine.

1 Like

FYI, I just added a small example: fix(cli): xorurl command generates its output without connecting to a network by bochaco · Pull Request #772 · maidsafe/sn_api · GitHub

1 Like

A simple sn_url lib is ready to go, download it from sn_url.zip

I’m not publishing it to github since it contains license/copyright etc with maidsafe in it. If you’d prefer not to publish that’s totally fine by me, just let me know so I can publish it under a different name/license for my own use (I’m using it locally and it works well for my needs).

Notes:

Removed logging (info! and warn! logs, still there but commented out)

Replaced tiny_keccak with xor_name since that’s all it was used for, no point duplicating xor_name logic here

Recreates Result and three Error copied from sn_api/src/api/errors.rs
Might not be the cleanest way to implement this?
Duplicated Errors are InvalidXorUrl InvalidInput InvalidMediaType

Some lint checks have been commented out, lint checks have been copied from self_encryption, would be good to fix or remove these.

Combined several files into lib.rs to keep the imports simple, first steps for refactoring sn_api:

  • sn_api/src/api/app/xorurl.rs completely remove
  • sn_api/src/api/app/xorurl_media_types.rs completely remove
  • sn_api/src/api/errors.rs might need tweaking for new errors?
  • sn_api/src/api/app/nrs.rs remove duplicate const NRS_MAP_TYPE_TAG
  • find places that depended on xorurl.rs and switch them to use sn_url

There’s some stuff in sn_api/…/nrs.rs that has not been moved over which seems like it probably does not fit within sn_url, but may do:

  • pub fn parse_url
  • pub fn parse_and_resolve_url
  • fn validate_nrs_name
  • fn sanitised_url

edit:

Also

  • xor_name in Cargo.toml needs changing to crates.io version instead of local (I had issues with OsRng so switched to my local copy)
  • add example binary as per post #6
2 Likes

Thanks @mav, we were discussing about this and we’ll be creating a repo next week. Also I think you were basing your changes on earlier version of sn_api, probably earlier than 7 days ago, if you look at master branch you’ll see how the safeurl mod contains all of it but split in three files, I think we should keep something like that in the upcoming safeurl crate and repo.

We can double check, but these are not supposed to belong to safeurl, they are more related to resolving NRS URLs and fetching content from xorurls.

1 Like

I’ve just created a branch (refactor(safeurl): moving out safe_url mod as a standalone crate by bochaco · Pull Request #779 · maidsafe/sn_api · GitHub) where we can work on this till we have the new repo, I just moved out the safe_url module as a crate at the root of the repo, if you want you can send your changes to that branch. Then we move that crate to the new sn_url repo as soon as is ready next week.

2 Likes

This is great. For now I’ll keep my local copy of sn_url for my own use and update to the official repo when it’s released later. I did a copy-paste from sn_api to sn_url, no logic or api changes, so I have nothing to add to the refactoring process that’s now underway.

Really amazing how fast the team at maidsafe is moving these days, thanks for everything.

2 Likes

I was reviewing this, and we actually need it for calculating the sha3 of a name when we want to calculate the xorname for an NRS name, that’s what we are using it for, remember an NRS name can be found at the xorname which is sha3(< nrs name >).

1 Like

Ah yes.

Even though the logic is the same for
fn xor_name_from_nrs_string
and
fn XorName::from_content
I think it probably makes sense to keep them separate from each other.

I had made this change to remove tiny_keccak from sn_url:

fn xor_name_from_nrs_string(name: &str) -> XorName {
    XorName::from_content(&[name.as_bytes()])
}
1 Like

ohh! i haven’t realised of what that from_content function was doing, you are right!!

FYI: sn_url crate just published: https://crates.io/crates/sn_url

3 Likes

what languages are you guys using it in? NodeJS is the standard? Which language for api is most current at now