How to upload a file to the network


#1

This will be a step-by-step guide for uploading a file to the network and be able to view it in SAFE browser.

You can always simply use web hosting manager to upload files to the network, however, if you go through this tutorial, you’ll learn a good chunk of the SAFE web api.

Download latest release of SAFE Browser: https://github.com/maidsafe/safe_browser/releases/tag/0.4.0

Clone this repository and cd into safe_web_api_playground: https://github.com/maidsafe/safe_examples/tree/master/safe_web_api_playground

Get the safe_web_api_playground running on localhost:3003 then access it in your SAFE Browser tab using localhost://p:3003.

Reference documentation: http://docs.maidsafe.net/beaker-plugin-safe-app/

The code examples I provide are meant for a step-by-step process and will be more verbose than is needed for ES6 in a program.

Overview

We’ll start with an example SAFE url where we’d like to be able to visit to view our uploaded html file:

safe://site.mypublicid

mypublicid is our public ID, and site is the service name.

A service can be an email address, chat ID, or in this case, site is a web service that holds file entries which are loaded into the browser.

We will:

  1. Create a mutable data structure with a randomly generated name, type tag 15002, and insert file entry
  • Create mutable data structure, named after the hashed version of our public id, mypublicid, with type tag 15001. Insert an entry where the key is `site, and the value is the name of the MD structure created in step 1.
  • Get _publicNames container and insert encrypted entry for key is the encrypted mypublicid and the value is the encrypted hash of mypublicid
  • Get _public container and insert entry where key is, in this case, _public/mypublicid/site-root and the value is name of the MD structure created in step 1.

Step 1: Initialise (cheers to Scotland), application

let appInfo = {
        id: 'net.maidsafe.test_site.webclient.1',
        name: 'Test site',
        vendor: 'MaidSafe Ltd.',
        scope: null
      };

      window.safeApp.initialise(appInfo)
      .then(function(res) {
        appHandle = res;
        return 'Returns app token: ' + res;
      });

There is no convention so far for id, name, and vendor, name them as you will.

Step 2: Ask for app authorization and permissions

window.safeApp.authorise(
      	appHandle,
      	{
          _public: [
            'Read',
            'Insert',
            'Update',
            'Delete'
          ],
          _publicNames: [
            'Read',
            'Insert',
            'Update',
            'Delete'
          ]
      	},
      	{own_container: true}
      ).then((res) => {
      	authUri = res;
      	return 'App was authorised and auth URI received: ' + res;
      });

own_container set to true will create private app container known as home container. We’ll take a took at it later.

_publicNames is a container created for your user account and will follow you around the network. In this tutorial, it will store the encrypted version of your public ID. It will other applications to lookup your public ID and other information intended to be shared across applications on the network.

_public container is created to hold non encrypted entries, in our case, a name reference to our file entry MD.

Step 3: Connect app to the network

window.safeApp.connectAuthorised(appHandle, authUri)
      .then(appHandle => {
      	return 'The app was authorised and a session was created with the network. App token returned: ' + appHandle;
      });

Step 4: Create MD to hold file entries

window.safeMutableData.newRandomPublic(appHandle, 15002)
      .then((res) => {
      	mdHandle = res;
      	return 'Returns handle to newly created, public, randomly named MutableData structure: ' + res;
      });

The 15002 tag is the network convention for a web service. In the case of an email inbox, 15003 will be used. These conventions are important for cross application compatibility.

Step 5: Setup MD

In order to save this MD we need three objects: permissions, permissions-set, and entries

window.safeMutableData.newPermissions(appHandle)
      .then((res) => {
      	permsHandle = res;
      	return 'Newly created permissions handle returned: ' + res;
      });

window.safeMutableData.newPermissionSet(appHandle)
      .then((res) => {
      	permSetHandle = res;
      	return 'Returns newly created PermissionsSet handle: ' + res;
      });

window.safeMutableData.newEntries(appHandle)
      .then((res) => {
      	entriesHandle = res;
      	return 'Returns an entries handle to be used with safeMutableDataEntries functions: ' + res;
      });

Here handles are returned to be able to run operations on our permissions, permissions-set, and entries objects.

We’ll now set allowable actions in our permissions-set:

      window.safeMutableDataPermissionsSet.setAllow(permSetHandle, "Insert")
      .then(() => window.safeMutableDataPermissionsSet.setAllow(permSetHandle, "Update"))
      .then(() => window.safeMutableDataPermissionsSet.setAllow(permSetHandle, "Delete"))

Obtain public signing key to insert that permissions-set into our permissions object:

window.safeCrypto.getAppPubSignKey(appHandle)
      .then((res) => {
      	signKeyHandle = res;

      	return 'Returns applications public signing key: ' + res;
      });

Insert permissions-set into our permissions object:

window.safeMutableDataPermissions.insertPermissionsSet(permsHandle, signKeyHandle, permSetHandle)
      .then(_ => {
      	return 'Finished inserting new permissions';
      });

Step 6: Save to the network

window.safeMutableData.put(mdHandle, permsHandle, entriesHandle)
      .then(_ => {
      	return 'Finished creating and committing MutableData to the network';
      });

Step 7: Insert files into service MD

window.safeMutableData.emulateAs(mdHandle, 'NFS')
      .then((res) => {
      	nfsHandle = res;
      	return 'Returns nfsHandle: ' + res;
      });

In the SAFE web API playground, you can choose a file with file explorer. Here we’ll simply insert our own string content:

 let fileContent = '<!DOCTYPE html><html><head><meta charset="utf-8"><title>SAFE web site</title></head><body>SAFE web site</body></html>';

      return window.safeNfs.create(nfsHandle, fileContent)
      .then((res) => {
        fileHandle = res;

      	return 'Returns the file handle of a newly created file: ' + res;
      });

Then commit the file to the network:

let fileName = 'index.html';
      return window.safeNfs.insert(nfsHandle, fileHandle, fileName)
      .then(res => {
      	fileHandle = res;

      	return 'Returns same fileHandle: ' + res;
      });

Step 8: Save name of randomly named MD

window.safeMutableData.getNameAndTag(mdHandle)
      .then((res) => {
        mdName = res.name.buffer;
      	return 'Name: ' + String.fromCharCode.apply(null, new Uint8Array(res.name.buffer)) + ', Tag: ' + res.tag;
      });

Step 9: Free handles from memory:

window.safeCryptoSignKey.free(signKeyHandle)
      .then(_ => {
        signKeyHandle = null;
      	return 'signKeyHandle freed from memory';
      });

window.safeMutableDataPermissionsSet.free(permSetHandle)
      .then(_ => {
        permSetHandle = null;
      	return 'permissionsSetHandle is freed from memory';
      });

window.safeMutableDataPermissions.free(permsHandle)
      .then(_ => {
        permsHandle = null;
      	return 'Frees permsHandle from memory'
      });

window.safeMutableDataEntries.free(entriesHandle)
      .then(_ => {
        entriesHandle = null;
        return 'Handle to MutableData entries freed from memory';
      });

window.safeMutableData.free(mdHandle)
      .then(_ => {
        mdHandle = null;
        return 'MutableData is freed from memory';
      });

All done with creating the service MD srtucture that holds our file entries.

Step 10: Create public ID and insert service

First, hash your desired public ID:

window.safeCrypto.sha3Hash(appHandle, 'mypublicid')
      .then((res) => {
        hashedString = res;
      	return 'SHA3 Hash generated: ', String.fromCharCode.apply(null, new Uint8Array(res));
      });

Create new mutable data structure named after your hashed public id:

window.safeMutableData.newPublic(appHandle, hashedString, 15001)
      .then((res) => {
      	mdHandle = res;
      	return 'Returns handle to newly created or already existing, public, explicitly named Mutable Data structure: ' + res;
      });

The 15001 tag designates this MD as a DNS lookup for services.

Now go through the same setup process as in Step 5 and come back here when you’re done.

We’ll now insert our service:

window.safeMutableDataEntries.insert(entriesHandle, 'site', mdName)
      .then(_ => 'New entry inserted');

Observe that the entry key is the service name, site, and the entry value is the name of our service MD, created in Step 4.

Repeat Step 6 to commit this MD to the network

Step 11: View file on network

At this point you can observe successful file upload in two ways:

  • Visit safe://site.mypublicid in your SAFE browser

or fetch the file:

window.safeApp.webFetch(
      	appHandle,
      	'safe://site.mypublicid/index.html' // the SAFE Network URL
      )
      .then((data) => {
        console.log(String.fromCharCode.apply(null, new Uint8Array(data)));
      	return String.fromCharCode.apply(null, new Uint8Array(data));
      });

Repeat Step 9 to free handles from memory


Step 12: Insert public ID into _publicNames container

Get _publicNames container:

let container = '_publicNames';
      return window.safeApp.getContainer(appHandle, container)
      .then((res) => {
      	mdHandle = res;
      	return 'Returns handle to Mutable Data behind ' + container + ' container: ' + res;
      });

Get mutation object:

window.safeMutableData.newMutation(appHandle)
      .then((res) => {
      	mutationHandle = res;
      	return 'Returns handle to be able to call safeMutableDataMutation functions: ' + res;
      });

Encrypt plain text version of public id:

window.safeMutableData.encryptKey(mdHandle, 'mypublicid')
      .then((res) => {
      	encryptedKey = res;
      	return 'Encrypted key: ' + res;
      });

Encrypt hashed version of public id. This should still be stored in your hashedString variable. Otherwise use window.safeCrypto.sha3Hash as we did in Step 10.

window.safeMutableData.encryptValue(mdHandle, hashedString)
      .then((res) => {
      	encryptedValue = res;
      	return 'Encrypted value: ' + res;
      });

Insert entry into mutation object:

window.safeMutableDataMutation.insert(mutationHandle, encryptedKey, encryptedValue)
      .then(_ => {
        return 'Registers an insert operation with mutation handle, later to be applied.';

        // You must now run safeMutableData.applyEntriesMutation(mdHandle, mutationHandle) to save changes.
      });

Apply mutation to _publicNames container:

window.safeMutableData.applyEntriesMutation(mdHandle, mutationHandle)
      .then(_ => {
      	return 'New entry was inserted in the MutableData and committed to the network';
      });

Free mdHandle and mutationHandle from memory

Step 13: Insert reference to service in _public container

Get _public container:

      let container = '_public';
      window.safeApp.getContainer(appHandle, container)
      .then((res) => {
      	mdHandle = res;
      	return 'Returns handle to Mutable Data behind ' + container + ' container: ' + res;
      });

Get mutation object:

window.safeMutableData.newMutation(appHandle)
      .then((res) => {
      	mutationHandle = res;
      	return 'Returns handle to be able to call safeMutableDataMutation functions: ' + res;
      });

Insert reference to service name:

window.safeMutableDataMutation.insert(mutationHandle, '_public/mypublicid/site-root', mdName)
      .then(_ => {
        return 'Registers an insert operation with mutation handle, later to be applied.';

        // You must now run safeMutableData.applyEntriesMutation(mdHandle, mutationHandle) to save changes.
      });

The entry key, '_public/mypublicid/site-root' is our convention for identifying our web service.
The entry value will be the name of our service MD created in Step 4, still held in mdName variable.

Apply mutation to _public container:

window.safeMutableData.applyEntriesMutation(mdHandle, mutationHandle)
      .then(_ => {
      	return 'New entry was inserted in the MutableData and committed to the network';
      });

Complete.

Reference over view of data structures and what we just created here: https://github.com/hunterlester/safe_dom_api_playground/blob/master/safe_dns.md


Lookup table of type tags
#2

Wow Hunter this looks awesome :slight_smile: I can’t wait to have a go at this, thanks.

Have you fixed the issue I had with building the dev-mid Browser? Or is there some way to get a working version?


#3

All fixed, yes (I think? I hope? LOL!)

I just made a build from scratch and everything went smoothly.

I’m just thinking about assumptions I may be forgetting.


#4

Great, I’ll try and find time to have a go with it today. Thanks.


#5

@hunterlester thanks for this great tutorial! One question: Is there a place where the available tag numbers are listed and explained? What effect does the tag number have exactly?


#6

The tag types haven’t been explained yet, I don’t think.

They are simply conventions for the network.
So far:
15001 -> public id
15002 -> web service
15003 -> email service

Since they are agreed upon conventions, the network can and will decide if it wants to keep these or use other numbers.


#7

Has anybody got the safe_dom_api working on Linux (I’m on Mint 18.1)?

  • ‘npm start’ works with the dev Safe Browser of @hunterlester
  • ‘npm run package’, fails, but maybe not really necessary?
  • Making Safe account and login does work.
  • the safe_dom_api_playground does work: gulp does start and I can surf to localhost:3003.
  • window.safeApp.initialise, authorise and connectAuthorise do work (also fu. call isRegistered and networkState).
  • but if I then try e.g. newRandomPrivate, the Safe Browser stops/crashes without saying why.

#8

I have written a summary here (until step 11 which is unsuccessful). It is very frustrating because everything works at each step, but when I do web fetch, it is blank, and the index.html is not found.

Let’s say that want to create a new public id and an new service with your tool. For instance, instead of safe://safeapi.playground
I would like to create
safe://bonnie.clyde
so clyde goes with tag 15001 and bonnie with tag 15002
and If I get it right, I reckon here that Bonnie is my service, and Clyde is my public id

These are all your steps:

1. first I create a newrandompublic mutable data structure with tag 15002, and then I use this mutable structre to emulate the network file system and upload the directory of my website.

safeApp

  • initialise (changed the appinfo accordingly)
  • authorise
  • connectAuthorised

safeMutableData

  • newRandomPublic (with tag 15002)
  • newPermissions
  • newPermissionSet

safeMutableDataPermissionsSet

  • setAllow (Update, Insert,Delete)

safeCrypto

  • getAppPubSignKey

safeMutableDataPermissions

  • insertPermissionsSet

safeMutableData

  • put
  • emulateAs

safeNfs

  • uploadDirectory (here I take my own Bonnie_Clyde directory with css, js, fonts and image, and my index.htlm at the root)
  • free

safeCryptoSignKey

  • free

safeMutableDataPermissionsSet

  • free

safeMutableDataPermissions

  • free

safeMutableDataEntries

  • free

safeMutableData

  • getNameAndTag
  • free

2. Then I create my public id Clyde so that it will hash

safeCrypto

  • sha3Hash ( I did type Clyde as a value )

safeMutableData

  • newPublic (with the hashedstring and tag 15001)
  • newPermissions
  • newPermissionSet
  • newEntries

safeMutableDataPermissionsSet

  • setAllow (Insert, update, delete)

safeCrypto
getAppPubSignKey

safeMutableDataPermissions

  • insertPermissionsSet

safeMutableDataEntries

  • insert (return window.safeMutableDataEntries.insert(entriesHandle, ‘Bonnie’, mdName)

safeMutableData

  • put

and then
safeApp
webfetch


#9

@hunterlester could you also explain how to run the script in one go instead of running all the snippets one by one ?

running snipets is good to learn, but very time consuming to run


#10

I’ll include a complete code example.

What you’re uncovering is good. You’re uncovering that the end user is not receiving enough useful information, if any, if an operation is unsuccessful.

According to the steps above, you didn’t create an entries object for your bonnie service, but regardless, authenticator should have let you know.


#11

the learning curve is a bit steep but your tool is fantastic.
I am a bit anxious for what comes next (let user insert dynamic content on my site must be tough ). But let’s proceed step by step. I have watched your vid countless times, started from scratch all over again with your write up, and couldn’t upload a site without webhosting manager yet :sob:


#12

The learning curve is indeed steep, but wouldn’t it be boring otherwise? :smiley:

@hunterlester I have two questions regarding handle freeing:

1.) In my example app I get the error “Uncaught (in promise) TypeError: Cannot read property ‘then’ of undefined” when freeing the mdHandle. It seems that free() doesn’t return anything? The error is thrown at this point of code:

window.safeMutableData.free(mdHandle).then(_ => {
mdHandle = null;
});

2.) Why is it necessary to free these resources? I mean, what benefit does it have? Just for memory management of the library (e.g. browser) or is the network also affected (e.g. are the handles otherwise being kept stored on the network?)


#13

Hi, is it possible to use this API into Electron for exemple (how?) or do we have to use the safe browser for now?

Since the “window” object contains the safeApi… :smiley:


#14

@SwissPrivateBanker I’ll get back as soon as I can. Instead of producing successes, I’m attempting all the ways in which the steps in this operation can go wrong.

@Mindphreaker Regarding first question, when did you clone the repos? I recently fixed that error you are receiving about the use of then. If you have the latest, I’ll take another look into what I missed.

For the second question, thank you to @bochaco for the explanation.
If I’m butchering this explanation, let it not reflect poorly on him.

In all cases, the allocated resources represented by a handle only have to do with local system heap and are not effecting network memory.

The good thing about the use of the SAFE browser is that even if calls are not made to free resources, those resources will be freed from system heap when the browser is closed and the JS garbage collector is triggered.

Same thing for use of safe_app_nodejs when making your own applications. Those resources will eventually be freed, when garbage collector is triggered.

I’m still learning our core Rust libraries, so I can’t speak very well to the low level but these libraries do have redundancies to ensure memory safety.

To your original question then, why do we then even need to manually free resources on the client?
When I’ve not freed resources, I’ve eventually run into generic errors and am no longer able to perform certain operations. So I think this is about managing the current client instance.

I’d like to replicate my errors and look more into this. Thank you for bringing it up.

@JesusTheHun Thank you for the question. You’re not stuck with the SAFE web API in the browser. You may use safe_app_nodejs API to allow applications, including electron applications, to interface with the network. See email app example


Upload, store and load image
#15

I’d just add that we are currently working on making sure that whenever a tab is closed the browser automatically frees all resources created by the contained app. As @hunterlester well explained, this is only a memory management thing on the browser, these handles only live in the browser’s main process.


#16

Thanks for the clarification!


#17

@SwissPrivateBanker hey, good to see you going beyond your banker ways and getting into code :wink: but I’m curious how you got the playground to work - I haven’t tried that step because according to the OP (quoted above) you first need the dev-mode browser, but I can’t get that to build.

Did you build the dev-mode Browser, or did you not need to?

I’m planning to play tomorrow, so a solution to this would be great, or I’ll just wing it! :slight_smile:


#18

replying to you in pm about this.

I have an idea about an app I have dreamt to build for years but can’t do on the clearnet.
Safe is the only way for me to see this through.
My interest in SAFE is not just speculative :wink:


#19

@SwissPrivateBanker is on Windows so we were able to figure some things out together.

I badly need a way to efficiently test and build for other platforms.

Working on it. Thankfully we have @srini on the team now :grinning:

Good news coming.

My mess of a browser build may be replaced by a very clean and simple toggle.


#20

Would be very handy if you could enter in the port to use