SAFE Drive - help with testing

drehb · October 18, 2018, 11:48am

Ah, didn’t realize that. Microsoft is much more Linux friendly these days indeed

lukas · October 19, 2018, 9:45am

I got breakpoints working.

Running latest master.

I can’t get it to work though.

And I can’t figure out what’s wrong in the code.

Think I’ve run out of steam on this.

I found that this is where the error message comes from, line 150 in root.js:

    // This handler's container is not mounted yet so attempt to mount it
    if (this._mountPath !== '/') {
      // This RootHandler is for a SAFE root container
      if (this._lazyInitialise) {
        // Make it on demand (lazy)
        return this.initContainer(rootContainerName)
      } else {
        // Either itemPath should not be handled here (so a bug elsewhere)
        // or perhaps the constructor has not finished creating containers
        throw new Error('getContainer() - no container object ready for ' + itemPath)

this._mountPath is "\/"
and this._lazyInitialise is false, making it throw that error.

happybeing · October 19, 2018, 10:34am

Excellent, well done. The error you found looks like the kind of thing I was expecting to have messed up, so very useful.

Can you see why the mountPath is “\/” and not “/”?

The code you show is in the RootHandler, which appears not to have this._container set, which should happen here:

github.com

happybeing/safenetwork-fuse/blob/master/src/safe-vfs/root.js#L182


      
           *
           * @param  {String} itemPath the mountPath of the FUSE item
           * @return {[type]}          a SafenetworkJs container
           */
          async getContainer (itemPath) {
            try {
              let defaultContainerName = '/' + itemPath.split('/')[1]
              if (await this._container) {
                // Root containers hold items which all start with '/'
                if (itemPath.indexOf(defaultContainerName) !== 0) {
                  let e = new Error('file does not exist')
                  debug(e)
                  throw e
                }
                return this._container
              }
          
              // Wasn't mounted by the constructor, so we do it on demand
              if (this._lazyInitialise && this._mountPath !== '/') {
                return this.initContainer(this._containerRef)
              }

That mirrors what you are seeing late in getContainer() so the issue might be that for some reason the RootHandler is being created with mountPath “\/” instead of “/”. This sounds very like a Windows v Linux thing so I reckon you are onto something if you can see where the RootHandler’s this._mountPath and change the code responsible to set it to “/” instead of “\/”. Probably here:

github.com

happybeing/safenetwork-fuse/blob/master/src/safe-vfs/index.js#L369


      
            }
          }
          
          async unmountFuse (mountPath) {
            return new Promise((resolve, reject) => {
              return Fuse.unmount(mountPath, err => {
                if (err) {
                  debug('Failed to unmount SAFE FUSE volume at: %s', mountPath)
                  debug(err)
                  reject(err)
                } else {
                  this._pathMap = new Map()
                  resolve()
                }
              })
            }).catch((error) => {
              debug(error.message)
            })
          }
          
          /**

If you fancy having another go, try setting a breakpoint there and watch the creation of the RootHandler’s container, a RootContainer object. Keep an eye on why this._mountPath doesn’t end up as “/”.

Hopefully I’m pointing in the right direction! A bit tricky debugging like this but I’m sure you are on the right track.

Update:

@lukas I’ve been looking around in safenetwork-fuse/src/safe-vfs/root.js and I have a hunch that you should try replacing every occurrence of path.sep with '/'. I don’t think it should be using path.sep at all.

happybeing · October 19, 2018, 6:57pm

Status Update

I’ve been testing the ability to create directories and copy files to your SAFE storage mounted as a SAFE Drive and so far it is going well!

using mock network, copied a 129M .git directory tree to a SAFE network directory, then did a file by file comparison with no differences
then did a commit on the original, repeated the compare, and those changes showed up

Anyone can try this if you pull the development branch.

Note, this works with live or mock networks, but if you want to use mock, you can set an additional environment variable to cause it to create a folder for testing: _public/tests/data1/. This saves you having to create one with WHM. To enable that, use SAFENETWORKJS_TESTS=testing along with NODE_ENV=dev (and a Dev build of SAFE Browser).

happybeing · October 20, 2018, 5:39pm

@lukas I’ve been looking around in safenetwork-fuse/src/safe-vfs/root.js and I have a hunch that you should try replacing every occurrence of path.sep with '/'. I don’t think it should be using path.sep at all.

happybeing · October 27, 2018, 12:39pm

UPDATE: new features available to test

I’ve just pushed some working [ahem] new features to the development branch, so if anyone would like to test them by running from node (ie not the packaged version) by all means give it a try.

You should git pull && git checkout development branch and use the instructions from the README to run under node. I’ll be working on a packaged version for release next week, but it would be good to get some more testing to run these new things through their paces.

New (buggy) features:

Speed. Things are now much faster, and you can use the file manager to browse your SAFE files without it appearing to hang.
Create and Update Files.
- You can create/update files into an existing folder (created using Web Hosting Manager) from the command line and piping into a file (e.g. echo Hello world > ~/SAFE/_public/mydomain/www-root/index.html).
- Copy file, make directory, recursive copy should also work so you should be able to copy websites too and from your SAFE Drive.
- Create a file and/or update its modification time using (Linux) touch
- You can also create/update files from some apps, but many don’t work properly yet (e.g. failing to save changes). I think vim works, but maybe doesn’t clean up temp files.
File Sharing via SAFE Web Mounts. Since all websites are public files, you can share files by saving them on a website domain, and others can access the files of any website from their SAFE drive by mounting its folder.
For example, to mount the websites at safe://heaven and safe://cat.ashi:
```
ls ~/SAFE/_webMounts/heaven
ls ~/SAFE/_webMounts/cat.ashi
ls ~/SAFE/_webMounts
tree ~/SAFE/_webMounts
# etc
```
No cats were harmed in the making of this feature

This feature also illustrates automount (see next bullet).
Automount Default Containers. Only _public and _publicNames are supported by SAFE Network just now, but later you will have _documents, _videos and so on. If these are not already mounted, you can force them to mount just by accessing them. You can test this by commenting out the default mount for _publicNames in bin.js and then trying:
```
ls ~/SAFE # shows _publicNames is not yet mounted
ls ~/SAFE/_publicNames # mounts and lists directory
ls ~/SAFE # show _publicNames is now mounted
```

If you do any testing please reply to this post to let me know, and if you find any issues please file them on github, mentioning the development branch.

See the OP for help if you are not sure where to start

Things I haven’t tested but if they are feasible for you to have a go:

Decentralised Git. Create a ‘headless’ (possibly the wrong term) git repository on your SAFE drive, set it as remote for your local git repo, and push changes to it, have others clone it and pull changes etc
Backup/Sync. Would be great if somebody would try using rsync to keep a public share of files updated, and to show that it can be used for backup. Also other apps than rsync but that’s one I know is awesome and will prioritise.

@lukas I made the changes I think might help get this to work on Windows so if you are still available, feel free to give it a go.

JPL · October 27, 2018, 1:05pm

Will try now! Just to preempt the inevitable, does it need a particular version of node?

happybeing · October 27, 2018, 1:12pm

Well reasonably up to date I guess, but should be no different to the other things like SAFE Browser. I’m not at my PC now so can’t check what I’m using. Sorry!

Thanks for checking it out again.

JPL · October 27, 2018, 3:09pm

Can you spell this out pls?

You should git pull && git checkout development branch

I downloaded the zip from the dev branch but ran into permissions issues with linking. I presume that’s a different problem but just checking.

Also the readme talks about SAFE Browser

NODE_ENV=dev safe-browser

I take it Peruse will also work (although that command failed when replacing safe-browser with peruse) ?

happybeing · October 27, 2018, 4:33pm

@JPL If you download from the development branch, you are good, otherwise you should go into the directory and issue:

git pull
git checkout development

And you may also need:

git checkout .

Normally, starting from scratch, I would do this:

git clone https://github.com/theWebalyst/safenetwork-fuse 
cd safenetwork-fuse 
git checkout development
git checkout .

And then the same for safenetworkjs

The linking issue looks like your system, so not sure about that.

Any recent SAFE/Peruse browser should do, with the proviso…

…be careful with NODE_ENV=dev. That is only to be used when you want to use the mock network, so requires a mock browser setup.

Unless you want to try saving quite a bit of data I suggest people use the live network, then you will be able to access other people’s website files too.

If anyone wants to try the git or rsync tests, I suggest starting with mock to see if there are obvious problems, and then switching to live.

Thanks a lot f your trying this out John!

JPL · October 27, 2018, 4:41pm

No probs it’s fun! Apart from this which I’m getting consistently in Debian and Fedora when issuing npm link from safenetworkjs folder:

user@user1:~/safe-fuse-test/safenetworkjs$ npm link
npm ERR! path /usr/lib/node_modules/safenetworkjs
npm ERR! code EACCES
npm ERR! errno -13
npm ERR! syscall unlink
npm ERR! Error: EACCES: permission denied, unlink '/usr/lib/node_modules/safenetworkjs'
npm ERR!  { Error: EACCES: permission denied, unlink '/usr/lib/node_modules/safenetworkjs'
npm ERR!   stack: 'Error: EACCES: permission denied, unlink \'/usr/lib/node_modules/safenetworkjs\'',
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'unlink',
npm ERR!   path: '/usr/lib/node_modules/safenetworkjs' }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator (though this is not recommended).

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/user/.npm/_logs/2018-10-27T16_34_11_808Z-debug.log

Using sudo works although I then run into (possibly related) issues running npm run build-mock from the safenetwork-fuse folder.


> safenetwork-fuse@0.1.0 build-mock /home/user/safe-fuse-test/safenetwork-fuse
> mkdir -p ./dist/mock/node_modules/@maidsafe/safe-node-app/src/native/mock/ && cp -R node_modules/@maidsafe/safe-node-app/src/native/mock/*.* ./dist/mock/node_modules/@maidsafe/safe-node-app/src/native/mock/ && pkg . -t host -o ./dist/mock/mount-safe

cp: cannot stat 'node_modules/@maidsafe/safe-node-app/src/native/mock/*.*': No such file or directory
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! safenetwork-fuse@0.1.0 build-mock: `mkdir -p ./dist/mock/node_modules/@maidsafe/safe-node-app/src/native/mock/ && cp -R node_modules/@maidsafe/safe-node-app/src/native/mock/*.* ./dist/mock/node_modules/@maidsafe/safe-node-app/src/native/mock/ && pkg . -t host -o ./dist/mock/mount-safe`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the safenetwork-fuse@0.1.0 build-mock script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/user/.npm/_logs/2018-10-27T16_45_49_947Z-debug.log

happybeing · October 27, 2018, 4:57pm

Firstly, you don’t need to build anything to try this, so skip that unless you want to try testing the build too.

You can instead just run using node bin.js in the safenetwork-fuse directory.

If you do want to build-mock, then it looks like you may have done the npm install step without NODE_ENV=dev because it isn’t finding the Maidsafe mock libraries. I think they plan to install them regardless, may do already in fact, but I’m still using the old versions for now.

PS I’m on Ubuntu now, but don’t recall having permissions problems with Debian, so not sure why you do there. A Web search should help you get those set correctly though, and would be preferable to using sudo

JPL · October 27, 2018, 5:02pm

Ah OK I’ll try again. Just checked and I did set NODE_ENV=dev so it’s not that. I probably missed some step but I had the same issues with Fedora too.

JPL · October 27, 2018, 5:11pm

Bah - not my day

user@user1:~/safe-fuse-test/safenetwork-fuse$ node bin.js
module.js:550
    throw err;
    ^

Error: Cannot find module 'debug'

happybeing · October 27, 2018, 5:20pm

John, you are doing valuable work. These look like bugs too - so that’s actually a win

It shows I don’t understand it all and/or don’t have time to test all the ways I can mess up. So, again: thank you.

Two things to try, first:

DEBUG=safenetworkjs*,safe-fuse* node bin

I’m curious to know if I don’t see that because I always run with DEBUG, although again it looks to me like the npm install was without NODE_ENV=dev though I know you said you definitely did that.

It would still be a bug though so I’m off to look at the package.json

Anyway, a quick fix might be to then try: npm install debug

JPL · October 27, 2018, 5:25pm

user@user1:~/safe-fuse-test/safenetwork-fuse$ DEBUG=safenetworkjs*,safe-fuse* node bin
module.js:550
    throw err;
    ^

Error: Cannot find module 'debug'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/user/safe-fuse-test/safenetwork-fuse/bin.js:3:15)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)

happybeing · October 27, 2018, 5:25pm

Yep, there’s a bug in the safenetworkjs/package.json so high five man

So in safenetworkjs, you need to do:

npm install --save debug

JPL · October 27, 2018, 5:28pm

user@user1:~/safe-fuse-test/safenetworkjs$ npm install --save debug
+ debug@4.1.0
added 16 packages from 3 contributors, updated 2 packages and audited 331 packages in 2.063s
found 1 high severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details

happybeing · October 27, 2018, 5:29pm

Yeh, that’s not one of ours as we say in the trade

Thanks for reporting it though. Nothing I can do for now.

JPL · October 27, 2018, 5:30pm

No worries. Think I’ll take the lazy option and wait for the zip package!