SAFE Drive - help with testing


#162
  1. Concerning the error I had: it only occurs on the account which owns safe://docs.to-do:
  • See safe://misc.draw2 for the output of the hanging safenetwork-fuse command and the find-output. I don’t know how to recreate the error (for another account), so no added value to put it in test-safedrive.sh. This error also occurred with different versions of Npm and Node.js.

  • find ~/SAFE/_webMounts/docs.to-do’ does work.

  • I then copied this folder like this:

$ rsync -a ~/SAFE/_webMounts/docs.to-do /tmp
$ cd /tmp/docs.to-do
$ ls -lRt
.:
total 20
drwxr-xr-x 2 draw draw 4096 Nov 22 23:25 js
drwxr-xr-x 2 draw draw 4096 Nov 22 23:25 css
-rw-r–r-- 1 draw draw 3140 Nov 1 17:33 index.html
-rw-r–r-- 1 draw draw 3111 Nov 1 17:33 favicon.png
-rw-r–r-- 1 draw draw 1150 Nov 1 17:33 favicon.ico
./js:
total 276
-rw-r–r-- 1 draw draw 4653 Nov 1 17:38 safe.js
-rw-r–r-- 1 draw draw 264100 Nov 1 17:33 vue.js
-rw-r–r-- 1 draw draw 4848 Nov 1 17:33 app.js
./css:
total 8
-rw-r–r-- 1 draw draw 8096 Nov 1 17:33 index.css

Ps the ‘-a’ arg of rsync stands for ‘archive’, which, among others, means recursive and keeping timestamps of files and directories. I see that the files keep the original timestamps, but the directories not. Is it possible to also keep the original timestamp of the dirs? That would be 2 november in this case.

I did the same for www.to-do. And then uploaded these 2 sites, with the web-hosting-manager to another account --> safe://www.te-de and safe://docs.te-de. Under this account ’find (or ls) ~/SAFE/_public/te-de/root-docs’ does work.

  1. Concerning Npm/Node.js version combinations, with the n command, I switched between the following 2 combinations:

Npm v6.4.1 and Node.js v11.2.0: doesn’t work (’nmp install’ of safenetworkjs gives an error).
Npm v5.6.0 and Node.js v8.11.3: works

I then switched back to npm v6.4.1 (with ‘sudo npm install npm@latest -g’ ), while keeping the same version of Node.js (v8.11.3), which worked like before.
-> Probably a problem with more recent Node.js versions.

  1. I tried your test-safedrive.sh 1 on Alpha 2 (nothing tested with mock) with npm v6.4.1 and node v8.11.3.
    I had to restart, because I had to install the tree cmd. Then, after +400 Account status and 18,5 minutes I got the ‘all tests passed’ :slight_smile: . I did see 1 error in the output logging of the script:
    ./test-safedrive.sh: line 196: /home/draw/SAFE/_public/tests/data1/testing-safedrive/del-dir/foo: No such file or directory

#163

Excellent work Ward, I’ve logged this and will go through it later. :+1:

Looks like this is revealing underlying issues that could be hard to isolate without this level of detail, so your experiments and detailed report are very helpful.

I hadn’t realised tree might not be installed so I might take it out, unless there’s a way to make it only be used if it is available.

Re: directory modification times: well spotted! Indeed, we don’t preserve all metadata but this is something that can be added later. Some things are going to be tricky or possibly affect performance though, such as directory size and modification time, especially as directories don’t exist! Ideally it would be standardised within the SAFE NFS API first, otherwise it will only work with SAFE Drive created stuff. <- @hunterlester should I raise an issue for this do you think? It seems more a fundamental design issue than a feature.

I pushed some bugfixes yesterday which might well improve some areas, so I recommend you git pull in safenetwork-fuse (no need to update safenetworkjs) before doing more tests. <- @isntism please note.

Thanks for this, and I also appreciate the detailed info, point about tree, and good catch supporting that test script error message! Really cool. Thanks. :clap:


#164

Useful link to check if a linux command is installed here.
Concerning the date (of publishing/or date the file had at publishing?): maybe the publisher doesn’t want that to be known in some cases (could reveal too much info)… So maybe an idea to make the saving of file/dir timestamp something optional? If it doesn’t make things to complicated.


#165

I already pulled both, safenetworkjs did change one file when doing so though, but now that I am here may as well share my preliminary findings.

Blender works very well, it spots the drive so you don’t even have to look for it, saving and loading both work as well, but I still have to try using external assets (images and the sort)

Krita works as well, though it lags while writing the filename in the saving prompt, not sure if that is something you can/should fix.

Finally, how would I share one of these files, or acces another public file, assuming the _public folder is public.


#166

Good point. I don’t have bandwidth to think about it atm so be good to have a list of feature ideas. Do you mind raising an issue - seems the simplest way for now - and prefix the title with ‘new feature:’ or something like that.

@isntism great news on Blender and Krita. Please can you keep a list of which apps you experimented with and roughly what you tried? Nothing onerous, just so we can pull things together in one go. Then at some point just send it to me and I’ll build up a list of things different people have tried out.

Cool. This is what I want to hear :slight_smile:

Well if you have put it in _public you are almost there. Anyone can access any file using SAFE Drive, if the file has been saved under a web service. So that means, for now, you need to create a public name and a trivial website using the Maidsafe Web Hosting Manager. Then find the corresponding folder under ~/SAFE/_public or ~/SAFE/_publicNames and save files there.

For example, if you create public name ‘isntism’ and it is accessible at safe://isntism you would copy files into ~/SAFE/_public/isntism/www-root (if I recall correctly - check that!).

Those files will then be available on SAFE Drive, to anyone, via a web mount. So:

ls ~/SAFE/_webMounts/isntism

Or if you chose to set up a different subname/service such as safe://files.isntism it would be available at:

ls ~/SAFE/_webMounts/files.isntism

Let us know how you get on and of course, ask if anything isn’t clear.

I already pulled both, safenetworkjs did change one file when doing so though, but now that I am here may as well share my preliminary findings.

That was probably something changed over the last few days, so if in doubt, yes pull both!


#167

Absolutely, especially to keep it on our radar and not lost in the forum.


#168

Done:


#169

Transfering a file from~/SAFE/_public/tests/data1 to ~/SAFE/_public/ism/root-files hangs for a while, jumps to 100% then gives throws error when getting information for file "/home/isntism/SAFE/_public/ism/root-files/blender_on_safe.blend":No such file or directory

Then I tried via desktop as mediator, same issue, then I tried sending a file to data1, which did work, it can see and read index.html and main.css, I used web hosting manager v0.5.0 with the standard layout.

safe://files.ism also works fine on the browser.

edit: Oh got an error trying to resave index.html from gedit, SAFE-Drive then didn’t respond:
(node:10536) UnhandledPromiseRejectionWarning: TypeError: this._safeVfs.vfsCache(...)._saveResultToCache is not a function at RootHandler.create (/home/isntism/Desktop/safe/safenetwork-fuse/src/safe-vfs/root.js:378:34) at <anonymous (node:10536) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 34) (node:10536) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.


#170

Last things first, yes I’m afraid gedit doesn’t work. vim seems OK, so some apps will be fine, but in general to get things done it is best to edit elsewhere, and then copy using the file manager or the command line, as they seem fairly reliable.

I’m not clear on what you did where you say ‘transferring file’, ‘tried via desktop’, ‘sending a file’ etc, and I would need to understand exactly what you did to help here, or to investigate the error. Looks like you are finding some limitations and bugs, but I need more detail to be sure. This is for future reference though, not essential at this stage, so up to you whether you go back and report in more detail. It is also helpful in this form, and great to see you trying these things out. I’m getting a real buzz from that :slight_smile:

It sounds like you may have got there in the end though?

Or if not, if you manually copy files to /home/isntism/SAFE/_public/ism/root-files instead of saving or editing them in place, you should be fine, and anyone will be able to access those files using their SAFE Drive via ~/SAFE/_webMount/files.ism, including yourself :slight_smile:

:clap: :clap:


#171

Ah right, I keep filtering things out that seem unimportant to prevent myself from writing a huge uncohesive monologue, I’ll try to stop doing that if it can be more helpful then annoying.
First things first though, I’m approaching this using the graphical interface instead of the console like any general user would.

After creating the website and the ism public name I attempted to move a .blend file to it by selecting it in nautilus and using copy, then paste in the root-files folder.

At first it froze and an icon-less file appeared as expected, but shortly after it jumped to 100% that error popped up and the file disappeared, reloading the folder did not fix it.

I then went ahead and assumed that it was a bug in the drive software so I dragged the file from the same nautilus page to the desktop, and then copy-pasted it using the keyboard shortcuts from there into the root-files folder, but the same thing happened.

Quickly I tested whether the git pull I did earlier broke the entire drive and dragged a file from one nautilus page, a folder on my desktop, to data1, the folder that worked before, and it got written there without a hitch, well apart from a small and expected freeze.

Now you responded and told me to to try using the console, after looking around the web for the right command to use I attempted using cp -i blender_on_safe.blend ~/SAFE/_public/ism/root-files/ from the data1 folder and it hung for a bit before allowing me to write down a new command, but when I refreshed the root-files folder it still did not show up.

These problems are not existent in the folder created by SAFE-drive, data1


#172

Prefect, that’s what I need, and great that you are using the graphical stuff which as you say, most people will.

I have tested basic copy type stuff with the file manager (not sure if it is nautilus - I’m using whatever the default is in Ubuntu Gnome desktop). I have not tested it much though, so really good that you are. By all means continue.

I suspect those little ‘hangs’ are where it is writing to the SAFE Network. So nothing to worry about so long as they complete and do what you expect, and of course they will take longer for bigger files, or when copying several files at once.

How big is that ‘.blend’ file?

I’m going to do some tricky coding, so will revise these in more detail once that’s done.

Thanks for the detailed report.

Edit:* a couple of questions:

  • which browser version are you using?

  • when you first tried to copy to ‘root-files’, did the browser pop up and ask you to authorise that? If not, that might be the issue, although I would expect that to hang indefinitely (until you clicked to say ‘yes’ it’s OK).


#173

The blend file is just 662KB, but my connection is fairly bad so I expected the hanging.
I’m using peruse-v0.7.0-linux-x64 and nautilus 1:3.26.4-0~ubuntu18.04.2, in case you meant file browser.

And I do not remember having to specifically click an authorize prompt from peruse when first trying to move files to that folder, but I may have just forgotten, either way, it doesn’t show up again.
Unless you mean the one that is required to actually get access to the files at all.

Interesting, I just tried again and it moved the .blend file into a new folder named es in its original directory, removing the original file, seems like there are some inconsistent issues.

edit, I also start the drive by going into the safenetwork-fuse folder and typing ./bin.js


#174

Thanks for the info.

It will only happen once each time you create a new web folder with Web Hosting Manager.

No, its a separate thing.

Yes indeed. Things that might seem similar from the user’s view can be very different in the software. This is why it can be hard to replicate issues, and why it helps to know the precise sequence of what was done.

Are you saying that moving the file this way worked - that is, from a folder on SAFE drive to a subfolder of the same folder?

Using ./bin.js won’t enable debug output, so if you want to see more detail of what is going on, use the DEBUG environment variable like this (although the specifics can be varied, this just happens to be what I’m using right now):

DEBUG=safe-fuse:ops*,safe-fuse:stubs*,safe-fuse:vfs-cache*,safe-vfs:*,safenetworkjs:containers*,safenetworkjs:cache*,safenetworkjs:file* node bin.js

#175

safenetwork-fuse-v0.1.0-rc2-linux-x64 <-- is that the latest?


#176

I tried moving the file from ~/SAFE/_public/test/data1 to ~/SAFE/_public/ism/root-files by dragging it over via two nautilus windows, just to see the error pop up again, but somehow the file ended up in an previously nonexistent directory at ~/SAFE/_public/test/data1/es with the original gone.

I’ll try running it with the debug environment for a while, that would be more useful.


#177

No afraid not and there isn’t a more up to date build than that - we’re all working using node to run from source at the moment. It’s not that hard, esp. on Ubuntu as that’s so standard, but some understandably prefer to wait for a build (which is fine, because that will need testing too :slight_smile: ).

So for anyone wanting to try before then, see this post.

@isntism oh right, that’s an interesting bug thinking about it (as if you didn’t know :slight_smile:). It will probably need info from the debug log to see exactly how nautilus tried to do this. You can capture that by adding the following to the end of the line when you start SAFE Drive: 2>&1 ¦ tee nautilus-move-file.log

Oh, and you will need the command to include the DEBUG= stuff I mentioned in a recent post too - that will make sure the info I need is in the log.

If you do this, and you’re OK with it, please create an issue on github, describe the sequence, and upload the nautilu-move-file.log there. Also note your OS version so the info is all in one place - or include a link to the post in this topic. That would be super helpful.

Thanks all. I’ve finished writing some new quite complicated code that I hope will get us closer to decentralised git but it has introduced bugs I’ll be trying to fix today. In some ways these changes make the code nicer too, because when I rewrite stuff I’ve written once and then debugged, I have a better understanding of what I’m are trying to do. :smile:


#178

Seems I can read files OK , make new dirs but cannot create or copy files?

willie@gagarin:~$ cd SAFE/
willie@gagarin:~/SAFE$ ls -al
total 0
drwxr-xr-x 1 willie willie 0 Nov 25 17:44 _public
drwxr-xr-x 1 willie willie 0 Nov 25 17:44 _publicNames
willie@gagarin:~/SAFE$ cd _public
willie@gagarin:~/SAFE/_public$ ls -al
total 0
drwxr-xr-x 1 willie willie 0 Nov 25 17:45 scotcoin
drwxr-xr-x 1 willie willie 0 Nov 25 17:45 southside
drwxr-xr-x 1 willie willie 0 Nov 25 17:45 tests
willie@gagarin:~/SAFE/_public$ mkdir newdir
willie@gagarin:~/SAFE/_public$ cd newdir/
willie@gagarin:~/SAFE/_public/newdir$ touch newfile1 newfile2
touch: cannot touch 'newfile1': Remote I/O error
touch: cannot touch 'newfile2': Remote I/O error
willie@gagarin:~/SAFE/_public/newdir$ ls -al
total 0
willie@gagarin:~/SAFE/_public/newdir$ cd ..
willie@gagarin:~/SAFE/_public$ cp -rv ~/projects/maidsafe/example-sample_sites/nostrils/ .
'/home/willie/projects/maidsafe/example-sample_sites/nostrils/' -> './nostrils'
'/home/willie/projects/maidsafe/example-sample_sites/nostrils/css' -> './nostrils/css'
'/home/willie/projects/maidsafe/example-sample_sites/nostrils/css/styles.css' -> './nostrils/css/styles.css'
cp: cannot create regular file './nostrils/css/styles.css': Remote I/O error
'/home/willie/projects/maidsafe/example-sample_sites/nostrils/index.html' -> './nostrils/index.html'
cp: cannot create regular file './nostrils/index.html': Remote I/O error
'/home/willie/projects/maidsafe/example-sample_sites/nostrils/wom-tut1.html' -> './nostrils/wom-tut1.html'
cp: cannot create regular file './nostrils/wom-tut1.html': Remote I/O error
'/home/willie/projects/maidsafe/example-sample_sites/nostrils/img' -> './nostrils/img'
'/home/willie/projects/maidsafe/example-sample_sites/nostrils/img/nostril.jpg' -> './nostrils/img/nostril.jpg'
cp: cannot create regular file './nostrils/img/nostril.jpg': Remote I/O error

Ubuntu 18.04 all dev versions as recommended

debug.txt is chunky…
willie@gagarin:~/projects/maidsafe/safenetwork-fuse$ ls -l debug.txt
-rw-r–r-- 1 willie willie 5201346 Nov 25 20:33 debug.txt

Do you want me to zip that and share it?


#179

Hey Willie, well done. That’s fantastic, and the behaviour is expected.

Everyone is going to find this confusing, so I think we will need to figure out a better way of handling this, but for now see JPL’s post below. Essentially he’s having the same issue, and my explanation in reply is immediately underneath:

TLDR; you can’t create files just anywhere, but a folder where you have already uploaded files using Web Host Manager, and which appears under _public will work. If it doesn’t, try the next folder deep until you find the folder that is the container. It will be the one with index.html in the case of an uploaded website.

I think Maidsafe might like to rethink having subfolders above that, and instead make every subfolder in _public be the start of a container. I think that will solve a few UX issues like the one you are experiencing. But it needs discussing.

Assuming you get the hang of that, have a play, and report back. Good luck :slight_smile:


#180

I stopped by step 5. Mount Your SAFE Drive, a) Assuming you are in the SAFE FUSE directory type: ./mount-safe

tomas@tomas-Lenovo-IdeaPad-Z510://home/tomas/Plocha/safenetwork-fuse-v0.1.0-linux-x64$ ./mount-safe
Server path not specified, so defaulting to ipc.config.socketRoot + ipc.config.appspace + ipc.config.id /tmp/app.6008
starting server on /tmp/app.6008
starting TLS server false
starting server as Unix || Windows Socket

socket connection to server detected

received event of : auth-uri { id: ‘6008-cli’,
message: ‘safe-c2fmzw5ldhdvcmstznvzzq:AQAAAAU0LTcAAAAAAAAAACAAAAAAAAAAsEMEn_RWABXicc7kzn7ggMcqGeDfmXCfyJhAskZSlssgAAAAAAAAABFiJSGfa5qM5G515ARlxMdkfQRbO6Cw0AAE9b9loUlJIAAAAAAAAAAqtxf_U54oXTuHNiLNUcUM_wuSO-Y-ROz5Ysky_Q1x6EAAAAAAAAAAaymR3vU_Z-YT-MEmBOESFRqrrgtF5ZA9SJtitQU4nM0qtxf_U54oXTuHNiLNUcUM_wuSO-Y-ROz5Ysky_Q1x6CAAAAAAAAAAHYDr_WtQsoLax31lwA-9TP1jYRzmmQTA4awJdOWb4y0gAAAAAAAAAPNYxYdJ9L1TnyGO-EfL2Bh4bqpbqduqLJWLLU2HfUj2GQAAAAAAAAAQAAAAAAAAADE3OC42Mi43Ni44OjU0ODMTAAAAAAAAADEzOC42OC4xODUuMjE4OjU0ODMSAAAAAAAAADEzOC42OC4xODEuNTc6NTQ4MxIAAAAAAAAAMTM4LjY4LjE4MS42MDo1NDgzEgAAAAAAAAAxMzguNjguMTgxLjg2OjU0ODMSAAAAAAAAADEzOC42OC4xODEuODc6NTQ4MxMAAAAAAAAAMTM4LjY4LjE4MS4xNjg6NTQ4MxMAAAAAAAAAMTM4LjY4LjE4MS4xNzY6NTQ4MxMAAAAAAAAAMTM4LjY4LjE4MS4xNzk6NTQ4MxMAAAAAAAAAMTM4LjY4LjE4MS4xODA6NTQ4MxMAAAAAAAAAMTM4LjY4LjE4MS4xODI6NTQ4MxMAAAAAAAAAMTM4LjY4LjE4MS4yNDI6NTQ4MxMAAAAAAAAAMTM4LjY4LjE4MS4yNDM6NTQ4MxMAAAAAAAAAMTM4LjY4LjE4MS4yNDk6NTQ4MxIAAAAAAAAAMTM4LjY4LjE4OS4xNDo1NDgzEgAAAAAAAAAxMzguNjguMTg5LjE1OjU0ODMSAAAAAAAAADEzOC42OC4xODkuMTc6NTQ4MxIAAAAAAAAAMTM4LjY4LjE4OS4xODo1NDgzEgAAAAAAAAAxMzguNjguMTg5LjE5OjU0ODMSAAAAAAAAADEzOC42OC4xODkuMzE6NTQ4MxIAAAAAAAAAMTM4LjY4LjE4OS4zNDo1NDgzEgAAAAAAAAAxMzguNjguMTg5LjM2OjU0ODMSAAAAAAAAADEzOC42OC4xODkuMzg6NTQ4MxIAAAAAAAAAMTM4LjY4LjE4OS4zOTo1NDgzEQAAAAAAAAA0Ni4xMDEuNS4xNzk6NTQ4MwFrFQAAAAAAAQcAAAAAAAAAYWxwaGFfMgDVlh0-jGsab_NUDY8IiCJu-VwEjonrNybrrbBGegWyLZg6AAAAAAAAGAAAAAAAAAAjzchaYyHotk3TTUw8SOJs7OOk3cw6JvgCAAAAAAAAAAwAAAAAAAAAX3B1YmxpY05hbWVz4EtvBO4G4ERE9hbpM7q6HtV6lbRkAFDZdEl8Rn2uBAWYOgAAAAAAAAEgAAAAAAAAANyYETTugClfKVFd4Kk0HjG1hVfhRxGsTBWY-qCsXDZ8GAAAAAAAAAAQxUAIZVMFvFkzOBdFuf4Myz-8QYcecWkABAAAAAAAAAAAAAAAAQAAAAIAAAADAAAABwAAAAAAAABfcHVibGljCe455GSxmG-3Hfk9I41zuBAxEIo4zAMTjaXFBU_Wv22YOgAAAAAAAAAABAAAAAAAAAAAAAAAAQAAAAIAAAADAAAA’ }
socket disconnected 6008-cli
(node:6008) UnhandledPromiseRejectionWarning: TypeError: Cannot read property ‘gt’ of undefined
at module.exports.statfs.safeVfs.getHandler.statfs.then (/snapshot/safenetwork-fuse-v0.1.0/src/fuse-operations/statfs.js:0:0)
at
(node:6008) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 11)
(node:6008) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I think, its may be because some errors by running of Peruse or Safe Browser:

tomas@tomas-Lenovo-IdeaPad-Z510:~/Plocha/Peruse-v0.7.0-linux-x64$ ./peruse
I 18-12-03 19:35:18.263956 Failed to Bootstrap with 138.68.181.87:5483: (ClientNotWhitelisted) Our Client is not whitelisted
I 18-12-03 19:35:19.742115 Failed to Bootstrap with 138.68.189.38:5483: (ClientNotWhitelisted) Our Client is not whitelisted
I 18-12-03 19:35:19.744016 Failed to Bootstrap with 138.68.189.39:5483: (ClientNotWhitelisted) Our Client is not whitelisted
I 18-12-03 19:35:28.344458 Failed to Bootstrap with 138.68.189.36:5483: (ClientNotWhitelisted) Our Client is not whitelisted
I 18-12-03 19:35:28.389736 Failed to Bootstrap with 138.68.181.168:5483: (ClientNotWhitelisted) Our Client is not whitelisted
I 18-12-03 19:35:28.391973 Failed to Bootstrap with 138.68.185.218:5483: (ClientNotWhitelisted) Our Client is not whitelisted
I 18-12-03 19:35:28.392487 Failed to Bootstrap with 138.68.189.38:5483: (ClientNotWhitelisted) Our Client is not whitelisted
I 18-12-03 19:35:28.441990 Bootstrapping(ad1cfc…) Lost connection to proxy PublicId(name: 7b6ee7…).

In http://invite.maidsafe.net is registerd my current IP adress and I can browse the SAFE Network. The step b) authorise SAFE FUSE to access my files, work well. I see, that SAFE Drive is mounted. But after ls ~/SAFE is nothing happened (loading process).

Thank you for your help
edit: Ubuntu 18.04.1


#181

You are running from the compiled version right? that one is very out of date, luckily it’s quite easy to get the development version, and last I checked that one does work fine, you should be able to piece together how to do so from recentish posts.

I think it’s around this post: SAFE Drive - help with testing?