Java API support on desktop platforms

lionel.faber · January 29, 2019, 3:19pm

You can run the tasks using gradle or gradlew (gradlew.bat on windows).
gradlew is a gradle wrapper that will help you run gradle tasks without installing gradle. StackOverflow has a good comparison.

This was because the ANDROID_HOME environment variable was set to ~/.android. So moving the CLI tools to that directory did the trick

Indeed there is. The safe_app_java project is a multi module gradle project for both desktop (safe-app) and android (safe-app-android)
If you want to build the libraries only for desktop without downloading the Android SDKs you can remove :safe-app-android from the settings.gradle file and then build the desktop JARs.

For some clarity on what the gradle tasks do:

The safe_app_java project uses native code that is written in Rust. This native code is compiled for the supported platforms in their respective formats. The download-nativelibs task downloads these native libraries into the project directories.

The native libraries and the required Java classes can be packaged into JARs and AARs for desktop and mobile respectively. The safe-app:pack task will build the required JARs for desktop and the safe-app-android:build task will build the AARs for mobile.

The API documentation for the Java API is available at docs.maidsafe.net/safe_app_java. However if you prefer to build it locally the safe-app-android:javadoc task will generate the documentation into the safe-app-android/build/docs/javadoc directory.

Traktion · February 14, 2019, 6:06pm

Ok, my son was sick again, so I got another day to play with this! ha!

So, I’m using createTestAppWithAccess() now and it seems to be getting further. However, there are a few things that aren’t as I’m expecting.

I’m attempting to use the NFS library to create a file and then read back from it. Using either mock or non-mock native libraries, this doesn’t seem to work for me - it just throws an exception when I try to read the data.

I have setup the app using the following basic settings:

AppExchangeInfo appExchangeInfo = new AppExchangeInfo("id", "scope", "name", "vendor");

ContainerPermissions[] containerPermissionsArray = new ContainerPermissions[1];
ContainerPermissions containerPermissions = new ContainerPermissions("_public", ALL_PERMISSIONS);
containerPermissionsArray[0] = containerPermissions;

AuthReq authReq = new AuthReq(appExchangeInfo, true, containerPermissionsArray, 1, 1);

Session session = Session.createTestAppWithAccess(authReq).get();
session.setOnDisconnectListener(o -> LOGGER.debug("Disconnected from network"));

Then some simplified code to read/write:

MDataInfo containerMDI = session.getContainerMDataInfo("apps/id").get();
File newFile = new File();
session.nfs.insertFile(containerMDI, path.getFileName().toString(), newFile).get();

NativeHandle writeHandle = session.nfs.fileOpen(containerMDI, newFile, NFS.OpenMode.OVER_WRITE).get();
session.nfs.fileWrite(writeHandle, byteBuffer.array()).get();
session.nfs.fileClose(writeHandle).get();

NativeHandle readHandle = session.nfs.fileOpen(containerMDI, newFile, NFS.OpenMode.READ).get();
session.nfs.fileRead(readHandle, 0, bufferLength).get();
session.nfs.fileClose(readHandle).get();

The exception message I get is:

java.util.concurrent.ExecutionException: java.lang.Exception: Core error: Routing client error -> Requested data not found : -103

From debugging, it looks like newFile is never populated with anything, when I was assuming it would be. Moreover, I’m wondering how I can open a file from just a file name (which I suspect may require MDataEntries manipulation of some sort).

Finally, I can’t see any temporary files being stored in my tmp directory, which I was expecting to see. I’m wondering if the two could be related?

I’ll keep playing with it and I’m pleased to have ported some of my old REST client code to the new native library. It would be great to get something writing/reading though!

vigneshwara · February 15, 2019, 7:19am

hi, @Traktion. Hope your son recovers soon.

When you are inserting the file, it is empty. And then when you are writing the file, the file doesn’t get updated. So you need to write the file, close it and use the returned file object to insert the file.
I’ll show you an example.

MDataInfo mDataInfo = session.getContainerMDataInfo("apps/id").get();
File file = new File();
NativeHandle fileHandle = session.nfs.fileOpen(mDataInfo, file, NFS.OpenMode.OVER_WRITE).get();
byte[] fileContent = Helper.randomAlphaNumeric(LENGTH).getBytes();
session.nfs.fileWrite(fileHandle, fileContent).get();
file = session.nfs.fileClose(fileHandle).get();
session.nfs.insertFile(mDataInfo, "sample.txt", file);

On doing this, we inserting a file that is already populated.

For reading the file, we open it and store the contents in byte array.

fileHandle = session.nfs.fileOpen(mDataInfo, file, NFS.OpenMode.READ).get();
byte[] readData = session.nfs.fileRead(fileHandle, 0, 0).get();

Thanks for asking and we hope to help you more.

Traktion · February 15, 2019, 11:50am

Thanks for the examples! It would be great to have them at the top of the NFS javadoc, as it isn’t obvious how to sequence these steps.

Maybe my brain just read it backwards, but it felt like I needed to insert the file before I could write to it!

If I don’t have the object for the inserted file, is there an easy way to retrieve it based on file name? Do I use nfs.updateFile or some such?

lionel.faber · February 15, 2019, 1:10pm

This should do what you’re looking for

NFSFileMetadata metadata = session.nfs.getFileMetadata(containerMDI, "filename").get();
NativeHandle readHandle = session.nfs.fileOpen(containerMDI, metadata, NFS.OpenMode.READ).get();
byte[] content = session.nfs.fileRead(readHandle, 0, session.nfs.getSize(readHandle).get()).get();
System.out.println(new String(content));
session.nfs.fileClose(readHandle).get();

This is something we have in mind We limited the javadoc to the API definitions for initial release. We’ll definitely add examples in the versions to come! And of course, pull requests welcome

Traktion · February 15, 2019, 2:22pm

Thanks! I should have some good examples in github to link to soon too, with any luck!

Traktion · February 15, 2019, 5:30pm

Ok, tried that out and got read/write working well!

The next challenge is directories… any pointers as to how to create/delete directories?

happybeing · February 15, 2019, 5:38pm

Directories don’t exist as entities - they are only ‘implied’ and there is no API in SAFE that understands a directory. So it is just an application convention that ‘/’ characters in the key under which file is stored are interpreted as directory separators.

Hope that makes sense, it feels hard to put into words!

Traktion · February 15, 2019, 5:42pm

Ah, right - so if I can have a key which is “/directory1/directory2/file.txt”?

EDIT: Or is it more like having 3 entries like:

/directory1/
/directory1/directory2/
/directory1/directory2/file.txt

The old REST API did have a concept of directories, so I’m guessing that was just an extra layer of abstraction. Interesting!

happybeing · February 15, 2019, 6:20pm

The first form is correct. There is no way (convention) at the moment to have an empty directory.

lionel.faber · February 15, 2019, 6:35pm

Yes. Having the entire path as a key would be one way to go about it.

Another way could be to have the folder name as the key and the value could be serialized MDataInfo of a different mutable data. This other MData can hold all the files inside that directory
So for instance,

In the app’s own container(App’s root folder):

Key	Value
file1	dataMap
file2	dataMap
folder1	serializedMdInfo(name: folder1, typetag: 15234)
file3	dataMap

And in mutable data with name: folder1 and typetag: 15234

Key	Value
file4	dataMap
file5	dataMap

You can even have an empty folder, which will be a mutable data without any entries. You can add files(entries) later, whenever required

P.S.: This just an approach, there can be multiple other possibilities too

Traktion · February 15, 2019, 6:41pm

Is there a convention for an approach? It sounds like one is an object store (entire path as key) and the other is a more traditional file/container hierarchy file system.

lionel.faber · February 15, 2019, 6:46pm

I don’t believe there is a convention as such. Both approaches have their own pros and cons. However, I believe that the WHM and the browser go with the latter to upload/fetch website files.

happybeing · February 15, 2019, 7:08pm

This is news to me! I’ve even discussed on here (and I think raised an issue) about the inability to have an empty folder in SAFE NFS, and this wasn’t mentioned so I don’t think it is well understood, and I’ve not seen it documented anywhere.

If like to have this specified so we can build on the convention. It has implications for everyone, so it would be very helpful to have SAFE NFS specified down to this level.

It will make it easier to develop, and help achieve cross app compatibility if there is a clear definition of what is expected of apps that want to inter operate.

happybeing · February 23, 2019, 1:03pm

So the points I raised above don’t get forgotten I’ve opened an issue requesting clarifying documentation wrt to this and other aspects of SAFE NFS:

lionel.faber · February 25, 2019, 9:13am

Sorry this took a while.

Over the weekend I played around with the NodeJS API to implement this approach for empty folders in NFS.

Here’s some brief context about the NFS implementation:

An NFS container is a mutable data that stores a list of entries where the key is the file name and the value is the location(dataMap) of a file which is stored as immutable data.

Since this container is a mutable data, we can perform general MData operations on it too. So, for an empty folder, we create a new mutable data instance, serialize it and store it as an entry with the folder name as key and the serialized instance as the value.

We can code the application to handle files and folders differently
Here’s a snippet using the NodeJS API:

// Initialize mutable data
  const typeTag = 15000;
  md = await safeApp.mutableData.newRandomPublic(typeTag);  
  await md.quickSetup({});

  // Emulate root folder as NFS and insert a file
  const nfs = await md.emulateAs('NFS');
  const content = '<html><body><h1>WebSite</h1></body></html>';
  const fileName = 'page1.html';
  const fileContext = await nfs.create(content);
  await nfs.insert(fileName, fileContext);

  // Initialize new mutable data. This will be an empty sub-folder
  let newMd = await safeApp.mutableData.newRandomPublic(typeTag);
  await newMd.quickSetup({});
  
  // Serialize the sub-folder info and store it in the root folder
  const folderName = "page2";
  const serialisedMD = await newMd.serialise();
  const mutation = await safeApp.mutableData.newMutation();
  await mutation.insert(folderName, serialisedMD);
  await md.applyEntriesMutation(mutation);
  
  // Fetch the sub-folder info and deserialize it to get the md object
  const entries = await md.getEntries();
  const value = await entries.get('page2');
  const emptyFolderMd = await safeApp.mutableData.fromSerial(value.buf);

  // Emulate the folder as NFS and add a file
  const cssFolder = await emptyFolderMd.emulateAs('NFS');
  const newFile = await nfs.create(content);
  await cssFolder.insert('page2.html', newFile);
  
  // Print the XOR URLs to view them in the MData viewer
  const rootInfo = await md.getNameAndTag();
  console.log("Root folder: " + rootInfo.xorUrl);
  const cssFolderInfo = await newMd.getNameAndTag();
  console.log("CSS folder info: " + cssFolderInfo.xorUrl);

I hope this helps!

happybeing · February 25, 2019, 9:30am

Thanks @lionel.faber it is indeed helpful. I’ve always known that it was feasible to have subcontainers but have never seen it suggested in documentation, discussion or indeed in code before (though that might well exist). Can you point to an RFC or documentation for this that maybe I’ve missed, or say whether it is specified in some private design doc? I’m curious why it had been so hard to discover!

I hadn’t thought of using it as a way to implement an empty folder, but I can see that this is in effect what we get on first creating a Web service - I.e. we get an entry in _public that points to an empty NFS container.

Things like this could do with being documented along with design guidance, as it is figuring out how to do things like this properly that takes devs a lot of time, and can lead to incompatible implementations.

Thanks very much for your help. I’m inclined not to act on it until it is clear in the API docs, and I’m sure things aren’t likely to change (eg in view of work in API Appendable Data and RDF).

lionel.faber · February 25, 2019, 10:06am

Exactly!
Although the WHM, browser etc. uses this structure to store sub-folders I’m afraid that there is no documentation for the current implementation. We are in the early stages of a documentation roadmap. We’ll be adding this information (and much more ) to project wikis.

lionel.faber · March 22, 2019, 8:11am

A heads-up on a few updates made to the safe_app_java project:

The Android SDK is no longer downloaded by default. So building the libs only for desktop is less of a pain.
The safe-app module now uses the gradle-java-flavours module to specify mock and non-mock flavours for the desktop build too! Thank you again @ogrebgr!
The safe-authenticator module has been removed and the Authenticator class has moved to the mock flavour of the desktop project.
The project now builds only 2 jars (mock and nonMock) that contain the native libs on all platforms. This comes with a cost of a larger jar size, but it seems to be the standard.

The OP with the build instructions and the examples have been updated

happybeing · June 5, 2019, 5:22pm

I’ve been looking at the webFetch() code in safe_app_nodejs and do not believe it allows subcontainers within an NFS container, which is what I thought you were implying above. See:

github.com

maidsafe-archive/safe_app_nodejs/blob/72d68f51f6885e07a24975d62f4746e90befcd86/src/web_fetch.js#L487-L492


      
          const emulation = content.emulateAs(DATA_TYPE_NFS);
          const { file, mimeType: fileMimeType } = await
                                  tryDifferentPaths(emulation.fetch.bind(emulation), path);
          const openedFile = await emulation.open(file, consts.pubConsts.NFS_FILE_MODE_READ);
          const data = await readContentFromFile(openedFile, fileMimeType, options);
          return data;

The above code has an NFS container (emulation) and it tries to find the file in this container. There is no code to handle the case where a path matches a subcontainer.

So while you might be able to represent an empty directory with an entry containing another NFS (sub)container, you can’t use the latter to hold files - which is what I took to be your meaning. It seems daft to use an empty subcontainer to hold an empty directory unless you can also use it to hold files, because you might just as well represent the empty folder with a ‘null’ value.

I don’t know if NFS is going to be supported in future APIs, so I’m not sure it is worth digging further into this now. And I’m glad I didn’t try implementing subcontainers in SAFE Drive and SafenetworkJs!