Java API support on desktop platforms

java
api
#4

Mission: Have flu; kill bugs.

Below, then works on Linux; a few typos fixed and after a struggle with the unfamiliar becomes a bit forced and saw the sdk command line tools installed, where perhaps there is still a hack to avoid needing that:


Check java version > 7
$ java -version
java version “1.8.0_191”

sudo apt-get install gradle

hiccup tempts then faking the license =>

However, I still got a run of “Warning: License for package Android SDK Platform ## not accepted.”
and a suggested solution like “sudo sdkmanager --licenses”, wasn’t simple to action.

Download Android Studio Command line tools only from
https://developer.android.com/studio/

unzip

cd ./tools/bin
./sdkmanager --licenses

answering yes to those.
is a route to accepting licenses but it had no effect.

So, I guessed the unzip wanted to be in ~/.android
moving the unzipped files and merge/copy over licenses to be within ~/.android

and trying the

$ ./gradlew safe-app:download-nativelibs

is successful.
as is then

$ ./gradlew :safe-app:pack


and then from https://github.com/maidsafe/safe_app_java - if you include the “./”

./gradlew :safe-app-android:download-nativelibs
./gradlew :safe-app-android:build
./gradlew :safe-app-android:javadoc

also work, though with a head full of glu, I have no idea what they do just yet :smiley:

1 Like
#5

You can run the tasks using gradle or gradlew (gradlew.bat on windows).
gradlew is a gradle wrapper that will help you run gradle tasks without installing gradle. StackOverflow has a good comparison.

This was because the ANDROID_HOME environment variable was set to ~/.android. So moving the CLI tools to that directory did the trick :slight_smile:

Indeed there is. The safe_app_java project is a multi module gradle project for both desktop (safe-app) and android (safe-app-android)
If you want to build the libraries only for desktop without downloading the Android SDKs you can remove :safe-app-android from the settings.gradle file and then build the desktop JARs.

For some clarity on what the gradle tasks do:

The safe_app_java project uses native code that is written in Rust. This native code is compiled for the supported platforms in their respective formats. The download-nativelibs task downloads these native libraries into the project directories.

The native libraries and the required Java classes can be packaged into JARs and AARs for desktop and mobile respectively. The safe-app:pack task will build the required JARs for desktop and the safe-app-android:build task will build the AARs for mobile.

The API documentation for the Java API is available at docs.maidsafe.net/safe_app_java. However if you prefer to build it locally the safe-app-android:javadoc task will generate the documentation into the safe-app-android/build/docs/javadoc directory.

2 Likes
#6

Ok, my son was sick again, so I got another day to play with this! ha!

So, I’m using createTestAppWithAccess() now and it seems to be getting further. However, there are a few things that aren’t as I’m expecting.

I’m attempting to use the NFS library to create a file and then read back from it. Using either mock or non-mock native libraries, this doesn’t seem to work for me - it just throws an exception when I try to read the data.

I have setup the app using the following basic settings:

AppExchangeInfo appExchangeInfo = new AppExchangeInfo("id", "scope", "name", "vendor");

ContainerPermissions[] containerPermissionsArray = new ContainerPermissions[1];
ContainerPermissions containerPermissions = new ContainerPermissions("_public", ALL_PERMISSIONS);
containerPermissionsArray[0] = containerPermissions;

AuthReq authReq = new AuthReq(appExchangeInfo, true, containerPermissionsArray, 1, 1);

Session session = Session.createTestAppWithAccess(authReq).get();
session.setOnDisconnectListener(o -> LOGGER.debug("Disconnected from network"));

Then some simplified code to read/write:

MDataInfo containerMDI = session.getContainerMDataInfo("apps/id").get();
File newFile = new File();
session.nfs.insertFile(containerMDI, path.getFileName().toString(), newFile).get();

NativeHandle writeHandle = session.nfs.fileOpen(containerMDI, newFile, NFS.OpenMode.OVER_WRITE).get();
session.nfs.fileWrite(writeHandle, byteBuffer.array()).get();
session.nfs.fileClose(writeHandle).get();

NativeHandle readHandle = session.nfs.fileOpen(containerMDI, newFile, NFS.OpenMode.READ).get();
session.nfs.fileRead(readHandle, 0, bufferLength).get();
session.nfs.fileClose(readHandle).get();

The exception message I get is:

java.util.concurrent.ExecutionException: java.lang.Exception: Core error: Routing client error -> Requested data not found : -103

From debugging, it looks like newFile is never populated with anything, when I was assuming it would be. Moreover, I’m wondering how I can open a file from just a file name (which I suspect may require MDataEntries manipulation of some sort).

Finally, I can’t see any temporary files being stored in my tmp directory, which I was expecting to see. I’m wondering if the two could be related?

I’ll keep playing with it and I’m pleased to have ported some of my old REST client code to the new native library. It would be great to get something writing/reading though!

#7

hi, @Traktion. Hope your son recovers soon.

When you are inserting the file, it is empty. And then when you are writing the file, the file doesn’t get updated. So you need to write the file, close it and use the returned file object to insert the file.
I’ll show you an example.

MDataInfo mDataInfo = session.getContainerMDataInfo("apps/id").get();
File file = new File();
NativeHandle fileHandle = session.nfs.fileOpen(mDataInfo, file, NFS.OpenMode.OVER_WRITE).get();
byte[] fileContent = Helper.randomAlphaNumeric(LENGTH).getBytes();
session.nfs.fileWrite(fileHandle, fileContent).get();
file = session.nfs.fileClose(fileHandle).get();
session.nfs.insertFile(mDataInfo, "sample.txt", file);

On doing this, we inserting a file that is already populated.

For reading the file, we open it and store the contents in byte array.

fileHandle = session.nfs.fileOpen(mDataInfo, file, NFS.OpenMode.READ).get();
byte[] readData = session.nfs.fileRead(fileHandle, 0, 0).get();

:heart::heart:

Thanks for asking and we hope to help you more.

4 Likes
#8

Thanks for the examples! It would be great to have them at the top of the NFS javadoc, as it isn’t obvious how to sequence these steps.

Maybe my brain just read it backwards, but it felt like I needed to insert the file before I could write to it! :slight_smile:

If I don’t have the object for the inserted file, is there an easy way to retrieve it based on file name? Do I use nfs.updateFile or some such?

2 Likes
#9

This should do what you’re looking for :slight_smile:

NFSFileMetadata metadata = session.nfs.getFileMetadata(containerMDI, "filename").get();
NativeHandle readHandle = session.nfs.fileOpen(containerMDI, metadata, NFS.OpenMode.READ).get();
byte[] content = session.nfs.fileRead(readHandle, 0, session.nfs.getSize(readHandle).get()).get();
System.out.println(new String(content));
session.nfs.fileClose(readHandle).get();

This is something we have in mind :slight_smile: We limited the javadoc to the API definitions for initial release. We’ll definitely add examples in the versions to come! And of course, pull requests welcome :wink: :wink:

2 Likes
#10

Thanks! I should have some good examples in github to link to soon too, with any luck! :slight_smile:

2 Likes
#11

Ok, tried that out and got read/write working well!

The next challenge is directories… any pointers as to how to create/delete directories?

#12

Directories don’t exist as entities - they are only ‘implied’ and there is no API in SAFE that understands a directory. So it is just an application convention that ‘/’ characters in the key under which file is stored are interpreted as directory separators.

Hope that makes sense, it feels hard to put into words!

3 Likes
#13

Ah, right - so if I can have a key which is “/directory1/directory2/file.txt”?

EDIT: Or is it more like having 3 entries like:

/directory1/
/directory1/directory2/
/directory1/directory2/file.txt

The old REST API did have a concept of directories, so I’m guessing that was just an extra layer of abstraction. Interesting!

#14

The first form is correct. There is no way (convention) at the moment to have an empty directory.

1 Like
#15

Yes. Having the entire path as a key would be one way to go about it.

Another way could be to have the folder name as the key and the value could be serialized MDataInfo of a different mutable data. This other MData can hold all the files inside that directory :slight_smile:
So for instance,

In the app’s own container(App’s root folder):

Key Value
file1 dataMap
file2 dataMap
folder1 serializedMdInfo(name: folder1, typetag: 15234)
file3 dataMap

And in mutable data with name: folder1 and typetag: 15234

Key Value
file4 dataMap
file5 dataMap

You can even have an empty folder, which will be a mutable data without any entries. You can add files(entries) later, whenever required :slight_smile:

P.S.: This just an approach, there can be multiple other possibilities too

4 Likes
#16

Is there a convention for an approach? It sounds like one is an object store (entire path as key) and the other is a more traditional file/container hierarchy file system.

#17

I don’t believe there is a convention as such. Both approaches have their own pros and cons. However, I believe that the WHM and the browser go with the latter to upload/fetch website files.

1 Like
#18

This is news to me! I’ve even discussed on here (and I think raised an issue) about the inability to have an empty folder in SAFE NFS, and this wasn’t mentioned so I don’t think it is well understood, and I’ve not seen it documented anywhere.

If like to have this specified so we can build on the convention. It has implications for everyone, so it would be very helpful to have SAFE NFS specified down to this level.

It will make it easier to develop, and help achieve cross app compatibility if there is a clear definition of what is expected of apps that want to inter operate.

2 Likes
#19

So the points I raised above don’t get forgotten I’ve opened an issue requesting clarifying documentation wrt to this and other aspects of SAFE NFS:

4 Likes
#20

Sorry this took a while.

Over the weekend I played around with the NodeJS API to implement this approach for empty folders in NFS.

Here’s some brief context about the NFS implementation:

An NFS container is a mutable data that stores a list of entries where the key is the file name and the value is the location(dataMap) of a file which is stored as immutable data.

Since this container is a mutable data, we can perform general MData operations on it too. So, for an empty folder, we create a new mutable data instance, serialize it and store it as an entry with the folder name as key and the serialized instance as the value.

We can code the application to handle files and folders differently :slight_smile:
Here’s a snippet using the NodeJS API:

// Initialize mutable data
  const typeTag = 15000;
  md = await safeApp.mutableData.newRandomPublic(typeTag);  
  await md.quickSetup({});

  // Emulate root folder as NFS and insert a file
  const nfs = await md.emulateAs('NFS');
  const content = '<html><body><h1>WebSite</h1></body></html>';
  const fileName = 'page1.html';
  const fileContext = await nfs.create(content);
  await nfs.insert(fileName, fileContext);

  // Initialize new mutable data. This will be an empty sub-folder
  let newMd = await safeApp.mutableData.newRandomPublic(typeTag);
  await newMd.quickSetup({});
  
  // Serialize the sub-folder info and store it in the root folder
  const folderName = "page2";
  const serialisedMD = await newMd.serialise();
  const mutation = await safeApp.mutableData.newMutation();
  await mutation.insert(folderName, serialisedMD);
  await md.applyEntriesMutation(mutation);
  
  // Fetch the sub-folder info and deserialize it to get the md object
  const entries = await md.getEntries();
  const value = await entries.get('page2');
  const emptyFolderMd = await safeApp.mutableData.fromSerial(value.buf);

  // Emulate the folder as NFS and add a file
  const cssFolder = await emptyFolderMd.emulateAs('NFS');
  const newFile = await nfs.create(content);
  await cssFolder.insert('page2.html', newFile);
  
  // Print the XOR URLs to view them in the MData viewer
  const rootInfo = await md.getNameAndTag();
  console.log("Root folder: " + rootInfo.xorUrl);
  const cssFolderInfo = await newMd.getNameAndTag();
  console.log("CSS folder info: " + cssFolderInfo.xorUrl);

I hope this helps!

8 Likes
NFS - i guess we will learn with the roadmap how the plan is here ..?
#21

Thanks @lionel.faber it is indeed helpful. I’ve always known that it was feasible to have subcontainers but have never seen it suggested in documentation, discussion or indeed in code before (though that might well exist). Can you point to an RFC or documentation for this that maybe I’ve missed, or say whether it is specified in some private design doc? I’m curious why it had been so hard to discover! :slight_smile:

I hadn’t thought of using it as a way to implement an empty folder, but I can see that this is in effect what we get on first creating a Web service - I.e. we get an entry in _public that points to an empty NFS container.

Things like this could do with being documented along with design guidance, as it is figuring out how to do things like this properly that takes devs a lot of time, and can lead to incompatible implementations.

Thanks very much for your help. I’m inclined not to act on it until it is clear in the API docs, and I’m sure things aren’t likely to change (eg in view of work in API Appendable Data and RDF).

7 Likes
#22

Exactly!
Although the WHM, browser etc. uses this structure to store sub-folders I’m afraid that there is no documentation for the current implementation. We are in the early stages of a documentation roadmap. We’ll be adding this information (and much more :smiley: ) to project wikis.

5 Likes
#23

A heads-up on a few updates made to the safe_app_java project:

  • The Android SDK is no longer downloaded by default. So building the libs only for desktop is less of a pain.
  • The safe-app module now uses the gradle-java-flavours module to specify mock and non-mock flavours for the desktop build too! Thank you again @ogrebgr!
  • The safe-authenticator module has been removed and the Authenticator class has moved to the mock flavour of the desktop project.
  • The project now builds only 2 jars (mock and nonMock) that contain the native libs on all platforms. This comes with a cost of a larger jar size, but it seems to be the standard.

The OP with the build instructions and the examples have been updated :slight_smile:

6 Likes