Java API support on desktop platforms

java
api

#1

Recently, we released support for native Android development for SAFE using a new Java API and we’re excited to see applications from the community!

Java, being platform independent, will run on any platform and since the recent release targeted only Android an interesting question arises, “What about desktop?”

The Java API are indeed supported on desktop too. But however, we are looking into certain aspects of packaging (following maven standards) and some IPC requirements before official release.

Nonetheless, by following the instructions below you can get started with the Java API on desktop. Note that these are not yet finalized and are likely to change.

Building the Java libraries:

The java libs for desktop can be built from the safe_app_java repository. The desktop libraries can be built via the following commands:

git clone https://github.com/maidsafe/safe_app_java
chmod +x gradlew
gradlew safe-app:download-nativelibs

Note: The above command will throw an error in case of missing Android SDKs due to the multi-module structure of safe_app_java. If this is the case for you run the following commands:

mkdir -p ~/.android/licenses
export $ANDROID_HOME=~/.android
echo "8933bad161af4178b1185d1a37fbf41ea5269c55" > $ANDROID_HOME/licenses/android-sdk-license
echo "d56f5187479451eabf01fb78af6dfcb131a6481e" >> $ANDROID_HOME/licenses/android-sdk-license

In other words, create a folder and set it’s path to the ANDROID_HOME environment variable and create a file with the following content at $ANDROID_HOME/licenses/android-sdk-license

8933bad161af4178b1185d1a37fbf41ea5269c55
d56f5187479451eabf01fb78af6dfcb131a6481e

Now run gradle :safe-app:download-nativelibs again.
This command will download all the native libraries that are required. You can now build the java libraries using the following command

gradlew :safe-app:pack

This command will build the java libraries and output 6 JAR files in the safe-app/build/libs folder:

safe-app-0.1.0-linux-x64
safe-app-0.1.0-osx-x64
safe-app-0.1.0-win-x64
safe-app-mock-0.1.0-linux-x64
safe-app-mock-0.1.0-osx-x64
safe-app-mock-0.1.0-win-x64

Each of these JAR files will contain the required classes and native libraries for the specific platform and variant (mock / non-mock). You can now include these libraries into your project and use the SAFE APIs :slight_smile:
For more information about the APIs check out this android tutorial and the API documentation.

Note: All the 6 JARs contain the same class files so adding more than one of the above libraries will throw an error.

Authentication with the SAFE browser

The application can authenticate to the SAFE network through the SAFE browser. You can create an auth request by following the authentication section of the tutorial. The only difference on desktop would be that the URL format to send the request to the authenticator would be:

safe-auth://<encoded_auth_request>

instead of

safe-auth://<app_id>/<encoded_auth_request>

On opening this URI the auth request will be sent to the authenticator in the SAFE browser.

After authentication, the browser will send the auth response back to the application through a custom URI registered for the application in the following format:

safe-<base_64_encoded_app_id>://<encoded_auth_response>

The application will need to register itself to open URLs of this type and use the auth response to connect to the network. This URI registration can be done across platforms using the system_uri crate.

With some experimentation I was able to create Java bindings for this crate using jnr-ffi as a proof of concept. The code is available in this repository. A lot more is to be done there in terms of error handling across the FFI but it’s a start :slight_smile: You can download the built jars (with native libs) from the releases.

Example CLI application

To demonstrate the authentication process with the SAFE browser I have created a simple Java CLI application which uses the SAFE browser to authenticate and connect to the network. The code is available is this GitHub repository.

Assuming you are on linux, to run the application on the mock network:

  • Download the latest mock browser from the GitHub releases
  • Clone the repository
  • Create a libs folder in the repository’s root and copy the built safe-app-mock-0.1.0-linux-x64.jar file from the safe_app_java/safe-app/build/libs folder. You will also need the system_uri-0.0.1-linux-x64.jar file from this release.
  • Now run the following command:
gradle pack-linux-x64-mock

This will generate a java-mock-0.0.1-linux-x64.jar file in the build/libs folder. This executable JAR is packaged along with the safe-app and system-uri packages. Now open the downloaded mock browser, create an account and login. On executing the CLI application via:

java -jar <path/to/executable.jar>

an auth request will be sent to the SAFE browser and on authentication the response will be sent back to the CLI application. :tada:
Now that you are connected to the network you can perform network operations of your choice by referring the rest of the tutorial or the API docs

Good luck!

OS X Limitation:

Unfortunately, we cannot map an executable JAR to a custom URI on OS X. It needs to be packaged as an application with an Info.plist file. So this CLI is limited to Windows and Linux(for now). More details about this issue on this thread.


#2

Thanks, @lionel.faber - really useful stuff there. I wish I had had this on Tuesday when I was working on my app! :slight_smile:

Are there plans to allow authentication directly (using safe_client_libs, I assume?) instead of via the browser? Could tokens be stored for repeat authorisation without having to re-authenticate each time? It would be good to see what has been authorised (outside of the browser) though.

For CLI apps, this would be essential in many cases, as using a browser may not be possible or desirable (IoT, services, etc).


#3

For development you can authenticate in a number of ways:

  • Using the mock authenticator APIs - like in the example application

  • Using the createTestAppWithAccess() API - Link to docs This will take an auth-request and return a Session object

Yes. You can store the Auth response from the browser and use it to connect the next time. However, if the app is revoked from the Authenticator then the string is of no use.

For now authentication to Alpha-2 is limited to the Authenticator(browser and mobile application) for tier-1 platforms. Technically, we can authenticate and fetch the authenticated apps using the API exposed by the safe-authenticator crate but the libraries currently expose them only for mock routing.


#4

Mission: Have flu; kill bugs.

Below, then works on Linux; a few typos fixed and after a struggle with the unfamiliar becomes a bit forced and saw the sdk command line tools installed, where perhaps there is still a hack to avoid needing that:


Check java version > 7
$ java -version
java version “1.8.0_191”

sudo apt-get install gradle

hiccup tempts then faking the license =>

However, I still got a run of “Warning: License for package Android SDK Platform ## not accepted.”
and a suggested solution like “sudo sdkmanager --licenses”, wasn’t simple to action.

Download Android Studio Command line tools only from
https://developer.android.com/studio/

unzip

cd ./tools/bin
./sdkmanager --licenses

answering yes to those.
is a route to accepting licenses but it had no effect.

So, I guessed the unzip wanted to be in ~/.android
moving the unzipped files and merge/copy over licenses to be within ~/.android

and trying the

$ ./gradlew safe-app:download-nativelibs

is successful.
as is then

$ ./gradlew :safe-app:pack


and then from https://github.com/maidsafe/safe_app_java - if you include the “./”

./gradlew :safe-app-android:download-nativelibs
./gradlew :safe-app-android:build
./gradlew :safe-app-android:javadoc

also work, though with a head full of glu, I have no idea what they do just yet :smiley:


#5

You can run the tasks using gradle or gradlew (gradlew.bat on windows).
gradlew is a gradle wrapper that will help you run gradle tasks without installing gradle. StackOverflow has a good comparison.

This was because the ANDROID_HOME environment variable was set to ~/.android. So moving the CLI tools to that directory did the trick :slight_smile:

Indeed there is. The safe_app_java project is a multi module gradle project for both desktop (safe-app) and android (safe-app-android)
If you want to build the libraries only for desktop without downloading the Android SDKs you can remove :safe-app-android from the settings.gradle file and then build the desktop JARs.

For some clarity on what the gradle tasks do:

The safe_app_java project uses native code that is written in Rust. This native code is compiled for the supported platforms in their respective formats. The download-nativelibs task downloads these native libraries into the project directories.

The native libraries and the required Java classes can be packaged into JARs and AARs for desktop and mobile respectively. The safe-app:pack task will build the required JARs for desktop and the safe-app-android:build task will build the AARs for mobile.

The API documentation for the Java API is available at docs.maidsafe.net/safe_app_java. However if you prefer to build it locally the safe-app-android:javadoc task will generate the documentation into the safe-app-android/build/docs/javadoc directory.


#6

Ok, my son was sick again, so I got another day to play with this! ha!

So, I’m using createTestAppWithAccess() now and it seems to be getting further. However, there are a few things that aren’t as I’m expecting.

I’m attempting to use the NFS library to create a file and then read back from it. Using either mock or non-mock native libraries, this doesn’t seem to work for me - it just throws an exception when I try to read the data.

I have setup the app using the following basic settings:

AppExchangeInfo appExchangeInfo = new AppExchangeInfo("id", "scope", "name", "vendor");

ContainerPermissions[] containerPermissionsArray = new ContainerPermissions[1];
ContainerPermissions containerPermissions = new ContainerPermissions("_public", ALL_PERMISSIONS);
containerPermissionsArray[0] = containerPermissions;

AuthReq authReq = new AuthReq(appExchangeInfo, true, containerPermissionsArray, 1, 1);

Session session = Session.createTestAppWithAccess(authReq).get();
session.setOnDisconnectListener(o -> LOGGER.debug("Disconnected from network"));

Then some simplified code to read/write:

MDataInfo containerMDI = session.getContainerMDataInfo("apps/id").get();
File newFile = new File();
session.nfs.insertFile(containerMDI, path.getFileName().toString(), newFile).get();

NativeHandle writeHandle = session.nfs.fileOpen(containerMDI, newFile, NFS.OpenMode.OVER_WRITE).get();
session.nfs.fileWrite(writeHandle, byteBuffer.array()).get();
session.nfs.fileClose(writeHandle).get();

NativeHandle readHandle = session.nfs.fileOpen(containerMDI, newFile, NFS.OpenMode.READ).get();
session.nfs.fileRead(readHandle, 0, bufferLength).get();
session.nfs.fileClose(readHandle).get();

The exception message I get is:

java.util.concurrent.ExecutionException: java.lang.Exception: Core error: Routing client error -> Requested data not found : -103

From debugging, it looks like newFile is never populated with anything, when I was assuming it would be. Moreover, I’m wondering how I can open a file from just a file name (which I suspect may require MDataEntries manipulation of some sort).

Finally, I can’t see any temporary files being stored in my tmp directory, which I was expecting to see. I’m wondering if the two could be related?

I’ll keep playing with it and I’m pleased to have ported some of my old REST client code to the new native library. It would be great to get something writing/reading though!


#7

hi, @Traktion. Hope your son recovers soon.

When you are inserting the file, it is empty. And then when you are writing the file, the file doesn’t get updated. So you need to write the file, close it and use the returned file object to insert the file.
I’ll show you an example.

MDataInfo mDataInfo = session.getContainerMDataInfo("apps/id").get();
File file = new File();
NativeHandle fileHandle = session.nfs.fileOpen(mDataInfo, file, NFS.OpenMode.OVER_WRITE).get();
byte[] fileContent = Helper.randomAlphaNumeric(LENGTH).getBytes();
session.nfs.fileWrite(fileHandle, fileContent).get();
file = session.nfs.fileClose(fileHandle).get();
session.nfs.insertFile(mDataInfo, "sample.txt", file);

On doing this, we inserting a file that is already populated.

For reading the file, we open it and store the contents in byte array.

fileHandle = session.nfs.fileOpen(mDataInfo, file, NFS.OpenMode.READ).get();
byte[] readData = session.nfs.fileRead(fileHandle, 0, 0).get();

:heart::heart:

Thanks for asking and we hope to help you more.


#8

Thanks for the examples! It would be great to have them at the top of the NFS javadoc, as it isn’t obvious how to sequence these steps.

Maybe my brain just read it backwards, but it felt like I needed to insert the file before I could write to it! :slight_smile:

If I don’t have the object for the inserted file, is there an easy way to retrieve it based on file name? Do I use nfs.updateFile or some such?


#9

This should do what you’re looking for :slight_smile:

NFSFileMetadata metadata = session.nfs.getFileMetadata(containerMDI, "filename").get();
NativeHandle readHandle = session.nfs.fileOpen(containerMDI, metadata, NFS.OpenMode.READ).get();
byte[] content = session.nfs.fileRead(readHandle, 0, session.nfs.getSize(readHandle).get()).get();
System.out.println(new String(content));
session.nfs.fileClose(readHandle).get();

This is something we have in mind :slight_smile: We limited the javadoc to the API definitions for initial release. We’ll definitely add examples in the versions to come! And of course, pull requests welcome :wink: :wink:


#10

Thanks! I should have some good examples in github to link to soon too, with any luck! :slight_smile:


#11

Ok, tried that out and got read/write working well!

The next challenge is directories… any pointers as to how to create/delete directories?


#12

Directories don’t exist as entities - they are only ‘implied’ and there is no API in SAFE that understands a directory. So it is just an application convention that ‘/’ characters in the key under which file is stored are interpreted as directory separators.

Hope that makes sense, it feels hard to put into words!


#13

Ah, right - so if I can have a key which is “/directory1/directory2/file.txt”?

EDIT: Or is it more like having 3 entries like:

/directory1/
/directory1/directory2/
/directory1/directory2/file.txt

The old REST API did have a concept of directories, so I’m guessing that was just an extra layer of abstraction. Interesting!


#14

The first form is correct. There is no way (convention) at the moment to have an empty directory.


#15

Yes. Having the entire path as a key would be one way to go about it.

Another way could be to have the folder name as the key and the value could be serialized MDataInfo of a different mutable data. This other MData can hold all the files inside that directory :slight_smile:
So for instance,

In the app’s own container(App’s root folder):

Key Value
file1 dataMap
file2 dataMap
folder1 serializedMdInfo(name: folder1, typetag: 15234)
file3 dataMap

And in mutable data with name: folder1 and typetag: 15234

Key Value
file4 dataMap
file5 dataMap

You can even have an empty folder, which will be a mutable data without any entries. You can add files(entries) later, whenever required :slight_smile:

P.S.: This just an approach, there can be multiple other possibilities too


#16

Is there a convention for an approach? It sounds like one is an object store (entire path as key) and the other is a more traditional file/container hierarchy file system.


#17

I don’t believe there is a convention as such. Both approaches have their own pros and cons. However, I believe that the WHM and the browser go with the latter to upload/fetch website files.


#18

This is news to me! I’ve even discussed on here (and I think raised an issue) about the inability to have an empty folder in SAFE NFS, and this wasn’t mentioned so I don’t think it is well understood, and I’ve not seen it documented anywhere.

If like to have this specified so we can build on the convention. It has implications for everyone, so it would be very helpful to have SAFE NFS specified down to this level.

It will make it easier to develop, and help achieve cross app compatibility if there is a clear definition of what is expected of apps that want to inter operate.