REST API Feedback

Having worked much more closely with the REST API recently, I thought I would share some thoughts:

  1. Use URL parameters instead of headers for command specific parameters (e.g. filters, ranges, sorts, etc).
  • It is much easier/obvious for developers to add parameters to a URL than to change headers.

  • As a REST response should be stateless and return a repeatable value for a request, URL parameters are consistent with this. In short, a data set as defined in the complete URL (with parameters) is unique to the URL.

  • Caching based on URL can be achieved, as the URL response to the request is predictable/repeatable without further input required.

  • Having a mixture of parameters and header variables is inconsistent. This adds complexity to the client, as there are more use cases to cater for.

  1. Splitting requests for files and directories adds complexity to the client.
  • Without knowledge of whether the URL is a file or a directory, you have to test for both. This adds complexity to the client and increases the number of requests required to derive the path type (i.e. file or directory).

  • There are common properties of files and directories (e.g. name, date created, date modified, permissions, size, etc). Having a common data structure simplifies the client, as files/directories just become specialisations of a path.

  • GETs to a specific path could return a response derived by the underlying type, e.g. a file GET could return the content, a directory GET could return the listing.

  • Along with the above, adding a parameter filter could allow a consistent response when needed (format=json to return properties/content as a json document, format=raw to just return the content as the response body, etc). This would cater for times when you just want the content in the response body (e.g. a web browser request) vs a JSON document (e.g. when you are writing a custom client of some sort).

  1. There should be a common way to query paths, rather than separate NFS and DNS commands.
  • Retrieving public data via DNS and private data via NFS seems odd. Both are retrieving data and the former is not a DNS operation.

  • Having two commands to retrieve data adds complexity to the client, which needs to derive the class of data before the correct command can be identified. We shouldnā€™t have to derive this information from the URL - the end point should be able to derive it instead.

  1. Ranges are required for reads and writes.
  • This appears to be in the works for reads, but is currently non-functional (and uses header variables - see above).

  • File systems expect to be able to define a file position to read/write from/to. Being able to define these ranges allows the client to control how much data needs to be downloaded/uploaded depending on the context.

  1. Appending/updating writes
  • Probably also in the works, but not having this functionality adds overhead for file changes (download full file, upload full modified file). The overhead becomes unworkable with large files which are changed frequently.

  • I understand parallel writes are a problem, but parallel reads should already be feasible (at a lower level) and this could be used by the client to optimise data retrieval externally to the launcher. This would allow more flexibility at the client end.

  1. Persistent application authentication at the network level
  • Permissions should not be dependent on the launcher restricting access. This should be done at the network level, which would avoid cross-application access issues (having to encrypt app data, not being able to restrict raw data access, etc).

  • Assuming access to the network can be granted on presentation of a persistent access token, the launcher could instead help to manage these lower level tokens instead of its higher level tokens. The UX could still be similar, but it would allow safe_core to handle this aspect and allow any other linked client apps to offload this responsibility.

  • Having this control at the network layer would also allow multiple devices/platforms to share the same permissions layer. You could then interrogate/modify permissions across all devices easily, without having to rely on separate launcher versions, etc.

I may have forgotten some other items, but I will amend this if they come to mind. I hope this feedback helps and is received in the constructive manner intended.

EDIT: Fixed bullet formatting

8 Likes

It very much is. You are raising some very important points and the input is highly appreciated for sure!

May I ask which platform/language youā€™ve been using for development? Javascript and in the browser? Did you use any helper libraries?

You know when editing you can specify a ā€œreasonā€ on the top (next to the title) that everyone can see in the history of the post, so you donā€™t have to do these Edit-Remarks for stuff like this :wink: .

5 Likes

Iā€™ll elaborate more on a dedicated thread(s), but Iā€™m developing a FUSE mounter for Safenet in Java. This is three different components, which exercise different aspects:

  1. Client Library - Interfaces with the REST API to provide a promise based, multithreaded, command based library. This implements Hystrix to provide thread pooling, fault tolerance and a standard promise-based command pattern. Jersey (jax-rs) is used for interfacing with the REST API end point, serialising/deserialising data etc.
  2. File System Provider - Implements the Java FileSystemProvider interface, using the above client library.
  3. FUSE Mounter - Integrate the above FileSystemProvider with a third party FUSE library (JNR) to allow Safenet paths to be mounted on Linux/Mac.

The FUSE mounter is now functional, but is limited to read-only functionality, currently. However, it can list directories, files, read text/binary files, etc (as of last night). It was fun copying/pasting from one window to another in Ubuntu and catting content in the console for the first time! :slight_smile:

The code is here for anyone interested: https://github.com/traktion0. Bare in mind it is in experimental state, the FileSystemProvider is particularly in need of refactoring and the mounter itself is pure hacked together as a POC! :wink:

7 Likes

Iā€™d very much like to try this. If I understand you correctly I could mount a read only version of SAFE NFS shared ā€˜driveā€™?

If so, can you present a step by step to go from ā€˜git cloneā€™ to mount (on Linux for me)? No worries if not, but if you do Iā€™ll definitely have a go. Great project. :slight_smile:

4 Likes

Thanks! It started as a play ground to refresh my Java experience, but has ended up as quite a nice wee project. Once I have all the basics in place, I will start to tidy the thing up and make it easy to install.

Re what it does, yes thatā€™s about right! You can mount your current ā€˜appā€™ drive too, but I suppose the FUSE mounter would have to identify itself as that app to be useful (e.g. impersonate it, until we get buckets/zones for different data).

It should become read/write shortly too, but I need to implement a few more methods to support this.

I havenā€™t got it into a nicely packaged state and was planning on creating a thread when it was all a bit more presentable! I have been just building the jar through my IntelliJ IDE, but you should be able to do a gradle build and create this. Iā€™m not at my home machine right now, but I can add some basic notes later. Once built, something like the below will work:

java -jar build/libs/safenet-java-fuse-all-0.0.1.jar safe://localhost/ /tmp/somemountpoint

This needs simplifying to a shell script and the URL is set to change (to easily switch between public/private/app content, etc), but it ā€˜worksā€™! :slight_smile:

5 Likes

Added a 6th suggestion to the OP.

2 Likes