RFC 42 – SAFE Launcher API v0.6

safe_launcher
safe_core

#1

Discussion topic for RFC 42 – SAFE Launcher API v0.6


RFC 41 – Low Level API
#2

4 posts were split to a new topic: Could we maybe put the content of the RFC right here?


#3

I have the feeling that a version parameter is missing in the API concerning SDs with type_tag above 15000. For example:

  • HEAD /structured-data/{Handle-Id} doesn’t return the version in its headers

  • PUT /structured-data/{Handle-Id} doesn’t need a version in its body

I needed to manage explicitly SD versions in my programs written in rust using core crates, so I suppose the same is needed using low level API.

For PUT endpoint, maybe it will be rendered transparent for the user (its implementation could issue a get request to get the version, add it to the SD and then issue the post request with the completed SD). If so, this should be explained in the RFC.

For HEAD endpoint having the ability to get SD version would still be useful.


#4

Thanks for your inputs @tfa.

For type tags above 15000, tge low level apis from core do not handle anything in specific. It is left to the developer to handle the internals completely.

Glad you brought this for discussion.

I split this into two apis. One normal metadata HEAD request and the other one for versions.

Fetching the versions is a slower process, while the other one needs only once call to the network. Invoking versions would be slower compared to the normal head request where the use wants to validate only the ownership.

I will also add the is-owner field to the versions HEAD request, if that is fine?

We do not allow a specific version to be updated. You can retrieve an old version, but when you update, the last updated will become the latest version.


Could we maybe put the content of the RFC right here?
#5

The versions HEAD request you are referring to concerns tag 501. I was talking about tags above 15000, which do not have any end point to get the version.

About PUT request, unless I didn’t understand the code, I think the version is managed automatically but not the way I described it: the user has first to:

  • either create a version 0 SD and then POST it

  • or fetch an existing SD with a GET request

After having done one of these, he can then modify it and then the next PUT request will automatically increase the version.


#6

So, what do you think of my previous suggestion for adding Version-Number in returned headers of Metadata request? (HEAD /structured-data/{Handle-Id})

This field would be present for any tag (500, 501 and > 15000) in addition to existing Is-Owner field.

For tag 501, I also wonder if the number of versions is always equal to version number plus one. If this is true, then my suggestion would have the added benefit to allow removal of Get Versions request. (HEAD /structured-data/versions/{Handle-Id})

This is not useful because Is-Owner is already returned by Metadata request, that would make 2 redundant ways to get this information.

Adding Version-Number to Metadata request seems more useful because it provides a new information currently not available and it may allow removal of another request.


#7

After implementing and playing with a mocked version of the API, I’ve to say it is a little odd (to say the list). I’ve mostly worked on the StructuredData part but from all I can see this feedback applies to the others, two.

First the in-depth details:

  • The API is inconsistent with the behaviour of the existing API calls. As in that it uses HTTP-Headers everywhere, including metadata-calls. While I understand the usage of the HEADERS when the content is a binary blob (as it has been done before, though I am not a fan), here the user has to always read the headers, while before at least directly called metadata was directly given in form of JSON-responses. Especially in the case of getting a handler this doesn’t make sense in any “resty” understanding. Same goes for posting content: before we send JSON, now it’s all headers. Not sure that is a good thing as headers allow for way less flexibility.
  • The API has inconsistent endpoints: POST /appendable-data/{Id} vs PUT /structured-data/{Handle-Id} –the same endpoint (/appendable-data/) having different keys depending on the method used on them is everything but resty. one endpoint is supposed to reference one resource and then the methods you do on them (POST, `GET``) have different outcomes but they are all done on the same reference. Having the handler in between there, but also not always is weird. And while I understand that it is for convenience (not having to get a handler first and then write it, but provide one that does both), this makes it very confusing when you just look at your API calls from the source code.
  • The whole Handler-ID things feels odd being done over a non-persistent Rest-API. You simply can’t guarantee the client is closing the handler properly and there is no way of knowing. With a persistent connection you could at least clean up from your side once the connection closes.

That last part leads me to the bigger question: should we really provide the lower level API this way over REST. It feels so unresty, primarily it is “session” state that we are trying to convey and aren’t properly cleaning up. I think we should either only offer this through a form of persistent connection (websockets?) that we clean up or generate an overall “session” based on the bearer-token that expires (and cleans up all left-over handlers) once there haven’t been any requests to the API in n-seconds.

Do we actually keep a server-side state for these handlers, btw? Or can’t we serialise it (like you can serialise a database-cursor) and hand over the serialised version and therefore keep our servers stateless?


Second general thing (but that is also an underlying API question), all these APIs feel like a misguided “open, read, close with FileHandle”. But for some reason those are different methods (rather than general one) on their respective structure (which I consider metadata or parsing information at this point). Wouldn’t it be more elegant to have an POST /handler/structured-data/{path} for ‘open’ that returns a (hopefully serialised) handlerId and then only do HEAD /handler/{ID} and POST /handler/{ID}, PUT /handler/{ID} (and DELETE /handler/{ID} to remove the handler – though with a serialised version, that wouldn’t be necessary) instead of name-spaced methods.

It is more RAW but also exposes the LOW_LEVEL-idea a bit more. Right now the API is a hybrid in between knowing and not knowing what it is about and handling and not handling it for the user of the API.


Just a thought though. We can also implement as is, play and experiment with it, see how it feels in usage and come up with a clean and concise design towards v1.0.


#8

Following a question I raised about unique ids on RFC 41, and considering I understand this right, I’d like to raise the following concern:

With unique IDs and removed sandboxing the LOW_LEVEL_APIs are under high likelihood of being mistakenly misused and that effecting the entire network. Just image any app – not realising what they are doing – writing to the userProfileData ID. Now this can’t be used again by any other app. That’s annoying and for 90% of cases not what the APP author intended. The only way to circumvent that is for them to explicitly adding a app-specific prefix and for single usage data to create crypto-collision-secure UUIDs. Both things easily overlooked by app developers (and lead them to making bad design decision they have live with for very long.)

There I’d propose the following changes to the “external” launcher API (safe_core can IMHO still provide the functions more raw):

  1. We should – by default – auto-prefix IDs given with a app-id given server-side. If an app doesn’t want that or wants to explicitly use a different prefix, they can send a NAMESPACE-HTTP-Header we’d use instead (or, if set to GLOBAL could also discard the prefix entirely).
  2. We should provide an id-less POST /structured-data/ that creates a prefixed (as in 1) and crypto-secure id (without collision) server-side and returns this as part of the header data so the app can store it for future reference – similarly as couchdb does it (a great source for API design BTW!). This lifts the burden of doing something unintentionally stupid from the user and provides a convenient layer that “just works” and in the right way.

RFC 41 – Low Level API
#9

I was trying to use low level API in a web page on a local safe network and I came across following error:

Request header field Encryption is not allowed by Access-Control-Allow-Headers in preflight response.

As Encryption header is optional I commented it out, but then same error appeared for Tag-Type header and I need this header because I use a tag with a value > 15000.

Here is a code excerpt:

    var request = new XMLHttpRequest();
    request.open("post", "http://api.safenet/structuredData/" + key + "/", true);
    request.setRequestHeader("Content-Type", "application/json");
    request.setRequestHeader('Authorization', 'Bearer ' + token);
    request.setRequestHeader('Tag-Type', tag);
    request.setRequestHeader('Encryption', 'NONE');

Is there a bug in low level API, or do I program headers in a wrong way?

I didn’t find solution in the Email app. This sample and its tutorial are not instructive enough because it is too complex and isn’t programmed in basic Javascript but uses high level libraries that:

  • hide the real underlying API calls

  • are not available in a web page

I think that a simple example of web page like this excellent one from @DavidMtl would be more enlightening.


#10

Another thing: The RFC states that “Unauthorised access is granted for reading public data using low level api by default”. This is what I need, but I am afraid that implementation don’t do that because for reading a SD both HEAD /structured-data/handle/{id} and GET /structured-data/{Handle-Id} have an Authorization header.


#11

No this is not true - version number of SD has nothing to do with versioning of data in data field of SD. The former is a network/back-end requirement while the latter is user-facing (representation of data in a particular way).

The version of SD will change on every POST - whether it is due to updating the data field or previous_owners field or any other field and is conceptually same irrespective of type-tag.


#12

Any change even in the keys held in the structured data can cause a increase in version number. @ustulation is planning to change the internals of Versioned structured data handling (tag type 501) to get the length of versions without making any additional network request. This will allow in exposing it as single API.

For any custom tag it becomes the responsibility of the application to manage its internals or how ever it gets implemented.

The present code in the master branch is not the final version of v0.6 of launcher API. This is still under development and will be iterated. Please feel free to give your feedback to keep improving!

At present API is exposed as utility functions abstracting the devs from knowing when to call Network PUT and Network POST to create and update the data types. I do agree, to expose the APIs as one to one mapping to the apis in safe_core.

For example,
At present to create a structured data, only one API has to be called.

POST /structured-data/{id} this will internally call two safe_core functions (create structured data and save it to the network)

Now if the application gets direct api access through one to one mapping via launcher then the apps will have to call two apis.

POST structured-data (json body) will return the handle (not saved in the network)
POST structured-data/handleId will save the Structured data referred by the handle to the network.

Can you please explain this little more.


#13

This is what I am doing. To recap my previous posts, I would need the following items:

  • Reading a SD without being authorized (as specified in RFC)

  • Reading SD version number (for any tags)

  • Correction of invalid Encryption / Tag-Type headers

  • A simple web page example using low level API (like @DavidMtl very helpful nfs sample)


#14

In spirit of building a dreamcode API, I think providing some higher level abstractions aren’t bad, but they must be consistent – and follow REST verbs. I’d really urge to take a look couchdb, which does that in an excellent manner.

I have to say, I still haven’t quite grasped, why we need to have the handleID exposed through the REST in the first place. Is it really that we have an open file handler in the backend/server side “open”? What does “open” mean in this case? Is there a lock around it or anything? Do we have a seeking position we keep track of? How?

The problem I have with it, is that it makes the REST-system super stateful all of the sudden. And I am wondering if we aren’t better off with an actual Resource-oriented interface and just use HTTP-Headers to provide specifying the wanted Version or seeking (through HTTP-Ranges) rather than keeping a stateful handlerId around…

I’ll ponder about this a bit more – I have limited concentration capacity at the moment :wink: .

I did so here: RFC 41 – Low Level API


#15

I have raised a pull request for updating the v0.6.

This is an implementation bug, will fix that.[quote=“tfa, post:13, topic:95”]
Reading SD version number (for any tags)
[/quote]

Safe_core handles versioning only for tag 500

I have corrected in the update. If you still find any mismatch then please do let me know, I will correct the same.

This can be done for sure.

@ben, the RFC is more abstract please do have a read through and suggest if it still needs improvement.

Thanks @tfa and @ben


#16

@Krishna will the 0.6 Launcher API (ie proxy) have any CSP header changes (such as this). Sorry to pester but haven’t seen a decision posted and don’t want it to be forgotten.


#17

actually 501, not 500

I think it’s the version field of SD that is wanted here.


#18

@happybeing, the proxy will be removed in the next two weeks from launcher. I hope, even in the next release we might remove the proxy from launcher.

As I informed you earlier, try building the apps using the SAFE Browser. Looking at the README, @joshuef has already integrated the NFS and DNS APIs. I have not tried it myself, but looking to pick it up later this weekend.


#19

@ustulation, is safe_core ready to expose the version field from Structured Data, then the same should be implemented for appendable data to make it consistent. @tfa is there any use case to explain how the version field from structured data is intended to be used? Am just trying to understand a practical use case so that @ustulation can be pushed to agree on this :wink:


#20

@Krishna, sorry if I didn’t grasp that the proxy would disappear so soon. I haven’t had time to look at beaker yet and want to get my RS demo (and others) working ASAP so was concerned if Launcher would be updated without these changes.

It does though seem like you might not get these changes in place (you “hope” to remove the proxy). We all know our plans don’t always go as expected. So, fingers crossed for beaker but if not please update the CSP if we’re still on the current setup during the next cycle.

Thanks.