NFS GetFile offset and length

API
1

I have been trying to get parts of files, but I get bad request responses when using offset/length when getting a file. This is via the launcher REST API on the alpha network.

Is this a known limitation or a bug? Has the syntax changed since documentation or is the functionality not present currently?

Also, are there any plans to provide writes to offsets for modifying or appending data to existing files?

1 Like
2

I have just found more recent docs here: https://api.safedev.org/nfs/file/get-file.html

Looks like it is a range header parameter instead of a URL parameter now. I will try that tomorrow!

Edit: it would still be good to know whether there are plans to modify files too though.

BTW, I plan to post some feedback later this week about the API. One thing that currently frustrates is getting files and directories using different URLs. It would be nice for there to just be a property defining what type it is (file, directory, link).

2 Likes
3

Can’t give you a reference but think this feature not implemented yet.

1 Like
4

Thanks for the heads up! I will give it a go and see how it pans out. I can work around it for now with a cache anyway and it will probably improve performance.

1 Like
5

I think this is the reference you were looking for : No modifyFile launcher/rest api

2 Likes
6

I haven’t had any joy with the range header. From looking at the source code, it doesn’t match the new documentation either (‘bytes=0-1024’, not ‘bytes 0-1024’).

TBH, I think using URL parameters is preferable to these header variables. Having an offset/length to filter the content is a standard REST API design and are straightforward to append to the request URL. Header variables are usually reserved for cross-cutting attributes, such as authentication tokens.

1 Like
7

You’re right, there should be an equal sign. I just updated the API Docs: https://api.safedev.org/nfs/file/get-file.html :slight_smile:

I tried a few tests and the range header works fine. The only difference is that the status code of the response header will be 206 (Partial Content) instead of 200 (OK).

4 Likes
8

Good stuff! Was that on test or alpha? I tried on alpha and didn’t have much joy.

1 Like
9

On TEST 11. But I just tried with Alpha 1 and it works fine too. Here’s the code sample:

var token = "YOUR_TOKEN_HERE";

var request = require('request');
var fs = require('fs');
var endpoint = 'http://localhost:8100/nfs/file/drive/test.png';

var onResponse = function(err, response, body) {
  if (err) {
    return console.error(err.message);
  }
  if (response.statusCode === 401) {
    return console.error('Failed to authorize');
  }
  if (response.statusCode === 206) {
    console.log(JSON.stringify(response.headers));
    return console.log('File downloaded');
  }
  console.error('Failed to download file.', body);
};


var filestream = fs.createWriteStream('./test.png'); // local path
request.get(endpoint, {
  auth: {
    bearer: token // pass the auth token
  },
  headers: {
    'Range': 'bytes=0-1000'
  }
}, onResponse).pipe(filestream);
2 Likes
10

Interesting! I must have done something stupid and will give it another go. Thanks for confirming!

2 Likes
11

I can see the Partial Content message coming through, but my client seems to be collecting a 500 Internal Server Error status in the response, causing an exception. However, I can consistently get the Partial Content message now, which makes me feel a solution is close!

1 Like
12

Ok, I think I have found the problem. A custom error filter looks to be causing exceptions for some 20x codes, which it shouldn’t be. I can see the 206 codes coming through now.

Thanks for feedback and will get my code updated!

EDIT: Patched/committed the code and it works nicely now!

2 Likes
13

I noticed that if you specify a range bigger than the available data size you receive an error (v0.9.2).
E.g. if the file size is 100 bytes, and I send “bytes=0-101” I get this response:

{
“errorCode”: 400,
“description”: “Invalid range specified”
}

Even that the file size can be retrieved form the response header Content-Range, I still think of two issues with this. Let’s suppose an app is reading files in chunks/pages, with a pagination size of 100:

  1. If a file’s length is 250 bytes, then the app can read the Content-Range from the first response, and realise that the last chunk is only 50 bytes, and adjust the Range value to “bytes=200-250” when sending the last request.
  2. If a file’s length is 50 bytes, the client app will need to know the size beforehand, forcing it to fetch the file’s metadata before reading the file.

Both situations add more work to the client app which can be avoided if this error is not retrieved and the last page is returned, even if the range provided is larger than the remaining bytes available. I can submit this as a bug in github if you agree, just let me know.

As an aside comment, I think the documentation should be clear that the Range defines the bytes to be returned in the following way: [ start, end ) i.e. a total of end - start bytes.

EDIT: I submitted it in github for further review.

1 Like
14

I agree with this and it is common practice to return what is available, if it is less than requested. This is a blunt use of error messages, IMO.

A good solution would be to return what is available and specify return length (as normal) in the header. This way, a client can either just blindly read whatever is available OR it can get the content length first, then only read that amount.

1 Like
15

I think it also needs to be updated in the DNS get file API Doc: https://api.safedev.org/dns/get-file.html

1 Like