SAFE Browser Fetch API best practice

Processing large files - such as the output of SPARQL queries is one that I’m personally interested in but it may be a more general issue as we shift processing from servers to client. In mobile this could be an even more useful feature but I don’t know how important.

I’m seeing streams being recommended and I suspect will become expected.

1 Like

@joshuef

I’m testing that now. I’m using Yew’s FetchService to retrieve a file at a path input by the user. When I input foo.txt the contents are as expected. However, when I input the XOR-URL of foo.txt it immediately fails with the error

GET safe://hbkygongu4rnhfc1zeermmhpcuxub9bz7owcgufmm3mxxuynof358onh6d/ net::ERR_BLOCKED_BY_CLIENT

fetch failed with error: ErrorMessage { msg: "408 Request Timeout: error getting file" }

When encountering this error in the past the solution was to refer to the file using its absolute path in the FilesContainer. That’s impossible here as files that can be read by the project won’t necessarily be in the FilesContainer. What is the solution?

Regarding the decision to support HTTP request/response emulation or not, such as enabling Fetch API support in SAFE Browser, I would highly recommend it on at least a temporary basis. Being able to get up and running quickly affords me the opportunity to learn the network as a whole and be able to contribute higher quality feedback on what optimal native support would look like. I believe this would apply to others interested in developing for SAFE as well.

4 Likes

Okay, I’ll have a read around!


@latch you’re getting that consistently eh? Weird that it’s a timeout error (and that it’s immediate…). Sounds buggy to me.

I’ll hopefully get into some more browser bugfixing today once I’ve tidied up some other work. Don’t have more ideas for you at the moment though, I’m afraid :frowning:

2 Likes

@joshuef

How could I further assist the resolution of the issue?

1 Like

@joshuef @latch it’s possible that the 408 is a RED HERRING, I took a look at the yew source code, in their fetch.rs file, here is the fetch implementation: https://github.com/yewstack/yew/blob/master/src/services/fetch.rs#L377

If you take a look on line 397 and onwards in the catch block, you’ll see that the default error catch for the fetch function throws a 408 Request Timeout.

@latch I would suggest modifying the fetch.rs source code to give more information about the e variable exposed in the catch block, that might give Josh more to work with.

2 Likes

@Shane

That’s interesting, thank you. I’ll investigate.

1 Like

@joshuef

That information is

Internal Fetch error (line 399): TypeError: Failed to fetch
2 Likes

Ah nice work @latch @Shane.


@latch can you provide a minimal repro case, if one exists on the shared vault already with the modified yew code that would suffice. Otherwise a wee gitlab repo I can clone and test with, that’d be amazing.


Hmmm, looking at the original error there, if we ignore the timeout, it looks like it could be being blocked by the browser. Not sure if yew is still attempting a HTTP fetch there somehow? Otherwise we have tests for the block / modification of urls in the browser, if you can get a repro / failing test case there, then that’s also a great help.

1 Like

@joshuef

I already have.

2 Likes

Awesome. Thanks @latch. Will get to this after finilasing some token work, and updating snapp tests to get that built and a new alpha out the door there. :+1:

3 Likes

The browser ‘TypeError’ is a vague catch all. On non SAFE web development recently I’ve been getting that in various situations, some seemed to be related to connection refused, some CORS preflight checks. You can test the latter if you enable the ‘CORS Everywhere’ add-on in Firefox.

2 Likes

@joshuef

FWIW after some research I found a suggestion to add webSecurity: false to instances of webPreferences. I tested that in a custom SAFE Browser build to no avail.

There are some missing FetchOptions currently in Yew’s implementation.

1 Like

@latch I’ve uploaded the project here (for the future, it’s always handy if you can upload a project to the shared vault for debugging :+1: ):
safe://hnyynyz1t6uezzn81kee3x6ikrasnho33h8f5txek3imchj14z53t69fwebnc

Using this browser branch, with updated electron which I’m working on:

I enter a URL of a json file: safe://hbkygony7b8kajf6i9pan37mxa5ujt9regiqg8o93wqtubk9pi8fintf6f (the fetchsafe.json file itself)

The error itself is a CORS error as @happybeing was suggesting :

Access to fetch at 'safe://hbkygony7b8kajf6i9pan37mxa5ujt9regiqg8o93wqtubk9pi8fintf6f/' from origin 'safe://hnyynyz1t6uezzn81kee3x6ikrasnho33h8f5txek3imchj14z53t69fwebnc' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

we’re not setting any AccessControl policy in our fetch.js func we use for returning data from the network. We could. I’m not sure we should… Or if we do, what that policy should be.

Would be happy to get suggestions on that, or PRs to improve this.


On the rust-http server front, this fetch.ts file above is essentially what we’d want to replace if we had such rust-server code. It wouldn’t be a bad project to get the logic setup in a simple rust server, tbh. We could then directly use this for the browser. Not sure we’ll get to that anytime soon though… :frowning:

2 Likes

@joshuef

What would be the disadvantage of disabling CORS with either no-cors or webSecurity: false? I’m somewhat familiar with CORS and the reasons behind its creation and I don’t know if they are relevant to SAFE.

Personally I’d prefer it if SAFE Browser was entirely Rust-based as appears[0][1] to be possible. For now, I’d be satisfied if I could fetch via XOR-URL and continue on with the SAFE app I’m developing.

1 Like

@joshuef

Using your fork and branch I’ve attempted to disable the CORS error with { mode: 'no-cors' } in both instances of fetch and webSecurity: false in the three instances of webPreferences. Alas, the CORS error remains.

Whether advisable or not, how can I modify the code to have the request allowed? I would like to continue developing my app, with a custom SAFE Browser build if necessary, while the exact official method is separately worked on.

Yeh a pure rust browser would be great. But practically speaking, that’s not going to happen for some time though.


@latch I think you should be able to add Access Control headers to the js code i linked to above to allow * , which should turn off CORS.

Another alternative would be to use the safe api itself to do your fetch for you, which won’t hit CORS. (I’m not sure that means we should turn CORS off though…).


here’s an explanation of electron’s webSecurity option: https://www.electronjs.org/docs/tutorial/security#5-do-not-disable-websecurity

I think this sort of thing is still reasonable to have even on SAFE.

@joshuef

Would you please show an example or snippet? I tried that with no success however I may have made a mistake.

If you recall two weeks ago I was trying that.

I’ve read that page. I’m aware of the potential security risks however being that I’m only accessing/testing code I write, and I’m now nearly three weeks of wasted time, if I don’t get this basic functionality in place soon I’ll have to stop developing for SAFE until considerable progress is made and I would very much prefer to avoid that.

@latch i’m sorry if you feel there’s wasted time here, some of this is as you’re blazing a trail here. which is massively useful in terms of shining a light on what’s missing.

What I recall is you trying to use the browser native fetch, for wasm. Which had issues in terms of the HTTP response being lacking. What I was trying to suggest above (although not that clearly I now see) was to use the javascript safe api to instantiate an app and then call safe.fetch(<url>). That would get your data without any CORS issues.

Though that in itself is more JS app level than using yew, which I know is what you’re aiming for here. It may be you could use the rust safe-api for this in your code directly. Though I’m not sure at all if this will compile for wasm. It may be worth a try though.


I’ve made this branch
https://github.com/joshuef/safe_browser/tree/AccessControlHeaderBranch and you can see the commit here to enable this. I think should get us around the CORS issue.


In general @latch if you feel the APIs aren’t intuitive or are missing something as you’re developing. We’re super open to feedback. You can feel free to raise an RFC much as @shane has done for missing apis over here so we can improve / add anything missing. HTTP responses, naming etc are all valid topics to be getting in to :+1:

3 Likes

@joshuef

I’ve built the SAFE Browser branch and the error remains.

With

FilesContainer created at: "safe://hnyynywy171fpi6xxptsbkz7cda5y7w3k7jtpp71h7oacy8ydbkwfgbgkhbnc"
+  file.css    safe://hbhyyyn5ox7wd6qyhri8cybrsj7azz4uxmuyhq4e1saxim7fsrjhsqc4fg
+  file.js     safe://hbkygydhuquhis85sbu9ek6jhidc7ynbumuxn8nf7s36fau4ob6ge1ho6q
+  file.wasm   safe://hbyyyyd5gf4kgeacnapdarhb7kq8nhitwwax3woc9an9xoatrtqt11tn9y
+  file.json   safe://hbkygonp7f5b84pdzu7yrqti9gr8hgtjtysyjswa55zpz7jbrfeku9g8hh
+  index.html  safe://hbhybyn7rws4pupisrocmx4rnb5k5oh7oawmmwaaton9b96zzrmta85mtc

I load the app, enter the file.json XOR-URL (safe://hbkygonp7f5b84pdzu7yrqti9gr8hgtjtysyjswa55zpz7jbrfeku9g8hh) and it fails to fetch with

Access to fetch at ‘safe://hbkygonp7f5b84pdzu7yrqti9gr8hgtjtysyjswa55zpz7jbrfeku9g8hh/’ from origin ‘safe://hnyynywy171fpi6xxptsbkz7cda5y7w3k7jtpp71h7oacy8ydbkwfgbgkhbnc’ has been blocked by CORS policy: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.

Ach, seems I’d missed a trick in the server there and we weren’t forwarding all headers :man_facepalming:

I’ve updated the branch now.

You can see that the access control header is set in the devtools network panel.

and then you can also see:

1 Like