Wasm would be a much better alternative to native/neon i think, as there’s no need to download a precompiled platform depended binary, we could maybe even use the same wasm file in a web browser. It might be easier to use wasm with the rust wasm_bindgen crate than neon.
But we would still need a way to communicate to the vault or some kind of service/daemon, maybe by using a simple tcp connection and use a simple text (or even binary) protocol to communicate to the safe browser?
Yielding to correction from my seniors, my understanding of the purpose of anonymous memory mapping in self_encryption is to prevent memory leaks.
Referencing libc docs on freeing memory allocated with malloc:
Occasionally,
freecan actually return memory to the operating system and make the process smaller. Usually, all it can do is allow a later call tomallocto reuse the space. In the meantime, the space remains in your program as part of a free-list used internally bymalloc.
Very large blocks (much larger than a page) are allocated with mmap (anonymous or via /dev/zero) by this implementation. This has the great advantage that these chunks are returned to the system immediately when they are freed. Therefore, it cannot happen that a large chunk becomes “locked” in between smaller ones and even after calling free wastes memory.
This is why a memory map is created for files over 50 MiB instead of utilizing a Vec which is allocated on the process’ heap segment, then at most 50 MiB of memory, even when freed, would remain in the process.
Can WASM provide the same memory efficiency?
As far as I can tell, so far no, with possible future consideration.
According to wee-alloc:
wee_allocwill never return freed pages to the WebAssembly engine / operating system. Currently, WebAssembly can only grow its heap, and can never shrink it. All allocated pages are indefinitely kept inwee_alloc's internal free lists for potential future allocations, even when running on unix targets.
@bzee has revived my interest in Neon and it looks to hold more promise than WASM for our needs and to remove our dependency on Node-FFI. Also significant performance gains over FFI, which would most likely be noticeable for file encryption.
@bochaco @Krishna Perhaps we can revisit this topic and look at how development could fit into our milestones and priorities.
in addition to my last post: i, for some reason, thought that there is no way to impl the safe client in wasm, but it should be doable, but it needs a few requirements:
but that’s just optimization, and as you’re writing, will be available in the future. the wee-alloc is also a very simplified allocator, its primary goal is to save on every bit of generated code, it shouldn’t be taken as a reference.
Do you mean implementing the stubs Rust uses for std::net etc?
This is interesting. Do you mean compiling/linking to WASM instead of Rust?
no, because we would need async io (crust is already using tokio for the network io, but this would also need async filesystem io), std::net is sync.
edit: we actually don’t need async io if we get multi threading instead.
i mean compiling the c deps (eg libsodium) to wasm and then linking it to the wasm-compiled rust, so that we end up with a statically linked (thus self contained) wasm file.
@hunterlester has a point, as this would need (i guess) quite some changes in the code base. Just compiling the code as it is now and adding the ffi layer to node on top might be a better/quicker solution for now.
It’s a very important aspect you brought up. This is actually the challenge I’ve been meaning to understand. A lot of crates depend on the libc crate, which links to the OS’s C Library.
The libsodium example you mention is interesting, Alex Crichton (Mozilla) tried to get it working for wasm32: wasm-sodium. It’s interesting to read how he solved it, though it’s a bit of a hack. Especially the libc stub that’s somehow needed by the real libsodium library that still depends on libc.
Anyway, the above highlights one of the challenges of depending on these libraries.
@torri I really appreciate your input. I’d be disappointed to put the nail in WASM’s coffin for our needs.
As it so happens, I’ve been informed that Rust 1.32.0 removes jemalloc, thank you @nbaksalyar, as the default allocator in favor of default system allocator.
Next I need to research common system allocators which hopefully behave as libc’s malloc, which upon meeting a threshold, will automatically make a system call for memory mapping instead of allocating to heap segment. In particular I want to study Windows’ CreateFileMappingA.
We may be able to remove that logic altogether from self_encryption, which would be a great way to start paving the way for WASM.
But this shouln’t change anything for wasm, as
there are no syscalls in wasm (at least not at the operating systems level), the whole point of wasm is to have a sandboxing layer integrated into the “language”/ISA, but there are “syscalls” to the runtime environment like grow_memory. Also i don’t get why a mmap would be better than just “growing” the memory (particularly in the case that the runtime environment can use mmap-allocs as an optimization).
I’m not sure I’m understanding but we may be getting around to the same point.
What I’ve been focused on is the point that self_encryption depends on the memmap crate which makes it not possible at the moment to compile for wasm32-unknown-unknown, as memmap requires the target platform to either be *nix or Windows.
What I’m saying is that self_encryption may not need to depend on memmap anymore with the release of Rust 1.32.0.
So self_encryption may be able to get rid of this logic, leaving heap alloc vs mmap decisions to be made by default system allocator, removing the memmap dependency and making it, hopefully, possible to cross-compile for WASM.
Good question. I don’t know, I’m trying to think through it.
Say a CLI app that’s only purpose is to run batch data (ImmutableData) uploads to the network and be a short-lived process, probably would be fine to just grow memory.
However, a safe_app instance could be a part of a long-lived process, such as in a background service/daemon, a network browser, or a binary that serves a browser extension, as I’ve been researching lately.
I’d think we’d need smart memory mapping to avoid memory leaks in those cases.
According to wee_allocator doc:
WebAssembly does not have any facilities for shrinking memory, at least right now.
I’m confused then, where is it you are reading that WASM can use mmap-allocs?
I’m no expert, but I think memory-mapping and allocating are distinct concepts. Swapping jemalloc for the system default does not influence memory-mapping. Memory-mapping is about virtual memory that is mapped to a file in a file system. Allocating is for the heap. So, on embedded systems you would allocate, but wouldn’t be likely to have memory-mapping at your disposal.
I think Tokio uses std::net internally. It just adds futures (tasking) so the code is cleaner in a way. Also, std::net was just an example. There is a whole lot more of standard library stuff that is stubbed for WASM. I’m afraid one way or the other a lot of dependencies depend on code that will be stubbed for WASM.
That means that even if you get everything to compile, it would not work when executing the code.
Ahh ok, so you’re referring to “real” file-mmapping, i have started writing a 2. post about that, then deleted it 
I think streaming the file in chunks should be enough for self_enc as it shouldn’t need to access the whole file at once? That would save some allocated memory. But we would still need to live with the copy-overhead, at least as long as there is no wasm OS like nebulet, which can inline and zero-copy IO.
And im confused too then ;D
Why shouldn’t it not be able to use mmap-allocs? it just means that the memory is allocated lazily on memory access. Freeing is an other part of the system, not the mmap by it self?
Why do you think that? If we stub everything for WASM, then there is nothing lift to not work
C deps can be a problem, as you already have pointed out in the other post, but how many c deps do we have? the c libsodium impl is critical in that regard, as that there is noone with the knowledge to port it to rust (crypto is hard!).
I’m just learning, so if I’ve got it right, in the case of self_encryption, anonymous memory mapping is used, so it’s not backed by a file on disk and therefore not dependent on an existent file system. It’s backed by a zero-allocated address space in-memory.
The way the allocator influences memory mapping is if it is programmed, like malloc, to create a memory mapping instead of allocating to heap segment, above a certain threshold, and second to return freed memory to the system. As far as I’ve read, jemalloc does not follow the same behavior, which is why self_encryption previously needed to contain that logic which jemalloc doesn’t handle.
Cool, I’ve read about that. I assumed because I saw some std::net modules being used in the Tokio code it was dependent on the TCP and UDP mechanisms too.
It’s an educated guess from looking at std implementation for WASM. They contain some stubs and stubs don’t always ‘work’. I’ve seen the amount of dependencies needed for safe_app, it’s a lot! I thought the chance is there at least a few of them use something from std:: that is stubbed (or panics) for WASM. Anyway, I can’t back it up with examples, it may be less worse than I thought looking at the stubs. Threading, syscalls and mutexes(?) won’t run on WASM, I assume — but I have no clue whether those components are used by safe_app or its dependencies.
My point is, the low-level libraries require OS-specific code. I assume WASM isn’t quite yet ready to be a full OS-environment yet, so to me it’s vague on how these low-level libraries translate to that context.
A lot of progress is made on wasm-bindgen, so if it doesn’t work now, it might in a few years. It’s exciting really.
Thanks for that explanation on jemalloc, @hunterlester. The more I read the less I understand.
This gives us portability, because each host can have their own implementation of wasi-core that is specifically written for their platform — from WebAssembly runtimes like Mozilla’s wasmtime and Fastly’s Lucet, to Node, or even the browser.
couldn’t we just compile the c code with clang (target = wasm32-unknown-wasi) and then link it to the rust code (also compiled to wasm+wasi)?
Suddenly saw that the tutorial now also includes an example for Rust! https://github.com/CraneStation/wasmtime/blob/master/docs/WASI-tutorial.md#from-rust
The wasi target in the Rust compiler seems quite cool already too! https://github.com/rust-lang/rust/blob/2477e2493e67527fc282c7239e019f7ebd513a1a/src/libstd/sys/wasi/fs.rs#L377
WASI is going to rock, I’m sure! 
I was looking at WASM in the Rust standard library again and spotted an implementation of the TcpListener::accept method.
Now, why is this interesting? I’ll try to explain as best I can: Most Rust applications use the standard library which is an abstraction over system libraries that do various things like networking and file system access. Such applications should work on most major operating systems. The Rust standard library takes care of calling the system-specific APIs. The TcpListener::accept method will differ between Windows and Linux. But, WASM is a little different from Windows and Linux.
WASM is a kind of virtual computer processor, like an Intel or AMD processor. In itself, this processor is a primitive component that does very low-level things. Adding up numbers, transfer data in memory and communicating with other components through buses. This is were operating systems like Linux and Windows come in; they implement all kinds of drivers so the processor can communicate with other components. And this is what Rust’s standard library ultimately relies on. So, how does WASM communicate with the network or file system?
WASM is different from your real processor because it is virtual. This means it actually has a ‘host’ that runs it. That means that if WASM were to want to access the network, it always has to go through the host. And this is exactly what WASI defines. It’s the system interface, similar to how Windows and Linux have interfaces for us to talk to the network or file system. So, just like you can use Linux to access the network, you can use WASI to access the network. But, WASI will ultimately rely on the host to provide it. The way this should work is that the host decides what access the program gets.
A very interesting talk I can recommend is this one:
The author explains the current way UNIX works and an improved way of thinking about capabilities (for networking sockets and files). By now, his implementations have been deprecated, and he refers to WebAssembly as a possible alternative. WebAssembly now builds on the ideas of the author.
Back to the TcpListener::accept example. As far as I understand: If you write a Rust application with a TcpListener and compile it to WASM-WASI, this program can be run with wasmtime. The accept call only accepts an already existing socket, so that means that the socket already has to be open. This is what wasmtime can do for you. You can tell it to open a socket, then run your program so that the program can accept incoming connections on that pre-existing socket.
This means the program is capability-based. It can only open sockets you provide it with. Files work similarly, the program will only open files you open for it. So, the program can’t open random files to read from and this is what it is all about: limiting the capabilities of a program to make sure it can’t do anything else than what it needs. This is for security purposes, so that if the program is hijacked by an attacker, it can’t do much harm. It can’t read from random important files etc.
There is a tracking issue that made this all possible here. Due respect for the people working on this!
I should perhaps have posted the following reply here.
. These are steps towards a WASI 1.0 W3 Standard.