2017 was a good year for Crust: the codebase was ported to Tokio Futures based networking which is becoming the default in Rust community, code became simpler and more maintainable, NAT traversal was greatly improved with external p2p crate and on top of that multiple transportation protocols became possible with current UDP and TCP implementations.
If you’re here, it means you’re wondering what’s next. Crust is already able to fly, but before seating hundreds of passengers, we need to put more effort to assure it’s robust. Hence, we will be giving a lot of attention to testing.
Unit and Integration Tests
Currently Crust has both unit and integration tests. Unit tests try to verify whether small pieces of code work as they are expected to. Whereas integration tests check if different pieces work together. kcov reports quite large code coverage by tests - 80.6%. As good as it sounds, it’s mainly because of high-level integration tests that exercise a lot of code. We would still love to have more unit tests - they help a lot to indicate and locate bugs and allow us to move faster while developing new features without regression.
uTP Testing
uTP is a transport protocol that works on top of UDP. It was first used in torrent clients which was a great success. Because of smart congestion control algorithm used by uTP, it’s very suitable for applications like torrent clients and SAFE Network vaults. The algorithm is able to dynamically adapt to network load at a given time and not disrupt internet connections, while still utilizing the unused bandwidth fully. So we are very excited to have uTP support in Crust as one of our transport protocols. This is still a very fresh work and still needs to be properly tested: we’ll do some more manual testing and write more automated integration tests. Eventually, it would be nice to go into the wild with this and test how it works with all sorts of network configurations. Hence, we’re planning to release some tools available to the community so that everybody interested could contribute. More on that later though.
Automated Soak Testing
We heavily rely on CI to avoid regression. We have builds for all major platforms: Windows, Linux and macOS. That works well for functional tests and to assure decent code quality.
But when it comes to performance, that’s a different story. Historically we have found bugs that would only start occurring after sending hundreds of megabytes over a connection. Ideally, we’d like to have automated soak tests. Unfortunately, it’s not feasible to run them on regular CI services like https://travis-ci.org/ or https://www.appveyor.com/, because running long lasting performance tests would hurt our regular testing experience. Hence, we need a separate testing environment for that and we’re planning to build it ourselves in MaidSafe HQ and run tests on our own machines. We’re planning to run soak tests regularly (maybe nightly) to detect any flaws as soon as possible.
Isolated Network Simulation
Crust implements a lot of NAT traversal techniques. Making sure that they all work as expected, especially hole punching, takes a lot of effort and, unfortunately, we haven’t found ways to do that automatically yet. Luckily, there’s a solution to this.
It’s possible to write a Rust library that allows launching a thread into it’s own isolated network environment where it only has access to a virtual network interface. From outside the thread we can then read/write packets directly to/from this interface in order to introduce latency, packet loss, or any other form of communications-mangling. Using iptables we can also set up these threads with their own NAT rules.
To showcase how this could be used, it could allow us to write test cases that look something like this:
// Example Crust test
#[test]
fn test_connect_to_peers() {
let (tx, rx) = mpsc::oneshot();
internet_simulator::run(&[
|| {
let service = Service::new();
service.start_listening();
let addr = expect_event!(service, Event::ListenerStarted(addr) => addr);
tx.send(addr);
let peer = expect_event!(service, Event::BootstrapAccept(peer) => peer);
// we connected! now exchange data or something.
},
|| {
let service = Service::new();
let addr = rx.recv();
service.add_contact(addr);
service.bootstrap();
let peer = expect_event!(service, Event::BootstrapConnect(peer) => peer);
// we connected! now exchange data or something.
},
]);
}
In the above code, the two Crust services would each have their own IP address, be behind their own NAT, and be communicating over a lossy connection with latency.
We already have basic proof-of-concept of this working. It’s Linux-only and isn’t yet fully developed so you can’t spin up a virtual internet with a single command, but all the essentials are in-place. Hence, that looks quite promising and exciting since nothing like this exists yet. Folks, what do you think? 
Testing Tools
The right tools for the right job might save you hours of work. So we are looking forward to enhance our testing toolbox.
One of the ideas we have is “router probing tool”. The problem is that routers in the wild exhibit all kind of weird and wacky behaviours which effect our ability to hole-punch and reliably maintain a connection. It would be good if we could catalogue these behaviours and their relative prevalence. This would help us know how Crust could be improved and how urgently certain improvements may be needed. A way to do this cataloguing would be to release a tool to our users which runs a set of tests while communicating with a set of external servers, then reports the test results back to us. Eventually, some form of this test could be integrated into the p2p library in order to avoid the need for the user to do their own manual configuration.
The second tool we’re thinking of has nothing to do with automation. We’re planning to build a small peer-to-peer application on top of Crust. Probably it’s going to be a secure p2p chat since chat applications are like hello world of networking and that would be perfect proof of concept that Crust is working properly. Such tool would also aid in testing how Crust acts in different network setups with different routers, etc. This app will be, of course, open source and binaries will be publicly available to the community. So everybody will have a chance not only to test drive it but also will be able to pitch their own ideas and contribute to development if they want to.
Finishing off for Alpha 3
Finally, we will build a small DHT layer on top of Crust to create a standalone application. This will require a functional bootstrap cache and secure serialisation. The bootstrap cache will help us retain peers to whom we could connect directly to before so that when we disconnect from the network and want to go online again, we will use those peers to proxy to our overlay network. In addition, those peers will also be used as rendezvous servers. Every message sent by Crust eventually will have to be encrypted, but that should not be tons of work since we already have secure_serialisation library. With this, we will have concluded the goals of Crust crate for alpha-3.