How to run a local test network


#1

To run a local testnet, you’ll need to download or build the SAFE Vault binary and modify or create the following config files which need to exist in the same folder as the binary:

safe_vault.crust.config
safe_vault.routing.config
safe_vault.vault.config

Download SAFE Vault

Downloading the latest prebuilt binary is the simplest way to get started. This will include the required config files, but they will need to be modified to include some developer options as detailed below.

Build SAFE Vault

If you want to try building the vault binary yourself, then you need to have Git and the latest stable release of Rust installed. If you don’t, see Installing Git and rustup to install these.

  1. Clone the repository:

     git clone https://github.com/maidsafe/safe_vault.git --depth 1
    
  2. Build SAFE Vault

     cd safe_vault
     cargo build --release
    

This should produce the SAFE Vault binary in the folder safe_vault/target/release. It will be called safe_vault or safe_vault.exe depending on your platform. This folder is where you should also create the config files.

Example contents for safe_vault.crust.config

{
  "hard_coded_contacts": [],
  "force_acceptor_port_in_ext_ep": false,
  "dev": {
    "disable_external_reachability_requirement": true
  }
}

Normally a vault is required to be able to allow a remote vault to connect to it; i.e. it must pass the “external reachability check”. For running a local testnet where all vaults are on the same machine or LAN, not only is this check useless, it could well fail since many routers don’t support hairpinning. Setting disable_external_reachability_requirement to true disables that check.

Further details on all Crust config options can be found in Crust’s config_handler.rs.

Example contents for safe_vault.routing.config

{
  "dev": {
    "allow_multiple_lan_nodes": true,
    "disable_client_rate_limiter": true,
    "disable_resource_proof": true,
    "min_section_size": 5
  }
}

Currently, we disallow more than one vault on a single LAN or machine. To disable this restriction, set allow_multiple_lan_nodes to true.

When a vault acts as a proxy for one or more clients, it limits the number and size of requests that it’ll forward on their behalf. To remove this limitation, set disable_client_rate_limiter to true. Setting this to true also allows a proxy to act on behalf of multiple clients with the same IP address, whereas this would not normally be allowed.

When a new vault connects to the network, each peer from the section it’s joining sends it a resource proof challenge. The challenge involves the new vault doing process-heavy work and sending a large message in response to confirm its upload capability. After a lengthy delay (several minutes) the peers all vote on whether to accept the new vault or not, and only if a quorum of positive votes are accumulated is the new vault allowed to join. If disable_resource_proof is set to true, the new vault is sent a challenge which involves minimal effort, time and traffic and the existing peers don’t delay their votes, enabling new vaults to join very much faster.

The network is comprised of sections of vaults, grouped together by IDs which are close to each other. These sections need to be comprised of a minimum number of vaults, currently defined as 8. Data is replicated across this number of vaults. This figure can be adjusted via the optional min_section_size field. As this figure reduces, the risk of data-loss increases (as there are fewer copies of each chunk of data). However, increasing this figure will increase the burden on each vault in terms of storage used, traffic sent and received, and network connections being maintained.

Further details on all Routing config options can be found in Routing’s config_handler.rs.

Example contents for safe_vault.vault.config

{
  "dev": {
    "disable_mutation_limit": true
  }
}

Currently, clients are only allowed a limited number of requests to mutate (store, modify or delete) data on the network. To remove this limitation, set disable_mutation_limit to true.

Further details on all SAFE Vault config options can be found in Vault’s config_handler.rs.

Set up logging

While the vault runs it can generate log messages if required. There are five levels of log message. From most severe to least these are: error, warning, info, debug and trace. By default, only error-level messages are output to the console.

You can modify the logging behaviour either via an environment variable or via a config file. The env var is probably simpler, but the config file supports more options. If you specify a certain level, that level and all those more severe will be output. E.g. for info-level logging you will get info, warning and error-level output.

Logging via an environment variable

You can set the modules and log-levels required in the env var RUST_LOG. For example (on Windows) to enable logging for all modules at info-level:

set RUST_LOG=info

To only log Crust messages at debug-level:

set RUST_LOG=crust=debug

To log Crust at debug-level, and Routing (including stats messages) and Vault at trace-level:

set RUST_LOG=crust=debug,routing,routing_stats,safe_vault=trace

For further details on using RUST_LOG, see the docs for the env_logger crate.

Logging via a config file

You can similarly affect what is logged by creating a file named log.toml in the same folder as the vault binary. This also allows you to adjust the actual contents and format of the individual log messages, and furthermore to choose to output to the console, or a logfile, or both.

For example, the following log.toml would cause all info-level messages to be logged to the console, and would write debug-level Crust messages and trace-level Routing and Vault messages to a logfile in the current dir with the name Vault-<timestamp>.log:

[appenders.async_console]
kind = "async_console"
pattern = "{({l}):1.1} {d(%y-%m-%d %H:%M:%S%.6f)} {m}\n"

[[appenders.async_console.filters]]
kind = "threshold"
level = "info"

[appenders.async_file]
kind = "async_file"
output_file_name = "Vault.log"
pattern = "{({l}):1.1} {d(%y-%m-%d %H:%M:%S%.6f)} [{M} #FS#{f}#FE#:{L}] {m}\n"
append = false
file_timestamp = true

[root]
level = "error"
appenders = ["async_console", "async_file"]

[loggers."crust"]
level = "debug"

[loggers."routing"]
level = "trace"

[loggers."routing_stats"]
level = "trace"

[loggers."safe_vault"]
level = "trace"

Be aware that if you run multiple instances of the same vault binary and you want more than one to log to a file, then the simplest approach is to set file_timestamp = true and ensure that you leave at least one second between starting any two vaults. This has the effect of adding a raw timestamp (in seconds) to the chosen filename; so you’ll end up with logfiles called e.g. Vault-1501614444.log, Vault-1501614446.log, Vault-1501614448.log, etc.

If you want to avoid using timestamped logfiles, you can instead copy the vault folder including the binary and config files to multiple different parent folders. Or you could update the output_file_name before starting each new vault.

Further details on using the log.toml config file can be found in the docs for the log4rs crate, the most useful being the details on patterns and files. Note that the examples there use YAML, whereas we only support the TOML format.

Run several vaults

With the config files mentioned above placed in the same folder as the vault binary, you should now be able to start multiple instances of the binary, either in multiple terminals on a single machine, or single instances on multiple machines on the same LAN, or a combination of these.

Note: The first vault must be run with the --first flag (or -f for short) so it knows there aren’t any peers to connect to.

You should run at least min_section_size instances of the vault binary before trying to connect any clients, since that’s the minimum number required to safely store any data (including client accounts) on the network. This value is currently set to 8 by default, but can optionally be changed via the min_section_size dev option of the Routing config file (see above).

Miscellaneous

Eventually the vault codebase will be updated to handle unexpected or malicious behaviour from their peers. However, until that’s in place, all vaults and clients connecting to a single testnet should use the same configuration options. An exception is that logging config files can all be different or omitted altogether.


How to develop for the SAFE Network (draft)
Trying to build Maidsafe
Running Test Network
What is the current max File size supported by NFS?
Creating an outside DigitalOcean Alpha Test Network
Creating an outside DigitalOcean Alpha Test Network
Creating an outside DigitalOcean Alpha Test Network
Routing client error -> Account does not exist for client
#2

Thank you for the guide, this is extremely helpful!

It looks like this only explains how to install and run a SAFE Vault, not an entire network. Based on the title I was hoping to run a Vault along with any other necessary peripherals in order for my SAFE Browser to connect and run the SAFE web API playground and do all sorts of experimentation completely offline.

Is this currently possible? Is there a different guide I can follow for this? Cheers!


#3

Make sure to start the first Vault using the --first flag.

See the “Run several vaults” section.

Yes, this should be possible :slightly_smiling_face: Just make sure to use the same config files for all vaults and clients connecting to your testnet.


#4

Thanks @frabrunelle; I had skipped that section because I didn’t realize I needed multiple vaults! My local network is working great now. I sent a PR to update the safe_vault README to help make it easier to set up.


#5

Has anyone managed to run multiple vaults on one (Windows) machine? I tried it by running the latest release v0.14.0 with the config options mentioned above but get several errors:

E 17-10-23 10:24:12.553838 Bootstrapping(f00d2c..) More than 1 routing node found on LAN. Currently this is not supported

Cannot start vault due to error: ChunkStore(Io(Error { repr: Os { code: 33, message: "The process cannot access the file because another process has locked a portion of the file." } }))

Is the latest build even meant to support the above options or do I need to build from master? Is it possible to modify the chunk store directory yet?

Btw.: If not, could anyone provide me with fresh windows binaries? I was only able to build it on linux yet.


#6

I finally managed to build safe_vault master on windows and successfully run a local test net! :slight_smile:

For anyone looking for a limitless coding experience - beyond mock routing - here is a full guide, including necessary vault binaries (for windows) and pre-configured config files:


Need help understanding an error thrown from safe_app.dll
#7

Some feedback on the local network;

Win10 on Hyper-V virtual machine.

First local network I setup was working fairly well. However, after a while I started seeing some problems.

  1. One of the main problems has been connecting to the network. In the beginning worked perfectly, connected every time, then it was enough to retry a couple of times, but then it became increasingly difficult. The error received was mostly:
    Failed to Bootstrap: (InvalidNameHash) Network name mismatch.
    This does not make sense to me since network is set to “local_1” everywhere. And why would it mismatch sometimes but not all times? Now I can almost never connnect, only getting this error.
    Took me around 15 retries to create an account just now. Then another 10 retries to login.

  2. After this I also started seeing NotEnoughSignatures when trying to connect. It was followed by Timed out waiting for Ack(.. So now either the error from 1. or this error was showing.

  3. A couple of times I also saw problems like Bootstrapper has no active children left - bootstrap has failed (3 nodes running, no sign of problems in the logs.) This was after creating an account and trying to login after it. (not sure it is supposed to go like that, but it wasn’t enough to just create the account at least to start interacting with network, so some login seemed necessary).

  4. Another problem (when it was still possible to connect) was that connecting new nodes and then dropping off the first nodes seemed to make SAFEBrowser lose connection, and making the data disappear. The data should be replicated over to the new nodes that are joining, no?

Would be really great to be able to run the local network again.
Q1: Anyone else have experienced these problems?
Q2: Anyone knowing if these problems are to be expected or if there might be some user handling error (i.e. I’m doing something wrong)?


#8

Really excited to try running my own local network! :grinning: Just a quick question… I assume that applications should be built without NODE_ENV=dev being set so that mock-routing support isn’t activated. Does this mean that when applications start, whether it be SAFE Browser or WebHostingManager, that it checks on your local network to see if there are vaults running? Then chooses those preferentially over the ‘public’ Alpha-2 test network?


#9

No, you have to specify a network name and nodes to whom you wish to connect to first. This can be done in the crust.config of SAFE Browser for instance. Also note: The network name specified in the crust.config file of SAFE Browser has to match the name in the config files of your vaults.


#10

Thanks for the pointer! Here is the setting in question for anyone who looks at this in the future. https://github.com/maidsafe/crust/blob/52b1e6e2ac7fb19a97f40d97d4aa43a09d9d82eb/src/main/config_handler.rs#L56


#11

@Mindphreaker, a quick question: you didn’t experience any of the problems I listed in the post above? Thanks


#12

I never experienced such problems but to be honest, I never let the network run for a longer period of time (> 1 day). Maybe the network gets unstable after some time, but then it should occur in Alpha 2 too I guess. Maybe someone from maidsafe can tell what these errors could mean. From the nature of the error messages it seems, that there might be an issue with churned nodes, section splits, etc.


#13

Hey @oetyng - I’m gonna try and get a look at this soon to see if I can figure out what happened. I know it’s a long shot, but you don’t happen to have any of the old logfiles from when you saw the failures do you?


#14

@oetyng I’ve had a look at the problems you saw and reviewed the code related to them, but I can’t see any obvious issues there. There’s probably not much more I can do unless you still have the logfiles and know what version of the code you were running?

If you can still replicate the problems then we could try getting on a hangout or something to maybe speed things up a bit?


#15

Hi @Fraser,thanks a lot for having a look and sorry for the late reply!
I will sit down with this in the weekend and try to get a good systematic approach with the setup to reproduce this and record everything. Then if I can gather the necessary information we could go through it, would be great!
So, hang on and I’ll get back and thanks again.


#16

I cannot currently reproduce this.
There has been environment changes*, but cannot say if they influenced the outcome.

I’ve been running 3 nodes, with the same config as in Mindphreakers repo.
Did try with Windows Defender turned on and off (it has been interfering with VisualStudio MS Tests among things) but made no difference.

I did however come into some other problems after starting up more nodes (total of 9). But I will unfortunately not be able to make a full account of it this weekend. But stay tuned, I’ll get back to it.

* Environment changes

I upgraded Windows 10 to Fall Creators Update 1709, installed new .Net Framework (4.7.1) and added a preview version of VisualStudio 2017 (running in parallel though, so not the one used in these tests), plus installed microservice framework ServiceFabric runtime on the machine + SDK and tools for VS. That’s pretty much it. (And the top two changes naturally would be the most likely to have anything to do with this. I have no clue how.)


#17

I wonder if anyone else has experienced this issue before…
It appears that I have to run exactly the min_section_size number of nodes for the network to function. So for example, if I set the min_section_size to 2 then ran four vaults (creating what I thought would be two complete sections) the network ceases to function. If I then stop two of the vaults it starts to work again… Is there a minimum number of sections you have to have running in order for the network to function (if you want multiple sections)? Or is there a requirement that the sections shouldn’t all be ran on a single computer?

Thanks in advance!


#18

Hi there,

The network should work with min_section_size set to 2, although I wouldn’t expect the network to split until there are at least 10 vaults running.

We have a SPLIT_BUFFER set to 3, which means that for a section to split, it would need to be able to split into two new sections so that each would have at least min_section_size + SPLIT_BUFFER vaults.

When I say the network should work - I mean that setting the size as low as 2 will be OK assuming you have very low or no churn. What are the symptoms for you when you mention the network ceases to function?

No, just one should be fine.

It should also be fine to run on one machine assuming you have set up the config files as per the details in the OP.


#19

Thank you very much for taking the time to reply! :grinning:

Thank you for the insight, I had no idea that is how it worked! Once I added the tenth vault the network split successfully.

As soon as I introduce more than min_section_size nodes onto the network then successful network connectivity stops. I have included a screen shot just to help illustrate my point. Here I have the same ten vaults as above running with min_section_size set to 2. I am trying to register an account but the SAFE Browser will just sit like this for a few minutes and then throw a Core error: Blocking operation was cancelled. I have tried a few different min_section_size values but this behavior appears to always exists when having a greater number of vaults than min_section_size.

Do you have any idea what is going wrong?


#20

I’m guessing you haven’t provided the same Routing config to the browser? The browser is probably connecting and expecting the normal 8 responses to requests, but the network is only providing 2 (or whatever you set the min_section_size to for the vaults)

If you copy the safe_vault.routing.config file to the folder where your browser binary is, then rename it to match the binary (probably to safe-browser.routing.config) that should hopefully fix this for you :slightly_smiling_face:

You don’t need to copy the safe_vault.vault.config by the way, just the routing one.


Error using Peruse dev browser v0.6.0