Evolving MockVault

#1

From what I have seen in the source, the vaults will write each chunk to its own file on disk. (When the code exits it would clean up the disk though.)

MockVault however, writes to a single file. This makes IO performance absolutely dismal, and the testing utility for it when storing anything above a couple of thousand stored values, diminishes.

Question:

Would it be possible to redesign the MockVault implementation, to be closer to the actual implementation, so that chunks are written to individual files? It wouldn’t be very complicated or invasive it seems to me, and the change would be rather well contained, with few implications if any.


I know there is also in-memory storage, not sure how I would enable that through the C# libs though?

Furthermore, I would actually want a persistence mechanism that is as closely resembling the actual implementation as possible, to increase the value of the tests.
I could device it myself of course, but as I would be circumventing the entire safe_app lib by doing that, it seems to me it would be making the value of such testing dubious.

4 Likes

#2

You have a couple of different options for enabling in-memory storage, see here. We run all of our tests internally with in-memory storage enabled because it is so much faster. Let me know if you have any questions.

We would like to make these configuration options more discoverable, since I assume you weren’t able to find that page easily, so if you have any ideas we’re all ears. Maybe adding a section about configuration below Features in the readme?

Sounds like a great idea to me. It makes sense to improve the performance of the default configuration, and at the same time to increase the accuracy with respect to simulating vaults. Do you mind writing an Issue so we can track this?

Edit: The advantage of the single MockVault file is that it can be easily shared between machines, but I’m thinking the individual files can all be stored in a folder which can easily be zipped up.

3 Likes

#3

Ah, I did actually see the example config file in the repo, and I tried to add one to the output folder of the binaries (with no effect, but I see now that should be because I didn’t name the file correctly), and so then I figured it needed to be modified/included when compiling the rust source.
It’s true I hadn’t seen that page. There’s also been some post here on how to setup the config files, which I did get more or less right some time ago when doing a local network, but I must admit it has been a little bit confusing to me, which config file is used when, what entries are valid in which one of them (I think I haven’t been sure which information is up to date), and where it is placed. So, I guess in that regard a section there, as you suggest, would be helpful.

EDIT: I still didn’t get it right it seems. Have SAFE.DataStore.Tests.safe_core.config in the SAFE.DataStore.Tests\bin\Debug\netcoreapp2.2 folder (as I’m trying to debug the tests). I wonder if the problem is that the output file is a .dll instead of an .exe …?

Super! Sure, I’ll go ahead and do that! :slight_smile:

2 Likes

#4

Are you sure there is no .exe? I’m going to call for @ravinderjangra’s help, he knows more about Windows and C# than I do :slight_smile:

But here is a snippet from our internal documentation, we haven’t officially released it yet but hopefully it’s helpful:

Yeah, we recognize it is a bit confusing, and we hope to release the internal doc soon. However, the relevant config file is this one:

3 Likes

#5

Thanks for writing up the Issue, linking it here:

I set the help-wanted tag so anyone from the community can feel free to jump in and give it a shot.

4 Likes

#6

The file naming should be in <executable name>.<library name>.config format.
You can get the executable name using Session.GetExeFileStemAsync() API. For .NET core it returns dotnet so the final name will be dotnet.safe_core.config.

Use Session.SetAdditionalSearchPathAsync(string path) API to add the path in configuration file search path list, where the path is the location for your config files.

2 Likes

#7

Aah, yes, now I remember… I actually did part of this long time ago when doing my version of the c# bindings (back when the c# bindings were very new).

Perfect, thanks @ravinderjangra!

I did this:

    async Task Configure(Session session)
    {
        var exeName = await Session.GetExeFileStemAsync();
        var configPath = $"{basePath}{exeName}.safe_core.config";
        await Session.SetAdditionalSearchPathAsync(configPath);
    }

And it works :slight_smile:

3 Likes

#8

agh, premature cheering.
I looked at the speed of which data was added, and though that I was now inmem, but what had happened was that the MockVault file (that was at 50Mb by then) had been overwritten when I did the above.

Hmm… to be continued :slight_smile:

EDIT: Or maybe not exactly caused by the above, but it happened just then. For a moment I wondered if explicitly setting "mock_vault_path" : null would overwrite the file, but it didn’t. So not sure why the old file was overwritten :thinking:

1 Like

#9

That’s definitely unexpected. I double-checked the code but there’s nothing that should be causing it to be overwritten. (The file gets opened here, with truncate set to false).

Did you manage to get in-memory storage working?

0 Likes

#10

Yes, indeed unexpected. I have no idea what could have happened, if there was some kind of filesystem crash (since the file got so big) or what happened. One minute it was there, the next, after I did those changes above, I started debugging, saw that every add was at a few ms each and thought “Yay, in memory storage!”, then saw every add get slower and slower, and got suspicious, tabbed back to the folder and saw a new MockVault file…

Unfortunately have not got the loading of config file working yet. But I have not been home today, so I will get back to this. But I don’t know right now what I might change. First of all I think I will try with .NET Framework instead of dotnetcore, just to check if there might be something about that.

1 Like

#11

Let me know if you have a reason to believe this to be a bug in our code, this definitely shouldn’t be happening!

Have you considered just setting the equivalent environment variables? It might be the simpler way to go here :slight_smile:

1 Like

#12

I don’t have any direct reason to believe so, but no indication of something else either. I’ve run maybe a couple of hundred times with MockVault, and have never seen that happen before. But it was the first time I had a file of 50Mb size (I had let it insert all night and day). As I have looked at the code I haven’t seen anything that could cause it. But I could try again and build a big file and see if I might trigger it to happen again.

Couldn’t get it working that way either :thinking: The SAFE_MOCK_UNLIMITED_MUTATIONS was already there, but adding SAFE_MOCK_IN_MEMORY_STORAGE with value true made no difference. I wonder what that might be about.

0 Likes

#13

hey @oetyng, I tried using mock_in_memory_storage option in the config file and also used SAFE_MOCK_IN_MEMORY_STORAGE environment variable, and both methods are working fine.

1 Like

#14

Have you confirmed that the variables are set properly (i.e. by running env | grep SAFE)? What OS are you on, by the way?

0 Likes

#15

About the config file, it seems that it doesn’t even load the file.
And the environment variable seems to have no effect. I have no idea what it’s about.

I’m on Win10. The variables are there, not sure how more to confirm them: doesn’t seem to be any trailing white space or anything like that.
safe_env_vars


A thought: Wouldn’t it be much easier if the API just took a config object (and naturally not require it, whereby default values used), and any users of the API can handle the config in anyway they need to.
Seems that there would be less strange things in the API, simplifying what it needs to do under the hood (looking for ExeFileStem, dealing with environment variables, setting search paths and trying to find the config etc.).
I think it is a little bit odd way to handle the config. Most API:s I use regularly just take some stuff as a ctor parameter, or use some fluent API with a builder (if they want to be fancy).

2 Likes

#16

Can you please try one more thing: please print out the contents of the configPath file in the code you posted above:

to make sure that what your application sees it is actually the same as what’s on the filesystem.

And for the environment variables, I’m suspecting the variables are not accessible to the application for some reason (due to some quirk with Windows or .NET, I haven’t worked with these in a while but i remember that env vars are not commonly used for configuration in this ecosystem). Can you try printing out the values of the env vars from within your application? Reference

I agree. We use config files throughout the backend – Crust and Routing both use them, allowing changing settings without recompiling. But it’s certainly possible to also support passing a config object to the SAFE App API – app_unregistered already accepts a crust config object as an optional parameter, and we could extend this to the safe_core config.

I believe that doing so would make things cleaner and would allow us to deprecate the environment variables. I was part of the design and implementation of the env vars but I have to admit that they are technically lacking: any app can change them and thus change the behavior of other apps. We also run into possible issues with multithreading: one thread can write to a var while another reads it – there is no concurrency guard on env vars, which is they are known to be a bad solution in these cases. We actually ran into this problem while writing tests because cargo test runs each test in a separate thread, so using env vars to test for certain things was tricky and we (well, I) ended up putting some dirty hacks in place.

Of course, we only use the config for the mock network, so the security points are moot. But passing the config in through the API would just be better, easier, and allow us to remove the hacks we use for testing.

3 Likes

#17

I agree with @marcin on that. @oetyng can you try setting env variable using Environment.SetEnvironmentVariable(string variable, string value) in you app/test. I used used the same.

2 Likes

#18

finally! That works :slight_smile:

I don’t know what is wrong with the other approach or execution, I did it through system settings.
Anyway this will be plenty good for now for creating larger test data sets.

Thanks @ravinderjangra and @marcin.

I’ll try remember leaving the computer on for a night and a day while building on the MockVault file as well, just to see if that file overwrite can be reproduced.

4 Likes

#19

Unfortunately, it seems I cannot manage to go back to mock vault file now.
I have set it (Machine, Process, User) through code, through system settings, can see it when fetching both through code and console. Also made sure there is no config file there. But it still uses in-memory store. It’s as if there’s some cache somewhere. :thinking:

EDIT: Holy smokes, and my MockVault files that I had (one named MockVault_1 as it was copied it to keep it) disappeared :exploding_head:
I had the folder open, I debugged one of the tests I always run, I tabbed back to the folder, and before I let go of it I could see the two files (as it had shown them there before), and when I released the keys they were gone (view got refreshed as window came into focus). Both at around 11 Mb. I don’t understand how the MockVault_1 can have disappeared. It should be unrecognizable as it has a different filename.
I’ve searched the entire disk for “MockVault”, nothing found. And they are not in the trashbin (older ones I deleted myself before are there).

0 Likes

#20

Okay, that’s weird. I’m running out of ideas. Could it be that some other process deleted the files, like an Antivirus? What path were the MockVault files in?

The setting must be coming from either a config file or an environment variable (is this what you mean by “system settings”?). Are you sure these aren’t set?

I think it would be helpful for us to add more logging in the code (e.g. “config found at path X with settings: […]”). That in combination with removing the environment variables (I wrote up an issue here by the way) should make these situations must easier to debug :confused:

1 Like