Evolving MockVault

#4

Are you sure there is no .exe? I’m going to call for @ravinderjangra’s help, he knows more about Windows and C# than I do :slight_smile:

But here is a snippet from our internal documentation, we haven’t officially released it yet but hopefully it’s helpful:

Yeah, we recognize it is a bit confusing, and we hope to release the internal doc soon. However, the relevant config file is this one:

3 Likes

#5

Thanks for writing up the Issue, linking it here:

I set the help-wanted tag so anyone from the community can feel free to jump in and give it a shot.

4 Likes

#6

The file naming should be in <executable name>.<library name>.config format.
You can get the executable name using Session.GetExeFileStemAsync() API. For .NET core it returns dotnet so the final name will be dotnet.safe_core.config.

Use Session.SetAdditionalSearchPathAsync(string path) API to add the path in configuration file search path list, where the path is the location for your config files.

2 Likes

#7

Aah, yes, now I remember… I actually did part of this long time ago when doing my version of the c# bindings (back when the c# bindings were very new).

Perfect, thanks @ravinderjangra!

I did this:

    async Task Configure(Session session)
    {
        var exeName = await Session.GetExeFileStemAsync();
        var configPath = $"{basePath}{exeName}.safe_core.config";
        await Session.SetAdditionalSearchPathAsync(configPath);
    }

And it works :slight_smile:

3 Likes

#8

agh, premature cheering.
I looked at the speed of which data was added, and though that I was now inmem, but what had happened was that the MockVault file (that was at 50Mb by then) had been overwritten when I did the above.

Hmm… to be continued :slight_smile:

EDIT: Or maybe not exactly caused by the above, but it happened just then. For a moment I wondered if explicitly setting "mock_vault_path" : null would overwrite the file, but it didn’t. So not sure why the old file was overwritten :thinking:

1 Like

#9

That’s definitely unexpected. I double-checked the code but there’s nothing that should be causing it to be overwritten. (The file gets opened here, with truncate set to false).

Did you manage to get in-memory storage working?

0 Likes

#10

Yes, indeed unexpected. I have no idea what could have happened, if there was some kind of filesystem crash (since the file got so big) or what happened. One minute it was there, the next, after I did those changes above, I started debugging, saw that every add was at a few ms each and thought “Yay, in memory storage!”, then saw every add get slower and slower, and got suspicious, tabbed back to the folder and saw a new MockVault file…

Unfortunately have not got the loading of config file working yet. But I have not been home today, so I will get back to this. But I don’t know right now what I might change. First of all I think I will try with .NET Framework instead of dotnetcore, just to check if there might be something about that.

1 Like

#11

Let me know if you have a reason to believe this to be a bug in our code, this definitely shouldn’t be happening!

Have you considered just setting the equivalent environment variables? It might be the simpler way to go here :slight_smile:

1 Like

#12

I don’t have any direct reason to believe so, but no indication of something else either. I’ve run maybe a couple of hundred times with MockVault, and have never seen that happen before. But it was the first time I had a file of 50Mb size (I had let it insert all night and day). As I have looked at the code I haven’t seen anything that could cause it. But I could try again and build a big file and see if I might trigger it to happen again.

Couldn’t get it working that way either :thinking: The SAFE_MOCK_UNLIMITED_MUTATIONS was already there, but adding SAFE_MOCK_IN_MEMORY_STORAGE with value true made no difference. I wonder what that might be about.

0 Likes

#13

hey @oetyng, I tried using mock_in_memory_storage option in the config file and also used SAFE_MOCK_IN_MEMORY_STORAGE environment variable, and both methods are working fine.

1 Like

#14

Have you confirmed that the variables are set properly (i.e. by running env | grep SAFE)? What OS are you on, by the way?

0 Likes

#15

About the config file, it seems that it doesn’t even load the file.
And the environment variable seems to have no effect. I have no idea what it’s about.

I’m on Win10. The variables are there, not sure how more to confirm them: doesn’t seem to be any trailing white space or anything like that.
safe_env_vars


A thought: Wouldn’t it be much easier if the API just took a config object (and naturally not require it, whereby default values used), and any users of the API can handle the config in anyway they need to.
Seems that there would be less strange things in the API, simplifying what it needs to do under the hood (looking for ExeFileStem, dealing with environment variables, setting search paths and trying to find the config etc.).
I think it is a little bit odd way to handle the config. Most API:s I use regularly just take some stuff as a ctor parameter, or use some fluent API with a builder (if they want to be fancy).

2 Likes

#16

Can you please try one more thing: please print out the contents of the configPath file in the code you posted above:

to make sure that what your application sees it is actually the same as what’s on the filesystem.

And for the environment variables, I’m suspecting the variables are not accessible to the application for some reason (due to some quirk with Windows or .NET, I haven’t worked with these in a while but i remember that env vars are not commonly used for configuration in this ecosystem). Can you try printing out the values of the env vars from within your application? Reference

I agree. We use config files throughout the backend – Crust and Routing both use them, allowing changing settings without recompiling. But it’s certainly possible to also support passing a config object to the SAFE App API – app_unregistered already accepts a crust config object as an optional parameter, and we could extend this to the safe_core config.

I believe that doing so would make things cleaner and would allow us to deprecate the environment variables. I was part of the design and implementation of the env vars but I have to admit that they are technically lacking: any app can change them and thus change the behavior of other apps. We also run into possible issues with multithreading: one thread can write to a var while another reads it – there is no concurrency guard on env vars, which is they are known to be a bad solution in these cases. We actually ran into this problem while writing tests because cargo test runs each test in a separate thread, so using env vars to test for certain things was tricky and we (well, I) ended up putting some dirty hacks in place.

Of course, we only use the config for the mock network, so the security points are moot. But passing the config in through the API would just be better, easier, and allow us to remove the hacks we use for testing.

3 Likes

#17

I agree with @marcin on that. @oetyng can you try setting env variable using Environment.SetEnvironmentVariable(string variable, string value) in you app/test. I used used the same.

2 Likes

#18

finally! That works :slight_smile:

I don’t know what is wrong with the other approach or execution, I did it through system settings.
Anyway this will be plenty good for now for creating larger test data sets.

Thanks @ravinderjangra and @marcin.

I’ll try remember leaving the computer on for a night and a day while building on the MockVault file as well, just to see if that file overwrite can be reproduced.

4 Likes

#19

Unfortunately, it seems I cannot manage to go back to mock vault file now.
I have set it (Machine, Process, User) through code, through system settings, can see it when fetching both through code and console. Also made sure there is no config file there. But it still uses in-memory store. It’s as if there’s some cache somewhere. :thinking:

EDIT: Holy smokes, and my MockVault files that I had (one named MockVault_1 as it was copied it to keep it) disappeared :exploding_head:
I had the folder open, I debugged one of the tests I always run, I tabbed back to the folder, and before I let go of it I could see the two files (as it had shown them there before), and when I released the keys they were gone (view got refreshed as window came into focus). Both at around 11 Mb. I don’t understand how the MockVault_1 can have disappeared. It should be unrecognizable as it has a different filename.
I’ve searched the entire disk for “MockVault”, nothing found. And they are not in the trashbin (older ones I deleted myself before are there).

0 Likes

#20

Okay, that’s weird. I’m running out of ideas. Could it be that some other process deleted the files, like an Antivirus? What path were the MockVault files in?

The setting must be coming from either a config file or an environment variable (is this what you mean by “system settings”?). Are you sure these aren’t set?

I think it would be helpful for us to add more logging in the code (e.g. “config found at path X with settings: […]”). That in combination with removing the environment variables (I wrote up an issue here by the way) should make these situations must easier to debug :confused:

1 Like

#21

This is a good suggestion. I have previously had Avast move executables under my nose when I started debugging. But two things: First of all I’ve never seen it happen with a regular file, just executables. Second this VM doesn’t have it, just Windows Defender (it is a little bit less aggressive with such things), and there’s nothing in quarantine history there.
Could still be something else, but I have no idea what it could be. It is a minimal environment, used only for development so there’s not much running on it.

The files were in AppData/Local/Temp. That’s where they end up.

There is no config file there. And I’ve checked Environment variables in 3 different ways, and I don’t know how to confirm it more. It’s almost the same as when I couldn’t set it to in-memory the first time. But then it worked with using Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", "true");
This time I’ve been trying setting it back to “false” and then use Environment.GetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE"); to check if it is set. And it is “false”, but it has still used in memory storage.
So, turns out when I pass in the bool variable.ToString(), it was setting it to “False”. Shouldn’t matter at all, but I tried adding .ToLowerInvariant(); as to make it “false”. Just tweaking whatever I see tweakable.
Also, there are 3 overloads, and so I made sure to set them all, i.e.:

Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", $"{inMem}".ToLowerInvariant(), EnvironmentVariableTarget.Machine);
Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", $"{inMem}".ToLowerInvariant(), EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", $"{inMem}".ToLowerInvariant(), EnvironmentVariableTarget.User);

Then, controlling the value of them all:

var env_0 = Environment.GetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", EnvironmentVariableTarget.Machine);
var env_1 = Environment.GetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", EnvironmentVariableTarget.Process);
var env_2 = Environment.GetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", EnvironmentVariableTarget.User);

And they are all “false”.

Additionally in cmd prompt, enter SET, and it gives this:
safe_env_vars

And the third and last check is via Control Panel\System and Security\System => Advanced system settings => Environment variablessafe_env_vars_2

And it shows the same.

So, this is what led me to think that there is maybe something cached somewhere? Well, I have no other guesses.

I did all of this the other day, made sure to check it one more time now before posting this. Same results.

1 Like

#22

Ah, “true” and “false” are just strings, they both indicate that the env vars are set (which is all safe_core checks, it doesn’t care about the value). You have to actually delete the env vars (unset on Unix, no idea about Windows).

Sorry for the confusion, just another reason to deprecate these pesky env vars :confused:

3 Likes

#23

Aaah… I see. :joy: Now it all makes sense.
I could remove Machine and User, but Process was still “false”, so then I had to do Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", null, EnvironmentVariableTarget.Process);
i.e. set the value to “null”, and then it worked, MockVault file created!

So, now I can let it run and see if I can find the source of this file disappearing… :thinking:

Thanks @marcin and don’t worry, there’s always stuff to get confused about and people to get confused about them :slight_smile:

4 Likes