Evolving MockVault

Core Libraries Vault
15

About the config file, it seems that it doesn’t even load the file.
And the environment variable seems to have no effect. I have no idea what it’s about.

I’m on Win10. The variables are there, not sure how more to confirm them: doesn’t seem to be any trailing white space or anything like that.
safe_env_vars

A thought: Wouldn’t it be much easier if the API just took a config object (and naturally not require it, whereby default values used), and any users of the API can handle the config in anyway they need to.
Seems that there would be less strange things in the API, simplifying what it needs to do under the hood (looking for ExeFileStem, dealing with environment variables, setting search paths and trying to find the config etc.).
I think it is a little bit odd way to handle the config. Most API:s I use regularly just take some stuff as a ctor parameter, or use some fluent API with a builder (if they want to be fancy).

2 Likes
16

Can you please try one more thing: please print out the contents of the configPath file in the code you posted above:

to make sure that what your application sees it is actually the same as what’s on the filesystem.

And for the environment variables, I’m suspecting the variables are not accessible to the application for some reason (due to some quirk with Windows or .NET, I haven’t worked with these in a while but i remember that env vars are not commonly used for configuration in this ecosystem). Can you try printing out the values of the env vars from within your application? Reference

I agree. We use config files throughout the backend – Crust and Routing both use them, allowing changing settings without recompiling. But it’s certainly possible to also support passing a config object to the SAFE App API – app_unregistered already accepts a crust config object as an optional parameter, and we could extend this to the safe_core config.

I believe that doing so would make things cleaner and would allow us to deprecate the environment variables. I was part of the design and implementation of the env vars but I have to admit that they are technically lacking: any app can change them and thus change the behavior of other apps. We also run into possible issues with multithreading: one thread can write to a var while another reads it – there is no concurrency guard on env vars, which is they are known to be a bad solution in these cases. We actually ran into this problem while writing tests because cargo test runs each test in a separate thread, so using env vars to test for certain things was tricky and we (well, I) ended up putting some dirty hacks in place.

Of course, we only use the config for the mock network, so the security points are moot. But passing the config in through the API would just be better, easier, and allow us to remove the hacks we use for testing.

3 Likes
17

I agree with @marcin on that. @oetyng can you try setting env variable using Environment.SetEnvironmentVariable(string variable, string value) in you app/test. I used used the same.

2 Likes
18

finally! That works :slight_smile:

I don’t know what is wrong with the other approach or execution, I did it through system settings.
Anyway this will be plenty good for now for creating larger test data sets.

Thanks @ravinderjangra and @marcin.

I’ll try remember leaving the computer on for a night and a day while building on the MockVault file as well, just to see if that file overwrite can be reproduced.

4 Likes
19

Unfortunately, it seems I cannot manage to go back to mock vault file now.
I have set it (Machine, Process, User) through code, through system settings, can see it when fetching both through code and console. Also made sure there is no config file there. But it still uses in-memory store. It’s as if there’s some cache somewhere. :thinking:

EDIT: Holy smokes, and my MockVault files that I had (one named MockVault_1 as it was copied it to keep it) disappeared :exploding_head:
I had the folder open, I debugged one of the tests I always run, I tabbed back to the folder, and before I let go of it I could see the two files (as it had shown them there before), and when I released the keys they were gone (view got refreshed as window came into focus). Both at around 11 Mb. I don’t understand how the MockVault_1 can have disappeared. It should be unrecognizable as it has a different filename.
I’ve searched the entire disk for “MockVault”, nothing found. And they are not in the trashbin (older ones I deleted myself before are there).

20

Okay, that’s weird. I’m running out of ideas. Could it be that some other process deleted the files, like an Antivirus? What path were the MockVault files in?

The setting must be coming from either a config file or an environment variable (is this what you mean by “system settings”?). Are you sure these aren’t set?

I think it would be helpful for us to add more logging in the code (e.g. “config found at path X with settings: […]”). That in combination with removing the environment variables (I wrote up an issue here by the way) should make these situations must easier to debug :confused:

1 Like
21

This is a good suggestion. I have previously had Avast move executables under my nose when I started debugging. But two things: First of all I’ve never seen it happen with a regular file, just executables. Second this VM doesn’t have it, just Windows Defender (it is a little bit less aggressive with such things), and there’s nothing in quarantine history there.
Could still be something else, but I have no idea what it could be. It is a minimal environment, used only for development so there’s not much running on it.

The files were in AppData/Local/Temp. That’s where they end up.

There is no config file there. And I’ve checked Environment variables in 3 different ways, and I don’t know how to confirm it more. It’s almost the same as when I couldn’t set it to in-memory the first time. But then it worked with using Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", "true");
This time I’ve been trying setting it back to “false” and then use Environment.GetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE"); to check if it is set. And it is “false”, but it has still used in memory storage.
So, turns out when I pass in the bool variable.ToString(), it was setting it to “False”. Shouldn’t matter at all, but I tried adding .ToLowerInvariant(); as to make it “false”. Just tweaking whatever I see tweakable.
Also, there are 3 overloads, and so I made sure to set them all, i.e.:

Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", $"{inMem}".ToLowerInvariant(), EnvironmentVariableTarget.Machine);
Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", $"{inMem}".ToLowerInvariant(), EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", $"{inMem}".ToLowerInvariant(), EnvironmentVariableTarget.User);

Then, controlling the value of them all:

var env_0 = Environment.GetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", EnvironmentVariableTarget.Machine);
var env_1 = Environment.GetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", EnvironmentVariableTarget.Process);
var env_2 = Environment.GetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", EnvironmentVariableTarget.User);

And they are all “false”.

Additionally in cmd prompt, enter SET, and it gives this:
safe_env_vars

And the third and last check is via Control Panel\System and Security\System => Advanced system settings => Environment variablessafe_env_vars_2

And it shows the same.

So, this is what led me to think that there is maybe something cached somewhere? Well, I have no other guesses.

I did all of this the other day, made sure to check it one more time now before posting this. Same results.

1 Like
22

Ah, “true” and “false” are just strings, they both indicate that the env vars are set (which is all safe_core checks, it doesn’t care about the value). You have to actually delete the env vars (unset on Unix, no idea about Windows).

Sorry for the confusion, just another reason to deprecate these pesky env vars :confused:

3 Likes
23

Aaah… I see. :joy: Now it all makes sense.
I could remove Machine and User, but Process was still “false”, so then I had to do Environment.SetEnvironmentVariable("SAFE_MOCK_IN_MEMORY_STORAGE", null, EnvironmentVariableTarget.Process);
i.e. set the value to “null”, and then it worked, MockVault file created!

So, now I can let it run and see if I can find the source of this file disappearing… :thinking:

Thanks @marcin and don’t worry, there’s always stuff to get confused about and people to get confused about them :slight_smile:

4 Likes
24

Did you see this (I assume you googled the problem)

They suggest some reasons

  • disk cleanup
  • antivirus
  • cc cleaner
25

If I had a swear jar, it would have been filled so many times over in my career due to things like this :rofl:

3 Likes
26

Sorry for adding to the swear jar :wink:

I can definitely see how it’s confusing that we use true and false in the config files, but a different scheme for the env vars. I don’t know when I’ll get around to reworking/removing the env vars, but I plan to move the documentation to the new wiki this week – so I’ll at least fix that to make things more clear.

27

Maybe instead of having “true” you could have “Remove_env_var_to_disable”

28

I’m not sure about removing env vars completely - unless we can achieve the same convenience, by for example having apps learn from SAFE Browser whether we’re using ‘dev’ or not (ie settings accessed through the API).

So any desktop or Web app can tell whether the Browser is Dev Browser or not, and also if experimental features are turned on.

You could still give control via an app supplied object if you want to, but that would not be required. If I understand that idea correctly.

29

It’s not that.
I don’t have CC Cleaner installed. In fact I barely have anything installed, except for the IDE. There’s no disk cleanup running. The MockVault files (nothing else) disappeared as I started debugging.
There’s no reason for random files in Temp folder to be deleted automatically by the system, since there’s no knowing what and for how long other applications will use what ever they store there.

The most likely IMO would be antivirus moving a file to quarantine, but I’m running a very “gentle” AV which hasn’t made itself known in any way (with other machines with more aggressive AV, I’ve had problems, since it interferes in all sorts of ways with the IDE and picks up executables and places them in quarantine), and there is nothing in quarantine history.

Other possibilities would maybe be that the IDE is buggy, since I’m running a preview. But I’m just guessing.

1 Like
30

Now usually previews/trials have limitations and could this be one of them? I am clutching at straws but it seems you are at that stage now as well

31

I’m a bit confused, what env vars are you referring to? The ones I want to get rid of are these: https://docs.rs/safe_core/0.32.0/safe_core/. This topic and @oetyng’s experiences have convinced me that they need to be deprecated.

Environment variables aren’t really a good way to get information about the Browser, anyway. Any app can change or unset them and they’re not guaranteed to be set in the first place.

32

@oetyng Have you run any Client Libs tests? I want to confirm that Client Libs isn’t responsible for deleting the file. The only place in the code where we remove the MockVault file is in a test, here. However, this test (serialisation_write_data) is ignored by default, so you would have to explicitly be running it, or you would have to run scripts/test-binary, so I don’t think this is the issue.

33

I moved the environment variable documentation to the wiki. I hope it is more clear this time around, let me know if not!

34

Apologies @marcin, I wasn’t paying attention. As you were! [in the military idiom sense]

1 Like