Does FilesContainer have a timestamp as key?

Hello, I’m still reading some code, and this is my second question, regarding FilesContainer, in sn_api/src/api/app/files.rs. In particular, the function files_container_create has this comment:

The FilesContainer is created as a Sequence with a single entry containing the timestamp as the entry’s key, and the serialised FilesMap as the entry’s value

Similarly, in append_version_to_files_container, it says

The FilesContainer is updated by adding an entry containing the timestamp as the entry’s key, and the serialised new version of the FilesMap as the entry’s value

I’m trying to see where and how this timestamp is actually created/stored, but from the high level code, it looks like only the serialized FilesMap is stored.

One possibility is that this timestamp as the “entry key” is a general property of Sequence, so perhaps it’s set up automatically, and I just haven’t dug deep enough to see it. Or, perhaps this is an outdated comment? Also a possibility. So being a newbie, I guess it’s better to just ask the question?

I know there’s a notion of “versions” in a sequence, but I assume this aforementioned “timestamp” is referring to an actual timestamp, right?

For reference, here’s code that I was talking about:

pub async fn files_container_create(
    &mut self,
    location: Option<&str>,
    dest: Option<&str>,
    recursive: bool,
    follow_links: bool,
    dry_run: bool,
) -> Result<(XorUrl, ProcessedFiles, FilesMap)> {
    // TODO: Enable source for funds / ownership
    // Warn about ownership?

    // Let's upload the files and generate the list of local files paths
    let (processed_files, files_map) = match location {
        Some(path) => {
            let mut processed_files =
                file_system_dir_walk(self, path, recursive, follow_links, dry_run).await?;

            // The FilesContainer is created as a Sequence with a single entry containing the
            // timestamp as the entry's key, and the serialised FilesMap as the entry's value
            // TODO: use RDF format
            let files_map = files_map_create(
                self,
                &mut processed_files,
                path,
                dest,
                follow_links,
                dry_run,
            )
            .await?;
            (processed_files, files_map)
        }
        None => (ProcessedFiles::default(), FilesMap::default()),
    };

    let xorurl = if dry_run {
        "".to_string()
    } else {
        let serialised_files_map = serde_json::to_string(&files_map).map_err(|err| {
            Error::Unexpected(format!(
                "Couldn't serialise the FilesMap generated: {:?}",
                err
            ))
        })?;

        // Store the FilesContainer in a Public Sequence
        let xorname = self
            .safe_client
            .store_sequence(
                serialised_files_map.as_bytes(),
                None,
                FILES_CONTAINER_TYPE_TAG,
                None,
                false,
            )
            .await?;

        XorUrlEncoder::encode_sequence_data(
            xorname,
            FILES_CONTAINER_TYPE_TAG,
            SafeContentType::FilesContainer,
            self.xorurl_base,
            false,
        )?
    };

    Ok((xorurl, processed_files, files_map))
}
4 Likes

The timestamp is populated in the two FileMeta fields created and modified (L58-59). The FileMeta is converted to a FileItem (of type BTreeMap<String,String>) before being added to the FilesMap.

The timestamp is created when the FileMeta is created, and depends if this file exists already (eg it’s an existing file in which case the timestamp is read from the existing file metadata L70) or if it’s not a file the timestamp is generated using gen_timestamp_secs() L149.

The timestamp is stored to the network when the FileMeta is converted to a FileItem using to_file_item().

FilesMap is a BTree<String,FileItem> L49.

Each FileItem is a BTreeMap<String,String> containing key/value pairs. Some of those FileItem key/value pairs will contain timestamp metadata.

So roughly in pseudocode this means timestamps can be accessed as my_files_map.my_filename.created

In the specific code you’ve quoted the timestamp is created in the call to files_map_create (the line immediately following the comment). There is a call to FileMeta::from_path (L1633) which will populate the timestamps using the existing file metadata in the filesystem.


Just fyi to save you some hassle you can include github code snippets here using these steps

This only works if the commit is specified in the url (eg if the url says ‘master’ it may change in the future so does not show a snippet).

This also makes it easier for people to view the surrounding code / context because it’s a link they can click on.

Ok so I see it truncates it now that I actually paste it here!! Thanks github!

4 Likes

Not sure i have much to add beyond @mav’s detailed response there :+1: re: this current api impl. :clap:

But it’s probably useful to note that this implementation will likely be deprecated (@treslumen you’re highlighting the need for some more documentation/clarifications in this recently-negleced repo), as @danda has in the last months whipped up sn_fs, which is a proof of concept of a much more competent FUSE based filesystem for SAFE, (this current API was a naive PoC to get us a the basics of files on SAFE for the CLI, but it’s not super performant, and definitely not a useful base for actually mounting a filesystem, which the new one does quite well!).

I can’t say when this will all get properly integrated into the APIs, but it’s almost certainly going to happen. (Maybe @danda can shed more light on any plans there… though I know he’s working on other things at the moment).

2 Likes

Yeah there are a bunch of things to do/solve before sn_fs can be properly integrated. Also I’m not able to work on it for at least a few weeks. So I would put it several months out at least.

2 Likes

Hi @mav, thanks for taking the time going through the details for me, and I appreciate your sharing the GitHub trick as well. Regarding my question though, I guess I interpreted the code comment in question differently, in terms of what kind of timestamp it’s talking about. It seems to me that it’s referring to a timestamp for the entire FilesContainer, which is an entry in a Sequence, right? And the comment is saying, this entry has a key, and a value, where the key is a timestamp.

So here’s my impression: each entry is like a “snapshot” of the entire FilesMap at a specific time, and when changes take place in that container, a new entry is appended to the sequence, with a new timestamp. This append op will increment the version of the sequence, but the comment seems to say that in addition to this intrinsic versioning mechanism, there’s also an explicit timestamp attached to each entry/snapshot of the container, and that timestamp I cannot see in the code.

So what do you guys think? How do you interpret the comment?

1 Like

I think you are correct @treslumen, that’s an old/outdated comment from the times when the FilesContainer was stored on a MutableData data type, which needed a key for each entry (it was a key-value type of store), thus we used to set the timestamp as the key just to create some unique value.

A FilesContainer is now stored on a Sequence data type, which doesn’t require a key, just the serialised snapshots of the FilesMaps are appended/stored.

3 Likes

Hey guys, I kept digging up such small things, but anyway, would it be helpful if I also submit a PR to update these code comments? (I assume creating an “issue” is not necessary?)

@joshuef you mentioned that files-related API will likely change, but I’m wondering if this FilesContainer stored as a Sequence will stay at least, even if the underlying FilesMap may change? I’m asking because if all of this is going away, then perhaps it’s not really useful to correct the comments. Just let me know.

2 Likes

Yeh, no need for issues for every small thing :+1:

I think a FilesContainer will likely go away, to be replaced with the sn_fs lib. I’m not too sure though as I haven’t been in and around that yet. And I’m not sure about the knock on for NRS… I’d presume we still need to be linking to files in that sort of fashion (which is almost the same as FilesContianer)… So perhaps we’ll be keeping that… But honestly not too sure here. @danda / @bochaco / @happybeing have you had any thoughts on this/ NRS w/ sn_fs ?

3 Likes

At present the basic thinking is that we will still have something like a FilesContainer that is SafeUrl addressable. (XorUrl or NrsUrl). It holds an sn_fs tree and local nodes can sync with it, eg when mounting. files and directories would still be accessible via path portion of SafeUrl. File content is still stored as Immutable data.and thus has its own XorUrl.

So basically, each FilesContainer becomes the root of an sn_fs mount point. Though we might want to call it something else.

And all of this is subject to change.

4 Likes