Register CRDT APIs

[rob]

A topic for discussing the Autonomi Register CRDT API.

This post is a Wiki so please edit it to include useful links to documentation, demos, code etc.

Original Topic created by @happybeing but became broken

[Traktion]

@Traktion

Users add an entry to the above register, with a format like ‘name,my_private_app_register’ (or json for flexibility), where name is the ‘host’ (in clear net speak)

From early hacking, it looks like registers aren’t append only atm (I thought they were). So, it looks like anyone with write access can merge the tree and delete the history.

That makes having a public writable register more like a variable that anyone can change. Unless I’m missing something (quite possible at this stage).

EDIT:

Specifically, someone with write access can run client.get_register(address).write_merging_branches("replace all entries with this").

[happybeing]

Register APIs need their own topic - shall we move your reply and this one to a topic for that “Register APIs”? cc @moderators

[EDIT: @rob I think you are a mod here, please can you move the above reply by Paul and everything below to the topic: Register CRDT APIs (fixing this topic up, sorry Happybeing) Thanks]

Registers are append only but still very raw and it isn’t clear how they are supposed to be used. I have a pretty good idea of what the underlying RegisterCRDT can and can’t do, but I don’t think Autonomi want to leave that open and fully flexible.

What the RegisterCRDT doesn’t do as far as I can see is handle history in the way you might expect, so there is in effect one history per variable (as you put it) and the latest ‘values’ will be the ‘heads’ (of the RegisterCRDT). But these histories are then independent and cannot be stepped through in sync, unless you keep state externally to record that.

To get at those histories though, requires access to the underlying RegisterCRDT tree, which I exposed in a PR to allow the examples to print the structure of a register. But I don’t think Autonomi want that left exposed, so I expect them to extend the higher level Register APIs to provide the features they want exposed while hiding the underlying MerkleReg.

Until they get to that I’m not sure if it will be a good idea or if best left open for developers. It could limit their usefulness but it may be better overall.

[Traktion]

Thanks - that’s really useful context. Agree this needs its own thread. Apologies for just firing out thoughts!

I’m in my mobile, but from poking around the code, I realised that client.get_register(name).read() returns the last entry and not the root.

EDIT: Looking again, the comment says it returns the last element, but looking at the code, I think it actually returns the whole tree. That means the following does remove the history:

client.get_register(address).write_merging_branches("replace all entries with this")

I tried calling register.merkle_reg().read().values (which you added as above, right?) to ensure I was seeing the full tree and it is the same after calling write_merging_branches (it has 1 entry).

Unless I’m missing something?

[happybeing]

@Traktion

Unless I’m missing something?

I’m in a different space right now but from memory merging writes a new entry on top of one or more concurrent entries. They may call it ‘replacing’ because it replaces the head(s) but they remain in the tree, just out of sight. The tricky bit is they are now hidden unless you access the MerkelTree. I noticed that Gab did expose a little more recently but haven’t looked at that.

You might like to look at how he’s using Registers (in the Folders API). From memory each directory is a register and he’s maintaining a separate history for each directory history (in a single RegisterCRDT per directory). Each file entry points to file metadata. Each directory entry points to the register for that subdirectory.

This means that the ‘history’ of each entry in a directory is separate though, and you can’t step through the history of the directory itself because the MerkleTree isn’t structured to do that. You would need to store state externally, if that would work IDK.

I came up with a design for doing this differently, but while that could store the whole history of a directory tree it had its own downsides. I’m not sure how useful history really is without that though. It’s a debate to be had, but probably best with examples to play with. I gave up on that area and switched to something else where there will be debate - hence my posts about web and NRS on this forum. But it will be better to have when there is something to play with.

[Traktion]

FWIW, from messing about with registers on my local test net:

println!("Adding entries for set1:");

let hash1 = register.write_merging_branches("entry1,val1".as_bytes()).expect("failed to add child");

for (entry) in register.merkle_reg().read().values() {
    let entry_str = String::from_utf8(entry.clone()).unwrap_or_else(|_| format!("{entry:?}"));
    println!("output set1: {entry_str}");
}

println!("write_atop new entries for set2:");

register.write_atop("entry2,val2".as_bytes(), &vec![hash1].into_iter().collect()).expect("failed to add child");
register.write_atop("entry3,val3".as_bytes(), &vec![hash1].into_iter().collect()).expect("failed to add child");
register.write_atop("entry4,val4".as_bytes(), &vec![hash1].into_iter().collect()).expect("failed to add child");
register.write_atop("entry5,val5".as_bytes(), &vec![hash1].into_iter().collect()).expect("failed to add child");

for (entry) in register.merkle_reg().read().values() {
    let entry_str = String::from_utf8(entry.clone()).unwrap_or_else(|_| format!("{entry:?}"));
    println!("output set2: {entry_str}");
}

println!("merging branches for set3");
register.write_merging_branches("entry6,val6".as_bytes()).expect("failed to replace register!");

for (entry) in register.merkle_reg().read().values() {
    let entry_str = String::from_utf8(entry.clone()).unwrap_or_else(|_| format!("{entry:?}"));
    println!("output set3: {entry_str}");
}

let entries = register.read();
println!("Register entries:");

// print all entries
for (entry) in entries.clone() {
    let (hash, bytes) = entry.clone();
    let data_str = String::from_utf8(bytes.clone()).unwrap_or_else(|_| format!("{bytes:?}"));
    println!("Entry - hash {hash}, data: {data_str}");
}

Outputs:

Adding entries for set1:
output set1: entry1,val1
write_atop new entries for set2:
output set2: entry4,val4
output set2: entry5,val5
output set2: entry2,val2
output set2: entry3,val3
merging branches for set3
output set3: entry6,val6
Register entries:
Entry - hash 373a8d.., data: entry6,val6

So, it looks like the ‘history’ is being removed due to branch merge. Again, I could be doing something wrong, but this feels like full write, rather than append only.

[happybeing]

I won’t be able to look at this for a while, but if you haven’t looked at the example in the SN readme which prints the structure it might help.

I’m not saying what you are doing isn’t destroying the history. If you are then that’s a kind of misuse. Merge itself shouldn’t do that. I can imagine that using the higher level APIs it looks like it though, because they don’t give access to the history, which is why I exposed the low level register.

Keep poking!

Off now, bacon calls!