How to use safe_client_libs in python?


#21

okay - minor update!

i added a function modifying FfiResults (…seriously… the toughest part was to find out how to make rust create a c-string in the correct form and assigning it the right way …and getting the pointer out of rust … with a lot of attempts i always only got a NUL-pointer … super-annoying -.- …)


creating the definition in python

image

and after assigning some initial values

calling the function and inspecting the result

=) so we’re getting closer i guess … :slight_smile:

(sorry if this is getting annoying … but since i’m learning here and it’s more one of the slower learning experiences i had so far in my life … i thought i might share the stuff i find out :smiley: maybe it helps someone else later on :wink: )

now i need to find out how those void (*o_cb)(… , … , …) stuff is supposed to work … and then we might approach the fun part of this adventure :slight_smile:


#22

Hi - great start on that! :slight_smile: I’ll try to help as much as I can:

Correct, functions don’t return values as usual – but they don’t modify the provided structures either. safe_authenticator was coded in that way first but after lots of iterations we have came to the current way of returning values through callbacks – which are that void (*o_cb)(...) thing you’re seeing in each and every FFI function. This approach turned out to be quite robust and doesn’t require you to manage memory manually. And it has an extra bonus of handling asynchronous calls automatically.

Yes, there is a difference between Rust strings (which are actually Vec<u8>s) and null-terminated C strings which can’t contain null bytes and don’t carry information about the string length with them – as opposed to Rust strings. But both can be converted, and we use C string for our FFI APIs exclusively.

To convert Rust string into a C string, you can use CString: its usage is not complicated and you can find some examples in the SAFE Client Libs code.

In this case you can also use CString, it will allocate the necessary memory and it will guarantee that the resulting string will be C-compatible.

I think that will be the hardest part as you will need to deal with data copying and asynchronous callbacks. I am not completely sure which is the idiomatic way to code that in Python, but from what I know starting from Python 3.5 there is a native support for asynchronous calls and await – so you can write more streamlined code:

r1 = await call1(a, b, c)
r2 = await call2(r1)
print(r2)

instead of something like

call1(a, b, c, callback1)

def callback1(r1):
  call2(r1, callback2)

def callback2(r2):
  print(r2)

Once you figure out the callbacks and data copying, it becomes a lot easier – you can automatically generate the language bindings, in the very same way as we do with C# and Java.

If you have more questions or something’s blocking you from experimenting, don’t heistate to ask! :slight_smile:


#23

ah Thanks!

(to be honest i already looked into this 2 times in the past … but now decided that i can’t do it ‘intuitively’ but need to investigate how to do it step by step … thats why i defined my own additional test functions and started with the really really small stuff :innocent: )

ohoh - i was afraid there might be something hiding :smiley: cb did indeed sound fishy from the start xD
but at least now the syntax makes sense … so those parameters are the callback functions themselves with the specified parameters … :slight_smile:

anyway! bring it on! :smile: next station callbacks from rust/c/ xD … and people will need to use python 3.5 then =D

…give me some time to dig into this … i’ll post an update once i know enough to ask qualified questions :stuck_out_tongue_winking_eye:


#24

haha @nbaksalyar

i need to check “if that’s the right way to do it” (at least roughly) but

i added a simple callback-function

image

and to python what i thought it might do the trick if i’m lucky xD

TaDAAAAA xD

python is just awesome :heart: :heart_eyes:

i’ll test some stuff with strings later too :thinking:

Ps: seriously… I didn’t expect it to just work :v::hugs:


#25

hmhmm - okay … you can even embed the the decorated function and everything inside of an object …

that would make using Safe from python feel like nothing special at all but pretty straight forward :thinking:

…but i’m feeling like that might be considered pretty bad programming style … i do evil stuff when programming all the time xD … but maybe someone else (more experienced and more professional than me) wants to speak up and tell me what would be a better way to do it …? (if nobody tells me to stop i will just go with the black magic because it’s pretty awesome to use i think :smiley: …and it’s easy for me to implement xD)

ps:

and here we go with a callback making use of a struct including a string


…now this only needs to be brought into a nicer form and the structs created and everything … but i guess this won’t happen tonight (at least it won’t be me tonight ; )

thank you so much @nbaksalyar ! I’ll try to find the time to push this forward! but i don’t really see what would be stopping me now :smiley:


#26

This looks nice - great idea with local functions! :slight_smile: Please share your further progress.

I believe there is another option to go for with Python: Client Libs can be embedded as a CPython extension rather than using FFI libs, but it has its own disadvantages – like confining to a particular Python implementation (and there are others like PyPy) and needing to compile the extension in the first place. But theoretically it should be a bit faster.


#27

=) glad you liked it - and good idea with the c extension :blush: but given that there is a project coming up at my jop really really soon that will probably require a lot of hours spent at work… I would try to create a first working interface to safe now as fast as possible with the ffi libs and try to resume to this topic then again in May when I will have some spare time again…

… At least that’s the plan in my mind for now


#28

Ah sure - I’ve just been thinking out loud :slight_smile: Eventually we might need to integrate support for Python into the binding generator, so your experiments are really helpful and valuable! And given your interest and progress, it might be integrated sooner than later.


#29

okay :smiley: i think i was a little bit too fast in celebrating xD you do have an impressive amount of structs defined in your code and when i try to extract those from the header files they are not in the right order - so they need to get ordered first :innocent:

and i thought a little bit about the binding generator thing
for the declarations with cffi i literally do need ‘standard c declaration’ stuff like

typedef unsigned char uint_fast8_t;
typedef unsigned long int uint_fast16_t;
typedef unsigned long int uint_fast32_t;
typedef unsigned long int uint_fast64_t;
typedef struct PermissionSet {
 _Bool read;
 _Bool insert;
 _Bool update;
 _Bool delete;
 _Bool manage_permissions;
} PermissionSet;
typedef struct ContainerPermissions {
 char* cont_name;
 PermissionSet access;
} ContainerPermissions;
typedef struct AppExchangeInfo {
 char* id;
 char* scope;
 char* name;
 char* vendor;
} AppExchangeInfo;
typedef struct AuthReq {
 AppExchangeInfo app;
 _Bool app_container;
 ContainerPermissions const* containers;
 uintptr_t containers_len;
 uintptr_t containers_cap;
} AuthReq;
typedef struct ContainersReq {
 AppExchangeInfo app;
 ContainerPermissions const* containers;
 uintptr_t containers_len;
 uintptr_t containers_cap;
} ContainersReq;

which is (close to) exactly what i get when running the gcc-compiler and redirect the output into a string :wink:

so with the ffi-util-structs getting exported to the c headers (but seriously: no pressure :wink: ) [and some automated reordering because cffi needs everything to be defined in the right order] we would automatically have the necessary cffi declarations then :thinking:

Yes it will still be a lot of work to create the representations of those data types for calling the functions later on …but I think with those declarations we’d have all we need

…i sometimes think a bit too complicated … but i’ll just let it all out since it’s just us here xD …
we can use some regular expression magic to extract and print out the definitions for the callback-decorators for the functions we want to call:

and extract the info necessary to construct all data types (if i’m not mistaken there are ordered dicts too in the newer versions of python … so when using those there shouldn’t be surprises)

and in the end just generate some initialization-frame-code

not everything would be generated automatically … but we’d have a pretty neat wireframe to fill with life afterwards (and if the variable names of the inner variables would be printed as comments too then initializing everything shouldn’t be too hard i guess) … would be easier than trying to really fully automate everything here and at the same time wouldn’t be too much additional workload for you :blush::hugs: (and to be honest … besides from trying to exactly those extractions in the bindings-generator i wouldn’t know what more you could do than to provide the c declarations :thinking: )

[yeah - it’s all a bit messy code … but i guess with some improvements (considering pointers) it would nearly be what would help a lot in creating something bindingy …]


#30

okay! change in plans!

I suddendly realized that it might be easier to start with an authenticator to develop the right way to communicate with SAFE because it’s a bit simpler than the app library (plus we can easily check there if locally defined callback functions keep working for all eternity or loose their validity at some point somehow by sending authentication requests from apps =) )

so i faked to know how core_tx and _core_joiner need to look like

And just went on with the callback functions as extracted with the code I showed above

After switching then to a mock version of the libs and realizing that in case of success I simply don’t get a description for the error (I suddenly got errors instead of result descriptions =/ )

I seem to have managed to make ‘first contact to safe’ with Python =) (though only on mock yet… But my internet died literally 10 minutes ago then and I’m just shortly writing this through my mobile because I’m so happy that this seems to be the way to go forward =D )


#31

I do have a question :smiley:

@nbaksalyar or anybody who happens to have an opinion on that matter …
…in the launcher version of the api i suggested to create a wonderful and slightly huge Safe-Class with initially no methods …

and then add the methods through monkey-patching into the class

like this you can add some documentation/explanation/examples in between of the declarations and it all looks nicer when looking at it in a jupyter notebook …

but of course it looks horrible if you convert the jupyter notebook into a plain python document and look at that …

do you think one super-large class definition with all the methods and with documentation somewhere else would be ‘nicer’ or a more distributed definition with included docs …?

short visualization:

wireframe generated from the function definition with included empty callback-functions

split into cells with some annotations

ps: yes the in-method-generation of c-variables and initialization with corresponding parameters are missing yet but one step at a time :wink: … i’m not entirely sure how exactly i would want to do this … so i thought i’d first look into this one here …

pps: damn - i forgot the self … anyway … another day :smiley:


#32

okay … i just played around a bit not trying to automate stuff … because i thought it might help to know what exactly i want to do before automating it any further …

my Idea would now be to split it up into larger chunks:

1.) the Data Types (and utils for that)

2.) the Authenticator / App Classes

and when using it you can do:

as always i just pushed everything to github - all you need to do to play around with the notebook is compiling yourself a version of the safe_client_libs and change the path when creating the Authenticator (obviously you need jupyter notebook installed on your pc + python 3 would help - i’m using 3.5 right now - for windows i would recommend the winPython suite / for linux i’m a huge fan of using anaconda environments … makes a lot of things easier =) - both of those options come including jupyter and should just work out of the box [anaconda after setting up a new environment with ‘conda create -n py35 python=3.5 anaconda’ … or something like that :smiley: ])

ps: soooo - and now i’m a bit on easter break xD … just wanted to push those last thoughts forward before vanishing for some days :smiley: … thinking this fully through and looking a little bit closer to where something is global and where local … will come later :wink:

pps: and if the datatypes-classes use the other data type classes for defining their inner variables even (not too large scale) changes of the structs in the libs don’t come with major changes in the python code, because they then just change automatically API-wide …


#33

Okay - obviously I have a hard time to just let go of this and not think about it xD

So here we go again - the user of the library obviously would want to add custom callback actions that interact with a user interface or just trigger other actions (if it’s a store to act on a new order coming in or updating the chat window or just updating a database / appending new data to the buffer… Etc etc…)

So every method that gets defined would need to come with one additional function for each callback that gets called in the locally defined callback action

Simple visualization of the process with additional setting method

Or by simply overwriting the local reference to the method from outside without an additional method definition… Not sure if that additional method really makes sense and has any benefits …

So by using the safe data class and assigning customized actions to the callbacks you can easily interact with safe then and it should be super easy to create well performing apps :slight_smile:


#34

Just to make life more complicated :wink: I note that Python also supports Promises, which is an alternative to callbacks for asynchronous function support.

https://www.qwant.com/?q=does+python+support+promises%3F


#35

Haha - thx! :hugs: I’ll look into it and think about how/if that can come in handy =)

Feedback and additional thoughts are always welcome :slight_smile:

Hmhmm or maybe it would be better to hand over the user defined callback action(s) when calling the method… So it wouldn’t be a modification of the safe data class but just a parameter that is connected to the method… Might be a better solution :thinking:

Ps: Japp I think that’s better - then there would be a default action for feedback if you don’t provide a customized one and if you do it gets linked and called on every callback later on - that comes with the benefit of not needing to pre-define additional methods and initializing everything… Basically then the method call looks exactly like the one from the api again xD… Leaving out the callback methods in the first place seems to have been short sighted by me…

To be honest the promise thing does look like it comes with more overhead @happybeing so I think I would go with the callbacks for now as they seem to be the simplest solution to me and should be fairly easy to auto-generate for all methods exposed by the api

Thanks for the heads up nonetheless!


#36

Okay - another status report

the general format of the methods I would suggest is like this:

I would write a script for generating this output for functions automatically from the declaration that comes from the gcc compiler output

Handed over numbers/strings get converted automatically into the corresponding c data types and then handed over to the api; structures need to be passed to the method as correct c data type (which shouldn’t be a problem because callback function delivers the relevant structures as outputs) and the doc string comes with information about the parameters and return values
(so at least when using ipython/jupyter Notebooks/an ide you can exactly see what is the necessary input and what will be the return data types you can expect for the callback functions)

Together with generated definitions for ‘empty’ entities of all structs that are known to the api you can look into them and validate your callback function behaviour before handing them to the api

I already implemented a first filtering/reordering of the gcc output from the c headers and generation of the corresponding Python class definitions in the file ‘SafeDataTypes_Concept_parser.ipynb’
(I don’t think it nicely designed or efficient but I think it might do the job and save me a lot of time after you did the hard work of creating fully defined header files that include all important data types =D )

There would be ‘datatypes’, ‘App’ and ‘Authenticator’ as main modules for pySafe

Thoughts on the concept or suggestions for improvements …?


ps: minor addition … i could add an additional parameter that defaults to ‘utf-8’ for the used encoding of data in case of strings … afaik python3 uses utf-8 by default …so as long as you don’t leave the python ecosystem you won’t have a problem … but the bytestring you end up with can be different for each encoding for one and the same string …

image

so if you ‘created your account in a different encoding’ you wouldn’t be able to log into it providing your login credetials with pySafe :thinking: (except of course by encoding it by hand into the correct bytes and providing those) … i think that option should be added from start …

pps: @nbaksalyar out of curiosity - do we happen to know which exact encoding the authenticator in peruse uses …? (not that it would have been of great interest to anybody not trying to create his own authenticator …)
[is it possible the self built client libs of the current master branch are not compatible with the alpha2 network? - because i get an ‘Core error: Unexpected: Could not connect to the SAFE Network’ error when trying to log into my account’]


#37

would it make sense to call the callback-functions together with a self?

image

if you do it like this one can access properties of the Authenticator-Class with self-defined callback-functions :thinking:

downside is that it’s an additional parameter again that needs to be included in every function you write and hand over …
(but without it you won’t be able to do self.AppState = newAppState in a callback-function … which could be very annoying in some situations … )

…not easy to find the right balance between modularity and simplicity of this thing …

(i guess if nobody has an argument against it we’ll start out with the self and get rid of it long term if it is just not needed and annoying)


#38

Hi @riddim, really great progress! Very pleased to see this shaping up :slight_smile:

@bochaco or @joshuef might correct me on this, but as far as I know the front-end defaults to UTF-8.

To my knowledge, Python 3 doesn’t use UTF-8 internally but is based on Unicode which is a bit different beast. UTF-8 or UTF-16 is just a binary representation or encoding of Unicode, but you are free to encode it in ASCII if it allows to. The actual binary encoding in the Python interpreter should not be important, it is an implementation detail that should not be relied upon, so all you need to consider is that you can encode it in different ways. And I would recommend to use UTF-8, it is a widely-adopted standard. :slight_smile:

The master branch is always compatible with alpha2, only dev is not guaranteed to be working with the current testnet.

The most probable cause for the error you’re seeing is a missing Crust configuration file. You can try to get one from the SAFE Browser package (it is named as safe_browser.crust.config) and put it along with the safe_authenticator.dll and safe_app.dll (or safe_app.so/safe_app.dylib, depending on your OS of choice). You can use this function to find the expected config file name base. E.g. if it returns binary_name, the config file has to be named as binary_name.crust.config.


#39

Damn - now you say it I’m pretty sure you are right xD

Kk - sounds reasonable

! Aaaaah thank you!

… I’m not sure when I will find the time to test it… But I’ll report back when I did!


#40

@riddim, It seems the strings passed to the authenticator libs are standard issue JS strings… What does that mean? I’ve been digging about to see (there’s no clear answer though… yay js!) and, while there’s a want to move to UTF-8 in the node world, most javascript engines, use UTF-16. (@bochaco correct me if I’m wrong here!) Including the v8 engine which is what chrome/electron are built upon.

So I think that’s what we’re passing right now.

You and @nbaksalyar raise a fine point though. It does appear that UTF-8 is more standardised, so perhaps we should be converting/passing this down to the authenticator for account creation.