At a recent meetup attendees were asking about how to work with real-time data and streaming.
I thought I read about a potential solution for MutableData structures to emit events when data is updated and clients would be able to listen for those events.
Is there an RFC for this?
I’m searching the forums but can’t find where I read that. Will someone point me in the right direction?
Maybe the mpid messaging system will be useful for this?
The MPID system appears to focus on client pair communications which may not scale data streams very well.
I’m wondering if a vault acting as a manager could coordinate a data stream to requesting clients.
Or in the case of a chat room where a client application needs real time data events, perhaps a client could directly listen to an MD for data update events or listen to a manager vault that emits events when it’s managed data is changed.
But these are not my ideas. I read them somewhere, just can’t find it again.
Streaming is an interesting one. Data rates can change as streaming occurs, for instance the source supplying the stream has their link incapable of maintaining the bit rate required for maximum quality, then it has to downgrade the data rate and thus the amount of data per second uploaded.
If there is one way streaming source -->> audience (multiple) then you can use simple reading and only poll if client is slightly faster or fast forwarding.
Lets say the stream is 4 Mbits/sec then that is 2 seconds per MD. Thus the listeners only need to be 4 seconds behind the source to allow double buffering. The source is filling up MDs while the clients are requesting MDs. If they are playing exactly in sync then no issue, if clients are slightly slower then no problems, and if the clients are slightly faster then every so often the client will “buffer” as happens now on the internet.
If the stream is being saved forever then replace the MDs with immutable chunks
And if the stream is not being saved then you could actually round robin the MDs thus saving on data storage in the network. The number of MDs in the round robin determines how far back a client can go. In the above example 1800 MDs allows 1 hour.
Real time 2 way streaming can be done by simply reducing the quantity of data being stored per write using a field for each block of streaming data (say 50 milli secs worth). So then its similar to above except instead of reading whole MDs it is reading the fields. Each time it moves onto a new MD it reads all the available fields into the client buffer then at an appropriate time it then reads more fields. And it simply keeps going.
The only time actual polling would occur is when the source does not write data (break in transmission) and the client polls waiting for new data.