Pinos ----- The idea is to make a DBus service where you can provide and consume media to/from. Some of the requirements are: - must be efficient for raw video using fd passing - must be able to provide media from any process - streaming media only (no seeking) - policy to restrict access to devices and streams Although an initial goal, the design is not limited to raw video only and should be able to handle compressed video and other streamable media as well. The design is in some part inspired by pulseaudio, hence its original name. We however are not concerned with playback of any of the media, this should be handled by a separate consumer rendering the media to a specific output device. DBus protocol ------------- The main daemon is registered on the session bus with name: org.pinos Various Source1 objects are registered in the server based on the available sources of content. Source1 has properties and has format descriptions of what it can provide. First a client needs to register with pinos by calling org.pinos.Daemon1.ConnectClient(). This creates a new Client1 object that the client must use for further communication. A client can then do org.pinos.Client1.CreateSourceOutput() to create a new SourceOutput1 to retrieve data from a source. It can specify a source explicitly or let the server choose a source. The client must provide a list of formats it can handle along with extra properties that can help with selecting an appropriate source. A client can then call org.pinos.SourceOutput1.Start() to negotiate the final media format and start the data transfer. A new fd is returned to the client along with the negotiated format and properties. All further media transport is then performed on the fd. The client will read from the fd to get data and metadata from the server. The wire format is generic and extensible and allows for inline serialized events such as property changes and format changes. fd management ------------- Pinos shares data between clients by using fd passing. Sometimes the memory referenced by the fd needs to be reused. It is important that all pinos clients lost their reference to the fd before it can be reused. What follows are some scenarios of how the lifecycle of the fds are managed. * server side v4l2src pinospay multisocketsink | | | | | | make buffer |--------->| | | | (1) | | |------------>| (2) -----> | | | | |<------------| | (4)| ............| (3) | | | |... | ... | | | | | |<------------| (5) <----- | (6)| | | | | (1) pinospay generates the pinos message for the v4l2 input buffer. It is assumed in the next steps that the payloader receives fd-memory from v4l2src and that the memory is only freed again when no clients are looking at the fd. (2) multisocketsink sends the buffer to N Pinos clients (3) for each client that is sent a buffer, multisocketsink sends an event with the client object and buffer in it. (4) pinospay uses the fdmanager object to map the fd-index that was sent, to the buffer and client. The buffer is reffed and kept alive for as long as the client is using the buffer. (5) when a message is received from a client, multisocketsink sends an event upstream. (6) pinospay parses the message and instructs the fdmanager to release the fd-index again. When all clients release the fd, the buffer will be unreffed and v4l2src can reuse the memory. * client consumer pinossrc xvimagesink | | -------> (1)|------------------->| (2) | | | (3) | |<-------------------| <------- (4)| | | | (1) pinossrc receives a buffer from Pinos and wraps the fd with data in an fd-memory. It remembers the fd-index as qdata on the memory. It has a special DestroyNotify on the qdata. (2) xvimagesink renders the buffer and frees the buffer. (3) freeing the buffer causes the qdata DestoyNotify to be called. (4) pinossrc constructs a release-fd message and sends it to Pinos * client producer videotestsrc pinossink | | (1)|------------------->| | | (2) -----> | | | (4) | (3) <----- |<-------------------| (1) videotestsrc produces a buffer (2) pinossink makes a PinosBuffer from the input buffer. It will also keep the buffer in a hash table indexed by the fd-index. (3) pinossink receives an fd-release message from Pinos. It removes the fd-index from the hashtable and the hashtable and the buffer is unreffed (4) the fd is returned to videotestsrc for reuse. * client producer, server side socketsrc pinospay multisocketsink | | | ------> (1) | (2)| | |----------->| | | |------------>| (3) --------> | | | .... | (5)|<------------| (4) | | ...... | | | | | (7)|<------------| (6) <-------- <------- (8)|<-----------| | | | | (1) pinos buffer arrives from a client. socketsrc wraps the fd (2) pinospay sets a weak-ref on the buffer to know when it is freed. (3) multisocketsink sends the buffer to the clients (4) for each buffer that is sent, an event is sent to the payloader (5) the payloader uses the fdmanager to map the fd-index to a buffer and a client. it keeps a ref on the buffer (6) release-fd is received from a client (7) pinospay removes the fd-index from the fdmanager. If all clients released the fd, the buffer will be freeds, triggering the DestroyNotify. This will then trigger an event with a release-fd message to the source. (8) the source sends the release-fd message to Pinos * client remove When a client disconnects from pinos, it must have released all fd-indexes that it received. Pinos will force a release and will reuse the fd-indexes when the client disconnects. Wire ---- Fixed header 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Version | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Version : 4 bytes : message version Length : 4 bytes : total message length Followed by 1 or more type-length-data sections 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Len ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : 1 byte : variable length, 7 bits, high bit is continuation marker : bytes, see below for contents based on Types: 1: continuation section 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Offset ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Size ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Rest of the commands can be found in the shared memory region at @offset and @size. A shared memory region is negotiated when the client connects to the server. : 8 bytes : offset : 8 bytes : size 2: header Header for payload 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | flags | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | seq | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PTS ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DTS-offset ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : 4 bytes : buffer flags : 4 bytes : sequence number : 8 bytes : presentation time : 8 bytes : dts-offset 3: fd-payload section Used to send a block of data between client and server. The type of fd and the possible operations on it are negotiated when the client connects. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | offset ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | size ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | fd-index | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : 4 bytes : id of the fd-payload : 8 bytes : offset : 8 bytes : size : 4 bytes : index of fd 4: release fd-payload Release a fd-payload with 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : 4 bytes : the id number of the released fd-payload 5: format change Perform an in-band format change. The following data blocks will be in this new format. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | id | format ..... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ...... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : 1 byte : format id : 0-terminated : contains serialized format 6: property changes Notify a property change. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | key .... .. | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ...... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | value .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ...... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : 0-terminated : key : 0-terminated : value ... : more key/values to fill length, 0 key or message length is end 7: refresh request Request a new refresh point 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | last-id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | request-type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PTS ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : the last seen id : the type of request 0 = keyframe 1 = keyframe+full headers : the timestamp when the refresh should be, 0 for as soon as possible