mirror of
https://gitlab.freedesktop.org/pipewire/pipewire.git
synced 2025-10-29 05:40:27 -04:00
3ace7e9648
Rework things so that we negotiate buffer pools beforehand and only pass buffer ids around We can then remove the refcount of buffers, events and commands. More work on buffer reuse Use the node state changes to trigger the next step in the configuration sequence. Move most of the client-node to a plugin Do buffer allocation in the port link.
456 lines
16 KiB
Text
456 lines
16 KiB
Text
Pinos
|
|
-----
|
|
|
|
The idea is to make a DBus service where you can provide
|
|
and consume media to/from.
|
|
|
|
Some of the requirements are:
|
|
|
|
- must be efficient for raw video using fd passing
|
|
- must be able to provide/consume/process media from any process
|
|
- streaming media only (no seeking)
|
|
- policy to restrict access to devices and streams
|
|
|
|
Although an initial goal, the design is not limited to raw video
|
|
only and should be able to handle compressed video and other
|
|
streamable media as well.
|
|
|
|
The design is in some part inspired by pulseaudio, hence its original
|
|
name. Increasinly we also seem to add functionality of jack.
|
|
|
|
|
|
Objects
|
|
-------
|
|
|
|
Daemon1: the main pinos daemon
|
|
/org/pinos/server
|
|
Client1: a connected client, the result object from call
|
|
Daemon1.ConnectClient
|
|
/org/pinos/client*
|
|
Device1: a physical device on the machine, devices can provide
|
|
processing nodes
|
|
/org/pinos/device*
|
|
Node1: a processing node, this can be a source, sink or transform
|
|
element.
|
|
/org/pinos/node*
|
|
Port1: a port on a Node1, ports can be input or output ports
|
|
/org/pinos/node*/port*
|
|
Channel1: a communication channel between a client and port
|
|
/org/pinos/client/channel*
|
|
Link1: a link between 2 ports
|
|
/org/pinos/link*
|
|
|
|
|
|
DBus protocol
|
|
-------------
|
|
|
|
The main daemon is registered on the session bus with name: org.pinos
|
|
|
|
Various Node1 objects are registered in the server based on the available
|
|
sources or sinks of content. Source1 has properties and has format descriptions of
|
|
what it can provide.
|
|
|
|
First a client needs to register with pinos by calling
|
|
org.pinos.Daemon1.ConnectClient(). This creates a new Client1 object that
|
|
the client must use for further communication.
|
|
|
|
A client can then do org.pinos.Client1.CreateChannel() to create a
|
|
new Channel to retrieve/send data from/to a node. It can specify a node/port
|
|
explicitly or let the server choose a port. The client must provide a list
|
|
of formats it can handle along with extra properties that can help with
|
|
selecting an appropriate port.
|
|
|
|
A client can then call org.pinos.Channel1.Start() to negotiate the final
|
|
media format and start the data transfer. A new fd is returned to the client
|
|
along with the negotiated format and properties.
|
|
|
|
All further media transport is then performed on the fd. The client will read
|
|
from the fd to get data and metadata from the server. The wire format is
|
|
generic and extensible and allows for inline serialized events such as
|
|
property changes and format changes.
|
|
|
|
|
|
fd management
|
|
-------------
|
|
|
|
Pinos shares data between clients by using fd passing. Sometimes the memory
|
|
referenced by the fd needs to be reused. It is important that all pinos
|
|
clients lost their reference to the fd before it can be reused.
|
|
|
|
What follows are some scenarios of how the lifecycle of the fds are
|
|
managed.
|
|
|
|
* server side
|
|
|
|
v4l2src pinossocketsink
|
|
| |
|
|
| |
|
|
make buffer |--------->|
|
|
| | (1)
|
|
| | (2) ----->
|
|
| | (3)
|
|
|... | ...
|
|
| |
|
|
| | (4) <-----
|
|
|<---------|
|
|
|
|
|
|
(1) pinossocketsink generates the pinos message from the v4l2 input
|
|
buffer. It is assumed in the next steps that the sink
|
|
receives fd-memory from v4l2src and that the memory is only
|
|
freed again when no clients are looking at the fd.
|
|
(2) pinossocketsink sends the buffer to N Pinos clients
|
|
(3) for each client that is sent a buffer, pinossocketsink uses the
|
|
fdmanager object to map the fd-index that was
|
|
sent, to the buffer and client. The buffer is reffed and kept
|
|
alive for as long as the client is using the buffer.
|
|
(4) when a message is received from a client, pinossocketsink
|
|
parses the message and instructs the fdmanager to release
|
|
the fd-index again. When all clients release the fd, the buffer
|
|
will be unreffed and v4l2src can reuse the memory.
|
|
|
|
* client consumer
|
|
|
|
pinossrc xvimagesink
|
|
| |
|
|
-------> (1)|------------------->| (2)
|
|
| |
|
|
| (3) |
|
|
|<-------------------|
|
|
<------- (4)| |
|
|
| |
|
|
|
|
|
|
(1) pinossrc receives a buffer from Pinos and wraps the fd with data
|
|
in an fd-memory. It remembers the fd-index as qdata on the memory.
|
|
It has a special DestroyNotify on the qdata.
|
|
(2) xvimagesink renders the buffer and frees the buffer.
|
|
(3) freeing the buffer causes the qdata DestoyNotify to be called.
|
|
(4) pinossrc constructs a release-fd message and sends it to Pinos
|
|
|
|
* client producer
|
|
|
|
|
|
videotestsrc pinossink
|
|
| |
|
|
(1)|------------------->|
|
|
| | (2) ----->
|
|
| |
|
|
| (4) | (3) <-----
|
|
|<-------------------|
|
|
|
|
|
|
(1) videotestsrc produces a buffer
|
|
(2) pinossink makes a PinosBuffer from the input buffer. It will also
|
|
keep the buffer in a hash table indexed by the fd-index.
|
|
(3) pinossink receives an fd-release message from Pinos. It removes
|
|
the fd-index from the hashtable and
|
|
the hashtable and the buffer is unreffed
|
|
(4) the fd is returned to videotestsrc for reuse.
|
|
|
|
|
|
* client producer, server side
|
|
|
|
socketsrc pinossocketsink
|
|
| |
|
|
------> (1) | (2)|
|
|
|----------->|
|
|
| | (3) -------->
|
|
| | ....
|
|
| (4)|
|
|
| | ....
|
|
| |
|
|
| (6)| (5) <--------
|
|
<------- (7)|<-----------|
|
|
| |
|
|
|
|
|
|
(1) pinos buffer arrives from a client. socketsrc wraps the
|
|
fd
|
|
(2) pinossocketsink sets a weak-ref on the buffer to know when it is
|
|
freed.
|
|
(3) pinossocketsink sends the buffer to the clients
|
|
(4) for each buffer that is sent, the sink uses the fdmanager to map the
|
|
fd-index to a buffer and a client. it keeps a ref on the buffer
|
|
(5) release-fd is received from a client
|
|
(6) pinossocketsink removes the fd-index from the fdmanager. If all
|
|
clients released the fd, the buffer will be freeds, triggering
|
|
the DestroyNotify. This will then trigger an event with a release-fd
|
|
message to the source.
|
|
(7) the source sends the release-fd message to Pinos
|
|
|
|
|
|
* client remove
|
|
|
|
When a client disconnects from pinos, it must have released all fd-indexes
|
|
that it received. Pinos will force a release and will reuse the fd-indexes
|
|
when the client disconnects.
|
|
|
|
|
|
|
|
Wire
|
|
----
|
|
|
|
Fixed header
|
|
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Version |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Flags |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Length |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
Version : 4 bytes : message version
|
|
Flags : 4 bytes : extra flags
|
|
Length : 4 bytes : total message length
|
|
|
|
|
|
Followed by 1 or more type-length-data sections
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Type | Len ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Data .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<type> : 1 byte
|
|
<length> : variable length, 7 bits, high bit is continuation marker
|
|
<data> : <length> bytes, see below for contents based on <type>
|
|
|
|
Types:
|
|
|
|
1: continuation section
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Offset ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Size ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
Rest of the commands can be found in the shared memory region at
|
|
@offset and @size. A shared memory region is negotiated when the client
|
|
connects to the server.
|
|
|
|
<offset> : 8 bytes : offset
|
|
<size> : 8 bytes : size
|
|
|
|
2: header
|
|
|
|
Header for payload
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| flags |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| seq |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| PTS ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| DTS-offset ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
|
|
<flags> : 4 bytes : buffer flags
|
|
<seq> : 4 bytes : sequence number
|
|
<pts> : 8 bytes : presentation time
|
|
<dts-offset> : 8 bytes : dts-offset
|
|
|
|
3: fd-payload section
|
|
|
|
Used to send a block of data between client and server. The type of fd and
|
|
the possible operations on it are negotiated when the client connects.
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| id |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| offset ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| size ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| fd-index |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<id> : 4 bytes : id of the fd-payload
|
|
<offset> : 8 bytes : offset
|
|
<size> : 8 bytes : size
|
|
<fd-index> : 4 bytes : index of fd
|
|
|
|
4: release fd-payload
|
|
|
|
Release a fd-payload with <id>
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| id |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<id> : 4 bytes : the id number of the released fd-payload
|
|
|
|
5: format change
|
|
|
|
Perform an in-band format change. The following data blocks will be in this
|
|
new format.
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| id | format ..... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| ...... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<format-id> : 1 byte : format id
|
|
<format> : 0-terminated : contains serialized format
|
|
|
|
6: property changes
|
|
|
|
Notify a property change.
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| key .... .. |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| ...... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| value .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| ...... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<key> : 0-terminated : key
|
|
<value> : 0-terminated : value
|
|
... : more key/values to fill length, 0 key or
|
|
message length is end
|
|
|
|
|
|
7: refresh request
|
|
|
|
Request a new refresh point
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| last-id |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| request-type |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| PTS ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<last-id> : the last seen id
|
|
<request-type> : the type of request
|
|
0 = keyframe
|
|
1 = keyframe+full headers
|
|
<PTS> : the timestamp when the refresh should be,
|
|
0 for as soon as possible
|
|
|
|
|
|
|
|
communication channel
|
|
---------------------
|
|
|
|
|
|
+-----+ +----+ +----+
|
|
| | | S | | |
|
|
| ----- ----- |
|
|
+-----+ +----+ +----+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+----+
|
|
| |
|
|
| C |
|
|
+----+
|
|
|
|
|
|
1) Update config C->S INIT
|
|
|
|
node-update
|
|
port-update
|
|
state change CONFIGURE
|
|
|
|
2) Set formats S->C CONFIGURE
|
|
|
|
set-property
|
|
enumerate ports
|
|
add-port
|
|
remove-port
|
|
enumerate formats
|
|
set-format
|
|
|
|
3) Buffer requirements update C->S
|
|
|
|
Update port status
|
|
state change READY if enough formats are set
|
|
|
|
4) Start S->C READY
|
|
|
|
read port memory requirements
|
|
add_mem
|
|
remove_mem
|
|
add_buffer
|
|
remove_buffer
|
|
command START/PAUSE
|
|
|
|
5) Pause S->C PAUSED
|
|
|
|
state change STREAMING
|
|
set-format to NULL -> state change to CONFIGURE
|
|
|
|
5) data transfer C->S STREAMING
|
|
|
|
need-input
|
|
have-output
|
|
|
|
process_buffer
|
|
reuse_buffer
|
|
state change PAUSED
|
|
|
|
6) data transfer S->C
|
|
|
|
process_buffer
|
|
reuse_buffer
|
|
|
|
7) format change C->S
|
|
|
|
port-update
|
|
state change CONFIGURE
|
|
|
|
8) format change S->C
|
|
|
|
Send set-format change on ports -> READY if new memory requirements
|
|
-> PAUSED/STREAMING if all ok
|
|
|
|
9) format-change to NULL
|
|
|
|
state change CONFIGURE
|
|
|
|
10) ERROR
|