mirrors/pipewire
1
0
Fork 0
mirror of https://gitlab.freedesktop.org/pipewire/pipewire.git synced 2025-12-23 08:56:47 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

doc: reorganize files

Browse source 
Separate various autogen files from the documentation .dox files.
Rename .dox files to match the intended tree structure.
This commit is contained in:
Pauli Virtanen 2023-11-18 15:13:05 +02:00 • committed by Wim Taymans
parent eca773fc12
commit 77fad4ee13
41 changed files with 60 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

126
doc/dox/internals/access.dox Normal file
View file

@ -0,0 +1,126 @@
/** \page page_access Access Control
This document explains how access control is designed and implemented.
PipeWire implements per client permissions on the objects in the graph.
Permissions include `R` (read), `W` (write), `X` (execute) and `M` (metadata).
- `R`: An object with permission `R` is visible to the client. The client will receive
registry events for the object and can interact with it.
- `W`: An object with permission `W` can be modified. This is usually done
through a method that modifies the state of the object. The `W` permission
usually implies the `X` permission.
- `X`: An object with permission `X` allows invoking methods on the object.
Some of those methods will only query state, others will modify the object.
As said above, modifying the object through one of these methods requires
the `W` permission.
- `M`: An object with `M` permission can be used as the subject in metadata.
Clients with all permissions set are referred to as "ALL" in the
documentation.
# Use Cases
## New Clients Need Their Permissions Configured
A new client is not allowed to communicate with the PipeWire daemon until
it has been configured with permissions.
## Flatpaks Can't Modify Other Stream/Device Volumes
An application running as Flatpak should not be able to modify the state of
certain objects. Permissions of the relevant PipeWire objects should not have
the `W` permission to avoid this.
## Flatpaks Can't Move Other Streams To Different Devices
Streams are moved to another device by setting the `target.node` metadata
on the node ID. By not setting the `M` bit on the other objects, this can be
avoided.
## Application Should Be Restricted In What They Can See
In general, applications should only be able to see the objects that they are
allowed to see. For example, a web browser that was given access to a camera
should not be able to see (and thus receive input data from) audio devices.
## "Manager" Applications Require Full Access
Some applications require full access to the PipeWire graph, including
moving streams between nodes (by setting metadata) and modifying properties
(eg. volume). These applications must work even when running as Flatpak.
# Design
## The PipeWire Daemon
Immediately after a new client connects to the PipeWire daemon and updates
its properties, the client will be registered and made visible to other
clients.
The PipeWire core will emit a `check_access` event in the \ref pw_context_events
context for the the new client. The implementer of this event is responsible
for assigning permissions to the client.
Clients with permission `R` on the core object can continue communicating
with the daemon. Clients without permission `R` on the core are suspended
and are not able to send more messages.
A suspended client can only resume processing after some other client
sets the core permissions to `R`. This other client is usually a session
manager, see e.g. \ref page_session_manager.
## The PipeWire Access Module
The \ref page_module_access hooks into the `check_access` event that is
emitted when a new client is registered. The module checks the permissions of
the client and stores those in the \ref PW_KEY_ACCESS
property on the client object. If this property is already set, the access
module does nothing.
If the property is not set it will go through a set of checks to determine
the permissions for a client. See the \ref page_module_access documentation
for details, particularly on the values documented below. Depending on the
value of the \ref PW_KEY_ACCESS property one the following happens:
- `"allowed"`, `"unrestricted"`: ALL permissions are set on the core
object and the client will be able to resume.
- `"restricted"`, `"flatpak"`, `"$access.force"`: No permissions are set on
the core object and the client will be suspended.
- `"rejected"`: An error is sent to the client and the client is
suspended.
As detailed above, the client may be suspended. In that case the session
manager or another client is required to configure permissions on the object
for it to resume.
## The Session Manager
The session manager listens for new clients to appear. It will use the
\ref PW_KEY_ACCESS property to determine what to do.
For clients that are suspended with `"restricted"`, `"flatpak"` or
`"$access.force"` access, the session manager needs to set permissions on the
client for the various PipeWire objects in the graph that it is allowed to
interact with. To resume a client, the session manager needs to set
permission `R` on the core object for the client.
Permissions of objects for a client can be changed at any time by the
session manager. Removing the client core permission `R` will suspend the
client.
The session manager needs to do additional checks to determine if the
manager permissions can be given to the particular client and then
configure ALL permissions on the client. Possible checks include
permission store checks or ask the user if the application is allowed
full access.
Manager applications (ie. applications that need to modify the graph) will
set the \ref PW_KEY_MEDIA_CATEGORY property in the client object to "Manager".
For details on the pipewire-media-session implementation of access control,
see \ref page_media_session.
*/

127
doc/dox/internals/audio.dox Normal file
View file

@ -0,0 +1,127 @@
/** \page page_audio Audio
This document explains how Audio is implemented.
# Use Cases
## Audio Devices Are Made Available As Processing Nodes/Ports
Applications need to be able to see a port for each stream of an
audio device.
## Audio Devices Can Be Plugged and Unplugged
When devices are plugged and unplugged the associated nodes/ports
need to be created and removed.
## Audio Port In Canonical Format
It must be possible to make individual audio channels available
as a single mono stream with a fixed format and samplerate.
This makes it possible to link any of the audio ports together
without doing conversions.
## Applications Can Connect To Audio Devices
Applications can create ports that can connect to the audio ports
so that data can be provided to or consumed from them.
It should be possible to automatically connect an application to
a sink/source when it requests this.
## Default Audio Sink and Sources
It should be possible to mark a source or sink as the default source
and sink so that applications are routed to them by default.
It should be possible to change the default audio sink/source.
## Application Should Be Able To Move Between Sinks/Sources
It should be possible to move an application from one device to
another dynamically.
## Exclusive Access
Application should be able to connect to a device in exclusive mode.
This allows the application to negotiate a specific format with the
device such as a compressed format.
Exclusive access means that only one application can access the device
because mixing is in general not possible when negotiating
compressed formats.
# Design
## SPA
Audio devices are implemented with an \ref spa_device "SPA Device" object.
This object is then responsible for controlling the \ref spa_node "SPA Nodes" that
provide the audio ports to interface with the device.
The nodes operate on the native audio formats supported by the device.
This includes the sample rate as well as the number of channels and
the audio format.
## Audio Adapter
An SPA Node called the "adapter" is usually used with the SPA device node as
the internal node.
The function of the adapter is to convert the device native format to
the required external format. This can include format or samplerate
conversion but also channel remixing/remapping.
The audio adapter is also responsible for exposing the audio channels
as separate mono ports. This is called the DSP setup.
The audio adapter can also be configured in passthrough mode when it
will not do any conversions but simply pass through the port information
of the internal node. This can be used to implement exclusive access.
Setup of the different configurations of the adapter can be done with
the PortConfig parameter.
## The Session Manager
The session manager is responsible for creating nodes and ports for
the various audio devices. It will need to wrap them into an audio
adapter so that the specific configuration of the node can be
decided by the policy mode.
The session manager should create name and description for the
devices and nodes.
The session manager is responsible for assigning priorities to the
nodes. At least \ref PW_KEY_PRIORITY_SESSION and \ref PW_KEY_PRIORITY_DRIVER
need to be set.
The session manager might need to work with other services to gain
exclusive access to the device (eg. DBus).
# Implementation
## PipeWire Media Session (alsa-monitor)
PipeWire media session uses the \ref SPA_NAME_API_ALSA_ENUM_UDEV plugin
for enumerating the ALSA devices. For each device it does:
- Try to acquire the DBus device reservation object to gain exclusive
access to the device.
- Create an SPA device instance for the device and monitor this device instance.
- For each node created by the device, create an adapter with
an ALSA PCM node in the context of the PipeWire daemon.
The session manager will also create suitable names and descriptions
for the devices and nodes that it creates as well as assign session
and driver priorities.
The session manager has the option to add extra properties on the
devices and nodes that it creates to control their behavior. This
is implemented with match rules.
*/

177
doc/dox/internals/daemon.dox Normal file
View file

@ -0,0 +1,177 @@
/** \page page_daemon PipeWire Daemon
The PipeWire daemon is the central process that manages data exchange between
devices and clients.
Typically general, users run one PipeWire daemon that listens for incoming
connections and manages devices. Clients (including the \ref
page_session_manager) are separate processes that talk to the daemon using the
PipeWire socket (default: `$XDG_RUNTIME_DIR/pipewire-0`). This approach
provides address-space separation between the privileged daemon and
non-privileged clients.
\dot
digraph pw {
compound=true;
node [shape="box"];
subgraph cluster_pw {
rankdir="TB";
label="PipeWire daemon";
style="dashed";
subgraph cluster_prot_native {
label="pipewire-module-protocol-native";
style="solid";
socket [label="$XDG_RUNTIME_DIR/pipewire-0"];
mod_impl [label="module implementation"];
socket -> mod_impl;
}
core [label="PipeWire Core"];
alsa [label="PipeWire ALSA support"];
mod_impl -> core;
core -> alsa;
}
kernel
client1 [ label="Media Player" ];
client2 [ label="Audio Software" ];
sm [ label="Session Manager", style="dotted" ];
client1 -> socket;
client2 -> socket;
sm -> socket;
alsa -> kernel;
}
\enddot
As shown above, the protocol is handled by the \ref
page_module_protocol_native. From PipeWire's point-of-view this module is just
another module.
# Configuration Files
On startup, the daemon reads a configuration file to configure itself.
It executes a series of commands listed in the config file. The lookup order
for configuration files are:
- `$XDG_CONFIG_HOME/pipewire/pipewire.conf` (usually `$HOME/.config/pipewire/pipewire.conf`)
- `$sysconfdir/pipewire/pipewire.conf` (usually `/etc/pipewire/pipewire.conf`)
- `$datadir/pipewire/pipewire.conf` (usually `/usr/share/pipewire/pipewire.conf`)
The first configuration file found is loaded as the base configuration.
Next, configuration sections (from files ending with a .conf extension) are collected
in the directories in this order:
- `$datadir/pipewire/pipewire.conf.d/` (usually `/usr/share/pipewire/pipewire.conf.d/`)
- `$sysconfdir/pipewire/pipewire.conf.d/` (usually `/etc/pipewire/pipewire.conf.d/`)
- `$XDG_CONFIG_HOME/pipewire/pipewire.conf.d/` (usually `$HOME/.config/pipewire/pipewire.conf.d/`)
They are applied to the global configuration file. Properties are overwritten
and array elements are appended. This makes it possible to make small custom customizations
or additions to the main configuration file.
The environment variables `PIPEWIRE_CONFIG_DIR`, `PIPEWIRE_CONFIG_PREFIX`,
and `PIPEWIRE_CONFIG_NAME`. Can be used to specify an alternative configuration
directory, subdirectory, and filename respectively.
## Configuration File Format
PipeWire's configuration file format is JSON. In addition to true JSON
PipeWire also understands a more compact JSON representation. Where
`"` can be omitted around strings, no trailing commas are required and
`:` or `=` can be used to separate object keys from their values.
Also, `#` can be used to start a comment until the end of the line.
The configuration file format is grouped into sections. A section is
either a dictionary (`{}`) or an array (`[]`). Dictionary and array entries
are separated by whitespace and may be simple value assignment, an array or
a dictionary. For example:
```
# A dictionary section
context.properties = {
# Keys often have a dot notation
core.daemon = true
}
# An array section containing three dictionary objects
context.modules = [
# a dictionary object with one key assigned to a string
{ name = libpipewire-module-protocol-native }
{ name = libpipewire-module-profiler }
# a dictionary object with two keys, one assigned to a string
# the other one to an array of strings
{ name = libpipewire-module-portal
flags = [ ifexists nofail ]
}
]
```
Allowed configuration file sections are:
- **context.properties** (dictionary): These properties configure the
pipewire instance.
- **context.spa-libs** (dictionary): Maps plugin features with globs to a
spa library.
- **context.modules** (array): Each entry in the array is a dictionary with
the name of the module to load, including optional args and flags. Most
modules support being loaded multiple times.
- **context.objects** (array): Each entry in the array is a dictionary con‐
taining the factory to create an object from and optional extra argu‐
ments specific to that factory.
- **context.exec** (array): Each entry in the array is dictionary containing
the path of a program to execute on startup and optional args. This ar‐
ray usually contains an entry to start the session manager.
# Logging
The `PIPEWIRE_DEBUG` environment variable can be used to enable
more debugging. This variable supports the following format:
- `PIPEWIRE_DEBUG=[<level>][,<glob1>:<level1>][,<glob2>:<level2>,...]` where the globs are
shell globs to match on log topics and the levels are the respective
log level to set for that topic. Globs are applied in order and a matching
glob overrides an earlier glob for that category. A level without a glob
prefix will set the global log level and is a more preformant version of
`*:<level>`. For example, `PIPEWIRE_DEBUG=E,mod.*:D,mod.foo:X` enables global error messages,
debugging on all modules but no messages on the foo module.
- `<level>` specifies the log level:
+ `X` or `0`: No logging is enabled.
+ `E` or `1`: Error logging is enabled.
+ `W` or `2`: Warnings are enabled.
+ `I` or `3`: Informational messages are enabled.
+ `D` or `4`: Debug messages are enabled.
+ `T` or `5`: Trace messages are enabled. These messages can be logged
from the realtime threads.
PipeWire uses a `category.topic` naming scheme, with the following categories:
- `pw.*`: PipeWire internal topics.
- `mod.*`: Module topics, for example `mod.foo` would usually refer to the
`foo` module.
- `ms.*`: Media session topics.
- `ms.mod.*`: Media session modules, for example `ms.foo` would usually refer
to the `media-session-foo` module.
- `conn.*`: Connection specific topics such as printing raw messages sent over
a communication socket. These are in a separate namespace as they are
usually vastly more verbose than the normal debugging topics.
This namespace must be explicitly enabled with a `conn.<glob>` glob.
The behavior of the logging can be further controlled with the following
environment variables:
- `PIPEWIRE_LOG_SYSTEMD=false`: Disable logging to the systemd journal.
- `PIPEWIRE_LOG=<filename>`: Redirect the log to the given filename.
- `PIPEWIRE_LOG_LINE=false`: Don't log filename, function, and source code line.
- `PIPEWIRE_LOG_COLOR=true/false/force`: Enable/disable color logging, and optionally force
colors even when logging to a file.
*/

70
doc/dox/internals/design.dox Normal file
View file

@ -0,0 +1,70 @@
/** \page page_design Design
A short overview of PipeWire's design.
PipeWire is a media server that can run graphs of multimedia nodes.
Nodes can run inside the server process or in separate processes,
communicating with the server.
PipeWire was designed to:
- Be efficient for raw video using fd passing and audio with
shared ringbuffers.
- Be able to provide/consume/process media from any process.
- Provide policy to restrict access to devices and streams.
- Be easily extensible.
Although an initial goal, the design is not limited to raw video
only and should be able to handle compressed video and other
media as well.
PipeWire uses the \ref page_spa "SPA plugin API" for the nodes in the graph.
SPA is designed for low-latency and efficient processing of any multimedia
format. SPA also provides a number of helper utilities that are not available
in the standard C library.
Some of the application we intend to build:
- v4l2 device provider: Provide controlled access to v4l2 devices
and share one device between multiple processes.
- gnome-shell video provider: GNOME Shell provides a node that
gives the contents of the frame buffer for screen sharing or
screen recording.
- Audio server: Mix and playback multiple audio streams. The design
is more like CRAS (Chromium audio server) than PulseAudio and with
the added benefit that processing can be arranged in a graph.
- Professional audio graph processing like JACK.
- Media playback backend.
# Protocol
The native protocol and object model is similar to
[Wayland](https://wayland.freedesktop.org) but with custom
serialization/deserialization of messages. This is because the data structures
in the messages are more complicated and not easily expressible in XML.
See \ref page_module_protocol_native for details.
# Extensibility
The functionality of the server is implemented and extended with modules and
extensions. Modules are server side bits of logic that hook into various
places to provide extra features. This mostly means controlling the processing
graph in some way. See \ref page_pipewire_modules for a list of current
modules.
Extensions are the client side version of the modules. Most extensions provide
both a client side and server side init function. New interfaces or new object
implementation can easily be added with modules/extensions.
Some of the extensions that can be written:
- Protocol extensions: A client/server side API (.h) together with protocol
extensions and server/client side logic to implement a new object or
interface.
- A module to check security of method calls.
- A module to automatically create, link or relink nodes.
- A module to suspend idle nodes.
*/

178
doc/dox/internals/dma-buf.dox Normal file
View file

@ -0,0 +1,178 @@
/** \page page_dma_buf DMA-BUF Sharing
PipeWire supports sharing Direct Memory Access buffers (DMA-BUFs) between
clients via the \ref SPA_DATA_DmaBuf data type. However properly negotiating
DMA-BUF support on both the producer and the consumer side require following
a specific procedure. This page describes said procedure by using events and
methods from the filter or stream API.
Note: This article focuses mostly on DMA-BUF sharing from arbitrary devices,
like discrete GPUs. For using DMA-BUFs created by v4l2 please refer to the
corresponding paragraph.
# Capability Negotiations
The capability negotiation for DMA-BUFs is complicated by the fact that a
usable and preferred optimal modifier for a given format can only be
determined by the allocator. This allocator has to be invoked with the intersection
of all supported modifiers for every client. As a result, the fixation of the
modifier is delegated from PipeWire to the node responsible for
allocating the buffers.
## pw_stream_connect
The stream parameters should contain two \ref SPA_PARAM_EnumFormat objects for
each format: one for DMA-BUFs, one for shared memory buffers as a fallback.
Query the list of all supported modifiers from your graphics API of choice.
Add a \ref SPA_FORMAT_VIDEO_modifier property to the first stream parameter with
the flags `SPA_POD_PROP_FLAG_MANDATORY | SPA_POD_PROP_FLAG_DONT_FIXATE`. The
value of the property should be set to a \ref SPA_CHOICE_Enum containing one
`long` choice per supported modifier, plus `DRM_FORMAT_MOD_INVALID` if the
graphics API supports modifier-less buffers.
Note: When a producer is only supporting modifier-less buffers it can omit
the \ref SPA_POD_PROP_FLAG_DONT_FIXATE (see param_changed hook, For producers).
The second stream parameter should not contain any \ref SPA_FORMAT_VIDEO_modifier
property.
To prioritise DMA-BUFs place those \ref SPA_PARAM_EnumFormat containing modifiers
first, when emitting them to PipeWire.
## param_changed Hook
When the `param_changed` hook is called for a \ref SPA_PARAM_Format the client
has to parse the `spa_pod` directly. Use
`spa_pod_find_prop(param, NULL, SPA_FORMAT_VIDEO_modifier)` to check
whether modifiers were negotiated. If they were negotiated, set the
\ref SPA_PARAM_BUFFERS_dataType property to `1 << SPA_DATA_DmaBuf`. If they were
not negotiated, fall back to shared memory by setting the
\ref SPA_PARAM_BUFFERS_dataType property to `1 << SPA_DATA_MemFd`,
`1 << SPA_DATA_MemPtr`, or both.
While consumers only have to parse the resulting \ref SPA_PARAM_Format for any
format related information, it's up to the producer to fixate onto a single
format modifier pair. The producer is also responsible to check if all clients
announce sufficient capabilities or fallback to shared memory buffers when
possible.
### For Consumers
Use \ref spa_format_video_raw_parse to get the format and modifier.
### For Producers
Producers have to handle two cases when it comes to modifiers wrt. to the
previous announced capabilities: Using only the modifier-less API, only the
modifier-aware one, or supporting both.
- modifier-less:
In this case only the modifier `DRM_FORMAT_MOD_INVALID` was announced with
the format.
It is sufficient to check if the \ref SPA_PARAM_Format contains the modifier
property as described above. If that is the case, use DMA-BUFs for screen-sharing,
else fall back to SHM, if possible.
- modifier-aware:
In this case a list with all supported modifiers will be returned in the format.
(using `DRM_FORMAT_MOD_INVALID` as the token for the modifier-less API).
On the `param_changed` event check if the modifier key is present and has the flag
\ref SPA_POD_PROP_FLAG_DONT_FIXATE attached to it. In this case, extract all modifiers
from the list and do a test allocation with your allocator to choose the preferred
modifier. Fixate on that \ref EnumFormat by announcing a \ref SPA_PARAM_EnumFormat with
only one modifier in the \ref SPA_CHOICE_Enum and without the
\ref SPA_POD_PROP_FLAG_DONT_FIXATE flag, followed by the previous announced
\ref EnumFormat. This will retrigger the `param_changed` event with an
\ref SPA_PARAM_Format as described below.
If the \ref SPA_PARAM_Format contains a modifier key, without the flag
\ref SPA_POD_PROP_FLAG_DONT_FIXATE, it should only contain one value in the
\ref SPA_CHOICE_Enum. In this case announce the \ref SPA_PARAM_Buffers accordingly
to the selected format and modifier. It is important to query the plane count
of the used format modifier pair and set `SPA_PARAM_BUFFERS_blocks` accordingly.
Note: When test allocating a buffer, collect all possible modifiers, while omitting
`DRM_FORMAT_MOD_INVALID` from the \ref SPA_FORMAT_VIDEO_modifier property and
pass them all to the graphics API. If the allocation fails and the list of
possible modifiers contains `DRM_FORMAT_MOD_INVALID`, fall back to allocating
without an explicit modifier if the graphics API allows it.
## add_buffer Hook
This is relevant for producers.
Allocate a DMA-BUF only using the negotiated format and modifier.
## on_event Hook
This is relevant for consumers.
Check the type of the dequeued buffer. If its \ref SPA_DATA_MemFd or
\ref SPA_DATA_MemPtr use the fallback SHM import mechanism.
If it's \ref SPA_DATA_DmaBuf
get the DMA-BUF FDs (the plane count is encoded in the `n_datas` variable of the
`spa_buffer` struct) and import them with the graphics API.
Note: Some graphics APIs have separated functions for the modifier-less case
(`DRM_FORMAT_MOD_INVALID`) or are omitting the modifier, since it might be used
for error handling.
## Example Programs
- \ref video-src-fixate.c "": \snippet{doc} video-src-fixate.c title
- \ref video-play-fixate.c "": \snippet{doc} video-play-fixate.c title
# DMA-BUF Mapping Warning
It's important to make sure all consumers of the PipeWire stream are prepared
to deal with DMA-BUFs. Most DMA-BUFs cannot be treated like shared memory in general
because of the following issues:
- DMA-BUFs can use hardware-specific tiling and compression as described by
modifiers. Thus, a `mmap(3)` on the DMA-BUF FD will not give a linear view of
the buffer contents.
- DMA-BUFs need to be properly synchronized with the asynchronous reads and
writes of the hardware. A `mmap(3)` call is not enough to guarantee proper
synchronization. (Maybe add link to linux syscall doc??)
- Blindly accessing the DMA-BUFs via `mmap(3)` can be extremely slow if the
buffer has been allocated on discrete hardware. Consumers are better off
using a proper graphics API (such as EGL, Vulkan or VA-API) to process the
DMA-BUFs.
# Size of DMA-BUFs
When importing a DMA-BUF with a proper graphics API the size of a single buffer plane
is no relevant property since it will be derived by the driver from the other properties.
Therefore consumers should ignore the field `maxsize` of a `spa_data` and the field
`size` of a `spa_chunk` struct. Producers are allowed to set both to 0.
In cases where mapping a single plane is required the size should be obtained locally
via the filedescriptor.
# SPA param video format helpers
SPA offers helper functions to parse and build a spa_pod object to/from the spa_video_info_*
struct. The flags \ref SPA_VIDEO_FLAG_MODIFIER and \ref SPA_VIDEO_FLAG_MODIFIER_FIXATION_REQUIRED
are used to indicate modifier usage with the format. `SPA_VIDEO_FLAG_MODIFIER` declares the
parsed/provided spa_video_info_* struct contains valid modifier information. For legacy
reasons `spa_format_video_*_build` will announce any modifier != 0 even when this flag is
unused. `SPA_VIDEO_FLAG_MODIFIER_FIXATION_REQUIRED` is exclusive to the parse helpers and
declares that the parsed spa_pod contains modifier information which needs to be fixated as
described above. The list of available modifiers has to be parsed manually from the spa_pod
object.
- \ref spa_video_info_raw, \ref spa_format_video_raw_parse, \ref spa_format_video_raw_build
- \ref spa_video_info_dsp, \ref spa_format_video_dsp_parse, \ref spa_format_video_dsp_build
# v4l2
Another use case for streaming via DMA-BUFs are exporting a camera feed from v4l2
as DMA-BUFs. Those are located in the main memory where it is possible to mmap them.
This should be done as follows: Neither producer nor consumer should announce a
modifier, but both should include `1 << SPA_DATA_DmaBuf` in the
`SPA_PARAM_BUFFERS_dataType` property. It's the the responsibility of the producer
while the `add_buffer` event to choose DMA-BUF as the used buffer type even though
no modifier is present, if it can guarantee, that the used buffer is mmapable.
Note: For now v4l2 uses planar buffers without modifiers. This is the reason for
this special case.
*/

26
doc/dox/internals/index.dox Normal file
View file

@ -0,0 +1,26 @@
/** \page page_internals Internals
# Internals
- \subpage page_design
- \subpage page_audio
- \subpage page_access
- \subpage page_portal
- \subpage page_midi
- \subpage page_objects_design
- \subpage page_library
- \subpage page_dma_buf
- \subpage page_scheduling
- \subpage page_native_protocol
# Components
- \subpage page_daemon
- \subpage page_session_manager
# Backends
- \subpage page_pulseaudio
*/

240
doc/dox/internals/library.dox Normal file
View file

@ -0,0 +1,240 @@
/** \page page_library PipeWire Library
There are two main components that make up the PipeWire library:
1. An implementation of a graph based media processing engine.
2. An asynchronous IPC mechanism to manipulate and introspect
a graph in another process.
There is usually a daemon that implements the global graph and
clients that operate on this graph.
The IPC mechanism in PipeWire is inspired by Wayland in that it
follows the same design principles of objects and methods/events
along with how this API is presented to the user.
PipeWire has a plugin architecture that allows new features to
be added (or removed) by the user. Plugins can hook into many
aspects of PipeWire and change the behaviour or number of
features dynamically.
# Principles
The PipeWire API is an object oriented asynchronous protocol.
All requests and replies are method invocations on some object.
Objects are identified with a unique ID. Each object implements an
interface and requests result in invocations of methods on the
interface.
The protocol is message based. A message sent by a client to the
server is called a method. A message from the server to the client
is called an event. Unlike Wayland, these messages are not (yet)
described in an external protocol file but implemented directly in
a protocol plugin. Protocol plugins can be added to add new
objects or even protocols when required.
Messages are encoded with \ref page_spa_pod, which make it
possible to encode complex objects with right types.
Events from the server can be a reply to a method or can be emitted
when the server state changes.
Upon connecting to a server, it will broadcast its state. Clients
should listen for these state changes and cache them. There is no
need (or mechanism) to query the state of the server.
The server also has a registry object that, when listening to,
will broadcast the presence of global objects and any changes in
their state.
State about objects can be obtained by binding to them and listening
for state changes.
# Versioning
All interfaces have a version number. The maximum supported version
number of an interface is advertized in the registry global event.
A client asks for a specific version of an interface when it binds
to them. It is the task of the server to adapt to the version of the
client.
Interfaces increase their version number when new methods or events
are added. Methods or events should never be removed or changed for
simplicity.
# Proxies and Resources
When a client connects to a PipeWire daemon, a new `struct pw_proxy`
object is created with ID 0. The `struct pw_core` interface is
assigned to the proxy.
On the server side there is an equivalent `struct pw_resource` with
ID 0. Whenever the client sends a message on the proxy (by calling
a method on the interface of the proxy) it will transparently result
in a callback on the resource with the same ID.
Likewise if the server sends a message (an event) on a resource, it
will result in an event on the client proxy with the same ID.
PipeWire will notify a client when a resource ID (and thus also proxy
ID) becomes unused. The client is responsible for destroying the
proxy when it no longer wants to use it.
# Interfaces
## struct pw_loop
An abstraction for a `poll(2)` loop. It is usually part of one of:
- `struct pw_main_loop`: A helper that can run and stop a `pw_loop`.
- `struct pw_thread_loop`: A helper that can run and stop a `pw_loop`
in a different thread. It also has some helper
functions for various thread related synchronization
issues.
- `struct pw_data_loop`: A helper that can run and stop a `pw_loop`
in a real-time thread along with some useful helper
functions.
## struct pw_context
The main context for PipeWire resources. It keeps track of the mainloop,
loaded modules, the processing graph and proxies to remote PipeWire
instances.
An application has to select an implementation of a `struct pw_loop`
when creating a context.
The context has methods to create the various objects you can use to
build a server or client application.
## struct pw_core
A proxy to a remote PipeWire instance. This is used to send messages
to a remote PipeWire daemon and to receive events from it.
A core proxy can be used to receive errors from the remote daemon
or to perform a roundtrip message to flush out pending requests.
Other core methods and events are used internally for the object
life cycle management.
## struct pw_registry
A proxy to a PipeWire registry object. It emits events about the
available objects on the server and can be used to bind to those
objects in order to call methods or receive events from them.
## struct pw_module
A proxy to a loadable module. Modules implement functionality such
as provide new objects or policy.
## struct pw_factory
A proxy to an object that can create other objects.
## struct pw_device
A proxy to a device object. Device objects model a physical hardware
or software device in the system and can create other objects
such as nodes or other devices.
## struct pw_node
A Proxy to a processing node in the graph. Nodes can have input and
output ports and the ports can be linked together to form a graph.
## struct pw_port
A Proxy to an input or output port of a node. They can be linked
together to form a processing graph.
## struct pw_link
A proxy to a link between in output and input port. A link negotiates
a format and buffers between ports. A port can be linked to many other
ports and PipeWire will manage mixing and duplicating the buffers.
# High Level Helper Objects
Some high level objects are implemented to make it easier to interface
with a PipeWire graph.
## struct pw_filter
A `struct pw_filter` allows you implement a processing filter that can
be added to a PipeWire graph. It is comparable to a JACK client.
## struct pw_stream
A `struct pw_stream` makes it easy to implement a playback or capture
client for the graph. It takes care of format conversion and buffer
sizes. It is comparable to Core Audio AudioQueue or a PulseAudio
stream.
# Security
With the default native protocol, clients connect to PipeWire using
a named socket. This results in a client socket that is used to
send messages.
For sandboxed clients, it is possible to get the client socket via
other ways, like using the portal. In that case, a portal will
do the connection for the client and then hands the connection socket
to the client.
All objects in PipeWire have per client permission bits, currently
READ, WRITE, EXECUTE and METADATA. A client can not see an object
unless it has READ permissions. Similarly, a client can only execute
methods on an object when the EXECUTE bit is set and to modify the
state of an object, the client needs WRITE permissions.
A client (the portal after it makes a connection) can drop permissions
on an object. Once dropped, it can never reacquire the permission.
Clients with WRITE/EXECUTE permissions on another client can
add and remove permissions for the client at will.
Clients with MODIFY permissions on another object can set or remove
metadata on that object.
Clients that need permissions assigned to them can be started in
blocked mode and resume when permissions are assigned to them by
a session manager or portal, for example.
PipeWire uses memfd (`memfd_create(2)`) or DMA-BUF for sharing media
and data between clients. Clients can thus not look at other clients
data unless they can see the objects and connect to them.
# Implementation
PipeWire also exposes an API to implement the server side objects in
a graph.
# Error Reporting
Functions return either NULL with errno set or a negative int error
code when an error occurs. Error codes are used from the SPA plugin
library on which PipeWire is built.
Some functions might return asynchronously. The error code for such
functions is positive and SPA_RESULT_IS_ASYNC() will return true.
SPA_RESULT_ASYNC_SEQ() can be used to get the unique sequence number
associated with the async operation.
The object returning the async result code will have some way to
signal the completion of the async operation (with, for example, a
callback). The sequence number can be used to see which operation
completed.
*/

103
doc/dox/internals/midi.dox Normal file
View file

@ -0,0 +1,103 @@
/** \page page_midi MIDI Support
This document explains how MIDI is implemented.
# Use Cases
## MIDI Devices Are Made Available As Processing Nodes/Ports
Applications need to be able to see a port for each stream of a
MIDI device.
## MIDI Devices Can Be Plugged and Unplugged
When devices are plugged and unplugged the associated nodes/ports
need to be created and removed.
## Applications Can Connect To MIDI Devices
Applications can create ports that can connect to the MIDI ports
so that data can be provided to or consumed from them.
## Some MIDI Devices Are Sinks Or Sources For MIDI Data
It should be possible to create a MIDI sink or source that routes the
MIDI events to specific MIDI ports.
One example of such a sink would be in front of a software MIDI
renderer.
An example of a MIDI source would be after a virtual keyboard or
as a mix from many MIDI input devices.
## Applications Should Auto-connect To MIDI Sinks Or Sources
An application should be able to be connected to a MIDI sink when
it wants to play MIDI data.
An application should be able to connect to a MIDI source when it
wants to capture MIDI data.
# Design
## SPA
MIDI devices/streams are implemented with an \ref spa_node with generic
control input and output Ports. These ports have a media type of
`"application/control"` and the data transported over these ports
are of type \ref spa_pod_sequence with the \ref spa_pod_control type set to
\ref SPA_CONTROL_Midi.
This means that every MIDI event is timestamped with the sample
offset against the current graph clock cycle to get sample accurate
midi events that can be aligned with the corresponding sample data.
Since the MIDI events are embedded in the generic control stream,
they can be interleaved with other control message types, such as
property updates or OSC messages.
## The PipeWire Daemon
Nothing special is implemented for MIDI. Negotiation of formats
happens between `"application/control"` media types and buffers are
negotiated in the same way as any generic format.
## The Session Manager
The session manager needs to create the MIDI nodes/ports for the available
devices.
This can either be done as a single node with ports per device/stream
or as separate nodes created by a MIDI device monitor.
The session manager needs to be aware of the various MIDI sinks and sources
in order to route MIDI streams to them from applications that want this.
# Implementation
## PipeWire Media Session
PipeWire media session uses the \ref SPA_NAME_API_ALSA_SEQ_BRIDGE plugin for
the MIDI features. This creates a single SPA Node with ports per
MIDI client/stream.
The media session will check the permissions on `/dev/snd/seq` before
attempting to create this node. It will also use inotify to wait
until the sequencer device node is accessible.
## JACK
JACK assumes all `"application/control"` ports are MIDI ports.
The control messages are converted to the JACK event format by
filtering out the \ref SPA_CONTROL_Midi types. On output ports, the JACK
event stream is converted to control messages in a similar way.
There is a 1 to 1 mapping between the JACK events and control
messages so there is no information loss or need for complicated
conversions.
*/

347
doc/dox/internals/objects.dox Normal file
View file

@ -0,0 +1,347 @@
/** \page page_objects_design Objects Design
This document is a design reference on the various objects that exist
in the PipeWire media and session management graphs. Explaining what these
objects are, how they are meant to be used, and how they relate to other
kinds of objects and concepts that exist in subsystems or other libraries.
# The Media Graph
The media graph represents and enables the media flow inside the PipeWire
daemon and between the daemon and its clients. It consists of nodes, ports
and links.
```
+------------+ +------------+
| | | |
| +--------+ Link +--------+ |
| Node | Port |--------| Port | Node |
| +--------+ +--------+ |
| | | |
+------------+ +------------+
```
## Node
A **node** is a media processing element. It consumes and/or produces buffers
that contain data, such as audio or video.
A node may operate entirely inside the PipeWire daemon or it may be operating
in a client process. In the second case, media is transferred to/from that
client using the PipeWire protocol.
In an analogy to GStreamer, a _node_ is similar (but not equal) to a
GStreamer _element_.
## Port
A **port** is attached on a **node** and provides an interface for input
or output of media on the node. A node may have multiple ports.
A port always has a direction, input or output:
- Input: it allows media input into the node (in other terms, it is a _sink_)
- Output: it outputs media out of the node (in other terms, it is a _source_)
In an analogy to GStreamer, a _port_ is similar (but not equal) to a
GStreamer _pad_.
## Link
A **link** connects two ports of opposite direction, making media flow from
the output port to the input port.
# The Session Management Graph
The session management graph is a virtual, higher level representation of the
media flow. It is created entirely by the session manager and it can affect
the routing on the media graph only through the session manager's actions.
The session management graph is useful to abstract the complexity of the
actual media flow both for the target user and for the policy management
codebase.
```
+---------------------+ +----------------------+
| | | |
| +----------------+ Endpoint Link +----------------+ |
| Endpoint |Endpoint Stream |-----------------|Endpoint Stream | Endpoint |
| +----------------+ +----------------+ |
| | | |
+---------------------+ +----------------------+
```
## Endpoint
An **endpoint** is a session management object that provides a representation
of user conceivable places where media can be routed to/from.
Examples of endpoints associated with hardware on a desktop-like system:
- Laptop speakers.
- USB webcam.
- Bluetooth headset microphone.
- Line out stereo jack port.
Examples of endpoints associated with hardware in a car:
- Speakers amplifier.
- Front right seat microphone array.
- Rear left seat headphones.
- Bluetooth phone voice gateway.
- Hardware FM radio device.
Examples of endpoints associated with software:
- Desktop screen capture source.
- Media player application.
- Camera application.
In most cases an endpoint maps to a node on the media graph, but this is not
always the case. An endpoint may be backed by several nodes or no nodes at all.
Different endpoints may also be sharing nodes in some cases.
An endpoint that does not map to any node may be useful to represent hardware
that the session manager needs to be able to control, but there is no way
to route media to/from that hardware through the PipeWire media graph. For
example, in a car we may have a CD player device that is directly wired to the
speakers amplifier and therefore audio flows between them without passing
through the controlling CPU. However, it is useful for the session manager to
be able to represent the *CD player endpoint* and the _endpoint link_ between
it and the amplifier, so that it can apply audio policy that takes into account
whether the CD player is playing or not.
### Target
An **endpoint** may be grouping together targets that can be reached by
following the same route and they are mutually exclusive with each other.
For example, the speakers and the headphones jack on a laptop are usually
mutually exclusive by hardware design (hardware mutes the speakers when the
headphones are enabled) and they share the same ALSA PCM device, so audio still
follows the same route to reach both.
In this case, a session manager may choose to group these two targets into the
same endpoint, using a parameter on the _endpoint_ object to allow the user
to choose the target (if the hardware allows configuring this at all).
## Endpoint Stream
An **endpoint stream** is attached to an **endpoint** and represents a logical
path that can be taken to reach this endpoint, often associated with
a _use case_.
For example, the "Speakers amplifier" endpoint in a car might have the
following streams:
- _Music_: A path to play music;
the implementation will output this to all speakers, using the volume
that has been configured for the "Music" use case.
- _Voice_: A path to play a voice message; such as a navigation message or
feedback from a voice assistant, the implementation will output this
to the front speakers only. Lowering the volume of the music (if any)
on these speakers at the same time.
- _Emergency_: A path to play an emergency situation sound (a beep,
or equivalent); the implementation will output this on all speakers.
Increasing the volume to a factory defined value if necessary (to ensure
that it is audible) while muting audio from all other streams at the
same time.
In another example, a microphone that can be used for activating a voice
assistant might have the following streams:
- _Capture_: A path to capture directly from the microphone; this can be used
by an application that listens for the assistant's wake-word in order
to activate the full voice recognition engine.
- _CaptureDelayed_: A path to capture with a constant delay (meaning that
starting capturing now will actually capture something that was spoken
a little earlier); this can be used by the full voice recognition engine,
allowing it to start after the wake-word has been spoken while capturing
audio that also includes the wake-word.
Endpoint streams may be mutually exclusive or they may used simultaneously,
depending on the implementation.
Endpoint streams may be implemented in many ways:
- By plugging additional nodes in the media graph that link to the device node
(ex. a simple buffering node linked to an alsa source node could implement
the _CaptureDelayed_ stream in the above microphone example).
- By using a different device node (ex. different ALSA device on the same card)
that has a special meaning for the hardware.
- By triggering switches on the hardware (ex. modify ALSA controls on the
same device).
## Endpoint Link
An **endpoint link** connects two streams from two different endpoints, creating
a logical representation of media flow between the endpoints.
An **endpoint link** may be implemented by creating one or more _links_ in the
underlying media graph, or it may be implemented by configuring hardware
resources to enable media flow, in case the flow does not pass through the
media graph.
### Constructing
Constructing an **endpoint link** is done by asking the _endpoint stream_
objects to prepare it. First, the source stream is asked to provide linking
information. When the information is retrieved, the sink stream is asked to
use this information to prepare and to provide its own linking information.
When this is done, the session manager is asked to create the link using the
provided information.
This mechanism allows stream implementations:
- To prepare for linking, adjusting hardware paths if necessary.
- To check for stream linking compatibility; not all streams can be connected
to all others (ex. streams with media flow in the hardware cannot be linked
to streams that are backed by nodes in the media graph).
- To provide implementation specific information for linking; in the standard
case this is going to be a list of _ports_ to be linked in the media graph,
but in a hardware-flow case it can be any kind of hardware-specific detail.
# Other Related Objects
## Device
A **device** represents a handle to an underlying API that is used to create
higher level objects, such as nodes, or other devices.
Well-known devices include:
| Device API | Description |
| :--- | :--- |
| alsa.pcm.device | A handle to an ALSA card (ex. `hw:0`, `hw:1`, etc). |
| alsa.seq.device | A handle to an ALSA Midi device. |
| v4l2.device | A handle to a V4L2 device (`/dev/video0`, `/dev/video1`, etc..). |
| jack.device | A JACK client, allowing PipeWire to slave to JACK for audio input/output. |
A device may have a _profile_, which allows the user to choose between
multiple configurations that the device may be capable of having, or to simply
turn the device _off_, which means that the handle is closed and not used
by PipeWire.
## Session
The **session** represents the session manager and can be used to expose
global properties or methods that affect the session management.
### Default Endpoints
The session is responsible for book-keeping the default device endpoints (one
for each kind of device) that is to be used to link new clients when
simulating a PulseAudio-like behavior, where the user can choose from the UI
device preferences.
For example, a system may have both "Speakers" and "HDMI" endpoints on the
"Audio Output" category and the user may be offered to make a choice within
the UI to select which endpoint they want to use by default for audio output.
This preference is meant to be stored in the session object.
### Multiple Sessions
It is not currently defined whether it is allowed to have multiple sessions
or not and how the system should behave if this happens.
# Mappings To Underlying Subsystem Objects
## ALSA UCM
This is a ***proposal***
| ALSA / UCM | PipeWire |
| :--- | :--- |
| ALSA card | device |
| UCM verb | device profile |
| UCM device | endpoint (+ target, grouping conflicting devices into the same endpoint) |
| UCM modifier | endpoint stream |
| PCM stream | node |
In UCM mode, an ALSA card is represented as a PipeWire device, with the
available UCM verbs listed as profiles of the device.
Activating a profile (ie. a verb) will create the necessary nodes for the
available PCM streams and at the same time it will also create one endpoint
for each UCM device. Optionally conflicting UCM devices can be grouped in
the same endpoint, listing the conflicting options as targets of the endpoint.
The available UCM modifiers for each UCM device will be added as streams, plus
one "default" stream for accessing the device with no modifiers.
## ALSA Fallback
| ALSA | PipeWire |
| :--- | :--- |
| card | device |
| PCM stream | node + endpoint |
In the case where UCM (or another similar mechanism) is not available,
ALSA cards are represented as PipeWire devices with only two profiles on/off.
When the on profile is activated, a node and an associated endpoint are created
for every available PCM stream.
Endpoints in this case have only one "default" stream, unless they are extended
by the session manager to have software-backed streams.
## V4L2
***FIXME***
| V4L2 | PipeWire |
| :--- | :--- |
| device | device + node |
# Relationship To Other API's
## PulseAudio
### Mapping PipeWire Objects For Access By PulseAudio Clients
| PipeWire | PulseAudio |
| :--- | :--- |
| device | card |
| device profile | card profile |
| endpoint (associated with a device) | sink / source |
| endpoint (associated with a client) | sink-input / source-output |
| endpoint target | port |
| endpoint stream | N/A, PA clients will be limited to the default stream |
### Mapping PulseAudio Clients To PipeWire
| PulseAudio | PipeWire |
| :--- | :--- |
| stream | client + node + endpoint (no targets, 1 default stream) |
## Jack
Note: This section is about JACK clients connecting to PipeWire through the
JACK compatibility library. The scenario where PipeWire connects to another
JACK server as a client is out of scope here.
### Mapping PipeWire Objects For Access By JACK Clients
| PipeWire | JACK |
| :--- | :--- |
| node | client |
| port | port |
| device | N/A |
| endpoint | N/A |
### Mapping JACK Clients To PipeWire
| JACK | PipeWire |
| :--- | :--- |
| client | client + node |
| port | port |
JACK clients do not create endpoints. A session manager should be JACK aware
in order to anticipate direct node linking.
*/

215
doc/dox/internals/portal.dox Normal file
View file

@ -0,0 +1,215 @@
/** \page page_portal Portal Access Control
This document explains how clients from the portal are handled.
The portal is a DBus service that exposes interfaces to
request access to the PipeWire daemon to perform a certain set of
functions. The PipeWire daemon runs outside the sandbox, the portal is a way
for clients inside the sandbox to connect to and use PipeWire.
The PipeWire socket is not exposed in the sandbox. Instead, The portal
connects to PipeWire on behalf of the client, informing PipeWire that this
client is a portal-managed client. PipeWire can detect and enforce
extra permission checks on the portal managed clients.
Once such portal is the [camera
portal](https://flatpak.github.io/xdg-desktop-portal/portal-docs.html#gdbus-org.freedesktop.portal.Camera)
that provides a PipeWire session to stream video from a camera.
# Use Cases
## New Portal Managed Clients Need Device Permissions Configured
When a new client is detected, the available objects need to be
scanned and permissions configured for each of them.
Only the devices belonging to the media_roles given by the
portal are considered.
## New Devices Need To Be Made Visible To Portal Managed Clients
Newly created objects are made visible to a client when the client
is allowed to interact with it.
Only the devices belonging to the media_roles given by the
portal are considered.
## Permissions For A Device Need To Be Revoked
The session manager listens to changes in the permissions of devices
and will remove the client permissions accordingly.
Usually this is implemented by listening to the permission store
DBus object. The desktop environment might provide a configuration panel
where these permissions can be managed.
# Design
## The Portal
A sandboxed client cannot connect to PipeWire directly. Instead, it connects
to the sandbox side of the portal which then connects the PipeWire daemon to
configure the session. The portal then hands the file descriptor of the
PipeWire connection to the client and the client can use this file descriptor
to interface with the PipeWire session directly.
When the portal connects, it will set the following properties on the
client object:
- `"pipewire.access.portal.is_portal" = true` for the connection of the
portal itself (as opposed to a client managed by the portal).
- `"pipewire.access.portal.app_id"` the [application ID](https://docs.flatpak.org/en/latest/conventions.html#application-ids) of the client.
- `"pipewire.access.portal.media_roles"` media roles of the client.
Currently only `"Camera"` is defined.
Before returning the connection to a client, the portal configures
minimal permissions on the client. No objects are initially visible. It is
the task of the \ref page_session_manager to make the objects in the graph
visible, depending on the client's `media_roles` (see also \ref
PW_KEY_MEDIA_ROLE).
## The PipeWire Portal Module
The PipeWire daemon uses the \ref page_module_portal to find the PID of the
processes that owns the DBus name `org.freedesktop.portal.Desktop`
(see the [XDG Desktop Portal](https://github.com/flatpak/xdg-desktop-portal)).
Client connections from this PID are tagged as \ref PW_KEY_ACCESS
`"portal"` (see \ref page_module_access). It will also set ALL permissions for
this client so that it can resume.
\dot
digraph pw {
compound=true;
node [shape="box"];
rankdir="TB";
dbus [label="org.freedesktop.portal.Desktop"];
portal_access [label="PipeWire (mod: Portal Access)"];
portal [label="xdg-desktop-portal"];
dbus -> portal_access [arrowhead="dot"];
dbus -> portal [arrowhead="dot"];
portal_access -> portal [label="pipewire.access = portal"];
{ rank="same"; portal_access; portal}
}
\enddot
## The Client
A client can ask the portal for a connection to the PipeWire daemon.
\dot
digraph pw {
compound=true;
node [shape="box"];
rankdir="LR";
pw [label="PipeWire"];
portal [label="xdg-desktop-portal"];
client [label="client"];
client -> portal;
portal -> pw [label="portal.is_portal=true", arrowhead="none"]
{rank="min"; pw};
{rank="max"; client};
}
\enddot
The portal maintains an (unrestricted) connection to the PipeWire daemon with
`"pipewire.access.portal.is_portal" = true` to identify the nodes the client
needs access to. It then creates a new restricted connection for the client,
tagged with additional information.
\dot
digraph pw {
compound=true;
node [shape="box"];
rankdir="LR";
pw [label="PipeWire"];
portal [label="xdg-desktop-portal"];
client [label="client"];
client -> portal [arrowhead="none"];
portal -> pw [label="portal.is_portal=true", arrowhead="none"]
portal -> pw [label="portal.app_id = $appid"]
{rank="min"; pw};
{rank="max"; client};
}
\enddot
The file descriptor for this restricted connection is passed back to the
client which can now make use of the resources it has been permitted to
access.
\dot
digraph pw {
compound=true;
node [shape="box"];
rankdir="LR";
pw [label="PipeWire"];
portal [label="xdg-desktop-portal"];
client [label="client"];
portal -> pw [label="portal.is_portal=true", arrowhead="none"]
pw->client [label="restricted connection"];
{rank="min"; pw};
{rank="max"; client};
}
\enddot
## The Session Manager
The session manager listens for new clients to appear. It will use the
\ref PW_KEY_ACCESS property to find portal connections. For client connections
from the portal the session manager checks the requested `media_roles` and
enables or disables access to the respective PipeWire objects.
It might have to consult a database to decide what is allowed, for example the
[org.freedesktop.impl.portal.PermissionStore](https://flatpak.github.io/xdg-desktop-portal/portal-docs.html#gdbus-org.freedesktop.impl.portal.PermissionStore).
\dot
strict digraph pw {
compound=true;
node [shape="box"];
rankdir="LR";
portal [label="xdg-desktop-portal"];
client [label="client"];
subgraph {
rankdir="TB";
permissions [label="PermissionStore"];
sm->permissions;
sm [label="Session Manager"];
pw [label="PipeWire"];
sm -> pw [headlabel="allow $media.roles"];
pw -> sm;
portal -> pw [label="portal.app_id = $appid"];
}
client -> portal [arrowhead="none"];
{rank="min"; sm, pw};
{rank="max"; client};
}
\enddot
In the case of the [XDG Desktop
Portal](https://github.com/flatpak/xdg-desktop-portal), the portal itself
queries the PermissionStore directly.
*/

1573
doc/dox/internals/protocol.dox Normal file
View file

File diff suppressed because it is too large Load diff

69
doc/dox/internals/pulseaudio.dox Normal file
View file

@ -0,0 +1,69 @@
/** \page page_pulseaudio PulseAudio Compatibility
# Internals - Mapping Between ALSA and Streams
This explains the mapping between alsa cards and streams and session manager
objects.
## ALSA Cards
An ALSA card is exposed as a PipeWire device.
## Streams
Each ALSA PCM is opened and a node is created for each PCM stream.
# Session Manager
## ALSA UCM
The mapping of the PipeWire object hierarchy to the ALSA object hierarchy is the following:
One PipeWire device is created for every ALSA card.
- For each UCM verb, a node is created for the associated PCM devices.
- For each UCM verb, an endpoint is created.
In a first step: For each available combination of UCM device and modifier,
a stream is created. Streams are marked with compatible other streams.
Streams with the same modifier and mutually exclusive devices are grouped
into one stream and the UCM devices are exposed on the endpoint as destinations.
## ALSA Fallback
Each PCM stream (node) becomes an endpoint. The endpoint references the
ALSA device ID.
Each endpoint has one stream (for now) called HiFi Playback / HiFi Capture.
More streams can be created depending on the format of the node.
## ALSA Pulse UCM
Using the ALSA backend of PulseAudio we can create the following streams.
## ALSA Pulse Fallback
The pulse ALSA backend will use the mixer controls and some probing to
create the following nodes and endpoints.
# PulseAudio
PulseAudio uses the session manager API to construct cards with profiles
and sink/source with ports.
If an endpoint references a device, a card object is created for the device.
Each endpoint becomes a sink/source.
Each Stream in the endpoint becomes a profile on the PulseAudio card. Because
only one profile is selected on the device, only one stream is visible on
the endpoint. This clashes with the notion that multiple streams can be
active at the same time but is a PulseAudio limitation.
Each Endpoint destination becomes a port on the sink/source.
*/

193
doc/dox/internals/scheduling.dox Normal file
View file

@ -0,0 +1,193 @@
/** \page page_scheduling Graph Scheduling
This document tries to explain how the PipeWire graph is scheduled.
Graph are constructed from linked nodes together with their ports. This
results in a dependency graph between nodes. Special care is taken for
loopback links so that the graph remains a directed graph.
# Nodes
Nodes are objects with 0 or more input and output ports.
Each node also has:
- an eventfd to signal the node that it can start processing
- an activation record that lives in shared memory with memfd.
```
eventfd
+-^---------+
| |
in out
| |
+-v---------+
activation {
status:OK, // bitmask of NEED_DATA, HAVE_DATA or OK
pending:0, // number of unsatisfied dependencies to be able to run
required:0 // number of dependencies with other nodes
}
```
The activation record has the following information:
- processing state and pending dependencies. As long as there are pending dependencies
the node can not be processed. This is the only relevant information for actually
scheduling the graph and is shown in the above illustration.
- Current status of the node and profiling info (TRIGGERED, AWAKE, FINISHED, timestamps
when the node changed state).
- Timing information, mostly for drivers when the processing started, the time, duration
and rate (quantum) etc..
- Information about repositions (seek) and timebase owners.
# Links
When two nodes are linked together, the output node becomes a dependency for the input
node. This means the input node can only start processing when the output node is finished.
This dependency is reflected in the required counter in the activation record. In below
illustration, B's required field is incremented with 1. The pending field is set to the
required field when the graph is started. Node A will keep a list of all targets (B) that it
is a dependency of.
This dependency update is only performed when the link is ready (negotiated) and the nodes
are ready to schedule (runnable).
```
eventfd eventfd
+-^---------+ +-^---------+
| | link | |
in A out ---------------------> in B out
| | | |
+-v---------+ +-v---------+
activation { target activation {
status:OK, --------------------> status:OK,
pending:0, pending:1,
required:0 required:1
} }
```
Multiple links between A and B will only result in 1 target link between A and B.
# Drivers
The graph can only run if there is a driver node that is in some way linked to an
active node.
The driver is special because it will have to initiate the processing in the graph. It
will use a timer or some sort of interrupt from hardware to start the cycle.
Any node can also be a candidate for a driver (when the node.driver property is true).
PipeWire will select the node with the highest priority.driver property as the driver.
Nodes will be assigned to the driver node they will be scheduled with. Each node holds
a reference to the driver and increments the required field of the driver.
When a node is ready to be scheduled, the driver adds the node to its list of targets
and increments the required field.
```
eventfd eventfd
+-^---------+ +-^---------+
| | link | |
in A out ---------------------> in B out
| | | |
+-v---------+ +-v---------+
activation { target activation {
status:OK, --------------------> status:OK,
pending:0, pending:0,
required:1 required:2
} }
| ^ ^
| | / /
| | / /
| | / /
| | / /
| | / /
v | /-------------/ /
activation { /
status:OK, V---------------/
pending:0,
required:2
}
+-^---------+
| |
| driver |
| |
+-v---------+
eventfd
```
As seen in the illustration above, the driver holds a link to each node it needs to
schedule and each node holds a link to the driver. Some nodes hold a link to other
nodes.
It is possible that the driver is the same as a node in the graph (for example node A)
but conceptually, the links above are still valid.
The driver will then start processing the graph by emitting the ready signal. PipeWire
will then:
- Check the previous cycle. Did it complete? Mark xrun on unfinished nodes.
- Perform reposition requests if any, timebase changes, etc..
- The pending counter of each follower node is set to the required field.
- It then loops over all targets of the driver and atomically decrements the required
field of the activation record. When the required field is 0, the eventfd is signaled
and the node can be scheduled.
In our example above, Node A and B will have their pending state decremented. Node A
will be 0 and will be triggered first (node B has 2 pending dependencies to start with and
will not be triggered yet). The driver itself also has 2 dependencies left and will not
be triggered (complete) yet.
## Scheduling node A
When the eventfd is signaled on a node, we say the node is triggered and it will be able
to process data. It consumes the input on the input ports and produces more data on the
output ports.
After processing, node A goes through the list of targets and decrements each pending
field (node A has a reference to B and the driver).
In our above example, the driver is decremented (from 2 to 1) but is not yet triggered.
node B is decremented (from 1 to 0) and is triggered by writing to the eventfd.
## Scheduling node B
Node B is scheduled and processes the input from node A. It then goes through the list of
targets and decrements the pending fields. It decrements the pending field of the
driver (from 1 to 0) and triggers the driver.
## Scheduling the driver
The graph always completes after the driver is triggered and scheduled. All required
fields from all the nodes in the target list of the driver are now 0.
The driver calculates some stats about cpu time etc.
# Remote nodes.
For remote nodes, the eventfd and the activation is transfered from the server
to the client.
This means that writing to the remote client eventfd will wake the client directly
without going to the server first.
All remote clients also get the activation and eventfd of the peer and driver they
are linked to and can directly trigger peers and drivers without going to the
server first.
## Remote driver nodes.
Remote drivers start the graph cycle directly without going to the server first.
After they complete (and only when the profiler is active), they will trigger an
extra eventfd to signal the server that the graph completed. This is used by the
server to generate the profiler info.
*/

46
doc/dox/internals/session-manager.dox Normal file
View file

@ -0,0 +1,46 @@
/** \page page_session_manager PipeWire Session Manager
The \ref page_daemon is primarily a framework that allows devices and
applications to exchange data.
It provides the mechanism to do so but the policy deciding which components
can talk to each other and when is controlled by the session manager. As
outlined in \ref page_objects_design, PipeWire provides a media graph
consisting of devices, nodes and ports. The session manager is the one that
decides on the links between those elements.
Two prominent session managers currently exist:
- [PipeWire Media Session](https://gitlab.freedesktop.org/pipewire/media-session), the
example session manager.
- [WirePlumber](https://gitlab.freedesktop.org/pipewire/wireplumber), a
modular session manager based on GObject.
[Documentation](https://pipewire.pages.freedesktop.org/wireplumber/)
This page describes some of the requirements for session managers in general.
# Client Management
PipeWire provides a \ref page_access "permission system" to limit client's
access to resources but only \ref page_module_access "basic permission
handling". The session manager is expected to decide whether clients may
access specific resources.
# Device Management
PipeWire's responsibility is to open devices, however the decision on which
devices should be opened is the job of a session manager, including the
configuration of those devices.
# Endpoint Grouping
An endpoint is, effectively, a group of nodes that are a logical unit that can
consume or produce media data. For example, a Bluetooth speaker may present as
several nodes but is only one logical unit to stream audio to.
See \ref page_objects_design for details on Endpoints.
*/