doc: add midi doc

Reorganize the docs a little. First a short intro, then list the use
cases, then the responsabilities of the various components, then
the implementation in various places.

doc/pipewire-access.dox

@@ -2,6 +2,57 @@
 
 This document explains how access control is designed implemented.
 
+PipeWire implements per client permissions on the objects in the graph.
+
+Permissions include R (read), W (write), X (execute) and M (metadata).
+
+An object with R permissions is visible to the client. The client will
+receive registry events for the object and can interact with it.
+
+Methods can be called on an object with X permissions. Some of those
+methods will only query state, others will modify the object.
+
+An object with W permission can be modified. This is usually done with
+a method that modifies the state of the object. The W permission usually
+implies the X permission.
+
+An object with M permission can be used as the subject in metadata.
+
+# Use cases
+
+## new clients need their permissions configured
+
+A new client is not allowed to communicate with the PipeWire daemon until
+it has been configured with permissions.
+
+## Flatpaks can't modify other stream/device volumes
+
+Flatpaks should not be able to modify the state of certain objects.
+
+Permissions of the relevant PipeWire objects should not have the W
+permission to avoid this.
+
+## Flatpaks can't move other streams to different devices
+
+Streams are moved to another device by setting the "target.node" metadata
+on the node id. By not setting the M bit on the other objects, this can
+be avoided.
+
+## Some application should be restricted in what they can see
+
+Application should only be able to see the objects that they are allowed
+to see. For example, a web browser that was given access to a camera should
+not be able to see audio devices.
+
+## Manager applications
+
+Some applications might be shipped in a flatpak but really require full
+access to the PipeWire graph. This includes moving streams
+(setting metadata) and modifying properties (volume).
+
+
+# Design
+
 ## The PipeWire daemon
 
 When a new client connects to the PipeWire daemon and right after it
@@ -84,32 +135,7 @@ Permissions of objects for a client can be changed at any time by the
 session manager. Removing the client core R permission will suspend the
 client completely.
 
-
+Manager applications will set the PW_KEY_MEDIA_CATEGORY property
-## Use cases
-
-### new clients need their permissions configured
-
-A new client that has been suspended by the PipeWire access module need
-to be resumed at some point with the right permissions.
-
-### Flatpaks can't modify other stream/device volumes
-
-Permissions of the PipeWire objects should not have the W permission to
-avoid this.
-
-### Flatpaks can't move other streams to different devices
-
-Streams are moved to another device by setting the "target.node" metadata
-on the node id. By not setting the M bit on the other objects, this can
-be avoided.
-
-### Manager applications
-
-Some applications might be shipped in a flatpak but really require full
-access to the PipeWire graph. This includes moving streams
-(setting metadata) and modifying properties (volume).
-
-These manager applications will set the PW_KEY_MEDIA_CATEGORY property
 in the client object to "Manager".
 
 The session manager needs to do additional checks to determine if the
@@ -119,7 +145,9 @@ permission store checks or ask the user if the application is allowed
 full access.
 
 
-### pipewire-media-session
+# Implementation
+
+## pipewire-media-session
 
 The example media session has an access-flatpak module that handles the
 clients that have a PW_KEY_ACCESS as "flatpak". Other clients are

doc/pipewire-midi.dox

@@ -0,0 +1,102 @@
+/** \page page_midi PipeWire MIDI
+
+This document explains how MIDI is implemented.
+
+# Use cases
+
+## MIDI devices are made available as processing nodes/ports
+
+Applications need to be able to see a port for each stream of a
+MIDI device.
+
+## MIDI devices can be plugged and unplugged
+
+When devices are plugged and unplugged the associated nodes/ports
+need to be created and removed.
+
+## Applications can connect to MIDI devices
+
+Applications can create ports that can connect to the MIDI ports
+so that data can be provided to or consumed from them.
+
+## Some MIDI devices are sinks or sources for midi data
+
+It should be possible to create a MIDI sink or source that routes the
+midi events to specific midi ports.
+
+One example of such a sink would be in front of a software midi
+renderer.
+
+An example of a MIDI source would be after a virtual keyboard or
+as a mix from many midi input devices.
+
+## Applications should autoconnect to MIDI sinks or sources
+
+An application should be able to be connected to a MIDI sink when
+it wants to play midi data.
+
+An application should be able to connect to a MIDI source when it
+wants to capture midi data.
+
+# Design
+
+## SPA
+
+MIDI devices/streams are implemented with an SPA Node with generic
+control input and output Ports. These ports have a media type of
+"application/control" and the data transported over these ports
+are of type spa_pod_sequence with the spa_pod_control type set to
+SPA_CONTROL_Midi.
+
+This means that every midi event is timestamped with the sample
+offset against the current graph clock cycle to get sample accurate
+midi events that can be aligned with the corresponding sample data.
+
+Since the MIDI events are embedded in the generic control stream,
+they can be interleaved with other control message types, such as
+property updates or OSC messages.
+
+## The PipeWire daemon
+
+Nothing special is implemented for MIDI. Negotiation of formats
+happens between "application/control" media types and buffers are
+negotiated in the same way as any generic format.
+
+## The session manager
+
+The session manager needs to create the MIDI nodes/ports for the available
+devices.
+
+This can either be done as a single node with ports per device/stream
+or as separate nodes created by a MIDI device monitor.
+
+The session manager needs to be aware of the various MIDI sinks and sources
+in order to route MIDI streams to them from applications that want this.
+
+
+# Implementation
+
+## pipewire-media-session
+
+PipeWire media session uses the SPA_NAME_API_ALSA_SEQ_BRIDGE plugin
+for the midi features. This creates a single SPA Node with ports per
+MIDI client/stream.
+
+The media session will check the permissions on /dev/snd/seq before
+attempting to create this node. It will also use inotify to wait
+until the sequencer device node is accessible.
+
+## JACK
+
+JACK assumes all "application/control" ports are midi ports.
+
+The control messages are converted to the JACK event format by
+filtering out the SPA_CONTROL_Midi types. On output ports, the JACK
+event stream is converted to control messages in a similar way.
+
+There is a 1 to 1 mapping between the JACK events and control
+messages so there is no information loss or need for complicated
+conversions.
+
+
+*/

doc/pipewire-portal.dox

@@ -2,24 +2,55 @@
 
 This document explains how clients from the portal are handled.
 
-## The portal
-
 The portal is a DBus service that exposes a couple of interfaces to
 request access to the PipeWire daemon to perform a certain set of
 functions.
 
 The portal makes it possible to use PipeWire without having to expose
-the local PipeWire socket in the sandboxed application. PipeWire can
+the local PipeWire socket in the sandboxed application. It is the
-detect and enforce extra permission checks on the Portal session.
+portal can connects to PipeWire on behalf of the client. The client
+becomes a portal managed client. PipeWire can detect and enforce
+extra permission checks on the portal managed clients.
 
 Once such portal is the Camera portal that provides a PipeWire session
 to stream video from a camera.
 
-In the portal case, it is the portal itself that will make a connection
+# Use cases
-to the PipeWire daemon to configure the session. It then hands the
+
-file descriptor of the PipeWire connection to the client. The client
+## new portal managed clients need device permissions configured
-can then use the file descriptor to interface with the PipeWire
+
-session.
+When a new client is detected, the available objects need to be
+scanned and permissions configured for each of them.
+
+Only the devices belonging to the media_roles given by the
+portal are considered.
+
+## new devices need to be made visible to portal managed clients
+
+Newly created objects are made visible to a client when the client
+is allowed to interact with it.
+
+Only the devices belonging to the media_roles given by the
+portal are considered.
+
+## permissions for a device need to be revoked
+
+The session manager listens to changes in the permissions of devices
+and will remove the client permissions accordingly.
+
+Usually this is implemented by listening to the permission store
+DBus object. The desktop environment might provide a configuration panel
+where these permissions can be managed.
+
+
+# Design
+
+## The portal
+
+The portal itself will make a connection to the PipeWire daemon to
+configure the session. It then hands the file descriptor of the PipeWire
+connection to the client. The client can then use the file descriptor
+to interface with the PipeWire session.
 
 When the portal connects, it will set the following properties on the
 client object:
@@ -70,33 +101,7 @@ allowed.
 The permission store can be used for this. Usually the portal also
 implements "org.freedesktop.impl.portal.PermissionStore" for this.
 
-
+# Implementation
-## Use cases
-
-### new portal managed clients need device permissions configured
-
-When a new client is detected, the available objects are scanned and
-permissions are configured for each of them.
-
-Only the devices belonging to the media_roles given by the
-portal are considered.
-
-### new devices need to be made visible to portal managed clients
-
-Newly created objects are made visible to a client when the client
-is allowed to interact with it.
-
-Only the devices belonging to the media_roles given by the
-portal are considered.
-
-### permissions for a device need to be revoked
-
-The session manager listens to changes in the permissions of devices
-and will remove the client permissions accordingly.
-
-Usually this is implemented by listening to the permission store
-DBus object. The desktop environment might provide a configuration panel
-where these permissions can be managed.
 
 ## pipewire-media-session