doc: add document about latency handling

doc/dox/internals/index.dox

@@ -11,6 +11,7 @@
 - \subpage page_library
 - \subpage page_dma_buf
 - \subpage page_scheduling
+- \subpage page_latency
 - \subpage page_native_protocol
 
 

doc/dox/internals/latency.dox

@@ -0,0 +1,233 @@
+/** \page page_latency Latency support
+
+This document explains how the latency in the PipeWire graph is implemented.
+
+# Use Cases
+
+## A node port has a latency
+
+Applications need to be able to query the latency of a port.
+
+Linked Nodes need to be informed of the latency of a port.
+
+## dynamically update port latencies
+
+It needs to be possible to dynamically update the latency of a port and this
+should inform all linked ports/nodes of the updated latency.
+
+## Linked nodes add latency to the signal
+
+When two nodes/ports have a latency, the signal is delayed by the sum of
+the nodes latencies.
+
+## Calculate the signal delay upstream and downstream
+
+A node might need to know how much a signal was delayed since it arrived
+in the node.
+
+A node might need to know how much the signal will be delayed before it
+exists the graph.
+
+## Detect latency mismatch
+
+When a signal travels through 2 different parts of the graph with different
+latencies and then eventually join, there is a latency mismatch. It should
+be possible to detect this mismatch.
+
+
+# Concepts
+
+## Port Latency
+
+The fundamental object for implementing latency reporting in PipeWire is the
+Latency object.
+
+It consists of a direction (input/output) and min/max latency values. The latency
+values can express a latency relative to the graph quantum, the samplerate or in
+nanoseconds. There is a mininum and maximum latency value in the Latency object.
+
+The direction of the latency object determines how the latency object propagates.
+
+- SPA_DIRECTION_OUTPUT Latency objects move from output ports downstream and contain
+  the latency from all nodes upstream. An output latency received on an input port
+  should instruct the node to update the output latency on its output ports related
+  to this input port. This corresponds to the JackCaptureLatency.
+
+- SPA_DIRECTION_INPUT Latency objects move from input ports upstream and contain
+  the latency from all nodes downstream. An input latency received on an output port
+  should instruct the node to update the input latency on its input ports related
+  to this output port. This corresponds to the JackPlaybackLatency.
+
+PipeWire will automatically propagate Latency objects from ports to all linked ports
+in the graph. Output Latency objects on output ports are propagated to linked input
+ports and input Latency objects on input ports are propagated to linked output ports.
+
+If a port has links with multiple other ports, the Latency objects are merged by
+taking the min of the min values and the max of the max values of all latency objects
+on the other ports. 
+
+This way, Output Latency always describes the aggragated total upstream latency of
+signal up to the port and Input latency describes the aggregated downstream latency
+of the signal from the port.
+
+
+## Node ProcessLatency
+
+This is a per node property and applies to the latency introduced by the node
+logic itself.
+
+This mostly works if (almost) all data processing ports (input/output) participate in
+the same data processing with the same latency, which is a common use case.
+
+ProcessLatency is mostly used to easily calculate Output and Input Latency on ports.
+We can simply add the ProcessLatency to all latency values in the ports Latency objects
+to obtain the corresponding Latency object for the other ports.
+
+
+# Latency updates
+
+Latency params on the ports can be updated as needed. This can happen because some upstream or
+downstream latency changed or when the ProcessLatency of a node changes.
+
+When the ProcessLatency changes, the procedure to notify of latency changes is:
+
+- Take output latency on input port, add new ProcessLatency, set new latency on output
+  port. This propagates the new latency downstream.
+
+- Take input latency on output port, add new ProcessLatency, set new latency on input
+  port. This propagates the new latency upstream.
+
+PipeWire will automatically aggregate latency from links and propagate the new latencies
+down and upstream.
+
+# Examples
+
+## A source node with a given ProcessLatency
+
+When we have a source with a ProcessLatency, for example, of 1024 samples:
+
+   +----------+     + Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+   |          FL ---+ Latency: [{ "direction": "input", "min-rate": 0, "max-rate": 0 } ]
+   | source   +
+   |          FR ---+ Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+   +----------+     + Latency: [{ "direction": "input", "min-rate": 0, "max-rate": 0 } ]
+        ^
+        |
+ ProcessLatency: [  { "quantum": 0, "rate": 1024, "ns": 0 } ]
+
+Both output ports have an output latency of 1024 samples and no input latency.
+
+## A sink node with a given ProcessLatency
+
+When we have a sink with a ProcessLatency, for example, of 512 samples:
+
+   Latency: [{ "direction": "output", "min-rate": 0, "max-rate": 0 } ]
+   Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
+     ^
+     |      +----------+
+     +---- FL          |
+            | sink     | <- ProcessLatency: [  { "quantum": 0, "rate": 1024, "ns": 0 } ]
+     +---- FR          |
+     |      +----------+
+     v
+   Latency: [{ "direction": "output", "min-rate": 0, "max-rate": 0 } ]
+   Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
+
+Both input ports have an input latency of 512 samples and no output latency.
+
+## A source and sink node linked together
+
+With the source and sink from above, if we link the FL channels, the input latency
+from the input port of the sink is propagated to the output port of the source
+and the output latency of the output port is propagated to the input port of the
+sink:
+
+
+      Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+      Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
+               ^   
+               |         Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+               |         Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
+               |              |
+   +----------+v              v+--------+
+   |          FL ------------ FL        |
+   | source   +                | sink   |
+   |          FR --+          FR        |
+   +----------+    |           +--------+
+                   v
+      Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+      Latency: [{ "direction": "input", "min-rate": 0, "max-rate": 0 } ]
+
+
+## Insert a latency node
+
+If we place a node with a 256 sample latency in the above source-sink graph:
+
+
+   Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+   Latency: [{ "direction": "input", "min-rate": 768, "max-rate": 768 } ]
+               ^   
+               |   Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+               |   Latency: [{ "direction": "input", "min-rate": 768, "max-rate": 768 } ]
+               |              ^
+               |              | Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
+               |              | Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
+               |              |          ^           
+               |              |          |           
+   +----------+v              v+--------+v            +-------+
+   |          FL ------------ FL        FL --------- FL       |
+   | source   +                | node   |           ^ | sink  |
+   |          .                |        .           | .       |
+   +----------+                +--------+           | +-------+
+                                                    v
+                             Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
+                             Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
+
+
+See how the output latency propagates and is incremented going downstream and the
+input latency is incremented and traveling upstream.
+
+
+## Link a port to two port with different latencies
+
+When we introduce a sink2 with different input latency and link this to
+the node FL output port, it will aggregate the two input latencies by
+taking the min of min and max of max.
+
+
+   Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+   Latency: [{ "direction": "input", "min-rate": 768, "max-rate": 2304 } ]
+               ^   
+               |   Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+               |   Latency: [{ "direction": "input", "min-rate": 768, "max-rate": 2304 } ]
+               |              ^
+               |              | Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
+               |              | Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 2048 } ]
+               |              |          ^          
+               |              |          |  Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
+               |              |          |  Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
+               |              |          |          ^
+               |              |          |          |
+   +----------+v              v+--------+v          v +-------+
+   |          FL ------------ FL        FL --------- FL       |
+   | source   +                | node   |  \          | sink  |
+   |          .                |        .   \         .       |
+   +----------+                +--------+    \        +-------+
+                                              \
+                                               \      +-------+
+						+--- FL       |
+                                                    ^ | sink2 |
+						    | .       .
+                                                    | +-------+
+                                                    v
+                             Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
+                             Latency: [{ "direction": "input", "min-rate": 2048, "max-rate": 2048 } ]
+
+
+The source node now knows that its output signal will be delayed between 768 amd 2304 samples
+depending on the path in the graph.
+
+We also see that node.FL has different min/max-rate input latencies. This information can be
+used to insert a delay node to align the latencies again. For example, if we delay the signal
+between node.FL and FL.sink with 1536 samples, the latencies will be aligned again.
+

doc/meson.build

@@ -58,6 +58,7 @@ extra_docs = [
   'dox/internals/index.dox',
   'dox/internals/design.dox',
   'dox/internals/access.dox',
+  'dox/internals/latency.dox',
   'dox/internals/midi.dox',
   'dox/internals/portal.dox',
   'dox/internals/daemon.dox',