pipewire/doc/dma-buf.dox

/** \page page_dma_buf DMA-BUF Sharing

PipeWire supports sharing Direct Memory Access buffers (DMA-BUFs) between
clients via the `SPA_DATA_DmaBuf` data type. However properly negotiating
DMA-BUF support on both the producer and the consumer side require following
a specific procedure. This page describes said procedure by using events and
methods from the filter or stream API.

Note: This article focuses mostly on DMA-BUF sharing from arbitrary devices,
like discrete GPUs. For using DMA-BUFs created by v4l2 please refer to the
corresponding paragraph.

# Capability Negotiations

The capability negotiation for DMA-BUFs is complicated by the fact that a
usable and preferred optimal modifier for a given format can only be
determined by the allocator. This allocator has to be invoked with the intersection
of all supported modifiers for every client. As a result, the fixation of the
modifier is delegated from PipeWire to the node responsible for
allocating the buffers.

## pw_stream_connect

The stream parameters should contain two `SPA_PARAM_EnumFormat` objects for
each format: one for DMA-BUFs, one for shared memory buffers as a fallback.

Query the list of all supported modifiers from your graphics API of choice.
Add a `SPA_FORMAT_VIDEO_modifier` property to the first stream parameter with
the flags `SPA_POD_PROP_FLAG_MANDATORY | SPA_POD_PROP_FLAG_DONT_FIXATE`. The
value of the property should be set to a `SPA_CHOICE_Enum` containing one
`long` choice per supported modifier, plus `DRM_FORMAT_MOD_INVALID` if the
graphics API supports modifier-less buffers.

Note: When a producer is only supporting modifier-less buffers it can omit
the `SPA_POD_PROP_FLAG_DONT_FIXATE` (see param_changed hook, For producers).

The second stream parameter should not contain any `SPA_FORMAT_VIDEO_modifier`
property.

To prioritise DMA-BUFs place those `SPA_PARAM_EnumFormat` containing modifiers
first, when emitting them to PipeWire.

## param_changed Hook

When the `param_changed` hook is called for a `SPA_PARAM_Format` the client
has to parse the `spa_pod` directly. Use
`spa_pod_find_prop(param, NULL, SPA_FORMAT_VIDEO_modifier)` to check
whether modifiers were negotiated. If they were negotiated, set the
`SPA_PARAM_BUFFERS_dataType` property to `1 << SPA_DATA_DmaBuf`. If they were
not negotiated, fall back to shared memory by setting the
`SPA_PARAM_BUFFERS_dataType` property to `1 << SPA_DATA_MemFd`,
`1 << SPA_DATA_MemPtr`, or both.

While consumers only have to parse the resulting `SPA_PARAM_Format` for any
format related information, it's up to the producer to fixate onto a single
format modifier pair. The producer is also responsible to check if all clients
announce sufficient capabilities or fallback to shared memory buffers when
possible.

### For Consumers

Use `spa_format_video_raw_parse` to get the format and modifier.

### For Producers

Producers have to handle two cases when it comes to modifiers wrt. to the
previous announced capabilities: Using only the modifier-less API, only the
modifier-aware one, or supporting both.

- modifier-less:
  In this case only the modifier `DRM_FORMAT_MOD_INVALID` was announced with
  the format.
  It is sufficient to check if the `SPA_PARAM_Format` contains the modifier
  property as described above. If that is the case, use DMA-BUFs for screen-sharing,
  else fall back to SHM, if possible.
- modifier-aware:
  In this case a list with all supported modifiers will be returned in the format.
  (using `DRM_FORMAT_MOD_INVALID` as the token for the modifier-less API).
  On the `param_changed` event check if the modifier key is present and has the flag
  `SPA_POD_PROP_FLAG_DONT_FIXATE` attached to it. In this case, extract all modifiers
  from the list and do a test allocation with your allocator to choose the preferred
  modifier. Fixate on that `EnumFormat` by announcing a `SPA_PARAM_EnumFormat` with
  only one modifier in the `SPA_CHOICE_Enum` and without the
  `SPA_POD_PROP_FLAG_DONT_FIXATE` flag, followed by the previous announced
  `EnumFormat`. This will retrigger the `param_changed` event with an
  `SPA_PARAM_Format` as described below.
  If the `SPA_PARAM_Format` contains a modifier key, without the flag
  `SPA_POD_PROP_FLAG_DONT_FIXATE`, it should only contain one value in the
  `SPA_CHOICE_Enum`. In this case announce the `SPA_PARAM_Buffers` accordingly
  to the selected format and modifier. It is important to query the plane count
  of the used format modifier pair and set `SPA_PARAM_BUFFERS_blocks` accordingly.

Note: When test allocating a buffer, collect all possible modifiers, while omitting
`DRM_FORMAT_MOD_INVALID` from the `SPA_FORMAT_VIDEO_modifier` property and
pass them all to the graphics API. If the allocation fails and the list of
possible modifiers contains `DRM_FORMAT_MOD_INVALID`, fall back to allocating
without an explicit modifier if the graphics API allows it.

## add_buffer Hook

This is relevant for producers.

Allocate a DMA-BUF only using the negotiated format and modifier.

## on_event Hook

This is relevant for consumers.

Check the type of the dequeued buffer. If its `SPA_DATA_MemFd` or
`SPA_DATA_MemPtr` use the fallback SHM import mechanism.
If it's `SPA_DATA_DmaBuf` 
get the DMA-BUF FDs (the plane count is encoded in the `n_datas` variable of the
`spa_buffer` struct) and import them with the graphics API.

Note: Some graphics APIs have separated functions for the modifier-less case
(`DRM_FORMAT_MOD_INVALID`) or are omitting the modifier, since it might be used
for error handling.

## Example Programs

- \ref video-src-fixate.c "": \snippet{doc} video-src-fixate.c title
- \ref video-play-fixate.c "": \snippet{doc} video-play-fixate.c title

# DMA-BUF Mapping Warning

It's important to make sure all consumers of the PipeWire stream are prepared
to deal with DMA-BUFs. Most DMA-BUFs cannot be treated like shared memory in general
because of the following issues:

- DMA-BUFs can use hardware-specific tiling and compression as described by
  modifiers. Thus, a `mmap(3)` on the DMA-BUF FD will not give a linear view of
  the buffer contents.
- DMA-BUFs need to be properly synchronized with the asynchronous reads and
  writes of the hardware. A `mmap(3)` call is not enough to guarantee proper
  synchronization. (Maybe add link to linux syscall doc??)
- Blindly accessing the DMA-BUFs via `mmap(3)` can be extremely slow if the
  buffer has been allocated on discrete hardware. Consumers are better off
  using a proper graphics API (such as EGL, Vulkan or VA-API) to process the
  DMA-BUFs.

# Size of DMA-BUFs

When importing a DMA-BUF with a proper graphics API the size of a single buffer plane
is no relevant property since it will be derived by the driver from the other properties.
Therefore consumers should ignore the field `maxsize` of a `spa_data` and the field
`size` of a `spa_chunk` struct. Producers are allowed to set both to 0.
In cases where mapping a single plane is required the size should be obtained locally
via the filedescriptor.

# SPA param video format helpers

SPA offers helper functions to parse and build a spa_pod object to/from the spa_video_info_*
struct. The flags `SPA_VIDEO_FLAG_MODIFIER` and `SPA_VIDEO_FLAG_MODIFIER_FIXATION_REQUIRED`
are used to indicate modifier usage with the format. `SPA_VIDEO_FLAG_MODIFIER` declares the
parsed/provided spa_video_info_* struct contains valid modifier information. For legacy
reasons `spa_format_video_*_build` will announce any modifier != 0 even when this flag is
unused. `SPA_VIDEO_FLAG_MODIFIER_FIXATION_REQUIRED` is exclusive to the parse helpers and
declares that the parsed spa_pod contains modifier information which needs to be fixated as
described aboath. The list of available modifiers has to be parsed manually from the spa_pod
object.

# v4l2

Another use case for streaming via DMA-BUFs are exporting a camera feed from v4l2
as DMA-BUFs. Those are located in the main memory where it is possible to mmap them.
This should be done as follows: Neither producer nor consumer should announce a
modifier, but both should include `1 << SPA_DATA_DmaBuf` in the
`SPA_PARAM_BUFFERS_dataType` property. It's the the responsibility of the producer
while the `add_buffer` event to choose DMA-BUF as the used buffer type even though
no modifier is present, if it can guarantee, that the used buffer is mmapable.

Note: For now v4l2 uses planar buffers without modifiers. This is the reason for
this special case.

*/