module-rtp: Add audio codec support to audio.c and replace opus.c with it

Opus was integrated as a completely separate code path to the PCM audio
processing found in audio.c. This is actually not ideal, since the only
part that actually is Opus specific is the part that en- and decodes from
and to PCM. The rest is 1:1 the same PCM handling.

For this reason, it is much better to instead add audio codec support to
audio.c, meaning that the code in there can now encode PCM audio right
before sending it out as RTP, and decode incoming packets to PCM right
before actually processing the decoded audio data.

This significantly modifies how stream.c initializes the PCM audio path,
since the audio codec feature is new. It now treats the Opus subtype
as an audio codec selector instead of a selector for an entirely
alternate code path (like how MIDI integration remains entirely separate).

Since audio codecs usually require their frames to be decoded in order,
this also integrates the RTP jitter buffer in the RTP module.

Opus is now integrated as such a codec in audio.c. When it is selected,
incoming packets in rtp_audio_receive() are first inserted into the
jitter buffer. That buffer then outputs packets in order, and then, these
packets are decoded to PCM. The rest of the processing chain goes as usual.
A similar route is used for when the jitter buffer signals packet loss
to be able to apply PLC.

For encoding, it is similar (except that no jitter buffer is involved);
in rtp_audio_flush_packets(), when Opus is active, the PCM data is
rerouted to be fed to Opus for encoding, and the Opus output is then
placed into the iovec array instead of the original PCM.

This also improves overall Opus support; it supports S16 PCM data in
addition to F32 data, correctly checks the ptime, sample rate etc. for
Opus compatibility, computes an ideal bitrate, allows for manual bitrate
selection and encoding complexity adjustment (via the new stream properties
"opus.encoder.bitrate" and "opus.encoder.complexity"), sets several other
Opus CTLs to fixed values, supports the Opus restricted-lowdelay mode
(sacrifices Speech code paths for lower latency, enabled by setting the
"opus.encoder.restricted-lowdelay" stream property to true), and also uses
Opus' PLC in case of packet loss.

The audio codec interface is designed such that adding other codecs in
the future is easily doable. New integrations need to implement the
function pointers found in the rtp_audio_codec structure, and expose
an instance of such a custom rtp_audio_codec structure instance (see
the get_rtp_opus_codec() implementation for an example).
This commit is contained in:
Carlos Rafael Giani 2026-06-24 18:48:36 +02:00
parent 31bb82e116
commit 6b524fd596
12 changed files with 1461 additions and 471 deletions

View file

@ -199,7 +199,8 @@ PW_LOG_TOPIC(mod_topic, "mod." NAME);
"( sess.min-ptime=<minimum packet time in milliseconds, default:2> ) " \
"( sess.max-ptime=<maximum packet time in milliseconds, default:20> ) " \
"( sess.media=<string, the media type audio|midi|opus, default audio> ) " \
"( audio.format=<format, default:"DEFAULT_RAW_AUDIO_FORMAT"> ) " \
"( audio.format=<format, default:"DEFAULT_RAW_AUDIO_FORMAT " for media type audio, " \
DEFAULT_OPUS_AUDIO_FORMAT " for media type opus, not used for MIDI> ) " \
"( audio.rate=<sample rate, default:"SPA_STRINGIFY(DEFAULT_RATE)"> ) " \
"( audio.channels=<number of channels, default:"SPA_STRINGIFY(DEFAULT_CHANNELS)"> ) " \
"( audio.position=<channel map, default:"DEFAULT_POSITION"> ) " \