Opus was integrated as a completely separate code path to the PCM audio
processing found in audio.c. This is actually not ideal, since the only
part that actually is Opus specific is the part that en- and decodes from
and to PCM. The rest is 1:1 the same PCM handling.
For this reason, it is much better to instead add audio codec support to
audio.c, meaning that the code in there can now encode PCM audio right
before sending it out as RTP, and decode incoming packets to PCM right
before actually processing the decoded audio data.
This significantly modifies how stream.c initializes the PCM audio path,
since the audio codec feature is new. It now treats the Opus subtype
as an audio codec selector instead of a selector for an entirely
alternate code path (like how MIDI integration remains entirely separate).
Since audio codecs usually require their frames to be decoded in order,
this also integrates the RTP jitter buffer in the RTP module.
Opus is now integrated as such a codec in audio.c. When it is selected,
incoming packets in rtp_audio_receive() are first inserted into the
jitter buffer. That buffer then outputs packets in order, and then, these
packets are decoded to PCM. The rest of the processing chain goes as usual.
A similar route is used for when the jitter buffer signals packet loss
to be able to apply PLC.
For encoding, it is similar (except that no jitter buffer is involved);
in rtp_audio_flush_packets(), when Opus is active, the PCM data is
rerouted to be fed to Opus for encoding, and the Opus output is then
placed into the iovec array instead of the original PCM.
This also improves overall Opus support; it supports S16 PCM data in
addition to F32 data, correctly checks the ptime, sample rate etc. for
Opus compatibility, computes an ideal bitrate, allows for manual bitrate
selection and encoding complexity adjustment (via the new stream properties
"opus.encoder.bitrate" and "opus.encoder.complexity"), sets several other
Opus CTLs to fixed values, supports the Opus restricted-lowdelay mode
(sacrifices Speech code paths for lower latency, enabled by setting the
"opus.encoder.restricted-lowdelay" stream property to true), and also uses
Opus' PLC in case of packet loss.
The audio codec interface is designed such that adding other codecs in
the future is easily doable. New integrations need to implement the
function pointers found in the rtp_audio_codec structure, and expose
an instance of such a custom rtp_audio_codec structure instance (see
the get_rtp_opus_codec() implementation for an example).
This makes it possible to dynamically add / remove receivers, which is
necesary for sending to multiple receivers. Mixed multi- and unicast
receivers are possible. Example pw-cli calls (56 is the ID of the RTP
sink node):
pw-cli c 56 User '{ extra="{ \"command.id\" : \"add-receiver\" , \"destination.ip\" : \"10.42.0.1\", \"destination.port\" : 55001 }" }'
pw-cli c 56 User '{ extra="{ \"command.id\" : \"remove-receiver\", \"destination.ip\" : \"10.42.0.1\" }" }'
pw-cli c 56 User '{ extra="{ \"command.id\" : \"clear-receivers\" }" }'
Commands and their arguments:
* "add-receiver" : Adds a receiver to the sink's list. If the given
IP address <-> port combination was already added, the command is
logged, but otherwise ignored. Arguments:
- "destination.ip" : IP address to send data to. Can be a uni- or
multicast address, but must be a valid address.
- "destination.port" : Port to send data to. Must be valid.
- "local.ifname", "source.ip", "net.ttl", "net.dscp", "net.loop" :
These are all optional, and work just like in the RTP sink
module's properties.
* "remove-receiver" : Removes a receiver from the sink's list. The
receiver is identified by the given IP address. A port can optionally
be specified as well. If it isn't, then the first receiver with that IP
address is removed. If no matching receiver is in the sink's list,
this command does nothing. Arguments:
- "destination.ip" : IP address to send data to. Can be a uni- or
multicast address, but must be a valid address.
- "destination.port" : Port to send data to. This is optional. But, if
it is set, it must be a valid port number.
* "clear-receivers" : Removes all receivers from the sink's list. If the
list is empty, this does nothing. This command has no arguments.
If the RTP sink module is created with the "destination.ip" and
"destination.port" properties set, it behaves as if "add-receiver" were
called right after the module was initialized. This means that if none
of these commands are used, the module behaves just as it did prior to
this patch. Note that the "remove-receivers" command can remove this
initial receiver as well.
If no receivers are added, the module continues to work normally.
Adding and removing receivers mid-operation is supported.
NOTE: "destination.ip") handling in stream_props_changed() is removed,
since it never really did anything other than change the param value.
In the past, it made no difference whether or not an RTP sink's
sess.ts-direct property was set to true. This changed in commit
6bf81ebe59 . Now, sess.ts-direct must be
set to true if an associated RTP source is using direct timestamp mode.
That's because when the RTP sink's direct timestamp mode is not enabled,
it will set the new internal ts_align offset to a nonzero value, which
then corrupts timestamps and causes out of sync playback in the RTP
sources.
Also, copy the sess.ts-direct property from the global RTP sink props
to the stream props to match the behavior of the RTP source module.
* Rename do_disconnect to do_disconnect_core, since the original name
is ambiguous (it can be interpreted as being about disconnecting
a network socket).
* impl->session_name is not used at all, so remove it.
The state_changed callbacks fulfill multiple roles, which is both a problem
regarding separation of concerns and regarding code clarity. De facto,
these callbacks cover error reporting, opening connections, and closing
connection, all in one, depending on a state that is arguably an internal
stream detail. The code in these callbacks tie these internal states to
assumptions that opening/closing callbacks is directly tied to specific
state changes in a common way, which is not always true. For example,
stopping the stream may not _actually_ stop it if a background send timer
is still running.
The notion of a "state_changed" callback is also problematic because the
pw_streams that are used in rtp-sink and rtp-source also have a callback
for state changes, causing confusion.
Solve this by replacing state_changed with three new callbacks:
1. report_error : Used for reporting nonrecoverable errors to the caller.
Note that currently, no one does such error reporting, but the feature
does exist, so this callback is introduced to preserve said feature.
2. open_connection : Used for opening a connection. Its optional return
value informs about success or failure.
3. close_connection : Used for opening a connection. Its optional return
value informs about success or failure.
Importantly, these callbacks do not export any internal stream state. This
improves encapsulation, and also makes it possible to invoke these
callbacks in situations that may not neatly map to a state change. One
example could be to close the connection as part of a stream_start call
to close any connection(s) left over from a previous run. (Followup commits
will in fact introduce such measures.)
Our current AES67 sender setup requires that that PTP driver drive the
entire graph. This adds support for allowing the AES67 RTP sink to be
driven by an arbitrary driver, while still using the PTP driver for
sending data on the network.
When aes67.driver-group is specified a pw_filter is created with no
ports, node.always-process = true and node.group set to the
aes67.driver-group. When set to PTP, this gives us process callbacks at
the PTP rate which we use to get the current PTP time in the RTP sender
by interpolating the clock snapshots from the pw-filter.
Implementation ideas from Wim Taymans. Co-authored with Sanchayan Maity.
For a detailed reference, refer the following papers by Fons Adriaensen.
- Using a DLL to filter time
(https://kokkinizita.linuxaudio.org/papers/usingdll.pdf)
- Controlling adaptive resampling
(http://kokkinizita.linuxaudio.org/papers/adapt-resamp.pdf)
This allows us to specify a static session ID, so that (Dante) receivers
can uniquely identify us as a sender. This prevents duplicate streams in
Dante Controller, for example.
Make a new function to set the rtp stream in the error state.
When we fail to start the stream, set the error state. Otherwise (like
when we try to use an invalid interface name) the socket create will
fail but the stream will still try to send data to the invalid socket.
Use `getaddrinfo` in `parse_address` instead of `inet_pton`.
Display Ipv6 addresses with scope identifiers correctly in `get_ip`
functions using `if_indextoname`.
Use the sess.latency.msec also for the sender and use it to control the
NODE_LATENCY. Make it a float to be in line with the other time values.
Set is to a default of ptime, which was what it used to be.
This makes it possible to set the ptime to a smaller value than the
sess.latency.msec so that we send out multiple packets per quantum.
This will result in some bursty output for now but with a timer that can
be improved later.
Update the docs a little, mention the new rtp.ptime and rtp.frametime.
The idea is to have the rtp-source and rtp-sink be separate modules that
can be loaded a static ip/port as well.
The SAP module would then expose existing rtp-sinks or additionally load
rtp-sources based on SAP messages.
This makes things a bit more flexible when we also want to expose the
rtp-sink using zeroconf, for example.
