Opus was integrated as a completely separate code path to the PCM audio
processing found in audio.c. This is actually not ideal, since the only
part that actually is Opus specific is the part that en- and decodes from
and to PCM. The rest is 1:1 the same PCM handling.
For this reason, it is much better to instead add audio codec support to
audio.c, meaning that the code in there can now encode PCM audio right
before sending it out as RTP, and decode incoming packets to PCM right
before actually processing the decoded audio data.
This significantly modifies how stream.c initializes the PCM audio path,
since the audio codec feature is new. It now treats the Opus subtype
as an audio codec selector instead of a selector for an entirely
alternate code path (like how MIDI integration remains entirely separate).
Since audio codecs usually require their frames to be decoded in order,
this also integrates the RTP jitter buffer in the RTP module.
Opus is now integrated as such a codec in audio.c. When it is selected,
incoming packets in rtp_audio_receive() are first inserted into the
jitter buffer. That buffer then outputs packets in order, and then, these
packets are decoded to PCM. The rest of the processing chain goes as usual.
A similar route is used for when the jitter buffer signals packet loss
to be able to apply PLC.
For encoding, it is similar (except that no jitter buffer is involved);
in rtp_audio_flush_packets(), when Opus is active, the PCM data is
rerouted to be fed to Opus for encoding, and the Opus output is then
placed into the iovec array instead of the original PCM.
This also improves overall Opus support; it supports S16 PCM data in
addition to F32 data, correctly checks the ptime, sample rate etc. for
Opus compatibility, computes an ideal bitrate, allows for manual bitrate
selection and encoding complexity adjustment (via the new stream properties
"opus.encoder.bitrate" and "opus.encoder.complexity"), sets several other
Opus CTLs to fixed values, supports the Opus restricted-lowdelay mode
(sacrifices Speech code paths for lower latency, enabled by setting the
"opus.encoder.restricted-lowdelay" stream property to true), and also uses
Opus' PLC in case of packet loss.
The audio codec interface is designed such that adding other codecs in
the future is easily doable. New integrations need to implement the
function pointers found in the rtp_audio_codec structure, and expose
an instance of such a custom rtp_audio_codec structure instance (see
the get_rtp_opus_codec() implementation for an example).
This makes it possible to dynamically add / remove receivers, which is
necesary for sending to multiple receivers. Mixed multi- and unicast
receivers are possible. Example pw-cli calls (56 is the ID of the RTP
sink node):
pw-cli c 56 User '{ extra="{ \"command.id\" : \"add-receiver\" , \"destination.ip\" : \"10.42.0.1\", \"destination.port\" : 55001 }" }'
pw-cli c 56 User '{ extra="{ \"command.id\" : \"remove-receiver\", \"destination.ip\" : \"10.42.0.1\" }" }'
pw-cli c 56 User '{ extra="{ \"command.id\" : \"clear-receivers\" }" }'
Commands and their arguments:
* "add-receiver" : Adds a receiver to the sink's list. If the given
IP address <-> port combination was already added, the command is
logged, but otherwise ignored. Arguments:
- "destination.ip" : IP address to send data to. Can be a uni- or
multicast address, but must be a valid address.
- "destination.port" : Port to send data to. Must be valid.
- "local.ifname", "source.ip", "net.ttl", "net.dscp", "net.loop" :
These are all optional, and work just like in the RTP sink
module's properties.
* "remove-receiver" : Removes a receiver from the sink's list. The
receiver is identified by the given IP address. A port can optionally
be specified as well. If it isn't, then the first receiver with that IP
address is removed. If no matching receiver is in the sink's list,
this command does nothing. Arguments:
- "destination.ip" : IP address to send data to. Can be a uni- or
multicast address, but must be a valid address.
- "destination.port" : Port to send data to. This is optional. But, if
it is set, it must be a valid port number.
* "clear-receivers" : Removes all receivers from the sink's list. If the
list is empty, this does nothing. This command has no arguments.
If the RTP sink module is created with the "destination.ip" and
"destination.port" properties set, it behaves as if "add-receiver" were
called right after the module was initialized. This means that if none
of these commands are used, the module behaves just as it did prior to
this patch. Note that the "remove-receivers" command can remove this
initial receiver as well.
If no receivers are added, the module continues to work normally.
Adding and removing receivers mid-operation is supported.
NOTE: "destination.ip") handling in stream_props_changed() is removed,
since it never really did anything other than change the param value.
rtp_stream_new() acquires a data loop with pw_context_acquire_loop() but
the out: error path never calls pw_context_release_loop(), leaking the loop
reference on every failure after acquisition.
Mirror rtp_stream_destroy() and other modules that pair acquire with release.
This fixes code duplication, since the checks are the same regardless
of payload type. hlen is also calculated the same across payload types,
so it is moved as well. The impl->receive_rtp() function pointer then
solely addresses the actual payload processing.
Rename format_info to rtp_format_info to make the purpose of that struct
and the associated variable clearer.
Document what info and stream_info are there for, since their purpose
is not immediately obvious.
Using find_audio_format_info() for Opus and MIDI makes no sense, since
both of these only have one info each, which is always found, so one might
as well just use that single, always-matching RTP format info directly.
This means that the audio_format_info array remains there solely for PCM,
so rename it to rtp_pcm_audio_format_info.
Keep the last relation between the sequence number and the timestamp
(ringbuffer position).
When a retransmission is requested for a given sequence number use the
relation to calculate the corresponding timestamp and retransmit the
packet from the ringbuffer again.
See #5276
They are emited from the streaming thread and therefore can be emitted
concurrently with the events on the main thread. This can cause crashes
when the hook list is iterated.
Instead, make those events into callbacks that are more efficient,
and threadsafe.
When the driver changes, the clock position can also change and there
would be a discont in the rtp_timestamp.
This is not usually a problem except in RAOP mode where the base rtp
timestamp is negotiated and anything that deviates too much is to be
discarded.
If we are not using direct_timestamp for the sender, make sure we always
keep the rtp_time aligned to avoid this problem.
See #5167
Clear the ringbuffer in stream_stop() when processing stops to prevent old invalid packets
from being sent when processing resumes via rtp_audio_flush_packets().
This ensures a clean state when the stream restarts.
The started boolean is insufficient to fully cover the possible internal
states. For this reason, it needs to be replaced by an enum that covers
these states.
Also, due to potential access by both the dataloop and the mainloop,
access to that internal state needs to be synchronized.
Finally, a variable "internal_state" makes for code that is easier to
read, since it emphasizes that this is state that is fully internal
inside the stream (and is not visible to the rtp-sink and rtp-source
modules for example).
The state_changed callbacks fulfill multiple roles, which is both a problem
regarding separation of concerns and regarding code clarity. De facto,
these callbacks cover error reporting, opening connections, and closing
connection, all in one, depending on a state that is arguably an internal
stream detail. The code in these callbacks tie these internal states to
assumptions that opening/closing callbacks is directly tied to specific
state changes in a common way, which is not always true. For example,
stopping the stream may not _actually_ stop it if a background send timer
is still running.
The notion of a "state_changed" callback is also problematic because the
pw_streams that are used in rtp-sink and rtp-source also have a callback
for state changes, causing confusion.
Solve this by replacing state_changed with three new callbacks:
1. report_error : Used for reporting nonrecoverable errors to the caller.
Note that currently, no one does such error reporting, but the feature
does exist, so this callback is introduced to preserve said feature.
2. open_connection : Used for opening a connection. Its optional return
value informs about success or failure.
3. close_connection : Used for opening a connection. Its optional return
value informs about success or failure.
Importantly, these callbacks do not export any internal stream state. This
improves encapsulation, and also makes it possible to invoke these
callbacks in situations that may not neatly map to a state change. One
example could be to close the connection as part of a stream_start call
to close any connection(s) left over from a previous run. (Followup commits
will in fact introduce such measures.)
config.h needs to be consistently included before any standard headers
if we ever want to set feature test macros (like _GNU_SOURCE or whatever)
inside. It can lead to hard-to-debug issues without that.
It can also be problematic just for our own HAVE_* that it may define
if it's not consistently made available before our own headers. Just
always include it first, before everything.
We already did this in many files, just not consistently.
There is no need to encode the potential format in the format.dsp of
control ports, this is just for legacy compatibility with JACK apps. The
actual format can be negotiated with the types field.
Fixes midi port visibility with apps compiled against 1.2, such as JACK
apps in flatpaks.
Idle the source when no packets are received and resume when new packets
arrive.
Add a stream.may-pause property to pause the stream when no packets are
received during the timeout window.
Make sure the rtp.streaming property is updated correctly and as soon as
we get the first packet.
Fixes#4456
When not using PTP as the driver, it is possible that packet receive and
the process() callback are out of sync, meaning that the target buffer
fill level might be off by upto one ptime's worth of samples
occasionally. This would make the DLL hunt for the target rate, and
cause a constantly varying delay.
Accounting for the delta between the packet receive time and the
process() time allows us to eliminate this jitter, resulting in much
more consistent rate matching.
Make a function that can initialize raw audio info from a dict and fill
in the defaults. We can use this in many of the modules when the audio
format is parsed.
Use the helper instead of duplicating the same code.
Also add some helpers to parse a json array of uint32_t
Move some functions to convert between type name and id.
Add spa_json_begin_array/object to replace
spa_json_init+spa_json_begin_array/object
This function is better because it does not waste a useless spa_json
structure as an iterator. The relaxed versions also error out when the
container is mismatched because parsing a mismatched container is not
going to give any results anyway.
Our current AES67 sender setup requires that that PTP driver drive the
entire graph. This adds support for allowing the AES67 RTP sink to be
driven by an arbitrary driver, while still using the PTP driver for
sending data on the network.
When aes67.driver-group is specified a pw_filter is created with no
ports, node.always-process = true and node.group set to the
aes67.driver-group. When set to PTP, this gives us process callbacks at
the PTP rate which we use to get the current PTP time in the RTP sender
by interpolating the clock snapshots from the pw-filter.
Implementation ideas from Wim Taymans. Co-authored with Sanchayan Maity.
For a detailed reference, refer the following papers by Fons Adriaensen.
- Using a DLL to filter time
(https://kokkinizita.linuxaudio.org/papers/usingdll.pdf)
- Controlling adaptive resampling
(http://kokkinizita.linuxaudio.org/papers/adapt-resamp.pdf)
32 bits are enough, and additionally this also fixes an incorrect
format string, which caused the default `audio.rate` to be
incorrectly set on some platforms, such as 32-bit arm ones.
Fixes#4080
Expose the acquire_loop/release_loop functions and use them in the
modules.
Make sure the nodes created from the module use the same data loop as
the module. We need to ensure this because otherwise, the nodes might
be scheduled on different data loops and the invoke or timer logic will
fail.
