This adds some infrastructure for canceller implementations to also
perform acoustic gain control. Cancellers now have a couple of new API
calls that allow them to get/set capture volume.
This is made slightly complex by the fact that cancellation happens in
thread context while most volume mangling needs to be done in main
context. To deal with this, while getting the volume we save source
volume updates as they are propagated to thread context and use this
cached value for queries. To set the volume, we send an async message to
main context and let that set the source volume.
This dumps out an additional file with each line having a command of the
form:
p <number of playback samples processed>
c <number of capture samples processed>
d <drift as passed to set_drift()>
The test program can be provided this file to "replay" the data exactly
as when it was run live.
The non-drift-compensation path is retained as-is since it is much
simpler.
This adds the ability for echo cancellers to provide their own drift
compensation, and hooks in the appropriate bits to implement this in the
WebRTC canceller.
We do this by introducing an alternative model for the canceller. So
far, the core engine just provided a run() method which was given
blocksize-sized chunks of playback and record samples. The new model has
the engine provide play() and record() methods that can (in theory) be
called by the playback and capture threads. The latter would actually do
the processing required.
In addition to this a set_drift() method may be provided by the
implementation. PA will provide periodic samples of the drift to the
engine. These values need to be aggregated and processed over some time,
since the point values vary quite a bit (but generally fit a linear
regression reasonably accurately). At some point of time, we might move
the actual drift calculation into PA and change the semantics of this
function.
NOTE: This needs further testing before being deemed ready for wider use.
This adds the WebRTC echo canceller as another module-echo-cancel
backend. We're exposing both the full echo canceller as well as the
mobile echo control version as modargs.
Pending items:
1. The mobile canceller doesn't seem to work at the moment.
2. We still need to add bits to hook in drift compensation (to support
sink and source from different devices).
The most controversial part of this patch would probably be the
mandatory build-time dependency on a C++ compiler. If the optional
--enable-webrtc-aec is set, then there's also a dependency on libstdc++.
This makes sure that we only perform any processing (resync or actual
cancellation) after the source provides enough data to actuall run the
canceller.
When a source-output isn't connected to our virtual source, we skip echo
cancellation altogether. This makes sense in general, and makes sure
that we don't end up adjusting for delay/drift when nothing is
connected. This should make convergence faster when the canceller
actually starts being used.
This increase the threshold for difference between the playback and
capture stream before samples are dropped from 1ms to 5ms (the
cancellers are generally robust to this much and higher). Also, we make
this a module parameter to allow easier experimentation with different
values.
Loading between a sink and its monitor causes a deadlock (while sending
messages for latency snapshots). It isn't a case that has any real
conceivable use, so let's just disallow it.
These are not used for anything at this point, but this
makes it easy to add ad-hoc debug prints that show the
memblockq name and to convert between bytes and usecs.
Uses the shared volume infrastructure by default with an option to
fallback on the old pretend-volume-sharing-that-kind-of-works if someone
wants it that way.
Users who keep left != right (or any sort of unbalanced channel volumes)
will likely want to disable shared volumes since it will cause their
master sink/source volume to be balanced.
This really isn't a very pleasant scenario since users would need to
manually set up echo cancellation in their config for this (until we
have a way to store module configuration). That said, the majority case
benefits from the volume sharing, so let's not wait for the
configuration infrastructure to be ready to use this.
When unloading, some module may end up trin to move a sink-input or
source-output back onto our virtual sink/source, causing an infinite
loop of us moving the stream away and having it moved back.
We prevent this from happening by preventing any stream from being
attached during unload.
I initially included put the Speex preprocessing assuming that we'd want
to use the digital gain control and noise suppression from Speex for all
echo cancelling implementations. In practice, we're probably going to
get entire implementations all processing in one package (WebRTC, custom
modules from various vendors, etc.).
This moves out this preprocessing and related knobs into the speex
implementation, which serves to clean out all implementation-specific
details from the module-echo-cancel core.
Some sink flags are really just a product of what callbacks
are set on the device. We still enforce a degree of sanity
that the flags match the callbacks set, but we also set the
flags automatically in our callback setter functions to
help ensure that a) people use them and b) flags & callbacks
are kept in sync.
This is not currently useful but future commits will make further
changes concerning automatic setting of flags and event delivery
that makes this structure necessary.
This picks sane defaults for the sample spec used (32 kHz, mono) and
preprocessing (on by default). This should make it unncessary to provide
additional parameters in the default desktop case.
The main exception would be decreasing the sample rate for hardware with
limited processing power (can bring it down to 16 or 8 kHz).
This piggy backs onto the previous changes for protocol 22 and
thus does not bump the version. This and the previous commits should be
seen as mostly atomic. Apologies for any bisecting issues this causes
(although I would expect these to be minimal)
We were using the block size in bytes instead of samples, which meant
preprocessing was broken. This fix makes a large-ish difference in the
quality of echo-cancellation with speex.
The echo suppress attenuation value was being incorrectly modified.
Fixed and added 2 arguments to change the attenuation of the residual
echo filter. Default values of the speex preprocessor will be used when
omitted.
This allows the selective enabling of speex' preprocessing algorithms
before running the echo-canceller -- for now this includes automatic
gain control, noise suppression and echo suppression. It's all off by
default for now, though at some point in the near future we might want
to enable at least denoising by default.
The denoising works pretty well, though we might want to add a way to
tweak the noise-suppression knob that libspeex provides.
The AGC option is just a stop-gap -- we need a real AGC mechanism that
tweaks the source volume rather than doing this in software.
The speex documentation mentions VAD and dereverb, but it appears that
these are not complete yet.
We don't do all this in a separate module from module-echo-cancel to
avoid the overhead of adding another virtual source. It makes more sense
to make a separate virtual source module that can be used for cases
where preprocessing is useful but AEC is not (for e.g. noise suppression
for fan noise in a recording application).
Another reason to keep this integrated with the AEC module is that the
echo suppression bits use the speex echo canceller state. This does leak
some information about the AEC implementation into module-echo-cancel,
but this is unavoidable.
This is the beginning of work to support compressed formats natively in
PulseAudio. This adds a pa_stream_new_extended() that takes a format
structure, sends it to the server (=> protocol extension) and has the
server negotiate with the appropropriate sink to figure out what format
it should use.
This is work in progress, and works only with PCM streams. Actual
compressed format support in some sink needs to be implemented, and
extensive testing is required.
More details on how this is supposed to work is available at:
http://pulseaudio.org/wiki/PassthroughSupport
(Based on Colin's review) We mark modules as being autoloaded so that
they can handle this as a special case if needed (which is required by
module-echo-cancel for now). This inverts how things were done and makes
using these modules manually less error-prone.
With automaticl filter loading by module-filter-apply, setting the
virtual sink/source to have the "phone" intended role will break routing
when you first connect a phone stream to an ALSA device and then turn on
your Bluetooth headset. This happens because module-intended-roles
doesn't move a stream if it is already on a device that provides the
required role.
This patch introduces a "manual_load" parameter that is meant to be used
when not using module-filter-apply for loading the AEC module. If this
parameter is set, the virtual devices are given the "phone" role, else
we count on module-filter-heuristics to do the right thing.
Mostly warnings about unused stuff.
Furthermore, the first hunk is a fix for the change in 177948a6.
Finally, comment in AEC_dtd was translated and the code simplified slightly.
CC module_bluetooth_device_la-module-bluetooth-device.lo
modules/bluetooth/module-bluetooth-device.c: In function ‘a2dp_process_render’:
modules/bluetooth/module-bluetooth-device.c:1335:30: warning: pointer targets in passing argument 6 of ‘sbc_encode’
differ in signedness [-Wpointer-sign]
../src/modules/bluetooth/sbc/sbc.h:92:9: note: expected ‘ssize_t *’ but argument is of type ‘size_t *’
CC module_rygel_media_server_la-module-rygel-media-server.lo
modules/module-rygel-media-server.c:383:13: warning: ‘append_property_dict_entry_object_array’ defined but not used [-Wunused-function]
CC module_echo_cancel_la-adrian-aec.lo
modules/echo-cancel/adrian-aec.h:360:15: warning: ‘AEC_getambient’ defined but not used [-Wunused-function]
modules/echo-cancel/adrian-aec.h:368:14: warning: ‘AEC_setgain’ defined but not used [-Wunused-function]
modules/echo-cancel/adrian-aec.h:374:14: warning: ‘AEC_setaes’ defined but not used [-Wunused-function]
modules/echo-cancel/adrian-aec.h:377:16: warning: ‘AEC_max_dotp_xf_xf’ declared ‘static’ but never defined [-Wunused-function]
CC module_echo_cancel_la-module-echo-cancel.lo
modules/echo-cancel/module-echo-cancel.c: In function ‘time_callback’:
modules/echo-cancel/module-echo-cancel.c:266:12: warning: variable ‘fs’ set but not used [-Wunused-but-set-variable]
CC module-virtual-sink.lo
modules/module-virtual-sink.c: In function ‘sink_input_pop_cb’:
modules/module-virtual-sink.c:206:15: warning: variable ‘current_latency’ set but not used [-Wunused-but-set-variable]
This change doesn't add any functionality in itself, but it will be useful in
the future for operating on chains of sinks or sources that are piggy-backing
on each other.
For example, the PA_PROP_DEVICE_MASTER_DEVICE property could
be handled in the core so that each virtual device doesn't have to maintain it
separately. By using the origin_sink and destination_source pointers the core
is able to see at stream creation time that the stream is created by a virtual
device, and then update that device's property list using the name of the
master device that the stream is being connected to. The same thing can be done
also when the stream is being moved from a device to another, in which case the
_MASTER_DEVICE property needs updating.
This is required to make sure that the source output between
module-echo-cancel and ALSA can't get plugged to the virtual source or
monitor of the virtual sink that we expose. This could be triggered by
changing the profile of the underlying ALSA device.
Optimises the core inner-product function, which takes the most CPU. The
SSE-optimised bits of the adrian echo canceller only if the CPU that PA
is running on actually supports SSE.
