We delay the audio a bit to keep packet intervals equal, which keeps
some data in buffers.
In theory the calculation keeps one buffer free, but it doesn't
explicitly keep "extra" buffer space so in theory might flush too late
and next process() might not have free buffers. However, as we encode
next packet right away this shouldn't really occur...
Try to keep one extra spare buffer free so that the flush time is
certainly early enough.