connection: Ensure buffer sizes do not exceed INT_MAX or PTRDIFF_MAX

Pointer arithmetic beyond PTRDIFF_MAX is broken, so buffer sizes
exceeding PTRDIFF_MAX (which is half of the address space!) are a bad
idea.  Furthermore, the code uses int for sizes in various places, so
buffer sizes exceeding INT_MAX are also a bad idea.  Therefore, limit
buffer sizes to (PTRDIFF_MAX / 2) + 1 or (INT_MAX / 2) + 1, whichever
is smaller.

Tests would require 2GiB of RAM and so have been omitted.  The test
would check for wl_connection_flush() flushing more than INT_MAX bytes
in one call, causing it to return a negative number and causing its
caller to wrongly believe an error occurred.

Signed-off-by: Demi Marie Obenour <demi@invisiblethingslab.com>
This commit is contained in:
Demi Marie Obenour 2024-08-05 12:17:02 -04:00
parent f8ecb6d531
commit 2ee9ef098b

View file

@ -32,6 +32,7 @@
#include <string.h>
#include <stdio.h>
#include <errno.h>
#include <limits.h>
#include <sys/uio.h>
#include <fcntl.h>
#include <unistd.h>
@ -70,7 +71,19 @@ struct wl_connection {
int fd;
int want_flush;
};
#define WL_BUFFER_MAX_SIZE_POT ((size_t)(8 * sizeof(size_t) - 1))
/* Pointer arithmetic beyond PTRDIFF_MAX is broken, so don't rely on it.
* ((ptrdiff_t)1 << (CHAR_BIT * sizeof(ptrdiff_t) - 1)) is undefined behavior
* (wrapping to PTRDIFF_MIN with -fwrapv), not PTRDIFF_MAX, so limit buffer size
* to ((size_t)1 << (CHAR_BIT * sizeof(ptrdiff_t) - 2)). int is used for
* sizes in various places, so for safety also use
* ((size_t)1 << (CHAR_BIT * sizeof(int) - 2)) as a limit.
*/
#if PTRDIFF_MAX < INT_MAX
# define WL_BUFFER_MAX_SIZE_POT ((size_t)(CHAR_BIT * sizeof(ptrdiff_t) - 2))
#else
# define WL_BUFFER_MAX_SIZE_POT ((size_t)(CHAR_BIT * sizeof(int) - 2))
#endif
static inline size_t
size_pot(uint32_t size_bits)