shm: memfd_create: fallback to not using MFD_NOEXEC_SEAL

MFD_NOEXEC_SEAL was introduced in linux 6.3. Kernels before that
will *reject* memfd_create() calls that set it.

This caused foot to exit (i.e. not start at all), when compiled on
linux >= 6.3, but run on linux < 6.3.

We _do_ want to use MFD_NOEXEC_SEAL, since a) our memory mapped really
shouldn't be executable, and b) to silence a warning on linux >= 6.3.

To handle all cases, first try *with* MFD_NOEXEC_SEAL. If that fails
with EINVAL, retry *without* it.

Closes #1514
This commit is contained in:
Daniel Eklöf 2023-10-12 16:16:11 +02:00
parent 7d7b48f104
commit c006ac3a07
No known key found for this signature in database
GPG key ID: 5BBD4992C116573F
2 changed files with 21 additions and 5 deletions

View file

@ -53,6 +53,11 @@
### Deprecated
### Removed
### Fixed
* Foot not starting on linux kernels before 6.3 ([#1514][1514]).
[1514]: https://codeberg.org/dnkl/foot/issues/1514
### Security
### Contributors

21
shm.c
View file

@ -27,10 +27,8 @@
#define MAP_UNINITIALIZED 0
#endif
#if defined(MFD_NOEXEC_SEAL)
#define FOOT_MFD_FLAGS (MFD_CLOEXEC | MFD_ALLOW_SEALING | MFD_NOEXEC_SEAL)
#else
#define FOOT_MFD_FLAGS (MFD_CLOEXEC | MFD_ALLOW_SEALING)
#if !defined(MFD_NOEXEC_SEAL)
#define MFD_NOEXEC_SEAL 0
#endif
#define TIME_SCROLL 0
@ -339,7 +337,20 @@ get_new_buffers(struct buffer_chain *chain, size_t count,
/* Backing memory for SHM */
#if defined(MEMFD_CREATE)
pool_fd = memfd_create("foot-wayland-shm-buffer-pool", FOOT_MFD_FLAGS);
/*
* Older kernels reject MFD_NOEXEC_SEAL with EINVAL. Try first
* *with* it, and if that fails, try again *without* it.
*/
errno = 0;
pool_fd = memfd_create(
"foot-wayland-shm-buffer-pool",
MFD_CLOEXEC | MFD_ALLOW_SEALING | MFD_NOEXEC_SEAL);
if (pool_fd < 0 && errno == EINVAL) {
pool_fd = memfd_create(
"foot-wayland-shm-buffer-pool", MFD_CLOEXEC | MFD_ALLOW_SEALING);
}
#elif defined(__FreeBSD__)
// memfd_create on FreeBSD 13 is SHM_ANON without sealing support
pool_fd = shm_open(SHM_ANON, O_RDWR | O_CLOEXEC, 0600);