SDL: wayland: Cleanup inset implementation

From 86b89cff405ed00f5fcf9dbb1843b693163efd79 Mon Sep 17 00:00:00 2001
From: Frank Praznik <[EMAIL REDACTED]>
Date: Thu, 24 Sep 2026 13:02:16 -0400
Subject: [PATCH] wayland: Cleanup inset implementation

Re-flow the documentation to avoid overly long lines, disable insets when display scaling is enabled, and skip a large inset-related code block when configuring window geometry if it's not needed.
---
 docs/README-wayland.md                | 15 +++++++++------
 src/video/wayland/SDL_waylandwindow.c |  9 ++++++---
 2 files changed, 15 insertions(+), 9 deletions(-)

diff --git a/docs/README-wayland.md b/docs/README-wayland.md
index f52499acf0732..3090eaa4064ae 100644
--- a/docs/README-wayland.md
+++ b/docs/README-wayland.md
@@ -25,12 +25,15 @@ encounter limitations or behavior that is different from other windowing systems
 ### Border insets for client-side decorations
 
 - Applications drawing client-side decorations (shadows or invisible resize borders) can declare their insets via the
-  `SDL_PROP_WINDOW_WAYLAND_BORDER_INSET_*` window properties when the window is created with the `SDL_PROP_WINDOW_CREATE_WAYLAND_ENABLE_INSETS_BOOLEAN` property set to `true`. SDL then sets the xdg window geometry to the surface minus
-  the insets (in points, regardless of pixel density) and translates configure sizes and size limits accordingly.
-  Maximized and fullscreen windows are always sized exactly as configured, while tiled windows keep their insets.
-  Changes take effect on the next configure. Only xdg-toplevel windows are affected; popup and custom-role surfaces
-  ignore the properties, as does libdecor, which manages the window geometry itself
-  (see `SDL_HINT_VIDEO_WAYLAND_ALLOW_LIBDECOR`).
+  `SDL_PROP_WINDOW_WAYLAND_BORDER_INSET_*` window properties when the window is created with the
+  `SDL_PROP_WINDOW_CREATE_WAYLAND_ENABLE_INSETS_BOOLEAN` property set to `true`. SDL then sets the xdg window geometry
+  to the surface minus the insets (in points, regardless of pixel density) and translates configure sizes and size
+  limits accordingly. Maximized and fullscreen windows are always sized exactly as configured, while tiled windows keep
+  their insets.  Changes take effect on the next configure. Only xdg-toplevel windows are affected; popup and
+  custom-role surfaces ignore the properties, as does libdecor, which manages the window geometry itself (see 
+  `SDL_HINT_VIDEO_WAYLAND_ALLOW_LIBDECOR`). Insets are also incompatible with display scaling mode, and will be
+  automatically disabled when `SDL_HINT_VIDEO_WAYLAND_SCALE_TO_DISPLAY` is enabled.
+  
 
 ### Windows do not appear immediately after creation
 
diff --git a/src/video/wayland/SDL_waylandwindow.c b/src/video/wayland/SDL_waylandwindow.c
index e3b0ebafec622..31559decddc49 100644
--- a/src/video/wayland/SDL_waylandwindow.c
+++ b/src/video/wayland/SDL_waylandwindow.c
@@ -644,7 +644,8 @@ static void ConfigureWindowGeometry(SDL_Window *window)
         data->mask.mapped = false;
     }
 
-    if (data->shell_surface_type == WAYLAND_SHELL_SURFACE_TYPE_XDG_TOPLEVEL && data->shell_surface.xdg.surface) {
+    if (data->shell_surface_type == WAYLAND_SHELL_SURFACE_TYPE_XDG_TOPLEVEL && data->shell_surface.xdg.surface &&
+        (data->enable_insets || !data->viewport)) {
         /* The window geometry is the surface minus the declared border insets. They don't apply
          * when an exact size is required (maximized/fullscreen).
          */
@@ -1189,7 +1190,7 @@ static void handle_xdg_toplevel_configure(void *data,
      * sizes adopted from the cached window size are converted back to geometry space for
      * the last_configure store.
      */
-    const bool insets_apply = !fullscreen && !maximized;
+    const bool insets_apply = wind->enable_insets && !fullscreen && !maximized;
 
     // When resizing, dimensions other than 0 are a maximum.
     const bool new_configure_size = width != wind->last_configure.width || height != wind->last_configure.height ||
@@ -3167,7 +3168,9 @@ bool Wayland_CreateWindow(SDL_VideoDevice *_this, SDL_Window *window, SDL_Proper
     }
 
     window->internal = data;
-    data->enable_insets = SDL_GetBooleanProperty(create_props, SDL_PROP_WINDOW_CREATE_WAYLAND_ENABLE_INSETS_BOOLEAN, false);
+    if (!c->scale_to_display_enabled) {
+        data->enable_insets = SDL_GetBooleanProperty(create_props, SDL_PROP_WINDOW_CREATE_WAYLAND_ENABLE_INSETS_BOOLEAN, false);
+    }
 
     if (window->x == SDL_WINDOWPOS_UNDEFINED) {
         window->x = 0;