SDL-gui-backend merged

SDL Development
1

The first piece of my SDL-gui-backend experiment has been merged into
the main SDL repository. I’ve been looking at what SDL needs to be
useful as a backend for GUI toolkits. This isn’t so much a question of
"how do I build a UI with SDL," but more like, “what would something
like GTK+ need to talk exclusively to SDL instead of X11?”

As a first target, we added what Unreal Engine 4’s Slate UI toolkit
would need to never talk directly to X11 on Linux, since the added
requirements were pretty modest.

Here’s the merge, modulo a few patches on top to get it to compile:

 https://hg.libsdl.org/SDL/rev/dc80dc0bd22e

The documentation on the new functions are right at the top of the
patch, if you scroll down a little. All of the new stuff works on Mac,
Windows, and X11, probably isn’t relevant to iOS/Android/etc, and still
needs support for Wayland, Mir, and Haiku.

This experiment will continue. I’m not sure what else, if anything else,
will be added, yet. There’s a delicate balance to SDL, and you don’t
want to go crazy adding things like this, especially when the primary
use is games, but I’m interested in finding cases like Slate–can mostly
use SDL, just needed X11 for a few special items–and seeing if we can
make those cases more portable.

–ryan.

2

2014-06-26 3:57 GMT+02:00 Ryan C. Gordon :

The first piece of my SDL-gui-backend experiment has been merged into the
main SDL repository. I’ve been looking at what SDL needs to be useful as a
backend for GUI toolkits. This isn’t so much a question of “how do I build
a UI with SDL,” but more like, “what would something like GTK+ need to talk
exclusively to SDL instead of X11?”

As a first target, we added what Unreal Engine 4’s Slate UI toolkit would
need to never talk directly to X11 on Linux, since the added requirements
were pretty modest.

Here’s the merge, modulo a few patches on top to get it to compile:

https://hg.libsdl.org/SDL/rev/dc80dc0bd22e

The documentation on the new functions are right at the top of the patch,
if you scroll down a little. All of the new stuff works on Mac, Windows,
and X11, probably isn’t relevant to iOS/Android/etc, and still needs
support for Wayland, Mir, and Haiku.

This experiment will continue. I’m not sure what else, if anything else,
will be added, yet. There’s a delicate balance to SDL, and you don’t want
to go crazy adding things like this, especially when the primary use is
games, but I’m interested in finding cases like Slate–can mostly use SDL,
just needed X11 for a few special items–and seeing if we can make those
cases more portable.

–ryan.

Interesting stuff!

SDL_GetGlobalMouseState(int *x, int *y)

What are you planning to do about platforms where this is impossible by
design (eg. Wayland)?

Jonas

3

What are you planning to do about platforms where this is impossible by
design (eg. Wayland)?

Why is that impossible by design?

–ryan.

4

2014-06-26 5:23 GMT+02:00 Ryan C. Gordon :

What are you planning to do about platforms where this is impossible by

design (eg. Wayland)?

Why is that impossible by design?

Wayland’s concept of being secure includes not exposing any global
information to clients,
including global mouse position and window positions. You only get mouse
events when the
cursor is inside one of you windows (or an implicit grab is in progress).

5

You just return the current mouse coordinates in global space when it’s
over one of your application windows.On Wed, Jun 25, 2014 at 8:53 PM, Jonas Kulla wrote:

2014-06-26 5:23 GMT+02:00 Ryan C. Gordon :

What are you planning to do about platforms where this is impossible by

design (eg. Wayland)?

Why is that impossible by design?

Wayland’s concept of being secure includes not exposing any global
information to clients,
including global mouse position and window positions. You only get mouse
events when the
cursor is inside one of you windows (or an implicit grab is in progress).

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

6

2014-06-26 6:01 GMT+02:00 Sam Lantinga :

You just return the current mouse coordinates in global space when it’s
over one of your application windows.

How do you define “global space” without knowing your window’s position on
the screen?>

On Wed, Jun 25, 2014 at 8:53 PM, Jonas Kulla <@Jonas_Kulla> wrote:

2014-06-26 5:23 GMT+02:00 Ryan C. Gordon :

What are you planning to do about platforms where this is impossible by

design (eg. Wayland)?

Why is that impossible by design?

Wayland’s concept of being secure includes not exposing any global
information to clients,
including global mouse position and window positions. You only get mouse
events when the
cursor is inside one of you windows (or an implicit grab is in progress).

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

7

You don’t know your window position in Wayland?? :)On Wed, Jun 25, 2014 at 9:29 PM, Jonas Kulla wrote:

2014-06-26 6:01 GMT+02:00 Sam Lantinga <@slouken>:

You just return the current mouse coordinates in global space when it’s

over one of your application windows.

How do you define “global space” without knowing your window’s position on
the screen?

On Wed, Jun 25, 2014 at 8:53 PM, Jonas Kulla wrote:

2014-06-26 5:23 GMT+02:00 Ryan C. Gordon :

What are you planning to do about platforms where this is impossible by

design (eg. Wayland)?

Why is that impossible by design?

Wayland’s concept of being secure includes not exposing any global
information to clients,
including global mouse position and window positions. You only get mouse
events when the
cursor is inside one of you windows (or an implicit grab is in progress).

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

8

2014-06-26 6:33 GMT+02:00 Sam Lantinga :

You don’t know your window position in Wayland?? :slight_smile:

That’s what I just said two emails ago… But hey, this time I actually have
a reference [1]
to point to! =P

[1] http://wayland.freedesktop.org/docs/html/sect-Protocol-Surface.html>

On Wed, Jun 25, 2014 at 9:29 PM, Jonas Kulla <@Jonas_Kulla> wrote:

2014-06-26 6:01 GMT+02:00 Sam Lantinga :

You just return the current mouse coordinates in global space when it’s

over one of your application windows.

How do you define “global space” without knowing your window’s position
on the screen?

On Wed, Jun 25, 2014 at 8:53 PM, Jonas Kulla <@Jonas_Kulla> wrote:

2014-06-26 5:23 GMT+02:00 Ryan C. Gordon :

What are you planning to do about platforms where this is impossible

by
design (eg. Wayland)?

Why is that impossible by design?

Wayland’s concept of being secure includes not exposing any global
information to clients,
including global mouse position and window positions. You only get
mouse events when the
cursor is inside one of you windows (or an implicit grab is in
progress).

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

9

While not strictly related to GUI toolkits, this does bring up a
question I have about the mouse state in SDL2.

Something that was recently brought up for both FNA and MonoGame was the
accuracy of MouseState. Basically, we still have the problem of the
mouse state only caring about the coordinates within the window, when
XNA4 can actually provide coordinates both <0 and >ClientBounds.

It looks like something similar to SDL_GetGlobalMouseState would work,
but with this call we would still need to get the mouse state relative
to the window and subtract to get XNA4 behavior… and if
SDL_GetGlobalMouseState is as inefficient as the docs note, this
probably isn’t something to use in addition to SDL_GetWindowPosition
every time the client asks for MouseState. On the other hand, something
like SDL_CaptureMouse would be pretty nice to get those negative
coordinates (for example) right away, but the rest of this call’s
behavior would make things messy really quickly…

TL;DR: Is there any use for something like SDL_GetGlobalMouseState
except that it provides the coordinates that SDL_CaptureMouse would
provide, or is this something that only weird projects like FNA would need?

-Ethan> Date: Wed, 25 Jun 2014 21:57:07 -0400

From: “Ryan C. Gordon”
To: sdl at lists.libsdl.org
Subject: [SDL] SDL-gui-backend merged…
Message-ID: <53AB7DF3.7010007 at icculus.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

The first piece of my SDL-gui-backend experiment has been merged into
the main SDL repository. I’ve been looking at what SDL needs to be
useful as a backend for GUI toolkits. This isn’t so much a question of
"how do I build a UI with SDL," but more like, “what would something
like GTK+ need to talk exclusively to SDL instead of X11?”

As a first target, we added what Unreal Engine 4’s Slate UI toolkit
would need to never talk directly to X11 on Linux, since the added
requirements were pretty modest.

Here’s the merge, modulo a few patches on top to get it to compile:

  https://hg.libsdl.org/SDL/rev/dc80dc0bd22e

The documentation on the new functions are right at the top of the
patch, if you scroll down a little. All of the new stuff works on Mac,
Windows, and X11, probably isn’t relevant to iOS/Android/etc, and still
needs support for Wayland, Mir, and Haiku.

This experiment will continue. I’m not sure what else, if anything else,
will be added, yet. There’s a delicate balance to SDL, and you don’t
want to go crazy adding things like this, especially when the primary
use is games, but I’m interested in finding cases like Slate–can mostly
use SDL, just needed X11 for a few special items–and seeing if we can
make those cases more portable.

–ryan.

10

It looks like something similar to SDL_GetGlobalMouseState would work,
but with this call we would still need to get the mouse state relative
to the window and subtract to get XNA4 behavior… and if
SDL_GetGlobalMouseState is as inefficient as the docs note, this

It’s not inefficient, but it does talk to the display server,
whereas SDL_GetMouseState() just returns whatever value SDL last saw for
the mouse without any further work done. I think it’s still safe to call
every frame.

probably isn’t something to use in addition to SDL_GetWindowPosition
every time the client asks for MouseState. On the other hand, something
like SDL_CaptureMouse would be pretty nice to get those negative
coordinates (for example) right away, but the rest of this call’s
behavior would make things messy really quickly…

Slate had a similar concern, but we decided doing the subtraction worked
fine, so we didn’t add an SDL_ClientToScreen() API.
SDL_GetWindowPosition() also does not talk to the display server, but
returns the last known window position, which is updated as the
appropriate events come from the OS.

I would say just call SDL_GetWindowPosition() and do the subtraction,
and we can revisit if this proves problematic.

–ryan.

11

That’s what I just said two emails ago… But hey, this time I actually
have a reference [1]
to point to! =P

That seems like a weird piece of security to enforce. Sort of feels like
that would break lots of things.

I’ll follow up with the Wayland people about this.

–ryan.

12

2014-06-26 9:48 GMT+02:00 Ryan C. Gordon :

That’s what I just said two emails ago… But hey, this time I actually

have a reference [1]
to point to! =P

That seems like a weird piece of security to enforce. Sort of feels like
that would break lots of things.

I’ll follow up with the Wayland people about this.

To be clear, the surface positions not being exposed is less of a security
thing, and more
of a “please don’t assume you’re on a flat rectangular display”. I think it
was argued
that the core design of Wayland should not prohibit different display
scenarios where the
concept of a Cartesian coordinate system makes no sense.

But do talk to the Wayland people, I’m sure they’ll have more accurate
explanations for
the hows and whys.

13

…seriously?

OK, someone totally needs to smack them all upside the head with a YAGNI-bat.? (This is a highly specialized form of clue-bat.)

MasonOn Thursday, June 26, 2014 11:44 AM, Jonas Kulla wrote:

To be clear, the surface positions not being exposed is less of a security thing, and more
of a “please don’t assume you’re on a flat rectangular display”. I think it was argued
that the core design of Wayland should not prohibit different display scenarios where the
concept of a Cartesian coordinate system makes no sense.

14

If you can still figure out what color the cursor is pointing at and
the like for a color picking tool, I’m happy. Otherwise, they should
be whipped with wet spaghetti noodles.

I’m all for “make few/no assumptions”. Just don’t break stuff in the
process, YAGNI violated or not.

JosephOn Thu, Jun 26, 2014 at 01:49:57PM -0700, Mason Wheeler wrote:

…seriously?

OK, someone totally needs to smack them all upside the head with a YAGNI-bat.? (This is a highly specialized form of clue-bat.)

15

2014-06-27 1:17 GMT+02:00 T. Joseph Carter :

If you can still figure out what color the cursor is pointing at and the
like for a color picking tool, I’m happy. Otherwise, they should be
whipped with wet spaghetti noodles.

You mean from anywhere on the screen (ie. not your own surface)? That’s not
possible
because in Wayland you have no access to the main framebuffer. Every
compositor will
have it’s own screenshooter utility that talks to it via a private protocol
to not expose this
to clients.

This one makes the most sense, really. It makes it infinitely harder to
steal someone’s
credit card information simply by stealthily taking screenshots and sending
them back.> I’m all for “make few/no assumptions”. Just don’t break stuff in the

process, YAGNI violated or not.

16

Hi Mason,

would you mind using another email address for the mailing list?
Your E-Mails are always filtered as SPAM by many mail providers, because
Yahoo sucks, see
http://www.ietf.org/mail-archive/web/ietf/current/msg87153.html for details.

(@Michael: I put you in CC because I just found mails from you from
another topic in my SPAM folder for the same reason)

Thanks,
Daniel

17

Apologies for resurrecting this thread.

I am currently taking some time to look into UE4 and SDL on Wayland.

The current mouse situation in UE4 is not great, as there are potentially a
few
issues because Slate “requires” global mouse co-ordinates and also warping
the
cursor.

I am mainly aiming at getting the Game working fully on Wayland but any
solution
needs to be compatible with the Editor and any strange Engine API use-cases
where a game developer may decide to call GetPosition and SetPosition
on the cursor.

As discussed before surfaces do not have an x and y co-ordinates. But you
can
create transient and popup surface which will be relative to a parent
surface
(and do not necessarily need to be inside the parent).

Personally looking at all the possible uses across all devices and
offscreen
rendering and rotatable windows, this seems to make sense. Even Mir seems
to also follow a similar direction.

Once the mouse cursor posts a leave event to the application it could be
anywhere. It may not even be on any screens (e.g. wii mote/pointer device).
Absolute mouse co-ordinates really do not make sense.

From the SDL point of view I can see a few alternatives:

  1. Create a parent/root surface which may (e.g. in the case of GNOME) be
    created
    at 0,0 display co-ordinates. Then any new window created with SDL would
    create
    a transient surface relative to that root window and report co-ordinates as
    relative to
    the top left of the root window.

  2. Probing the display with large popup surfaces to check to see if they
    will clip and
    making some assumptions based on that.

  3. Create the option for a fully virtualized desktop where SDL is only
    really
    creating one onscreen surface and then handing out offscreen surfaces to
    the
    application then performing internal compositing.
    I created a test using the Dummy video driver and then internally looping
    back to
    the main driver and forwarding/overriding any calls and hooking getproc
    address for
    glBindFramebuffer to alter calls to bind 0 to a pre-set render to texture
    framebuffer.

However this brings in the question, what layer is responsible for
"simulating"
a traditional x,y mouse-based desktop?

SDL, UE4 Engine, Slate, Application developer, XWayland, etc.

Is it really SDL’s job to be a desktop GUI toolkit or compositor?

18

Currently I have SlateViewer working on GNOME Shell Wayland Session using
Intel Mesa OpenGL 3.3.

The hittest api works great to allow the user to move the window about.

Drop-down/popup windows are working somewhat by making them transient and
relative to a parent surface.

It would be good for SDL to have some kind of parent/child/transient/sub
window/surface support.
Some assumptions like it would not show in the taskbar and be guaranteed to
be in that relative place always (even when using either a tiling or
expos?/GNOME activities view).