tvOS swipe left/right gesture support (patch enclosed)

I found that events weren’t being generated for the simple directional swipes on the Apple TV touch remote.

While it’s possible to use the touch press/release/motion events to detect simple swipes of this nature (and I had this working well enough) it struck me that this was re-inventing the wheel.

As such, I have written a patch that recognises the proper swipe up/down/left/right events issued in UIKit and passes these on as directional key presses. This also means that the touch remote operates the same way as the old-style IR remote in this respect.

Code:

diff -r 007dfe83abf8 src/video/uikit/SDL_uikitview.m
— a/src/video/uikit/SDL_uikitview.m Wed Oct 19 20:50:33 2016 -0700
+++ b/src/video/uikit/SDL_uikitview.m Sun Dec 18 14:57:42 2016 +0300
@@ -48,7 +48,20 @@
#if !TARGET_OS_TV
self.multipleTouchEnabled = YES;
#endif-
+#if TARGET_OS_TV

  •    UISwipeGestureRecognizer *swipeUp = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
    
  •    swipeUp.direction = UISwipeGestureRecognizerDirectionUp;
    
  •    [self addGestureRecognizer:swipeUp];
    
  •    UISwipeGestureRecognizer *swipeDown = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
    
  •    swipeDown.direction = UISwipeGestureRecognizerDirectionDown;
    
  •    [self addGestureRecognizer:swipeDown];
    
  •    UISwipeGestureRecognizer *swipeLeft = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
    
  •    swipeLeft.direction = UISwipeGestureRecognizerDirectionLeft;
    
  •    [self addGestureRecognizer:swipeLeft];
    
  •    UISwipeGestureRecognizer *swipeRight = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
    
  •    swipeRight.direction = UISwipeGestureRecognizerDirectionRight;
    
  •    [self addGestureRecognizer:swipeRight];
    

+#endif
touchId = 1;
SDL_AddTouch(touchId, “”);
}
@@ -56,6 +69,27 @@
return self;
}

+#if TARGET_OS_TV
± (void)swipeGesture:(UISwipeGestureRecognizer *)sender
+{

  • switch (sender.direction)
  • {
  • case UISwipeGestureRecognizerDirectionUp:
  •    SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_UP);
    
  •    break;
    
  • case UISwipeGestureRecognizerDirectionDown:
  •    SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_DOWN);
    
  •    break;
    
  • case UISwipeGestureRecognizerDirectionLeft:
  •    SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_LEFT);
    
  •    break;
    
  • case UISwipeGestureRecognizerDirectionRight:
  •    SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_RIGHT);
    
  •    break;
    
  • }
    +}
    +#endif
  • (void)setSDLWindow:(SDL_Window *)window
    {
    SDL_WindowData *data = nil;

SDL is already supposed to send left/right/up/down key presses when ?press? events are detected on the Apple TV Remote ( https://developer.apple.com/library/content/documentation/General/Conceptual/AppleTV_PG/DetectingButtonPressesandGestures.html https://developer.apple.com/library/content/documentation/General/Conceptual/AppleTV_PG/DetectingButtonPressesandGestures.html ) ? does that work for you? Does your extra gesture recognizer code interfere with that at all?> On Dec 18, 2016, at 8:15 AM, oviano wrote:

I found that events weren’t being generated for the simple directional swipes on the Apple TV touch remote.

While it’s possible to use the touch press/release/motion events to detect simple swipes of this nature (and I had this working well enough) it struck me that this was re-inventing the wheel.

As such, I have written a patch that recognises the proper swipe up/down/left/right events issued in UIKit and passes these on as directional key presses. This also means that the touch remote operates the same way as the old-style IR remote in this respect.

Code:

diff -r 007dfe83abf8 src/video/uikit/SDL_uikitview.m
— a/src/video/uikit/SDL_uikitview.m Wed Oct 19 20:50:33 2016 -0700
+++ b/src/video/uikit/SDL_uikitview.m Sun Dec 18 14:57:42 2016 +0300
@@ -48,7 +48,20 @@
#if !TARGET_OS_TV
self.multipleTouchEnabled = YES;
#endif

+#if TARGET_OS_TV

  •    UISwipeGestureRecognizer *swipeUp = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
    
  •    swipeUp.direction = UISwipeGestureRecognizerDirectionUp;
    
  •    [self addGestureRecognizer:swipeUp];
    
  •    UISwipeGestureRecognizer *swipeDown = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
    
  •    swipeDown.direction = UISwipeGestureRecognizerDirectionDown;
    
  •    [self addGestureRecognizer:swipeDown];
    
  •    UISwipeGestureRecognizer *swipeLeft = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
    
  •    swipeLeft.direction = UISwipeGestureRecognizerDirectionLeft;
    
  •    [self addGestureRecognizer:swipeLeft];
    
  •    UISwipeGestureRecognizer *swipeRight = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
    
  •    swipeRight.direction = UISwipeGestureRecognizerDirectionRight;
    
  •    [self addGestureRecognizer:swipeRight];
    

+#endif
touchId = 1;
SDL_AddTouch(touchId, “”);
}
@@ -56,6 +69,27 @@
return self;
}

+#if TARGET_OS_TV
± (void)swipeGesture:(UISwipeGestureRecognizer *)sender
+{

  • switch (sender.direction)
  • {
  • case UISwipeGestureRecognizerDirectionUp:
  •    SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_UP);
    
  •    break;
    
  • case UISwipeGestureRecognizerDirectionDown:
  •    SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_DOWN);
    
  •    break;
    
  • case UISwipeGestureRecognizerDirectionLeft:
  •    SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_LEFT);
    
  •    break;
    
  • case UISwipeGestureRecognizerDirectionRight:
  •    SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_RIGHT);
    
  •    break;
    
  • }
    +}
    +#endif
  • (void)setSDLWindow:(SDL_Window *)window
    {
    SDL_WindowData *data = nil;

SDL mailing list
SDL at lists.libsdl.org <mailto:SDL at lists.libsdl.org>
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

That’s great, but it doesn’t work and no such events come through.

If you can point me in the direction of where the directional swipe gestures are handled in the code I can try and fix it, but I couldn’t see it.

If you’re talking about the old (non touch) remote then yes that generates the events correctly.

I’m talking about the swipes on the touch remote.

https://hg.libsdl.org/SDL/file/f6cd81aab88e/src/video/uikit/SDL_uikitview.m#l204 https://hg.libsdl.org/SDL/file/f6cd81aab88e/src/video/uikit/SDL_uikitview.m#l204> On Dec 18, 2016, at 10:44 AM, oviano wrote:

That’s great, but it doesn’t work and no such events come through.

If you can point me in the direction of where the directional swipe gestures are handled in the code I can try and fix it, but I couldn’t see it.

If you’re talking about the old (non touch) remote then yes that generates the events correctly.

I’m talking about the swipes on the touch remote.


SDL mailing list
SDL at lists.libsdl.org <mailto:SDL at lists.libsdl.org>
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

I just tested on the tvOS Simulator and the key press events work for me with the latest SDL code from hg.> On Dec 18, 2016, at 10:45 AM, Alex Szpakowski <@Alex_Szpakowski> wrote:

https://hg.libsdl.org/SDL/file/f6cd81aab88e/src/video/uikit/SDL_uikitview.m#l204 https://hg.libsdl.org/SDL/file/f6cd81aab88e/src/video/uikit/SDL_uikitview.m#l204

On Dec 18, 2016, at 10:44 AM, oviano <ovcollyer at mac.com <mailto:ovcollyer at mac.com>> wrote:

That’s great, but it doesn’t work and no such events come through.

If you can point me in the direction of where the directional swipe gestures are handled in the code I can try and fix it, but I couldn’t see it.

If you’re talking about the old (non touch) remote then yes that generates the events correctly.

I’m talking about the swipes on the touch remote.


SDL mailing list
SDL at lists.libsdl.org <mailto:SDL at lists.libsdl.org>
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Ok I’ll try the simulator later but you’re testing the swipe gestures, correct?

The Apple TV Remote touchpad ?tap? actions, as described here: https://developer.apple.com/tvos/human-interface-guidelines/remote-and-controllers/#buttons https://developer.apple.com/tvos/human-interface-guidelines/remote-and-controllers/#buttons

The ?tap? actions map pretty cleanly to button presses, in terms of what their behaviour describes. SDL doesn?t really have analogous user-facing APIs that describe OS-handled swipe gestures in the same manner, however you can access the Remote?s touchpad as joystick axes in order to get a more granular description of what the user is doing with it.> On Dec 18, 2016, at 11:55 AM, oviano wrote:

Ok I’ll try the simulator later but you’re testing the swipe gestures, correct?


SDL mailing list
SDL at lists.libsdl.org <mailto:SDL at lists.libsdl.org>
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

The up/down/left/right swipe gestures on Apple TVs are the primary method to navigate around button/menu systems on the device so assuming that some SDL apps will have button or menu systems that require the user to navigate from one button to another then it’s far better to hook into these gestures…

Otherwise you are expecting the user to re-implement these themselves and inevitably there will be subtle differences in “feel” compared to the official ones leading to a substandard user experience.

2 out of the 3 gesture types outlined here are implemented as key presses (click and tap) https://developer.apple.com/tvos/human-interface-guidelines/remote-and-controllers/#gestures https://developer.apple.com/tvos/human-interface-guidelines/remote-and-controllers/#gestures

Given the fact that key presses can?t represent swipe strength which is one of the main purposes of a swipe, and the fact that the direction key presses are already used by the tap gestures, how do you propose swipes should be exposed in SDL?s APIs?> On Dec 18, 2016, at 12:32 PM, oviano wrote:

The up/down/left/right swipe gestures on Apple TVs are the primary method to navigate around button/menu systems on the device so assuming that some SDL apps will have button or menu systems that require the user to navigate from one button to another then it’s far better to hook into these gestures…

Otherwise you are expecting the user to re-implement these themselves and inevitably there will be subtle differences in “feel” compared to the official ones leading to a substandard user experience.


SDL mailing list
SDL at lists.libsdl.org <mailto:SDL at lists.libsdl.org>
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Given the fact that key presses can?t represent swipe strength which is one of the main purposes of a swipe

Actually I suppose the Apple documentation is a bit misleading about this. :slight_smile:

I pushed a commit which should do what you want: https://hg.libsdl.org/SDL/rev/a6babd973955> On Dec 18, 2016, at 12:41 PM, Alex Szpakowski <@Alex_Szpakowski> wrote:

Given the fact that key presses can?t represent swipe strength which is one of the main purposes of a swipe

Actually I suppose the Apple documentation is a bit misleading about this. :slight_smile:

Well the swipe gesture I’m talking about hooking into, and which my patch uses is just a simple standard swipe with direction only. There is concept of strength:

https://developer.apple.com/reference/uikit/uiswipegesturerecognizer

So it maps pretty easily to the arrow key presses.

It’s funny - I actually had no idea (despite owning a touch-based Apple TV since they were available) that you could tap to produce the direction “buttons”, I’d always swiped which is why it seems to me such a big omission. I simply had no way to navigate within the UI of my application (a video player) :slight_smile:

There is no concept of strength, I should say.

Thanks for making the change - I haven’t tested your slight variation on my code yet but I will at some point and let you know if I encounter any problems.