SDL linux console / remote session

I’m sure there is an easy answer to this but I’m buggered if I can find it (I’m equally
sure I tried to find an answer a few years ago but didn’t get very far).

The problem is this: I have a computer (in this case a R-PI) connected to a screen over
HDMI. I want to run a small SDL server to display patterns on the screen but since
the computer is essentially headless I want to run my SDL program remotely from a
ssh session.

I (think) I’m using the linux framebuffer and can run my test program from the
console fine. I can run it from an ssh login as root fine, I can run it using sudo fine
(and I could setuid it).

However, I would prefer to find the correct permissions / group memberships to run
the program as an ordinary ID without root permissions.

My user is a member of the video, audio and input groups and I have toyed with
changing the perms of /dev/console and /dev/tty0 and DISPLAY="" but whatever I do
my test program will not run from an ssh session without the invetitable error:

SDL_Init failed: Unable to open a console terminal

Surely there is something dead obvious that I can do!

I managed to run SDL-based graphics on Raspberry Pi 3 remotely without sudo. Graphics is implemented in the framebuffer (no need to run X server) using Pygame library. Application runs as a systemd service.
NOTE: I had no need in mouse or keyboard events, so I can’t say are they working fine with this approach or not.
See details in this StackExchange answer