[SOLVED] Drawing an OSD-like texture on top of a YUV video texture

Hi there. I need to draw a graphic layer over a YUV video texture using SDL2.
The YUV texture is YUV12, and the OSD texture that should be drawn on top of it is BGR24.
Think of it as a TV OSD on top of a film or TV program.

I am trying to use SDL_SetTextureBlendMode() on the graphics layer with SDL_BLENDMODE_BLEND.
Then, what I do is SDL_RenderCopy() the YUV texture first, and then then I SDL_RenderCopy() the texture
containing the graphics layer, which is BGR24.

I know I am missing something here… According to:
https://wiki.libsdl.org/SDL_SetTextureBlendMode
SDL_BLENDMODE_BLEND should add the colors on the 2nd texture, and since black pixels are 0x0, 0x0, 0x0, they shouldn’t be adding anything, so if I set SDL_SetTextureAlphaMod() to 255 on the 2nd texture, and 0 on the YUV video texture, the 2nd texture should have it’s not-black pixels drawn on the first without any transparent effect, but it’s not the case.

Any ideas, please?

EDIT: In case someone’s wondering, this is for implementing overlays to the Hypseus (previously Daphne, the Laserdisc emulator) on this repository -> https://github.com/btolab/hypseus/commits/master, so if you help me we will all win :slight_smile:

With 24-bit RGB textures you don’t really have much choice in what gets alpha blended and what not. Either it’s completely opaque or everything has the same alpha value (set with SDL_SetTextureAlphaMod). I recommend you use 32-bit textures with an alpha channel. That way you can select exactly what has to be transparent and what not.

There is one specific case that doesn’t need an alpha channel: If your overlay only has black and white pixels and you only want to draw the white ones use the ADD blend mode. It should look the same as with an alpha channel.

If you draw the YUV texture and then the OSD texture, this is what the BLEND blend mode calculates.

             R    G    B    A
framebuf =   0,   0,   0,   0       // The default frame buffer that will be shown on screen.
yuvpixel = 142,  42,  45, 255       // Some pixel from the video converted to RGBA.
osdpixel =   0,   0,   0, 255       // Pixel from the OSD.

// Drawing the YUV texture to the screen.
framebuf = (framebuf.R * (255 - yuvpixel.A) + yuvpixel.R * yuvpixel.A) / 255,
           (framebuf.G * (255 - yuvpixel.A) + yuvpixel.G * yuvpixel.A) / 255,
           (framebuf.B * (255 - yuvpixel.A) + yuvpixel.B * yuvpixel.A) / 255,
           (framebuf.A * (255 - yuvpixel.A) / 255 + yuvpixel.A

// And with the numbers from above.
framebuf = (0 * (255 - 255) + 142 * 255) / 255,
           (0 * (255 - 255) +  42 * 255) / 255,
           (0 * (255 - 255) +  45 * 255) / 255,
           (0 * (255 - 255)) / 255 + 255
         = 142,  42,  45, 255
 
// Drawing the OSD texture to the screen. Same as above with osdpixel instead.
framebuf = (framebuf.R * (255 - osdpixel.A) + osdpixel.R * osdpixel.A) / 255,
           (framebuf.G * (255 - osdpixel.A) + osdpixel.G * osdpixel.A) / 255,
           (framebuf.B * (255 - osdpixel.A) + osdpixel.B * osdpixel.A) / 255,
           (framebuf.A * (255 - osdpixel.A) / 255 + osdpixel.A

// And with the numbers again.
framebuf = (142 * (255 - 255) + 0 * 255) / 255,
           ( 42 * (255 - 255) + 0 * 255) / 255,
           ( 45 * (255 - 255) + 0 * 255) / 255,
           (255 * (255 - 255)) / 255 + 255
         =   0,   0,   0, 255

(Using 0 to 255 just as an example. It’s 0 to 1 in a floating point format on the graphics card.)

Indeed the zeros do not add anything. However, because the RGB texture is opaque, it will set the destination pixel to black.

I created a little program that shows how 24-bit and 32-bit textures behave in a situation like this. Run it with yuvandoverlay crew.y4m. The window should show the video 4 times, each with their own overlay texture. I used target textures on the top and surfaces on the bottom. The left side is RGB and the right is RGBA. Initially, the left side should only show the overlay and the right should show the video beneath the overlay as expected. Note that some renderers do not support 24-bit textures and automatically use 32-bit textures. That’s why the top-left video might also show correctly. Try the ADD blend mode (F3) and it will show up correctly for all of them. I did only use white in this example though. Additional Controls:

  • Spacebar: Play/Pause
  • F1 to F4: NONE, BLEND, ADD, MOD blend modes of the overlay textures.
  • 1 to 5: alpha mod value of the overlay textures.

yuvandoverlay.zip (585.2 KB)

yuvandoverlay

@ChliHug: Thanks a lot for the response and the code example!
I have found it VERY usefull, and I already have things half-working. Currently, I have a test OSD texture blending mode set to SDL_BLENDMODE_BLEND, and alpha set to 255 on this test texture.
This is an RGBA8888 texture, so the surface it gets it’s pixels from is also RGBA8888. You use ARGB8888 in your example, but the idea is the same (with the alpha component moved to the first byte in my case).
Of course, pixels with a 0x00000000 are totally transparent as intended, and pixels with any other color, like a pure-red 0xff0000ff are opaque, and it looks good ON TOP on the full motion video YUV texture.

The problem is that, aside from this test case I am using, Hypseus (Daphne) uses an 8bpp surface for the OSD-like overlay. I am trying to blit this 8bpp to a 32bpp RGBA8888 texture, so I can use it as intended, but I don’t get expected results at all, with strange values in the pixels of the destination 32bpp surface.
So, what do you recommend me to do? Is converting the 8bpp pixels “manually” to 32bpp the only option or does SDL2 have any function that could help here?

I chose ARGB8888 because most renderers can support this format natively. If you create a texture with a different format than the ones listed in the SDL_RendererInfo.texture_formats array, SDL may automatically create an intermediate surface and convert the data for you.

Paletted formats are not supported, though, so you have to go with two surfaces instead. The conversion from a 8-bit surface to a 32-bit surface could look something like this.

palettetorgba.c
#include <SDL.h>

static SDL_Window * window;
static SDL_Renderer * renderer;
static SDL_Texture * texture_background;
static SDL_Texture * texture_rgba;
static SDL_Surface * surface_8bit;
static SDL_Surface * surface_32bit;

static void SetRect(SDL_Rect * r, int x, int y, int w, int h)
{
	r->x = x;
	r->y = y;
	r->w = w;
	r->h = h;
}

static double GetNow()
{
	return (double)SDL_GetPerformanceCounter() / (double)SDL_GetPerformanceFrequency();
}

static int LoadAndDisplay(int argc, char**argv)
{
	int i, done = 0, window_width, window_height;
	SDL_RendererInfo info;
	double lastupdate = 0;

	int surfacebpp;
	Uint32 Rmask, Gmask, Bmask, Amask, texformat;

	window = SDL_CreateWindow("Palette to RGBA", 100, 100, 200, 200, 0);
	if (window == NULL) {
		return 100;
	}

	renderer = SDL_CreateRenderer(window, -1, 0);
	if (renderer == NULL) {
		return 110;
	}

	SDL_GetRendererInfo(renderer, &info);


	surface_8bit = SDL_CreateRGBSurface(0, 20, 20, 8, 0, 0, 0, 0);
	if (surface_8bit == NULL) {
		SDL_Log("Could not create 8-bit surface");
		return 120;
	}
	SDL_PixelFormatEnumToMasks(SDL_PIXELFORMAT_ARGB8888, &surfacebpp, &Rmask, &Gmask, &Bmask, &Amask);
	surface_32bit = SDL_CreateRGBSurface(0, 20, 20, 32, Rmask, Gmask, Bmask, Amask);
	if (surface_32bit == NULL) {
		SDL_Log("Could not create 32-bit surface");
		return 130;
	}

	texture_rgba = SDL_CreateTexture(renderer, SDL_PIXELFORMAT_ARGB8888, SDL_TEXTUREACCESS_STREAMING, 20, 20);
	if (texture_rgba == NULL) {
		SDL_Log("Could not create RGBA texture");
		return 140;
	}
	SDL_SetTextureBlendMode(texture_rgba, SDL_BLENDMODE_BLEND);
	texture_background = SDL_CreateTexture(renderer, SDL_PIXELFORMAT_ARGB8888, SDL_TEXTUREACCESS_STATIC, 200, 200);
	if (texture_background == NULL) {
		SDL_Log("Could not create background RGBA texture");
		return 150;
	} else {
		Uint8 * data = SDL_malloc(200 * 200 * 4);
		if (data == NULL) {
			SDL_Log("Memory allocation error");
			return 160;
		}
		for (i = 0; i < 200 * 200 * 4; i += 4) {
			data[i] = 0xff;
			data[i + 1] = (i % 800) / 4;
			data[i + 2] = 0xff - (i % 800) / 4;
			data[i + 3] = 0xff;
		}
		SDL_UpdateTexture(texture_background, NULL, data, 200 * 4);
		SDL_free(data);
	}

	SDL_QueryTexture(texture_rgba, &texformat, NULL, NULL, NULL);
	if (texformat != surface_32bit->format->format) {
		SDL_Log("Texture format (%s) didn't match surface format (%s)", SDL_GetPixelFormatName(texformat), SDL_GetPixelFormatName(surface_32bit->format->format));
		return 170;
	}

	SDL_GetWindowSize(window, &window_width, &window_height);

	while (!done) {
		SDL_Event e;
		SDL_Rect dst;
		double now = GetNow();

		while (SDL_PollEvent(&e)) {
			if (e.type == SDL_QUIT) {
				done = 1;
			} else if (e.type == SDL_KEYUP) {
				Uint32 sym = e.key.keysym.sym;
				if (sym == SDLK_ESCAPE) {
					done = 1;
				}
			} else if (e.type == SDL_WINDOWEVENT) {
				if (e.window.event == SDL_WINDOWEVENT_RESIZED || e.window.event == SDL_WINDOWEVENT_SIZE_CHANGED) {
					window_width = e.window.data1;
					window_height = e.window.data2;
				}
			}
		}


		if (lastupdate + 1.5 < now) {
			int x, y;
			Uint8 * data = surface_8bit->pixels;
			SDL_Palette * palette = surface_8bit->format->palette;
			Uint32 state = (Uint32)(now * 1000000.);
			lastupdate = now;

			/* Scrambling palette */
			for (i = 0; i < palette->ncolors; i++) {
				palette->colors[i].r = state & 0xff;
				palette->colors[i].g = (state >> 8) & 0xff;
				palette->colors[i].b = (state >> 16) & 0xff;
				palette->colors[i].a = (state >> 24) & 0xff;
				state = (state * 1103515245) + 12345;
			}

			/* Scrambling image */
			for (y = 0; y < 20; y++) {
				for (x = 0; x < 20; x++) {
					data[x] = state % palette->ncolors;
					state = (state * 1103515245) + 12345;
				}
				data += surface_8bit->pitch;
			}

			/* Converting to RGBA */
			SDL_BlitSurface(surface_8bit, NULL, surface_32bit, NULL);

			/* Uploading RGBA */
			SDL_UpdateTexture(texture_rgba, NULL, surface_32bit->pixels, surface_32bit->pitch);
		}

		SDL_SetRenderDrawColor(renderer, 0, 0, 0, 0);
		SDL_RenderClear(renderer);

		SetRect(&dst, -50, -50, 300, 300);
		SDL_RenderCopyEx(renderer, texture_background, NULL, &dst, (Sint32)(now * 360) % 360, NULL, SDL_FLIP_NONE);

		SetRect(&dst, 50, 50, 100, 100);
		SDL_RenderCopy(renderer, texture_rgba, NULL, &dst);

		SDL_RenderPresent(renderer);

		SDL_Delay(1);
	}

	return 0;
}

int main(int argc, char* argv[])
{
	int result;
	window = NULL;
	renderer = NULL;
	texture_background = NULL;
	texture_rgba = NULL;
	surface_8bit = NULL;
	surface_32bit = NULL;

	if (SDL_Init(SDL_INIT_VIDEO | SDL_INIT_EVENTS | SDL_INIT_TIMER) < 0) {
		SDL_Log("Could not initialize SDL");
		return 1;
	}

	result = LoadAndDisplay(argc, argv);

	if (surface_8bit != NULL)
		SDL_FreeSurface(surface_8bit);
	if (surface_32bit != NULL)
		SDL_FreeSurface(surface_32bit);

	if (texture_background != NULL)
		SDL_DestroyTexture(texture_background);
	if (texture_rgba != NULL)
		SDL_DestroyTexture(texture_rgba);

	if (renderer != NULL)
		SDL_DestroyRenderer(renderer);
	if (window != NULL)
		SDL_DestroyWindow(window);
	SDL_Quit();

	return result;
}

I can’t say anything regarding the performance of this conversion. I’m guessing it’s just a simple loop that copies the palette values into the RGBA surface. If you spot a pattern in the overlay data, you might be able to create your own, faster algorithm for the 8-bit to 32-bit conversion.

I can’t think of a way to somehow make the renderer display the 8-bit data directly. With shaders sure, but they’re not available with the SDL2 renderers. There are other projects like SDL_gpu that provide access to them, if the surface conversion solution is not adequate.

Well, this was sorted out and I got to implement working overlays on SDL2 version on Daphne (Hypseus).
You can get the sources here:


I am marking this as solved, thanks a lot, ChliHug! :wink: