A need help to use bitmap array on my SDL game

I’m writing a visual novel engine. It’s actually a re-implementation of an engine used by a Japanese company. I’m trying to use libass (alpha substation) as subtitles, but I don’t know how to convert the ASS bitmap array to an SDL texture.

I attach here my code and also the ASS struct for the subtitle bitmap

int playText(void *import)
    {
        kotonohaData::acessMapper *importedTo = static_cast<kotonohaData::acessMapper *>(import);
        importedTo->root->log0->appendLog("(Text) - Start");
        kotonohaTime::delay(1000);

        char file[64] = "test.ass";
        char *subfile = file;
        ASS_Library *ass_library;
        ASS_Renderer *ass_renderer;
        int frame_w = 1280;
        int frame_h = 720;
        ass_library = ass_library_init();
        if (!ass_library)
        {
            printf("ass_library_init failed!\n");
            exit(1);
        }
        ass_set_extract_fonts(ass_library, 1);
        ass_renderer = ass_renderer_init(ass_library);
        if (!ass_renderer)
        {
            printf("ass_renderer_init failed!\n");
            exit(1);
        }
        ass_set_storage_size(ass_renderer, frame_w, frame_h);
        ass_set_frame_size(ass_renderer, frame_w, frame_h);
        ass_set_fonts(ass_renderer, NULL, "sans-serif", ASS_FONTPROVIDER_AUTODETECT, NULL, 1);
        ASS_Track *track = ass_read_file(ass_library, subfile, NULL);
        if (!track)
        {
            printf("track init failed!\n");
            return 1;
        }
        importedTo->root->log0->appendLog("(Text) - End");
        while (importedTo->control->outCode == -1)
        {
            if (importedTo->control->display[2])
            {
                ASS_Image *img = ass_render_frame(ass_renderer, track, (int)(5 * 1000), NULL);
                SDL_Texture *texture = nullptr;
                texture = SDL_CreateTexture(importedTo->root->renderer, SDL_PIXELFORMAT_RGBA8888, SDL_TEXTUREACCESS_STREAMING, img->w, img->h);
                SDL_UpdateTexture(texture, nullptr, img->bitmap, img->w * sizeof(Uint32));
                SDL_Rect rect = {0, 0, frame_h, frame_w};
                SDL_RenderCopy(importedTo->root->renderer, texture, &rect, NULL);
                SDL_DestroyTexture(texture);
                importedTo->control->display[2] = false;
                importedTo->control->display[3] = true;
            }
        }
        ass_free_track(track);
        ass_renderer_done(ass_renderer);
        ass_library_done(ass_library);
        importedTo->root->log0->appendLog("(Text) - End");
        return 0;
    };```

/*

  • A linked list of images produced by an ass renderer.

  • These images have to be rendered in-order for the correct screen

  • composition. The libass renderer clips these bitmaps to the frame size.

  • w/h can be zero, in this case the bitmap should not be rendered at all.

  • The last bitmap row is not guaranteed to be padded up to stride size,

  • e.g. in the worst case a bitmap has the size stride * (h - 1) + w.
    */
    typedef struct ass_image {
    int w, h; // Bitmap width/height
    int stride; // Bitmap stride
    unsigned char bitmap; // 1bpp strideh alpha buffer
    // Note: the last row may not be padded to
    // bitmap stride!
    uint32_t color; // Bitmap color and alpha, RGBA
    // For full VSFilter compatibility, the value
    // must be transformed as described in
    // ass_types.h for ASS_YCbCrMatrix
    int dst_x, dst_y; // Bitmap placement inside the video frame

    struct ass_image *next; // Next image, or NULL

    enum {
    IMAGE_TYPE_CHARACTER,
    IMAGE_TYPE_OUTLINE,
    IMAGE_TYPE_SHADOW
    } type;

    // New fields can be added here in new ABI-compatible library releases.
    } ASS_Image;

Welcome! libass produces 1bpp bitmaps with static color, you need to convert this bitmap to suitable pixel format first, then update your texture, don’t forget to draw all bitmaps, ASS_Image is a linked list, every individual bitmap has specific RGBA color, you can render them one after one (SDL renderer will blend them for you), or blend them to single pixmap by yourself, and render it.
Also, looks like you initialize libass every drawing call, doesn’t look good, it’s better to store it in your rendering context.
May I ask you, why are you using libass for text rendering? You may use nice SDL_ttf library for such task.

The link shows a video of the application running

Right, let me get this straight. the unsigned char *bitmap has apha, it should be treated with the value color. I looked up some examples.

What I understand is that I have to go through the image line by line and convert the bitmap to the sdl texture.

if (importedTo->control->display[2])
            {
                ASS_Image *img = ass_render_frame(ass_renderer, track, (int)(5 * 1000), NULL);
                SDL_Texture *texture = NULL;
                texture = SDL_CreateTexture(importedTo->root->renderer, SDL_PIXELFORMAT_RGBA8888, SDL_TEXTUREACCESS_STREAMING, img->w, img->h);
                void *pixels = nullptr;
                int pitch = img->stride;
                SDL_LockTexture(texture, NULL, &pixels, &pitch);
                uint32_t *pixelArray = (uint32_t *)pixels;
                for (int y = 0; y < img->h; y++)
                {
                    for (int x = 0; x < img->w; x++)
                    {
                        uint8_t alpha = img->bitmap[y * img->stride + x];
                        uint32_t color = img->color;
                        uint8_t red = (color >> 24) & 0xff;
                        uint8_t green = (color >> 16) & 0xff;
                        uint8_t blue = (color >> 8) & 0xff;
                        pixelArray[y * img->w + x] = (alpha << 24) | (red << 16) | (green << 8) | blue;
                    }
                }
                SDL_UnlockTexture(texture);
                SDL_Rect rect = {0, 0, frame_h, frame_w};
                SDL_RenderCopy(importedTo->root->renderer, texture, &rect, NULL);
                SDL_DestroyTexture(texture);
                importedTo->control->display[2] = false;
                importedTo->control->display[3] = true;
            }

When I asked the copilot, he suggested using this structure, but the image came out red with a black background that was completely out of shape

It’s just a test, I loaded a test.ass to just create the text header, so the text is called statically, according to this code it’s in a loop in a separate thead, as you can see there is an array in the if that starts the drawing. following this rule image → video → subtitle → ui → renderPresent

Here is the link to the complete code
Ruaneri-Portela/Kotonoha: Visual Novel engine basead on SDL and FFMPEG, reimplement of engine FORIS used in Days series games (producted by 0verflow) (github.com)

I am aware of the sdl source libraries, however it is much more convenient to use libass, as the application and engine for visual novels from a company that uses animation (videos) to compose, and libass and consensus within the community anime sub. I think it works better than creating a new subtitling system.

I already have the texts, they are received in another function with a start and end time. I will make a function for each text to become an ass_object within ass_track

Above I posted an example of code, when applied it gave me this. which is better than the sprinkles I’ve had trying to apply the last few times.

From what I understand, correct me, does each char in the segment have the bytes for RGBA? Or just A? So if I just create a new segment adding the color (the int color value) to each of the bytes corresponding to the RGB space? It is vital to me that the empty apha channel is transparent, as each texture is superimposed on the other.

[https://imgur.com/a/GwCzVYu]

The src rect is invalid here and you forgot about destination rect, img’s w and h are not same as frame size, same for position: it’s not 0:0.
SDL_Rect dst = {img->dst_x, img->dst_y, img->w, img->h};

What’s the type of pixelArray?

That’s not an alpha, you can think it’s color or transparent flag, and this is 1 bit bitmap, so you need look it by bit, not by byte.

I successfully manged to render my .ass file:

#include <SDL.h>
#include <ass/ass.h>
#include <cassert>
#include <array>
#include <string_view>
#include <cstdint>
#include <thread>
#include <chrono>

inline std::pair<size_t, std::uint8_t> x2pos_mask(int x) {
	size_t pos = x / CHAR_BIT;
	size_t r = x % CHAR_BIT;

	std::uint8_t mask = 0b10000000;
	mask >>= r;

	return {x, mask};
}

int main(int, char **) {
	constexpr auto w = 320;
	constexpr auto h = 200;

	SDL_Init(SDL_INIT_VIDEO);

	auto window = SDL_CreateWindow("Subtitles", SDL_WINDOWPOS_CENTERED, SDL_WINDOWPOS_CENTERED, w, h, 0);
	assert(window);
	auto renderer = SDL_CreateRenderer(window, -1, 0);
	assert(renderer);

	int frame_w = 0;
	int frame_h = 0;
	assert(SDL_GetRendererOutputSize(renderer, &frame_w, &frame_h) == 0);

	auto ass = ass_library_init();
	assert(ass);
	auto ass_renderer = ass_renderer_init(ass);
	assert(ass_renderer);

	ass_set_storage_size(ass_renderer, frame_w, frame_h);
	ass_set_frame_size(ass_renderer, frame_w, frame_h);

	constexpr std::string_view file_sv = "test.ass";
	std::array<char, file_sv.size() + 1> file;
	auto it = std::copy(file_sv.begin(), file_sv.end(), file.begin());
	*it = '\0';

	ass_set_fonts(ass_renderer, nullptr, "sans-serif", ASS_FONTPROVIDER_AUTODETECT, nullptr, 1);
	auto track = ass_read_file(ass, file.data(), nullptr);
	int detect = 0;
	ASS_Image *img = ass_render_frame(ass_renderer, track, 0, &detect);

	for (; img != nullptr; img = img->next) {
		if (img->w == 0 || img->h == 0) {
			continue;
		}

		auto texture = SDL_CreateTexture(renderer, SDL_PIXELFORMAT_RGBA8888, SDL_TEXTUREACCESS_STREAMING, img->w, img->h);
		assert(texture);


		std::uint32_t *pixels = nullptr;
		int pitch = 0;
		assert(SDL_LockTexture(texture, nullptr, reinterpret_cast<void **>(&pixels), &pitch) == 0);

		auto bitmap = img->bitmap;

		// It's better to unroll this loop.
		for (int y = 0; y < img->h; y++) {
			for (int x = 0; x < img->w; x++) {
				auto pos_mask = x2pos_mask(x);

				auto filled = bitmap[pos_mask.first] & pos_mask.second;

				pixels[x] = (filled) ? img->color : 0;
			}

			pixels = reinterpret_cast<std::uint32_t *>(reinterpret_cast<std::uintptr_t>(pixels) + pitch);
			bitmap += img->stride;
		}

		SDL_UnlockTexture(texture);

		SDL_Rect dst = {img->dst_x, img->dst_y, img->w, img->h};
		SDL_RenderCopy(renderer, texture, nullptr, &dst);

		SDL_DestroyTexture(texture);
		
	}

	SDL_RenderPresent(renderer);

	std::this_thread::sleep_for(std::chrono::seconds(10));

	ass_free_track(track);
	ass_renderer_done(ass_renderer);
	ass_library_done(ass);

	SDL_Quit();

	return 0;
}

I just copied your usage of libass, it’s first time I touch this library, so if there any error related to libass usage, you need to find and fix it by yourself.
So, your error is miss-use of bitmap and loosing some points like positioning.
Good luck with your engine!

One more thing, that’s a linked list, you can refer to my code how to handle it.

About the performance: such conversions is hard for compiler to optimize, so you can use lookup tables for bitmap mask and unroll the loop.

Ah… Now I understand, in ASS it creates b&w images where the color is defined by the color variable. And it treats it as a list because it can have captions of different colors and formats, where the table and the combination of them all are properly output.

I’m still confused about segmenting the bits, and how to do it, but I’ll use your example.

I applied your code and it looks like this… I have to find out how to apply the alpha channel so that it’s transparent… Now I found it strange because it had a strange font, in the example given in the asslib git the image comes out using the same functions with the subtitle in good quality

In fact, I had a perennial bug where I couldn’t resize the screen during playback, but now it’s working! I don’t know how it was solved hahaha, mysteries of programming!

In addition, should I combine the caption in an image? to get transparency?

You should set texture blend mode for alpha.

Try to export pixels to PNG first, to verify there’s no problem with rendering.

Sorry, I’ve got it completely wrong. Despite libass states bitmap as 1bpp, the pixels are not packed (in fact they’re 8bpp). Here’s reworked code with no render glitches:

#include <SDL.h>
#include <ass/ass.h>
#include <cassert>
#include <array>
#include <string_view>
#include <cstdint>
#include <thread>
#include <chrono>

int main(int, char **) {
	constexpr auto w = 640;
	constexpr auto h = 480;

	SDL_Init(SDL_INIT_VIDEO);

	auto window = SDL_CreateWindow("Subtitles", SDL_WINDOWPOS_CENTERED, SDL_WINDOWPOS_CENTERED, w, h, 0);
	assert(window);
	auto renderer = SDL_CreateRenderer(window, -1, 0);
	assert(renderer);

	int frame_w = 0;
	int frame_h = 0;
	assert(SDL_GetRendererOutputSize(renderer, &frame_w, &frame_h) == 0);

	auto ass = ass_library_init();
	assert(ass);
	auto ass_renderer = ass_renderer_init(ass);
	assert(ass_renderer);

	ass_set_storage_size(ass_renderer, frame_w, frame_h);
	ass_set_frame_size(ass_renderer, frame_w, frame_h);

	constexpr std::string_view file_sv = "test.ass";
	std::array<char, file_sv.size() + 1> file;
	auto it = std::copy(file_sv.begin(), file_sv.end(), file.begin());
	*it = '\0';

	ass_set_fonts(ass_renderer, nullptr, "sans-serif", ASS_FONTPROVIDER_AUTODETECT, nullptr, 1);
	auto track = ass_read_file(ass, file.data(), nullptr);
	int detect = 0;
	ASS_Image *img = ass_render_frame(ass_renderer, track, 0, &detect);

	for (; img != nullptr; img = img->next) {
		if (img->w == 0 || img->h == 0) {
			continue;
		}

		auto texture = SDL_CreateTexture(renderer, SDL_PIXELFORMAT_RGBA8888, SDL_TEXTUREACCESS_STREAMING, img->w, img->h);
		assert(texture);

		SDL_SetTextureBlendMode(texture, SDL_BLENDMODE_BLEND);

		std::uint32_t *pixels = nullptr;
		int pitch = 0;
		assert(SDL_LockTexture(texture, nullptr, reinterpret_cast<void **>(&pixels), &pitch) == 0);

		auto bitmap = img->bitmap;

		// It's better to unroll this loop.
		for (int y = 0; y < img->h; y++) {
			for (int x = 0; x < img->w; x++) {
				auto alpha = bitmap[x];

				pixels[x] = img->color & (0xffffff00 | alpha);
			}

			pixels = reinterpret_cast<std::uint32_t *>(reinterpret_cast<std::uintptr_t>(pixels) + pitch);
			bitmap += img->stride;
		}

		SDL_UnlockTexture(texture);

		SDL_Rect dst = {img->dst_x, img->dst_y, img->w, img->h};
		SDL_RenderCopy(renderer, texture, nullptr, &dst);

		SDL_DestroyTexture(texture);
		
	}

	SDL_RenderPresent(renderer);

	std::this_thread::sleep_for(std::chrono::seconds(10));

	ass_free_track(track);
	ass_renderer_done(ass_renderer);
	ass_library_done(ass);

	SDL_Quit();

	return 0;
}

Oh I think I’ve finally understood, libass generates a bitmap that represents alpha and another that represents color. In the case of white text with a black background, it creates two images of the respective colors and then applies the 1-bit bpp to the relevant pixels.

I tested your new example and I was able to understand it, it applies the color value and putting in the bits referring to alpha the value that tells if the bitmap exists.

However, testing the same code here, nothing appeared, I found it strange and changed the background color… That’s what I ended up with;

If you have a github I’d be happy to mention you as a contributor, even though I haven’t managed to apply the subtitles yet I’ve managed to understand thanks to your help.

I see, this doesn’t work with white subtitles at all, due to color profile issue, my bad. You need to find the right one and map img->color to it.

I tried the following, using SDL Color blend I made the rectangle when not filled have a color with alpha defined in hard coded.

What happens is the first and drawn with color. but the subsequent ones do not, they only render the boxes where the text is located. In case (3)

In the case (2) using the code with the color coming from the img, in the case below using the color defined in code

As you can see, it’s very close to what I want, but when I use your code on the spot, the colors are wrong. Is it an SDL bug? Because this error only happens when SDL_Blend is applied. When not, he assembles each layer with its colors and black boxes around it

Finally I generated the caption without being white, I used the ass or aegisub manipulation program, but still without fruit. Img (4)

Finally, I managed to implement

I saw that there was an error in the color variable. Pulling it out of the aligned bitmap construction form was perfect

Below is the code for the ASSLIB implementation in the final SDL, many thanks for Yataro’s help

/**
 * @file Text.h
 * @brief Contains the definition of the textObject class and the playText function.
 *
 * This file defines the textObject class, which is used to create and export text to an ASS file.
 * It also defines the playText function, which is used to render the text on the screen using SDL and ASS.
 */
namespace kotonoha
{
    class textObject
    {
    public:
        kotonohaData::acessMapper *exportTo = NULL;
        void push(std::string text, std::string timeStart, std::string timeEnd, std::string type, std::string sceneName)
        {
            // Start ASS Header
            if (!exportTo->text.init)
            {
                exportTo->text.stream << "[Script Info]\nTitle:" << sceneName << "\nScriptType: v4.00+\nWrapStyle: 0\nScaledBorderAndShadow: yes\nYCbCr Matrix: None\n\n[V4+ Styles]\nFormat: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding\nStyle: Default, Arial, 24, &H00FFFFFF, &H000000FF, &H00000000, &H80000000, -1, 0, 0, 0, 100, 100, 0, 0.00, 1, 2, 2, 2, 10, 10, 10, 1\n\n[Events]\nFormat: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text\n";
                exportTo->text.init = true;
            }
            // Convert  (MM:SS:DD) to ASS format
            std::vector<std::string> timeEndMMSSDD;
            std::vector<std::string> timeStartMMSSDD;
            std::string token;
            if (timeEnd.size() > 9)
            {
                timeEnd.erase(timeEnd.size() - 3);
            }
            else
            {
                timeEnd.erase(timeEnd.size() - 1);
            }
            std::istringstream stringStream(timeEnd);
            while (std::getline(stringStream, token, ':'))
            {
                timeEndMMSSDD.push_back(token);
            }
            stringStream.str("");
            stringStream = std::istringstream(timeStart);
            while (std::getline(stringStream, token, ':'))
            {
                timeStartMMSSDD.push_back(token);
            }
            // Push to ASS File, this line append one track sub to ASS
            exportTo->text.stream << "Dialogue:0,0:" << timeStartMMSSDD[0] << ":" << timeStartMMSSDD[1] << "." << timeStartMMSSDD[2] << ",0:" << timeEndMMSSDD[0] << ":" << timeEndMMSSDD[1] << "." << timeEndMMSSDD[2] << ",Default,,0,0,0,," << text << std::endl;
        };
    };
    int playText(void *import)
    {
        kotonohaData::acessMapper *importedTo = static_cast<kotonohaData::acessMapper *>(import);
        int h, w;
        // Init ASS
        ASS_Library *ass_library;
        ASS_Renderer *ass_renderer;
        ass_library = ass_library_init();
        ass_renderer = ass_renderer_init(ass_library);
        ass_set_extract_fonts(ass_library, 1);
        ass_set_fonts(ass_renderer, NULL, "sans-serif", ASS_FONTPROVIDER_AUTODETECT, NULL, 1);
        ass_set_hinting(ass_renderer, ASS_HINTING_NATIVE);
        // Create new ASS Track
        ASS_Track *track = ass_new_track(ass_library);
        std::string subSs = importedTo->text.stream.str();
        char *str_c = new char[subSs.length() + 1];
        strcpy(str_c, subSs.c_str());
        ass_process_data(track, str_c, subSs.length() + 1);
        // Start
        kotonohaTime::delay(1000);
        importedTo->root->log0->appendLog("(Text) - Start");
        SDL_Texture *texture = NULL;
        ASS_Image *img = NULL;
        while (importedTo->control->outCode == -1)
        {
            if (importedTo->control->display[2])
            {
                // Get window size to update
                SDL_GetWindowSize(importedTo->root->window, &w, &h);
                ass_set_storage_size(ass_renderer, w, h);
                ass_set_frame_size(ass_renderer, w, h);
                img = ass_render_frame(ass_renderer, track, kotonohaTime::convertToMs(importedTo->control->timer0.pushTime()), NULL);
                // This for loop send all tracks to display
                for (; img != nullptr; img = img->next)
                {
                    SDL_Rect dst = {img->dst_x, img->dst_y, img->w, img->h};
                    texture = SDL_CreateTexture(importedTo->root->renderer, SDL_PIXELFORMAT_RGBA8888, SDL_TEXTUREACCESS_STREAMING, img->w, img->h);
                    SDL_SetTextureBlendMode(texture, SDL_BLENDMODE_BLEND);
                    std::uint32_t *pixels = NULL;
                    int pitch = 0;
                    SDL_LockTexture(texture, NULL, reinterpret_cast<void **>(&pixels), &pitch);
                    uint32_t cor = (img->color & 0xffffff00) | 0xff;
                    for (int y = 0; y < img->h; y++)
                    {
                        for (int x = 0; x < img->w; x++)
                        {
                            if (img->bitmap[x] != 0)
                            {

                                pixels[x] = cor;
                            }
                            else
                            {
                                pixels[x] = 0x00000000;
                            }
                        }
                        pixels = reinterpret_cast<std::uint32_t *>(reinterpret_cast<std::uintptr_t>(pixels) + pitch);
                        img->bitmap += img->stride;
                    }
                    SDL_UnlockTexture(texture);
                    SDL_RenderCopy(importedTo->root->renderer, texture, NULL, &dst);
                }
                // End frame sub draw
                free(img);
                importedTo->control->display[2] = false;
                importedTo->control->display[3] = true;
            }
        };
        // End text engine
        SDL_DestroyTexture(texture);
        ass_free_track(track);
        ass_renderer_done(ass_renderer);
        ass_library_done(ass_library);
        importedTo->root->log0->appendLog("(Text) - End");
        return 0;
    };
}

This code is part of Kotonoha engine, Text.h is a header to manager subtitles on screen, see complete solucion is here Ruaneri-Portela/Kotonoha: Visual Novel engine basead on SDL and FFMPEG, reimplement of engine FORIS used in Days series games (producted by 0verflow) (github.com)