SDL RLE load/save

SDL Development
1

I’ve been playing around with this idea of saving and loading the internal
sdl rle internal format, in order to improve load time, as well as keeping
on disk size down…

Does this make any kind of sense?

Well, it’s quite messy (writing routines to mimic the compress/decompress
internal to sdl, poking pointer values into yet more private structures like
SDL_BlitMap, setting the sdl_rle flags on the surfaces, hack hack hack…
).

I’m wondering if there’s a cleaner way, perhaps via a custom loader via
SDL_image?

2

[…]

Not really… RLE (Run-Length Encoding) can indeed be used for data
compression, but the internal SDL RLE encoding is designed for fast
rendering (no per pixel transparency checking required for colorkey
and alpha blits) rather than data size.

You’re probably much better off using PNG, or (in some cases) JPG.

PNG features effective non-destructive compression, and supports an
alpha channel for translucency effects and antialiazing. Just save
with maximum compression for minimal size. (Some applications use a
lower degree of compression by default, probably because it can take
quite a while to compress a high resolution digital photo with the
maximum setting…)

JPG is destructive, but for some types of images (typically “normal”
photos and similar material), it tends to generate much smaller files
than PNG or similar. Just try it and see if you can compress enough
without unacceptable artifacts. (Any serious image editing software
should have a preview feature that gives you the final result and the
file size as you adjust the compression settings.)

The downside of JPG is that it doesn’t support an alpha channel, and
due to the compression method, it doesn’t really work with
colorkeying either - so in games, it’s kind of hard to use for
anything but tiles, background images and the like.

You might get away with saving the alpha channel out as a separate PNG
(should compress really well, as it’s mostly fully opaque or fully
transparent, except around antialiazed edges), and then merging it
back with the color channels from the JPG into an RGBA surface at
load time. I suppose that’s basically what JNG
(http://mng.internet.bs/spec/jng.html) does…?

//David Olofson - Programmer, Composer, Open Source Advocate

.------- http://olofson.net - Games, SDL examples -------.
| http://zeespace.net - 2.5D rendering engine |
| http://audiality.org - Music/audio engine |
| http://eel.olofson.net - Real time scripting |
'-- http://www.reologica.se - Rheology instrumentation --'On Tuesday 21 October 2008, Frank Grime wrote:

I’ve been playing around with this idea of saving and loading the
internal sdl rle internal format, in order to improve load time, as
well as keeping on disk size down…

Does this make any kind of sense?

3

Well, the thought process here is reducing the decompression time during the
load. I’m on an embedded platform so it’s a fairly underpowered cpu with
low memory constraints. I want fast load times, as well as fast blitting,
and reasonable on disk size.

For loading PNG or JPG, it essentially decodes them into surface with the
proper display format, and then I rle the surface. I’m trying to cut out a
step of the loading.

Though the internal sdl rle encoding is optimized for fast blitting, I’ve
found that for most images, the data size is smaller than the raw data. In
some cases, it’s larger, but not by much.

Rather than having to decode and then rle, I’d like to be able to load (or
memory map) the rle data directly. The idea here is that the rle is smaller
than just the raw data (most of the time), it’s already optimized for
blitting, and the loaded images also uses less memory.

Yes, JPG or PNG would take up less space, but I’m trying to find the happy
medium between disk size, memory usage, loading, as well as fast blitting.On Tue, Oct 21, 2008 at 10:42 AM, David Olofson wrote:

On Tuesday 21 October 2008, Frank Grime wrote:

I’ve been playing around with this idea of saving and loading the
internal sdl rle internal format, in order to improve load time, as
well as keeping on disk size down…

Does this make any kind of sense?
[…]

Not really… RLE (Run-Length Encoding) can indeed be used for data
compression, but the internal SDL RLE encoding is designed for fast
rendering (no per pixel transparency checking required for colorkey
and alpha blits) rather than data size.

You’re probably much better off using PNG, or (in some cases) JPG.

PNG features effective non-destructive compression, and supports an
alpha channel for translucency effects and antialiazing. Just save
with maximum compression for minimal size. (Some applications use a
lower degree of compression by default, probably because it can take
quite a while to compress a high resolution digital photo with the
maximum setting…)

JPG is destructive, but for some types of images (typically “normal”
photos and similar material), it tends to generate much smaller files
than PNG or similar. Just try it and see if you can compress enough
without unacceptable artifacts. (Any serious image editing software
should have a preview feature that gives you the final result and the
file size as you adjust the compression settings.)

The downside of JPG is that it doesn’t support an alpha channel, and
due to the compression method, it doesn’t really work with
colorkeying either - so in games, it’s kind of hard to use for
anything but tiles, background images and the like.

You might get away with saving the alpha channel out as a separate PNG
(should compress really well, as it’s mostly fully opaque or fully
transparent, except around antialiazed edges), and then merging it
back with the color channels from the JPG into an RGBA surface at
load time. I suppose that’s basically what JNG
(http://mng.internet.bs/spec/jng.html) does…?

//David Olofson - Programmer, Composer, Open Source Advocate

.------- http://olofson.net - Games, SDL examples -------.
| http://zeespace.net - 2.5D rendering engine |
| http://audiality.org - Music/audio engine |
| http://eel.olofson.net - Real time scripting |
‘-- http://www.reologica.se - Rheology instrumentation --’

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

4

If you’re on an embedded platform, then yeah, go for it. The problem with doing what you’re proposing is that the data at that level is optimized for the specific video hardware you’re using, and so you’re not going to create something that’s all that portable. Apparently sometimes you can even create incompatibilities within the same hardware platform.

But if you’re on something like a phone, where the hardware is the exact same every single time, that’s not a problem.> ----- Original Message -----

From: frankgrime@gmail.com (Frank Grime)
To: A list for developers using the SDL library. (includes SDL-announce)
Sent: Tuesday, October 21, 2008 8:52:21 AM
Subject: Re: [SDL] SDL RLE load/save

Well, the thought process here is reducing the decompression time during the load. I’m on an embedded platform so it’s a fairly underpowered cpu with low memory constraints. I want fast load times, as well as fast blitting, and reasonable on disk size.

For loading PNG or JPG, it essentially decodes them into surface with the proper display format, and then I rle the surface. I’m trying to cut out a step of the loading.

Though the internal sdl rle encoding is optimized for fast blitting, I’ve found that for most images, the data size is smaller than the raw data. In some cases, it’s larger, but not by much.

Rather than having to decode and then rle, I’d like to be able to load (or memory map) the rle data directly. The idea here is that the rle is smaller than just the raw data (most of the time), it’s already optimized for blitting, and the loaded images also uses less memory.

Yes, JPG or PNG would take up less space, but I’m trying to find the happy medium between disk size, memory usage, loading, as well as fast blitting.

On Tue, Oct 21, 2008 at 10:42 AM, David Olofson wrote:

On Tuesday 21 October 2008, Frank Grime wrote:

I’ve been playing around with this idea of saving and loading the
internal sdl rle internal format, in order to improve load time, as
well as keeping on disk size down…

Does this make any kind of sense?
[…]

Not really… RLE (Run-Length Encoding) can indeed be used for data
compression, but the internal SDL RLE encoding is designed for fast
rendering (no per pixel transparency checking required for colorkey
and alpha blits) rather than data size.

You’re probably much better off using PNG, or (in some cases) JPG.

PNG features effective non-destructive compression, and supports an
alpha channel for translucency effects and antialiazing. Just save
with maximum compression for minimal size. (Some applications use a
lower degree of compression by default, probably because it can take
quite a while to compress a high resolution digital photo with the
maximum setting…)

JPG is destructive, but for some types of images (typically “normal”
photos and similar material), it tends to generate much smaller files
than PNG or similar. Just try it and see if you can compress enough
without unacceptable artifacts. (Any serious image editing software
should have a preview feature that gives you the final result and the
file size as you adjust the compression settings.)

The downside of JPG is that it doesn’t support an alpha channel, and
due to the compression method, it doesn’t really work with
colorkeying either - so in games, it’s kind of hard to use for
anything but tiles, background images and the like.

You might get away with saving the alpha channel out as a separate PNG
(should compress really well, as it’s mostly fully opaque or fully
transparent, except around antialiazed edges), and then merging it
back with the color channels from the JPG into an RGBA surface at
load time. I suppose that’s basically what JNG
(http://mng.internet.bs/spec/jng.html) does…?

//David Olofson - Programmer, Composer, Open Source Advocate

.------- http://olofson.net - Games, SDL examples -------.
| http://zeespace.net - 2.5D rendering engine |
| http://audiality.org - Music/audio engine |
| http://eel.olofson.net - Real time scripting |
‘-- http://www.reologica.se - Rheology instrumentation --’

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

5

Well, the thought process here is reducing the decompression time
during the load. I’m on an embedded platform so it’s a fairly
underpowered cpu with low memory constraints. I want fast load
times, as well as fast blitting, and reasonable on disk size.

Is CPU power the only real restriction, or is loading bandwidth also a
factor? Do you actually need to compress the files at all?

Yes, JPG or PNG would take up less space, but I’m trying to find the
happy medium between disk size, memory usage, loading, as well as
fast blitting.

I think there are various “real” compression algorithms that will do a
much better job than a simple RLE that isn’t even tuned for the job.

I’d suggest looking for “executable packers”, as that’s one area where
you want efficiency and fast decompression, but don’t care much about
compression speed. Most of the widely used compression algorithms
(zip, gzip, bzip2, rar etc) focus on size, decompression speed and
compression speed with roughly equal priority, which means you’re
paying a penalty for “reasonably fast” compression that you don’t
really need.

//David Olofson - Programmer, Composer, Open Source Advocate

.------- http://olofson.net - Games, SDL examples -------.
| http://zeespace.net - 2.5D rendering engine |
| http://audiality.org - Music/audio engine |
| http://eel.olofson.net - Real time scripting |
'-- http://www.reologica.se - Rheology instrumentation --'On Tuesday 21 October 2008, Frank Grime wrote:

6

[…]

For loading PNG or JPG, it essentially decodes them into surface
with the proper display format, and then I rle the surface. I’m
trying to cut out a step of the loading.
[…]

Another thought: Have you verified that the actual RLE encoding step
has significant impact on loading time? (That is, verified that it’s
not storage bandwidth or other steps that consume time.)

I don’t think it takes much pixel shuffling in a loader to end up
slower than the SDL RLE encoder…

//David Olofson - Programmer, Composer, Open Source Advocate

.------- http://olofson.net - Games, SDL examples -------.
| http://zeespace.net - 2.5D rendering engine |
| http://audiality.org - Music/audio engine |
| http://eel.olofson.net - Real time scripting |
'-- http://www.reologica.se - Rheology instrumentation --'On Tuesday 21 October 2008, Frank Grime wrote:

7

Is CPU power the only real restriction, or is loading bandwidth also a
factor? Do you actually need to compress the files at all?

Loading bandwidth is a restriction, and so is memory usage, so using the rle
format solves many problems simultaneously.

Yes, JPG or PNG would take up less space, but I’m trying to find the
happy medium between disk size, memory usage, loading, as well as
fast blitting.

I think there are various “real” compression algorithms that will do a
much better job than a simple RLE that isn’t even tuned for the job.

That is true, but that would also involve writing my own blitter that
understands how to blit the compressed format…>

I’d suggest looking for “executable packers”, as that’s one area where
you want efficiency and fast decompression, but don’t care much about
compression speed. Most of the widely used compression algorithms
(zip, gzip, bzip2, rar etc) focus on size, decompression speed and
compression speed with roughly equal priority, which means you’re
paying a penalty for “reasonably fast” compression that you don’t
really need.

8

That’s a good point; I’m not sure how much of an impact that the rle
encoding step does, but rather than load then encode, I thought it’d be
simpler just to load directly into a format that the blitter already
understands.

I was thinking why do the rle at run-time if we can just load it pre-rle’d.
It cuts my in-memory and on disk usage almost in half, though it isn’t very
portable…

I it working, but I was just trying to find a cleaner way to do it.On Tue, Oct 21, 2008 at 12:11 PM, David Olofson wrote:

On Tuesday 21 October 2008, Frank Grime wrote:
[…]

For loading PNG or JPG, it essentially decodes them into surface
with the proper display format, and then I rle the surface. I’m
trying to cut out a step of the loading.
[…]

Another thought: Have you verified that the actual RLE encoding step
has significant impact on loading time? (That is, verified that it’s
not storage bandwidth or other steps that consume time.)

I don’t think it takes much pixel shuffling in a loader to end up
slower than the SDL RLE encoder…

//David Olofson - Programmer, Composer, Open Source Advocate

.------- http://olofson.net - Games, SDL examples -------.
| http://zeespace.net - 2.5D rendering engine |
| http://audiality.org - Music/audio engine |
| http://eel.olofson.net - Real time scripting |
‘-- http://www.reologica.se - Rheology instrumentation --’

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

9

[…]

Yes, JPG or PNG would take up less space, but I’m trying to find
the happy medium between disk size, memory usage, loading, as
well as fast blitting.

I think there are various “real” compression algorithms that will
do a much better job than a simple RLE that isn’t even tuned for
the job.

That is true, but that would also involve writing my own blitter
that understands how to blit the compressed format…

Well, I was thinking compressed files for fast loading, using a format
that decompresses fast enough. That is, if you have enough CPU power
to make that pay off with any reasonably efficient compression.

The compressed files could contain actual SDL RLE encoded data, so you
can just load, decompress (most of these algorithms allow in-place
decompression BTW, so no need for intermediate buffers) - but then
you still have this “messing with SDL internals” issue, of course;
just smaller files and (hopefully) faster loading…

//David Olofson - Programmer, Composer, Open Source Advocate

.------- http://olofson.net - Games, SDL examples -------.
| http://zeespace.net - 2.5D rendering engine |
| http://audiality.org - Music/audio engine |
| http://eel.olofson.net - Real time scripting |
'-- http://www.reologica.se - Rheology instrumentation --'On Tuesday 21 October 2008, Frank Grime wrote:

10

Well, I was thinking compressed files for fast loading, using a format
that decompresses fast enough. That is, if you have enough CPU power
to make that pay off with any reasonably efficient compression.

There’s definitely a fine line – i/o speed vs cpu, or load speed vs
decompress speed. It’s easy enough to use the gzip wrappers (gzopen,
gzread, etc) and load the data on disk, but I think that mmapping the rle
data would probably give the fastest percieved load time. I think that’s
the next step.

The compressed files could contain actual SDL RLE encoded data, so you> can just load, decompress (most of these algorithms allow in-place

decompression BTW, so no need for intermediate buffers) - but then
you still have this “messing with SDL internals” issue, of course;
just smaller files and (hopefully) faster loading…

//David Olofson - Programmer, Composer, Open Source Advocate

11

That’s a good point; I’m not sure how much of an impact that the
rle encoding step does, but rather than load then encode, I thought
it’d be simpler just to load directly into a format that the blitter
already understands.

I was thinking why do the rle at run-time if we can just load it
pre-rle’d. It cuts my in-memory and on disk usage almost in half,
though it isn’t very portable…

Well, it’s probably no big deal if you can make sure users can’t mix
up SDL versions and data files. You may need to generate platform
specific data files, but that may not be much of an issue. Depends on
how you’re distributing…

Also note that the RLE data format also depends on the display pixel
format, so on a platform with different display modes - like anytihng
but handheld devices and old consoles, basically - you can’t really
use this approach at all. Well, you could convert as needed, but that
sort of defeats the whole point.

I it working, but I was just trying to find a cleaner way to do it.

I don’t think there is one under these circumstances… Platforms that
are basically low on everything don’t lend themselves very well to
truly portable solutions.

//David Olofson - Programmer, Composer, Open Source Advocate

.------- http://olofson.net - Games, SDL examples -------.
| http://zeespace.net - 2.5D rendering engine |
| http://audiality.org - Music/audio engine |
| http://eel.olofson.net - Real time scripting |
'-- http://www.reologica.se - Rheology instrumentation --'On Tuesday 21 October 2008, Frank Grime wrote:

12

Hmm. I think gzip would be too slow.On Tue, Oct 21, 2008 at 1:44 PM, george pee wrote:

Well, I was thinking compressed files for fast loading, using a format

that decompresses fast enough. That is, if you have enough CPU power
to make that pay off with any reasonably efficient compression.

There’s definitely a fine line – i/o speed vs cpu, or load speed vs
decompress speed. It’s easy enough to use the gzip wrappers (gzopen,
gzread, etc) and load the data on disk, but I think that mmapping the rle
data would probably give the fastest percieved load time. I think that’s
the next step.

The compressed files could contain actual SDL RLE encoded data, so you

can just load, decompress (most of these algorithms allow in-place
decompression BTW, so no need for intermediate buffers) - but then
you still have this “messing with SDL internals” issue, of course;
just smaller files and (hopefully) faster loading…

//David Olofson - Programmer, Composer, Open Source Advocate

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

13

Ya, it’d end up getting convoluted if I were to start making a header which
includes the display format info… We’d end up with our own file format
like png.

A blitter that uses the png format as the source format might work, but I
doubt it.On Tue, Oct 21, 2008 at 1:46 PM, David Olofson wrote:

On Tuesday 21 October 2008, Frank Grime wrote:

That’s a good point; I’m not sure how much of an impact that the
rle encoding step does, but rather than load then encode, I thought
it’d be simpler just to load directly into a format that the blitter
already understands.

I was thinking why do the rle at run-time if we can just load it
pre-rle’d. It cuts my in-memory and on disk usage almost in half,
though it isn’t very portable…

Well, it’s probably no big deal if you can make sure users can’t mix
up SDL versions and data files. You may need to generate platform
specific data files, but that may not be much of an issue. Depends on
how you’re distributing…

Also note that the RLE data format also depends on the display pixel
format, so on a platform with different display modes - like anytihng
but handheld devices and old consoles, basically - you can’t really
use this approach at all. Well, you could convert as needed, but that
sort of defeats the whole point.

I it working, but I was just trying to find a cleaner way to do it.

I don’t think there is one under these circumstances… Platforms that
are basically low on everything don’t lend themselves very well to
truly portable solutions.

//David Olofson - Programmer, Composer, Open Source Advocate

.------- http://olofson.net - Games, SDL examples -------.
| http://zeespace.net - 2.5D rendering engine |
| http://audiality.org - Music/audio engine |
| http://eel.olofson.net - Real time scripting |
‘-- http://www.reologica.se - Rheology instrumentation --’

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org