Embedding resources inside your program

I read this question in the archive a little while ago, and I remember
a funny hack from, uh, a very long time ago, that is actually more
portable than putting your resource file at the end of your executable
(which requires knowing the format of the header for your executable
format), but I didn’t have posting access at the time (was too lazy to
subscribe just for that!). Now I do, so there you go…

First, you need to make an object file from your resource file. This
is the least portable way, but it seems that every platform in the
universe has a way of doing that, going all the way back to CP/M, DOS
and Apple DOS. On Linux, binutils makes it pretty easy, with the
following command-line:

objcopy -I binary -O elf32-i386 -B i386 --rename-section
.data=.rodata,alloc,load,readonly,data,contents README.txt readme.o

The “-I binary” tells it that the source is binary. binutils will make
a set of three symbols, called _binary_README_txt_start,
_binary_README_txt_end and _binary_README_txt_size. The last one is
special, because it is a symbol with a fake address that can be used
to add to the start one (if you actually read it, it will probably
cause a segfault).

The “-O elf32-i386” specifies the format. Weirdly enough, you have to
also specify the architecture with “-B i386”. If you use “file” on an
object file, the first is the beginning part (“ELF 32-bit LSB”), and
the -B is the second part (“Intel 80386”).

The "–rename-section .data=.rodata,alloc,load,readonly,data,contents"
part is optional. It put the data in the read-only text segment,
instead of the writable data segment (the default). This both protects
you from writing to it accidentally (it will segfault), but also makes
it possible for the operating system to share that data between
multiple instances of the same process, using less physical memory).
In both cases, modern operating systems demand-paging will make it
that there is only physical memory consumed for pages that were used
recently (you could put a gigabyte file in there, and it will (more or
less) happily run on a system with 128 megs of memory).

To use it, you do as follow:

#include <stdio.h>

extern char _binary_README_txt_start[];
extern char _binary_README_txt_end[];
/* The following is not really needed, it’s there just for the demo. */
extern char _binary_README_txt_size[];

int main() {
printf(“start: %p\n end: %p\n size: %p\n”, _binary_README_txt_start,
_binary_README_txt_end, _binary_README_txt_size);

    fwrite(_binary_README_txt_start, _binary_README_txt_end -

_binary_README_txt_start, 1,
stdout);

    return 0;

}

In actual use, you can wrap this in a number of ways to make it more
portable. If you have a function that returns a pointer to the data,
for example, you could have it mmap() the file if it is external,
return &_binary_README_txt_start, or load it with stdio (or something)
and return a pointer to malloc()'d memory (you’ll need a "release"
function too, to munmap() or free() the memory, of course, which would
be in a no-op when using this trick).

If you’re clever, you can have resources in formats that are
pre-converted as much as possible. If your game is fixed resolution,
pre-compiled run-length encoding and such things might be possible. If
you’re bordering on the insane (which might apply when coding for some
consoles and hand-helds), you could even have SDL_Surface structures
and such baked right in there! Make a “resource creating program” that
loads the images the normal way, stuffs them in SDL surfaces, then
serialize them into your resource file (unportable, but okay, because
it will be baked into your executable, which isn’t portable either in
many cases). You’ll want to do some pointer swizzling (look it up on
Wikipedia), then you’re good to go!

I can’t really guarantee that your sanity will survive using those
tricks, though. :wink:

Have fun!–
http://pphaneuf.livejournal.com/

Quoting Pierre Phaneuf :

I read this question in the archive a little while ago, and I remember
a funny hack from, uh, a very long time ago,

i belive this was probably me going on about this :wink: most things i was
freaking
people out with was having an changable part of the exe so new highscores are
kept, eventually i decided to hold detailed high score type info and used
sqlite3, so i have the exe file and a highscore dtatabse file

my music man has given me about 10 meg of sound files ( i decided that
sometimes
the same sound isnt the same sound, so for one sound there are actually four
sounds that get played randomly ) so i think with that amount of data having
everything in the exe isnt really a great idea, might still do it, not like a
15 meg exe is exactly big these days…

and while i have an audience, i am also thinking of having skins for the game,
which would be a zip file containing all the sounds and gfx files, zips stored
in a directory of which the contents are read on program loading, any problems
with the contents of the folder or contents of the zip it just defaults to the
original that are built in the exe file.

sounds that get played randomly ) so i think with that amount of data having
everything in the exe isnt really a great idea, might still do it, not like a
15 meg exe is exactly big these days…

Well, the important thing is the overall disk footprint, no? A 1 meg
executable with a 14 megs data file or a 15 megs stand-alone
executable is pretty much the same, but some people prefer a single
file for easy of deployment purpose.

and while i have an audience, i am also thinking of having skins for the game,
which would be a zip file containing all the sounds and gfx files, zips stored
in a directory of which the contents are read on program loading, any problems
with the contents of the folder or contents of the zip it just defaults to the
original that are built in the exe file.

PhysFS could be interesting for that (http://icculus.org/physfs/).
Never used it myself, but Ryan knows his stuff, usually. ;-)On Thu, Apr 24, 2008 at 3:00 PM, wrote:


http://pphaneuf.livejournal.com/

Not sure about this…

Under Linux (as it has MMIO) it doesn’t really matter.

Under Windows though, I think it does - I believe the OS loads the exe
entirely into memory, so it’s gonna kill your start-up times. Any
Windows Guru got a definitive answer here?

Jitsu Love,

Eddy

Pierre Phaneuf wrote:> On Thu, Apr 24, 2008 at 3:00 PM, wrote:

sounds that get played randomly ) so i think with that amount of data having
everything in the exe isnt really a great idea, might still do it, not like a
15 meg exe is exactly big these days…

Well, the important thing is the overall disk footprint, no? A 1 meg
executable with a 14 megs data file or a 15 megs stand-alone
executable is pretty much the same, but some people prefer a single
file for easy of deployment purpose.

and while i have an audience, i am also thinking of having skins for the game,
which would be a zip file containing all the sounds and gfx files, zips stored
in a directory of which the contents are read on program loading, any problems
with the contents of the folder or contents of the zip it just defaults to the
original that are built in the exe file.

PhysFS could be interesting for that (http://icculus.org/physfs/).
Never used it myself, but Ryan knows his stuff, usually. :wink:

Under Windows though, I think it does - I believe the OS loads the exe
entirely into memory, so it’s gonna kill your start-up times. Any Windows
Guru got a definitive answer here?

I’m fairly sure that, for all its suckiness, Windows has
demand-paging. On some hardware that doesn’t have MMU, though, this
might not be the case. These days, that could be the case of some
mobile platforms (cell phones, portable music players, handheld gaming
systems), platforms using ?Clinux, and some platforms using Windows CE
(http://msdn.microsoft.com/en-us/library/aa915332.aspx).

That said, I’d probably point to putting things outside of the executable. :slight_smile:

My biggest piece of missing functionality in SDL, I’d say, would be a
way to get the pathname of the appropriate data directories (plural,
because of platform dependent and independent data, which might be in
the same place, but not necessarily). Finding the appropriate
directory on Mac OS X, for example, isn’t obvious, and is fairly
stable from one application to the other, due to the bundle packaging.
Maybe a pair of functions returning read-only SDL_RWops?On Fri, May 9, 2008 at 5:40 AM, Eddy Cullen wrote:


http://pphaneuf.livejournal.com/

Under Windows though, I think it does - I believe the OS loads the exe
entirely into memory, so it’s gonna kill your start-up times. Any
Windows Guru got a definitive answer here?

Windows won’t load data at the end of an .exe into memory when starting
the program. This is why those > 2 gigabyte self-extracting .rar files
can work on Win32.

–ryan.

Windows won’t load data at the end of an .exe into memory when starting the
program. This is why those > 2 gigabyte self-extracting .rar files can work
on Win32.

I think he was referring to the objcopy-based approach, where the data
goes in the application’s text segment and is addressable using a few
special symbols, rather than the "put the data at the end of the .EXE"
trick.

The amount of virtual memory consumed (as opposed to physical memory,
and delays in start-up times) is a very real issue, though, at a
certain point (a few hundred megabytes would be okay, but a gigabyte
would be start to be bad!).

Again, sanity points to something in the fashion of PhysFS instead. :-)On Mon, May 12, 2008 at 2:16 PM, Ryan C. Gordon wrote:


http://pphaneuf.livejournal.com/

shrug

I know very little about Windows internals, I mean, who wants to? grin

Ryan C. Gordon wrote:>

Under Windows though, I think it does - I believe the OS loads the exe
entirely into memory, so it’s gonna kill your start-up times. Any
Windows Guru got a definitive answer here?

Windows won’t load data at the end of an .exe into memory when starting
the program. This is why those > 2 gigabyte self-extracting .rar files
can work on Win32.

–ryan.


SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Embedding resources is nice, but it doesn’t work all that well cross-platform. What I’d really like to see is an SDL_Archive library, something optimized for storage and quick retrieval of media for games.> ----- Original Message -----

From: eac203@ecs.soton.ac.uk (Edward Cullen)
To: A list for developers using the SDL library. (includes SDL-announce)
Sent: Monday, May 12, 2008 3:58:34 PM
Subject: Re: [SDL] embedding resources inside your program

shrug

I know very little about Windows internals, I mean, who wants to? grin

Ryan C. Gordon wrote:

Under Windows though, I think it does - I believe the OS loads the exe
entirely into memory, so it’s gonna kill your start-up times. Any
Windows Guru got a definitive answer here?

Windows won’t load data at the end of an .exe into memory when starting
the program. This is why those > 2 gigabyte self-extracting .rar files
can work on Win32.

–ryan.


SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org


SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org