Thoughts on the new filesystem API

SDL Development
21
  1. With this setup it may be better to provide the following API instead of
    the current one (or as well as it):

char const *SDL_get_location (char const *abstract_location_name);

This uses a text string for the location name. The advantage is that if
the setup file in (2) is read it can provide any extra non-standard
directory locations the application requires.

I, for one, like this idea, but I have a question:
Are you willing to provide the needed data structure(s) under the zlib
license?

We’re talking about half a dozen keys so a linear array with a sequential
search is just fine. It’s not like it will be queried many times either.

But you could use C std library bsearch to do a binary chop if the elements
were sorted … hardly seems worth it though.On 22/08/2013, at 1:32 PM, Jared Maddox wrote:

–
john skaller
@john_skaller
http://felix-lang.org

22

four separate directories:

  1. Installation directory
  2. Application Data directory
  3. User Data directory
  4. Application Invocation directory

Let’s not over-complicate this. In practice, it’s only two things (“where do I read?” and “where do I write?”).

Where do I read an write what?

char const *SDL_get_location (char const *abstract_location_name);
I, for one, like this idea, but I have a question:

The goal isn’t to build an abstract means to configure the way the system thinks about file layout, the goal is to understand where files go, as dictated by the standards for that platform, using platform-specific APIs behind the scenes that obtain this information so the app doesn’t have to.

The base function above (together with a SDL_set_location for configuration) obeys (Bertrand
Meyer) Open/Closed Principle.

It’s a fixed API which can provide the functionality you want right now,
but it is extensible in that YOU (SDL developer) can add a new location easily
if users jump up and down and say what is provided isn’t enough (and you know,
it never will be … :slight_smile:

Or you can tell the user to go jump and extend it themselves.

And you can still provide

char const *SDL_GetBaseDirectory() { return SDL_get_location ("base"); }

if you like.

There is rule in computer science a lot of people never grokked:
when I here someone talk about Singletons I get out my machine gun.
There is no such thing as something there is only one of.

And that includes places to read and write files :slight_smile:

Consider also … how would you TEST your application prior to installation
if the filesystem API were’t programmable?On 22/08/2013, at 2:26 PM, Ryan C. Gordon wrote:

From my perspective, the above three snippets actually describe up to

–
john skaller
@john_skaller
http://felix-lang.org

23

Potentially three things: “where do I read static data?”, “where do I
write user-specific settings?”, and “where do I write shared settings?”.
Shared settings could be used, for example, to implement local
highscore files shared between users. Or DLC, which cannot be stored
with the static data because you probably don’t have write permission
for that directory.On 22.08.2013 06:26, Ryan C. Gordon wrote:

From my perspective, the above three snippets actually describe up to
four separate directories:

  1. Installation directory
  2. Application Data directory
  3. User Data directory
  4. Application Invocation directory

Let’s not over-complicate this. In practice, it’s only two things
(“where do I read?” and “where do I write?”).

–
Rainer Deyke (rainerd at eldwood.com)

24

I’ve been using binreloc in all of my projects to handle the “where the heck is the exe”…

This code came out of the old autopackage (now listaller http://listaller.tenstral.net/ )On Aug 21, 2013, at 5:21 , Sam Lantinga wrote:

Wow, looking at that code takes me back… :slight_smile:

On Wed, Aug 21, 2013 at 1:18 AM, Ryan C. Gordon wrote:

It makes much more sense to me to stick a shell script in the PATH that
cd’s to the proper directory first than a symlink.

It’s also an option, probably a better one.

What the Loki games would do, near startup, is figure out where the real binary was from argv[0]: if it had a ‘/’ in it, you knew exactly where the file was, and if not, they searched the $PATH for it. This was before Linux offered /proc/self/exe.

You can see exactly what Loki’s code did here (look for “loki_initpaths”)…

https://svn.icculus.org/*checkout*/loki_utils/trunk/loki_paths.c?content-type=text%2Fplain

–ryan.

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Edward Rudd
OutOfOrder.cc
Skype: outoforder_cc
317-674-3296

25

So developers don’t have to think as much about platform differences…
Otherwise this function becomes really useless on OS X (and iOS) as I
have to create my own code for “those” platforms… (And I’m assuming
you means Contents/Resources :smiley: )…

(oops, yes, Contents/Resources, sorry.)

Just so I’m clear, you’re asking for a hint so you can do this…

// we put the data files somewhere else here, but only on Mac OS X!
#if defined(STEAM_BUILD) && defined(APPLE)
SDL_SetHint(SDL_HINT_FILESYSTEM_OSX_USE_RESOURCE_DIR, “1”);
#endif
std::string mypath = SDL_GetBaseDir();

…instead of this…

std::string mypath = SDL_GetBaseDir();
// we put the data files somewhere else, but only on Mac OS X!
#if defined(STEAM_BUILD) && defined(APPLE)
mypath += “Contents/Resources/”;
#endif

…or am I misunderstanding the problem?

Well… it would be more like this in the second case the way things currently are in your code

std::string mypath = SDL_GetBaseDir();
// we put the data files somewhere else, but only on Mac OS X!
#if defined(APPLE)
#if defined(STEAM_BUILD)
mypath += “…/”; // I don’t like appenting …'s I prefer realpaths!
#else
mypath += “Contents/Resources/”;
#endif
#endif

Now, one “nicety” of the way the SDL hint system works… Is I do not have to set my hints via the SDL_SetHint… I can set them via the env…

So I can add this to my Info.plist

LSEnvironment

SDL_FILESYSTEM_OSX_USE_RESOURCE_DIR
1

Now for we can detect whether the OS X app is packaged in a bundle or not and make the best guess as to the data dir (in bundle resource out of bundle exedir)… an alternative approach might be

char* SDL_GetDataDir(const char* existing_file);

and have it check platform specific known dirs and find the file that should be in the data dir. (it could be a relative dir/file e.g. audio/main.pak)

My main issue with this API is that it works cleanly on all platforms except OS X which makes it useless as I essentially have to resort to what I’m currently doing which is platform specific code to use binreloc on linux NSBundle resourcePath on Mac. And if users have to resort to that, there really isn’t much point to having the API in the first place. Being able to simply put a two liner of

std::string g_datapath, g_userpath;

g_datapath = SDL_GetDataDir();
g_userpath = SDL_GetUserDir();

is what people would expect. and would keep in line with the way the rest of SDL is written. (keep the #ifdefs away !!)

Edward Rudd
OutOfOrder.cc
Skype: outoforder_cc
317-674-3296On Aug 20, 2013, at 21:39 , Ryan C. Gordon wrote:

26

Silly question, but can’t SDL detect whether an OSX program is running
from within a bundle or as standalone? That should be enough to
determine whether to look for program assets within the bundle or
elsewhere.

27

Silly question, but can’t SDL detect whether an OSX program is running
from within a bundle or as standalone? That should be enough to
determine whether to look for program assets within the bundle or
elsewhere.

The Cocoa API we’re using for SDL_GetBasePath() already does this.

–ryan.

28

My main issue with this API is that it works cleanly on all platforms
except OS X which makes it useless as I essentially have to resort to

You keep using the word “useless” because this requires an #ifdef with
an extra strcat instead of spinning off a separate .m file in your
project with a handful of Objective-C calls. An #ifdef you need anyhow,
because different builds expect the datafiles in different places.

So I wish you’d stop throwing that word around.

is what people would expect. and would keep in line with the way the
rest of SDL is written. (keep the #ifdefs away !!)

Let me think on this a little more. I’m sort of coming around on the
Contents/Resources thing. I could see a world where Apple starts
rejecting apps from their Store because they don’t have data files
there, because that sounds like the sort of thing they’d do.

–ryan.

29

The base function above (together with a SDL_set_location for configuration) obeys (Bertrand
Meyer) Open/Closed Principle.

John Carmack put it more succinctly than I could:

 https://twitter.com/ID_AA_Carmack/status/53512300451201024

Consider also … how would you TEST your application prior to installation
if the filesystem API were’t programmable?

Probably with the two lines I wrote for testfilesystem.c:

 SDL_Log("base path: '%s'\n", SDL_GetBasePath());
 SDL_Log("pref path: '%s'\n", SDL_GetPrefPath("libsdl",

“testfilesystem”));

There’s only ever one right answer, and it printed it to stdout. I don’t
see what value a configuration mechanism (that parses from a RWops!)
adds to this.

Instead of the Wikipedia page for “Open/Closed Principle,” read this
next time:

 http://sebastiansylvan.com/2013/08/16/the-perils-of-future-coding/

–ryan.

30

Consider also … how would you TEST your application prior to installation
if the filesystem API were’t programmable?

Probably with the two lines I wrote for testfilesystem.c:

SDL_Log(“base path: ‘%s’\n”, SDL_GetBasePath());
SDL_Log(“pref path: ‘%s’\n”, SDL_GetPrefPath(“libsdl”, “testfilesystem”));

I’m sorry I wasn’t plain enough. I’m not asking how you test SDL_GetBasePath,
I’m asking how you test your application, which is not installed, if it tries to get
data from the SDL_GetBasePath() which returns the location the data WILL HAVE
when it is installed … only it isn’t yet installed.

I have this problem all the time. I don’t know a good answer.

Instead of the Wikipedia page for “Open/Closed Principle,” read this next time:

I’ve never seen the Wikipedia page. I read Meyers book … years ago.

https://www.sebastiansylvan.com/post/the-perils-of-future-coding/

Yes, I understand all that already. I gave up reading “pop culture”
a long time ago.

There’s an “engineering tradeoff” in all choices. (I hate that word
“engineering” applied to software).

It’s your choice of course :)On 23/08/2013, at 3:42 PM, Ryan C. Gordon wrote:

–
john skaller
@john_skaller
http://felix-lang.org

31

2013/8/23, john skaller :

I’m sorry I wasn’t plain enough. I’m not asking how you test
SDL_GetBasePath,
I’m asking how you test your application, which is not installed, if it
tries to get
data from the SDL_GetBasePath() which returns the location the data WILL
HAVE
when it is installed … only it isn’t yet installed.

I have this problem all the time. I don’t know a good answer.

Does any filesystem API in existence provide a good answer for this
one? Not as far as I know…

Of course, an option could be to have SDL peek at the program’s
location. If it’s running from where programs are usually installed,
then use the relevant directory it’d use when installed, otherwise use
the program’s own directory.

Sounds confusing though :stuck_out_tongue:

Yes, I understand all that already. I gave up reading “pop culture”
a long time ago.

There’s an “engineering tradeoff” in all choices. (I hate that word
“engineering” applied to software).

It’s your choice of course :slight_smile:

This stuff rarely changes, it’s just a few paths and as you can see
not all of them would be processed immediately (though SDL_Init could
take care of most of that). Separate functions will be probably easier
to understand and use in the long term.

Also, you missed a golden opportunity to just reuse SDL’s hint system :stuck_out_tongue:

32

I’m sorry I wasn’t plain enough. I’m not asking how you test SDL_GetBasePath,
I’m asking how you test your application, which is not installed, if it tries to get
data from the SDL_GetBasePath() which returns the location the data WILL HAVE
when it is installed … only it isn’t yet installed.

GetBasePath returns where the program is running from, not where it will
be installed.

–ryan.

33

I’m sorry I wasn’t plain enough. I’m not asking how you test SDL_GetBasePath,
I’m asking how you test your application, which is not installed, if it tries to get
data from the SDL_GetBasePath() which returns the location the data WILL HAVE
when it is installed … only it isn’t yet installed.

I have this problem all the time. I don’t know a good answer.

What I do is having all my assets defined by a relative path and then
have a list of directories which are searched in a specific order. One
entry is the platform-specific data directory, another (with higher
priority) is a directory that can be specified on the command line,
which can be supplied during testing and omitted when running the
installed package.

The question that bugs me, however: with the details of platform
specific data locations hidden within SDL, how do I know where to
install data in the first place? In my case, the platform specific
stuff is handled by configure/cmake at build time, so make install
will put the data where it belongs and passed to the code via config.h
or a -D switch.

Using the SDL file API, I might get the correct place to put stuff for
each platform (and not what I might believe to be correct), but I
would have no easy way to make sure stuff actually got installed in
the proper place.

If there was a way to use those paths returned by the SDL file API
during a configure step (i.e. via pkgconfig variables or some such),
I’d be hooked. Otherwise, I’ll keep my own solution, even if it might
be non-standard.

KaiOn Fri, Aug 23, 2013 at 12:58 PM, john skaller wrote:

34

I’m sorry I wasn’t plain enough. I’m not asking how you test SDL_GetBasePath,
I’m asking how you test your application, which is not installed, if it tries to get
data from the SDL_GetBasePath() which returns the location the data WILL HAVE
when it is installed … only it isn’t yet installed.

GetBasePath returns where the program is running from, not where it will be installed.

So the directory containing the executable?

Er … so what does it return if you’re running SDL from a Python script?

What does it do if SDL is in a shared library which is loaded at run time?
Which is the case for all my stuff by default.On 24/08/2013, at 12:14 AM, Ryan C. Gordon wrote:

–
john skaller
@john_skaller
http://felix-lang.org

35

I’m sorry I wasn’t plain enough. I’m not asking how you test SDL_GetBasePath,
I’m asking how you test your application, which is not installed, if it tries to get
data from the SDL_GetBasePath() which returns the location the data WILL HAVE
when it is installed … only it isn’t yet installed.

I have this problem all the time. I don’t know a good answer.

What I do is having all my assets defined by a relative path and then
have a list of directories which are searched in a specific order.

Yes, I tend to do that too. It isn’t really satisfactory.

Another method I use is to have a single file which lists where
everything is, and then point at it by considering in order:

(a) command line switch
(b) environment variable
(c) default value set at build time

If there was a way to use those paths returned by the SDL file API
during a configure step (i.e. via pkgconfig variables or some such),
I’d be hooked. Otherwise, I’ll keep my own solution, even if it might
be non-standard.

From the discussion, I think the intent is for SDL to only provide
absolutely minimal support. After all this is the approach of the rest
of the system: just provide basic “getting started” stuff much of which
would be replaced down the track in a sophisticated system. So it is
useful to satisfy initial requirements and for simpler (aspects of a)
system.On 24/08/2013, at 2:05 AM, Kai Sterker wrote:
On Fri, Aug 23, 2013 at 12:58 PM, john skaller <@john_skaller> wrote:

–
john skaller
@john_skaller
http://felix-lang.org

36

2013/8/23, john skaller :

Er … so what does it return if you’re running SDL from a Python script?

Ideally the directory of the script, but I guess that depends on how
the binding is implemented…

What does it do if SDL is in a shared library which is loaded at run time?
Which is the case for all my stuff by default.

Should that be an issue? I mean, it should be exactly the same deal as
when using a DLL at load-time, i.e. it’s the host program that
matters, not the library.

37

2013/8/23, john skaller <@john_skaller>:

Er … so what does it return if you’re running SDL from a Python script?

Ideally the directory of the script, but I guess that depends on how
the binding is implemented…

What’s it got to do with bindings? The SDL_* filesystem API is
part of the library, a binding just translates from C to Python.

What does it do if SDL is in a shared library which is loaded at run time?
Which is the case for all my stuff by default.

Should that be an issue?

It is for me.

I mean, it should be exactly the same deal as
when using a DLL at load-time, i.e. it’s the host program that
matters, not the library.

Not with my system, since the default operation uses a single
executable to run all programs code as DLLs. The location of
that executable is irrelevant. This is similar to Python having
a single executable. The location of the executable is irrelevant
since the content is in the loaded DLL.

The thing is on my system you can also statically link exactly the same
program, i.e. you can get the mainline object file and link against what
would by default be a DLL, to create a stand alone executable.
[You can even statically link plugins without modifying the code
that loads them]

And in my system by default, the location of the DLL or statically linked
executable is in a cache of no relevance at all: the actual program is
considered the source code, like in Python.

So the file system API isn’t going to work unless I can intervene and
program it (if the defaults are inappropriate). The biggest worry is
probably that someone will write an add-on library that actually uses
the API, eg some image or font handling library that tries to find data
using the API instead of requiring the locations be passed explicitly
(Principle of Explicit Coupling, again Bertrand Meyer, OOSC).On 24/08/2013, at 2:46 AM, Sik the hedgehog wrote:

–
john skaller
@john_skaller
http://felix-lang.org

38

Er … so what does it return if you’re running SDL from a Python script?

Wherever the Python binary is running from.

What does it do if SDL is in a shared library which is loaded at run time?
Which is the case for all my stuff by default.

Wherever the thing with main() is running from.

–ryan.

39

Wherever the thing with main() is running from.

(or WinMain() or whatever.)

–ryan.

40

2013/8/23, john skaller :

What’s it got to do with bindings? The SDL_* filesystem API is
part of the library, a binding just translates from C to Python.

The problem is that from SDL’s viewpoint it’s always a standard C
program. There’s nothing it can do about it, and will happen with
every binding out there, not just SDL. So unless Python does some
hacking to trick bindings into using the correct name, forget it.

Of course, being a binding means it may as well just rewrite these few
functions in Python instead of calling SDL. Redundant, but it’d work.

Not with my system, since the default operation uses a single
executable to run all programs code as DLLs. The location of
that executable is irrelevant. This is similar to Python having
a single executable. The location of the executable is irrelevant
since the content is in the loaded DLL.

Then you’re messing with the whole concept about how programs work and
there’s nothing SDL can do about it.

So the file system API isn’t going to work unless I can intervene and
program it (if the defaults are inappropriate). The biggest worry is
probably that someone will write an add-on library that actually uses
the API, eg some image or font handling library that tries to find data
using the API instead of requiring the locations be passed explicitly

So basically you want SDL to not have a filesystem API at all, period.

(Principle of Explicit Coupling, again Bertrand Meyer, OOSC).

Funny thing is, hardcoding filename structures like that is a bad idea
from a design standpoint :stuck_out_tongue: