Thoughts on the new filesystem API

[slouken]

Yes, it sounds like for an interpreted language, you want the language
binding to provide this functionality, eclipsing the SDL binding for that
API. That way the language binding can do smart things about where the
modules are located and where the runtime path is.On Fri, Aug 23, 2013 at 10:23 AM, Sik the hedgehog < sik.the.hedgehog at gmail.com> wrote:

2013/8/23, john skaller :

What’s it got to do with bindings? The SDL_* filesystem API is
part of the library, a binding just translates from C to Python.

The problem is that from SDL’s viewpoint it’s always a standard C
program. There’s nothing it can do about it, and will happen with
every binding out there, not just SDL. So unless Python does some
hacking to trick bindings into using the correct name, forget it.

Of course, being a binding means it may as well just rewrite these few
functions in Python instead of calling SDL. Redundant, but it’d work.

Not with my system, since the default operation uses a single
executable to run all programs code as DLLs. The location of
that executable is irrelevant. This is similar to Python having
a single executable. The location of the executable is irrelevant
since the content is in the loaded DLL.

Then you’re messing with the whole concept about how programs work and
there’s nothing SDL can do about it.

So the file system API isn’t going to work unless I can intervene and
program it (if the defaults are inappropriate). The biggest worry is
probably that someone will write an add-on library that actually uses
the API, eg some image or font handling library that tries to find data
using the API instead of requiring the locations be passed explicitly

So basically you want SDL to not have a filesystem API at all, period.

(Principle of Explicit Coupling, again Bertrand Meyer, OOSC).

Funny thing is, hardcoding filename structures like that is a bad idea
from a design standpoint

---

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

[john_skaller]

Er … so what does it return if you’re running SDL from a Python script?

Wherever the Python binary is running from.

Right, which is totally … er … useless

What does it do if SDL is in a shared library which is loaded at run time?
Which is the case for all my stuff by default.

Wherever the thing with main() is running from.

How does it do that? Just curious. I have no idea how that
could be organised. Is there a Posix API call to find where
the main program is running from? Or are you just checking
argv0 somehow?On 24/08/2013, at 3:18 AM, Ryan C. Gordon wrote:

–
john skaller
@john_skaller
http://felix-lang.org

[john_skaller]

2013/8/23, john skaller <@john_skaller>:

What’s it got to do with bindings? The SDL_* filesystem API is
part of the library, a binding just translates from C to Python.

The problem is that from SDL’s viewpoint it’s always a standard C
program. There’s nothing it can do about it, and will happen with
every binding out there, not just SDL. So unless Python does some
hacking to trick bindings into using the correct name, forget it.

Of course, being a binding means it may as well just rewrite these few
functions in Python instead of calling SDL. Redundant, but it’d work.

That depends what you mean by “work”. If your Python has its own
version of the call that tells where that .py file is, but you also have
a CPython module that calls SDL_ the two will return different
values.

The best you could do is add a new function to the Python
binding that “does the right thing” … which is what you said.

Not with my system, since the default operation uses a single
executable to run all programs code as DLLs. The location of
that executable is irrelevant. This is similar to Python having
a single executable. The location of the executable is irrelevant
since the content is in the loaded DLL.

Then you’re messing with the whole concept about how programs work and
there’s nothing SDL can do about it.

Dynamic linkage “messes” with the whole concept of how
programs work, I’m just leveraging it. Python, Java, etc also
"mess" with this concept, having bytecode compilers and
interpreters and even JiTs. Messing about with stuff is what
programmers do.

My point is that there is something SDL can do about it:
provide a way to set the values returned by the API.On 24/08/2013, at 3:23 AM, Sik the hedgehog wrote:

–
john skaller
@john_skaller
http://felix-lang.org

[slime]

How does it do that? Just curious. I have no idea how that
could be organised. Is there a Posix API call to find where
the main program is running from? Or are you just checking
argv0 somehow?

SDL is open source, you know.
http://hg.libsdl.org/SDL/file/tip/src/filesystem

[icculus]

Wherever the Python binary is running from.

Right, which is totally … er … useless

Presumably the binding between SDL and a scripting language interpreter
can provide insight that SDL couldn’t about what script it is running.

Wherever the thing with main() is running from.

How does it do that? Just curious. I have no idea how that
could be organised. Is there a Posix API call to find where
the main program is running from? Or are you just checking
argv0 somehow?

Platform-specific magic. Details are here:

 http://hg.libsdl.org/SDL/file/cf99258f905c/src/filesystem

–ryan.

[Gabriel_Jacobo]

2013/8/23 john skaller

Dynamic linkage “messes” with the whole concept of how
programs work, I’m just leveraging it. Python, Java, etc also
"mess" with this concept, having bytecode compilers and
interpreters and even JiTs. Messing about with stuff is what
programmers do.

My point is that there is something SDL can do about it:
provide a way to set the values returned by the API.

–
john skaller
skaller at users.sourceforge.net
http://felix-lang.org

I’m sure that if you provide a patch for this and push it with the same
enthusiasm as you’ve been discussing this thread it will land in no time :)–
Gabriel.

[Sik_the_hedgehog]

2013/8/23, john skaller :

Dynamic linkage “messes” with the whole concept of how
programs work, I’m just leveraging it. Python, Java, etc also
"mess" with this concept, having bytecode compilers and
interpreters and even JiTs. Messing about with stuff is what
programmers do.

May as well say that programs in general mess with the whole concept.

What I meant is that even when you load a DLL, the process properties
are still based around the program that was originally booted. When
you move the program into one of those DLLs instead you break this
assumption, effectively rendering all those properties useless, as
they refer to whatever loaded the DLL instead.

Granted, forking in itself already muddles the situation…

My point is that there is something SDL can do about it:
provide a way to set the values returned by the API.

And how do you determine the values you’d use to set them? Oh right -
platform-specific code, i.e. the very thing you’re trying to avoid.
So you effectively rendered the whole purpose of the API completely
useless, no matter how it’s implemented. The whole point of the API is
that SDL figures it all, if you do it on your own then you don’t need
SDL for it.

[Jared_Maddox]

Message-ID: <521592F9.3020407 at icculus.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

four separate directories:

1. Installation directory
2. Application Data directory
3. User Data directory
4. Application Invocation directory

Let’s not over-complicate this. In practice, it’s only two things
(“where do I read?” and “where do I write?”).

Where I work, we have multi-user computers that run time card
software. For obvious reasons we don’t want people to have the ability
to modify any of the associated files, thus the program needs to be
available from a user account that doesn’t have the permissions to
modify it. However, what if it downloads an update (I don’t think it
has this ability, mind you, this is just a very easy leap to make)?
You’ll want to be able to apply the update from any user account
(because you don’t know which will be used when), but you still want
tour program directory protected, AND you want to store any
unimportant settings wherever is appropriate. Thus, three directories,
all of which may be in a different location, depending on the OS:

1. Installation directory (shared execute/read; on Windows, Program Files),
2. Application Data directory (shared read/write),
3. User Data directory (user read/write; the home directory, My Documents).

The invocation directory was mentioned for completeness, other than an
image viewer I can’t really think of why you would care about this
from SDL. The other three, in comparison, matter for any multi-user
environment.

Another use-case:
Your game has player accounts. You want any player account to be
usable from any OS account, but you also want OS accounts to have a
player account that will load automatically. Solution? Store the
account data in the shared directory, and an “auto-load” file in the
user directory. If the user wants to use a different account, they can
back out of the one they’re currently in.

char const *SDL_get_location (char const *abstract_location_name);
I, for one, like this idea, but I have a question:

The goal isn’t to build an abstract means to configure the way the
system thinks about file layout, the goal is to understand where files
go, as dictated by the standards for that platform, using
platform-specific APIs behind the scenes that obtain this information so
the app doesn’t have to.

Point. And if someone really cares, they can write it themselves.
However, none of this diminishes the value of directories 1-3.> Date: Thu, 22 Aug 2013 00:26:33 -0400

From: “Ryan C. Gordon”
To: SDL Development List
Subject: Re: [SDL] Thoughts on the new filesystem API

From my perspective, the above three snippets actually describe up to

Date: Thu, 22 Aug 2013 02:08:28 -0300
From: Sik the hedgehog <sik.the.hedgehog at gmail.com>
To: SDL Development List
Subject: Re: [SDL] Thoughts on the new filesystem API
Message-ID:
<CAEyBR+VXHKYCq-xCtNWO_XKW1KpBHcHh+wOegwi4HqTtd2Lb1g at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

2013/8/22, Jared Maddox <@Jared_Maddox>:

4. Application Invocation directory

Isn’t that just the current directory?

Yes. I considered actually wording it like that, but I decided clarity
was of more value than succinctness.

Date: Thu, 22 Aug 2013 02:10:01 -0300
From: Sik the hedgehog <sik.the.hedgehog at gmail.com>
To: SDL Development List
Subject: Re: [SDL] Thoughts on the new filesystem API
Message-ID:
<CAEyBR+VxckwASX-nEjmVHFGdxAibnohFYvwKVqeuMqQxFKVA1A at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Whoops, sorry for the empty e-mail (clicked the wrong thing). But
yeah, now I see where the list comes from, you need separate
directories for global settings and user-specific settings. So yeah,
that indeed makes it three directories (the fourth item is still
pointless though, you don’t care from where your program runs after
all).

Oh, I agree, but as I best recall I actually got than from one of
Sam’s descriptions: I honest-to-goodness read four different
descriptions between those three posts.

Date: Thu, 22 Aug 2013 16:09:09 -0400
From: Edward Rudd
To: SDL Development List
Subject: Re: [SDL] Thoughts on the new filesystem API
Message-ID:
Content-Type: text/plain; charset=“us-ascii”

Now, one “nicety” of the way the SDL hint system works… Is I do not have to
set my hints via the SDL_SetHint… I can set them via the env…

So I can add this to my Info.plist

LSEnvironment

SDL_FILESYSTEM_OSX_USE_RESOURCE_DIR
1

Certainly an upshot of shoehorning any possible configurability into
the hint system. It would (just as an example) allow developers to use
different invocation scripts to test version compatibility: for each
new “external” version make a new directory, and a new invocation
script that redirects the filesystem API to that directory. Testing an
executable for compatibility with multiple data & plugin versions
becomes as simple as going down a list of scripts.

[slime]

Is #2 even possible on all platforms? Where would that be located?On 2013-08-24, at 12:44 AM, Jared Maddox wrote:

Where I work, we have multi-user computers that run time card
software. For obvious reasons we don’t want people to have the ability
to modify any of the associated files, thus the program needs to be
available from a user account that doesn’t have the permissions to
modify it. However, what if it downloads an update (I don’t think it
has this ability, mind you, this is just a very easy leap to make)?
You’ll want to be able to apply the update from any user account
(because you don’t know which will be used when), but you still want
tour program directory protected, AND you want to store any
unimportant settings wherever is appropriate. Thus, three directories,
all of which may be in a different location, depending on the OS:

1. Installation directory (shared execute/read; on Windows, Program Files),
2. Application Data directory (shared read/write),
3. User Data directory (user read/write; the home directory, My Documents).

[Drake_Wilson]

Quoth Jared Maddox , on 2013-08-23 22:44:36 -0500:

You’ll want to be able to apply the update from any user account
(because you don’t know which will be used when),

This is straying off-topic somewhat, but doing this securely with
only filesystem-level setup ranges from finicky and easy to get wrong
to impossible, in the presence of multiple users that should have any
isolation. (If not, why not just put updates in the original
installation directory?)

Windows, IIRC, solves this by providing a way for installation of
add-on or update signed MSI packages with a known-related key to be
triggered at the correct privilege level from any account, while
denying normal writes. Some applications whose updates can’t be fit
into this model just make updates per-user, or even default to
installation into user directories for that reason. Legacy
compatibility can also cause writes back to the program directory to
be redirected to a per-user directory by Win32, that being a practice
which was common in the pre-NT consumer Windows era where filesystem
permissions didn’t exist.

Your game has player accounts. You want any player account to be
usable from any OS account, but you also want OS accounts to have a
player account that will load automatically. Solution?

This is also hazardous: little slip-ups in the reading code (easier to
do in the case of games where files are not commonly exchanged and
performance can be critical), or not locking a file and assuming it’s
the same as it was a moment ago, or such, can open up holes between OS
accounts if one isn’t careful. This is not an invitation for someone
to start a subthread about language memory safety.

Some Unixy systems have games installed setgid to the ‘games’ group in
order to allow systemwide high score files without letting users too
easily trample all over them. This requires the installer and game
code to know about this and juggle GIDs and filesystem bits to work
without breaking other interprocess interactions or trampling group
ownerships of actual user-specific files.

Naturally, one then gets things like
http://www.gentoo.org/security/en/glsa/glsa-200603-23.xml and
https://forums.gentoo.org/viewtopic-t-446415.html (Gentoo’s divergence
regarding the meaning of the ‘games’ group partially aside—the main
pattern I’ve seen on something like Windows is similar to the Gentoo
case).

Conclusion: while “read system data” and “read/write user data” are
mostly portable concepts, “read/write shared data” is very problematic
and requires more integration hoops than a “show me directories” API
can reasonably hope to provide. Making it look too convenient might
well lead developers into traps.

—> Drake Wilson

[Sik_the_hedgehog]

On systems without concept of multiuser, #2 and #3 would point to the
same place. On systems without any concept of proper organization at
all, #1 would also point to the same place ^^;

2013/8/24, Alex Szpakowski :> Is #2 even possible on all platforms? Where would that be located?

On 2013-08-24, at 12:44 AM, Jared Maddox wrote:

Where I work, we have multi-user computers that run time card
software. For obvious reasons we don’t want people to have the ability
to modify any of the associated files, thus the program needs to be
available from a user account that doesn’t have the permissions to
modify it. However, what if it downloads an update (I don’t think it
has this ability, mind you, this is just a very easy leap to make)?
You’ll want to be able to apply the update from any user account
(because you don’t know which will be used when), but you still want
tour program directory protected, AND you want to store any
unimportant settings wherever is appropriate. Thus, three directories,
all of which may be in a different location, depending on the OS:

1. Installation directory (shared execute/read; on Windows, Program
   Files),
2. Application Data directory (shared read/write),
3. User Data directory (user read/write; the home directory, My
   Documents).

---

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

[Rainer_Deyke]

On 24.08.2013 07:20, Drake Wilson wrote:> This is also hazardous: little
slip-ups in the reading code (easier to

do in the case of games where files are not commonly exchanged and
performance can be critical), or not locking a file and assuming it’s
the same as it was a moment ago, or such, can open up holes between OS
accounts if one isn’t careful. This is not an invitation for someone
to start a subthread about language memory safety.

This is not such a difficult problem.

First possible solution: write to a new file, perform an atomic file
rename to replace the old file. This also protects against corrupted
files if the power is cycled while writing to the file, and is therefore
good practice even when writing user-specific files. In the worst case,
you’ll lose writes, which is usually not that big of a deal.

Second possible solution: use SQLite, which provides ACID guarantees.
This also protects against corrupted files if the power is cycled while
writing to the file, and is therefore also good practice even when
writing user-specific files.

Third possible solution: do nothing, and assume that only one user will
run the game concurrently. This is usually a pretty safe assumption,
since most games are not played (or even playable) over a network
connection. I suspect that it’s more likely that one user runs two
concurrent instances of the same game than that two different users run
two concurrent instances of the same game.–
Rainer Deyke (rainerd at eldwood.com)

[marcusva]

On, Wed Aug 21, 2013, Ryan C. Gordon wrote:

Doesn’t that mean the code is not fully portable anymore?

SDL can’t psychically know where your app chooses to store its data.

On the other hand, if the user runs the program from the symlink,
chances are it may also expect the program to behave like it was
running from there… Not sure how this pans out in practice.

There’s a long history Linux games, back to Loki at least, installing a
symlink to some place in your $PATH, so you can launch the game from the
command line without having to know the full location or add every game
to your $PATH.

There is a long history in commercial Linux games - not to be confused
with the even greater amount of open source games, which rely on sane
build tools to take the burden of defining a base path (which in that
case is almost always different from the binary path) and all that
nilly-willy from the shoulders of some framework or library. There is a
reason, why nearly all build tools for Win32 and Unix-like platforms
feature synonyms for prefix, datadir, bindir and so on.

In my opinion, it should be up to the build tools, but that’s a
different discussion.

Regarding the API itself, I find the “SDL_GetBasePath” quite confusing,
since it is not the base path of the application, but the application
binary path - which can be different. If we’re sticking to Steam and
Loki games, it probably won’t, if we think about all the other
applications and games, it most certainly will. Thus, if the API should
stay as is, I’d really love to the SDL_GetBasePath to be renamed to what
it actually delivers: SDL_GetBinaryPath().

Cheers
Marcus
-------------- next part --------------
A non-text attachment was scrubbed…
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: http://lists.libsdl.org/pipermail/sdl-libsdl.org/attachments/20130824/b7f2c57b/attachment.pgp

[Edward_Rudd]

is what people would expect. and would keep in line with the way the
rest of SDL is written. (keep the #ifdefs away !!)

Let me think on this a little more. I’m sort of coming around on the Contents/Resources thing. I could see a world where Apple starts rejecting apps from their Store because they don’t have data files there, because that sounds like the sort of thing they’d do.

And usually it’s windows that is the “oddball” operating system that causes us grief in our designs.

We could just have a custom SDL Info.plist setting for “bundled” mac apps to alter the behavior (for the oddball cases)… As the general case should be Contents/Resources. That way it’s a data configuration options not a code setting. Since this whole mess is OS X specific it fits as an OSX/IOS"y" way to set it up.

Edward Rudd
OutOfOrder.cc
Skype: outoforder_cc
317-674-3296On Aug 23, 2013, at 1:29 , Ryan C. Gordon wrote:

[Drake_Wilson]

Quoth Rainer Deyke , on 2013-08-24 10:47:02 +0200:> On 24.08.2013 07:20, Drake Wilson wrote:

the same as it was a moment ago, or such, can open up holes between OS
accounts if one isn’t careful. This is not an invitation for someone
to start a subthread about language memory safety.

This is not such a difficult problem.

Sigh. Okay, I should have said that differently.

It is solvable (not that I think your description really addresses it)
if you already recognize the need for being careful and are willing
to pay the cost. That, not the technical problem, is the first weak
link. Removing the natural PricklyHedge makes it worse. And it won’t
be effective without more platform-specific juggling anyway. You’re
not going to inject that setresgid() on Unixy games-group systems via
path APIs, or make the initial setup work, or anything like that.

If people really want this, I’d say spec it out properly, give it a
nice case and some blinkenlights, and make it a separate library, at
least. Don’t put it in SDL proper as a half-working attractive trap.

Now I’m done with that topic.

—> Drake Wilson

[Sik_the_hedgehog]

2013/8/24, Rainer Deyke :

First possible solution: write to a new file, perform an atomic file
rename to replace the old file. This also protects against corrupted
files if the power is cycled while writing to the file, and is therefore
good practice even when writing user-specific files. In the worst case,
you’ll lose writes, which is usually not that big of a deal.

Honestly, I don’t get why operating systems haven’t tried to make this
the default way of writing files and requiring programs to go out of
their way if they want to overwrite directly.

2013/8/24, Marcus von Appen :

I’d really love to the SDL_GetBasePath to be renamed to what
it actually delivers: SDL_GetBinaryPath().

Personally I think SDL_GetExecutablePath() is clearer… although
maybe it should be SDL_GetExecutableDirPath(), since it returns the
directory, not the filename of the executable (really, “path” should
be “directory”).