Android SDL_RWops safety - how should it be fixed?

SDL Development
1

Sam has asked that some Android devs who have an idea of JNI (which doesn’t include myself, sadly) might have a look at how to solve http://bugzilla.libsdl.org/show_bug.cgi?id=1422 sanely.

PROBLEM:

The problem is that SDL_RWops on Android cannot be used from any thread other than the main thread, so it is e.g. impossible to use them from the sound thread for streaming sounds from the assets, as an example (since I think SDL_RWops is the only way to access the assets on Android without extracting them to the sdcard first or some other ugly workaround).

For my personal situation (I am trying to do streamed sound on Android from the assets), required would be:

  • SDL_RWops can be created from any thread
  • Once created, an SDL_RWop object can be used from any thread (not just the one that created it), but calling it at the same time from multiple threads is not necessarily safe

Nice to have would be probably to avoid crashes and other nasty things even if read is done on an SDL_RWops even at the same time from multiple threads.

SOLUTION?

Now, there seems to be a patch that solves some or all of this: http://bugzilla.libsdl.org/show_bug.cgi?id=1159

However, the patch doesn’t cleanly apply (and I failed with my limited JNI knowledge to make it work - while I managed to make it compile in a way that seemed to be more or less senseful to me, it simply didn’t work at runtime). Also, according to Sam, the patch adds a large performance penalty.

So a nice solution that can go into mainline without performance penalty for those who don’t use threading would be good. But personally, I’d also love a compile-time switch enabled, slow performance version that can be enabled by people like me who need it - which that patch would be essentially, in case it worked.

I tried to work around this limitation purely application-side with lots of mutexes and locking and redirecting file access to the main thread, but it doesn’t work too well (very, very slow and delayed). Something inside SDL that makes threading SDL_RWops work sanely on Android would be much better.

Is there someone with JNI knowledge floating around who might want to share a few ideas or suggestions?------------------------
-Jonas Thiem
My Blog: http://gamedev-couch.blogspot.com/

2

Well, if this was Delphi, I’d do it with a call to Synchronize.? That pushes an event containing a function pointer onto the event queue for the main Application object’s poll loop (which runs on the main thread) to pick up? and execute the associated function, and the Synchronize call blocks until the main thread is finished with it.? Not sure if JNI or the Android NDK has something similar built in, but it wouldn’t surprise me.

There are two tricky aspects here, though.? The first is that trying something like Synchronize when you’re already on the main thread can deadlock your program or cause other nasty problems, so your RW routines would need to check for that.? The second is that the Synchronize process has a fair amount of overhead to it, which means you want to do it as little as possible to avoid noticeable slowdowns.? So if you’re just reading from your RWops a few bytes at a time, you should rewrite the code so instead you blockread the whole thing into memory at once.________________________________
From: MrJones
To: sdl at lists.libsdl.org
Sent: Sunday, April 8, 2012 8:52 AM
Subject: [SDL] Android SDL_RWops safety - how should it be fixed?

Sam has asked that some Android devs who have an idea of JNI (which doesn’t include myself, sadly) might have a look at how to solve http://bugzilla.libsdl.org/show_bug.cgi?id=1422 sanely.

PROBLEM:

The problem is that SDL_RWops on Android cannot be used from any thread other than the main thread, so it is e.g. impossible to use them from the sound thread for streaming sounds from the assets, as an example (since I think SDL_RWops is the only way to access the assets on Android without extracting them to the sdcard first or some other ugly workaround).

For my personal situation (I am trying to do streamed sound on Android from the assets), required would be:

  • SDL_RWops can be created from any thread
  • Once created, an SDL_RWop object can be used from any thread (not just the one that created it), but calling it at the same time from multiple threads is not necessarily safe

Nice to have would be probably to avoid crashes and other nasty things even if read is done on an SDL_RWops even at the same time from multiple threads.

SOLUTION?

Now, there seems to be a patch that solves some or all of this: http://bugzilla.libsdl.org/show_bug.cgi?id=1159

However, the patch doesn’t cleanly apply (and I failed with my limited JNI knowledge to make it work - while I managed to make it compile in a way that seemed to be more or less senseful to me, it simply didn’t work at runtime). Also, according to Sam, the patch adds a large performance penalty.

So a nice solution that can go into mainline without performance penalty for those who don’t use threading would be good. But personally, I’d also love a compile-time switch enabled, slow performance version that can be enabled by people like me who need it - which that patch would be essentially, in case it worked.

I tried to work around this limitation purely application-side with lots of mutexes and locking and redirecting file access to the main thread, but it doesn’t work too well (very, very slow and delayed). Something inside SDL that makes threading SDL_RWops work sanely on Android would be much better.

Is there someone with JNI knowledge floating around who might want to share a few ideas or suggestions?

-Jonas Thiem
My Blog: http://gamedev-couch.blogspot.com/

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

3

Hi, please don’t forget about Android ICS jni compatibility.

Current version is crashing. I already wrote about it but nobody answeredFrom: sdl-bounces@lists.libsdl.org [mailto:sdl-bounces at lists.libsdl.org] On
Behalf Of MrJones
Sent: Sunday, April 08, 2012 9:53 PM
To: sdl at lists.libsdl.org
Subject: [SDL] Android SDL_RWops safety - how should it be fixed?

Sam has asked that some Android devs who have an idea of JNI (which doesn’t
include myself, sadly) might have a look at how to solve
http://bugzilla.libsdl.org/show_bug.cgi?id=1422 sanely.

PROBLEM:

The problem is that SDL_RWops on Android cannot be used from any thread
other than the main thread, so it is e.g. impossible to use them from the
sound thread for streaming sounds from the assets, as an example (since I
think SDL_RWops is the only way to access the assets on Android without
extracting them to the sdcard first or some other ugly workaround).

For my personal situation (I am trying to do streamed sound on Android from
the assets), required would be:

  • SDL_RWops can be created from any thread
  • Once created, an SDL_RWop object can be used from any thread (not just the
    one that created it), but calling it at the same time from multiple threads
    is not necessarily safe

Nice to have would be probably to avoid crashes and other nasty things even
if read is done on an SDL_RWops even at the same time from multiple threads.

SOLUTION?

Now, there seems to be a patch that solves some or all of this:
http://bugzilla.libsdl.org/show_bug.cgi?id=1159

However, the patch doesn’t cleanly apply (and I failed with my limited JNI
knowledge to make it work - while I managed to make it compile in a way that
seemed to be more or less senseful to me, it simply didn’t work at runtime).
Also, according to Sam, the patch adds a large performance penalty.

So a nice solution that can go into mainline without performance penalty for
those who don’t use threading would be good. But personally, I’d also love a
compile-time switch enabled, slow performance version that can be enabled by
people like me who need it - which that patch would be essentially, in case
it worked.

I tried to work around this limitation purely application-side with lots of
mutexes and locking and redirecting file access to the main thread, but it
doesn’t work too well (very, very slow and delayed). Something inside SDL
that makes threading SDL_RWops work sanely on Android would be much better.

Is there someone with JNI knowledge floating around who might want to share
a few ideas or suggestions?

-Jonas Thiem
My Blog: http://gamedev-couch.blogspot.com/

4

Mason Wheeler wrote:

Well, if this was Delphi, I’d do it with a call to Synchronize. That pushes an event containing a function pointer onto the event queue for the main Application object’s poll loop (which runs on the main thread) to pick up and execute the associated function, and the Synchronize call blocks until the main thread is finished with it. Not sure if JNI or the Android NDK has something similar built in, but it wouldn’t surprise me.

That was what I am basically doing for application-side fix. Believe me: it is the biggest. hack. ever. Right now I am using mutexes which is probably the wrong tool for the job and making performance a bit worse even, but it doesn’t really matter since I think it’s the wrong point to fix it anyway - it will never work well that way. Mainly since, obviously, the main thread might have better things to do than read stuff for the sound thread.

Mason Wheeler wrote:

There are two tricky aspects here, though. The first is that trying something like Synchronize when you’re already on the main thread can deadlock your program

Oh yes. And finding out and tracing through all the code if you are on the main thread was a HUGE effort and I spend lots of wasted code lines on this.

There should be a proper solution, so a hack as you suggest (and which I implemented in a pretty much similar way I suppose) isn’t needed.------------------------
-Jonas Thiem
My Blog: http://gamedev-couch.blogspot.com/

5

Well, the “proper solution” is to fix Android so it isn’t limited to only doing I/O on the main thread.? That’s where the real problem lies.? And if you can’t do that, I don’t know if there is a better workaround.? Maybe there’s room for improvement in your specific implementation of it, but conceptually I don’t think there’s a better way.________________________________
From: MrJones
To: sdl at lists.libsdl.org
Sent: Sunday, April 8, 2012 1:52 PM
Subject: Re: [SDL] Android SDL_RWops safety - how should it be fixed?

Mason Wheeler wrote:
Well, if this was Delphi, I’d do it with a call to Synchronize. That pushes an event containing a function pointer onto the event queue for the main Application object’s poll loop (which runs on the main thread) to pick up and execute the associated function, and the Synchronize call blocks until the main thread is finished with it. Not sure if JNI or the Android NDK has something similar built in, but it wouldn’t surprise me.
That was what I am basically doing for application-side fix. Believe me: it is the biggest. hack. ever. Right now I am using mutexes which is probably the wrong tool for the job and making performance a bit worse even, but it doesn’t really matter since I think it’s the wrong point to fix it anyway - it will never work well that way. Mainly since, obviously, the main thread might have better things to do than read stuff for the sound thread.

Mason Wheeler wrote:
There are two tricky aspects here, though. The first is that trying something like Synchronize when you’re already on the main thread can deadlock your program
Oh yes. And finding out and tracing through all the code if you are on the main thread was a HUGE effort and I spend lots of wasted code lines on this.

There should be a proper solution, so a hack as you suggest (and which I implemented in a pretty much similar way I suppose) isn’t needed.

-Jonas Thiem
My Blog: http://gamedev-couch.blogspot.com/

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org