Mix_PlayChannel Android lag

Hi!

I’m using SDL_Mixer (the newest available on Mercurial). On my PC the sound is okay, but when running the game on Android, the sound lags around half a second. Currently I’m using the following code:

Code:
Mix_OpenAudio(44100, AUDIO_S16SYS, 2, 512)

for initializing and

Code:
Mix_PlayChannel(-1, m_Sfx[0], 0)

for playing a small sound. I’ve tried setting different values for the initialization with frequency 22050, using 1 channel and varying the samples from 512 to 4096, but the result hasn’t changed much. I’m using mono sounds.

Any tips would be appreciated.

Thank you!------------------------
Runic Girls - a hexa match3 game with girls created using SDL 2.0:
http://www.facebook.com/RunicGirls

Gameplay trailer:

[…]

I’m using SDL_Mixer (the newest available on Mercurial). On my PC the sound is okay, but when running the game on Android, the sound lags around half a second.
[…]

Android is notorious for awful audio latency… Haven’t investigated
it deeply, but by the looks of it, the API is fundamentally broken by
design. (It doesn’t technically need to be callback based - though all
serious realtime audio APIs are - but it simply has to play by the
same rules as a callback based API, to avoid intermediate buffering
behind the scenes.)

You might want to try some simple test using the SDL audio API
directly, but I wouldn’t think SDL_Mixer is part of this problem. Only
the sample rate and buffer size should matter. (Higher sample rates
and smaller buffers decrease latency.)

Are you getting better latencies with apps that use the Android audio
API directly? (Possible if they use other APIs, as those might bypass
the problematic API that SDL is using. I suspect that would be of no
use to SDL, as the APIs I’m aware of are not streaming APIs - but if
you really have to, you might be able to use that instead of
SDL_Mixer…)On Thu, Apr 3, 2014 at 6:52 AM, LBandy wrote:


//David Olofson - Consultant, Developer, Artist, Open Source Advocate

.— Games, examples, libraries, scripting, sound, music, graphics —.
| http://consulting.olofson.net http://olofsonarcade.com |
’---------------------------------------------------------------------’

I agree with David that a simple test should be done. But David is
also correct that Android audio is famously terrible. I usually have a
list of links I refer people to, but I misplaced it. I have been
fighting Android audio for years now and I always get blamed, but it’s
completely Android’s fault.

Yes, their APIs are terrible. They have like 4. SoundPool, AudioTrack,
MediaPlayer, and OpenSL ES. The first 3 are Java, the last one is C.
SoundPool is a massive hack that historically had significantly lower
latency, but the API is so nerfed, it wasn’t really usable for any
real world cases. But this is why people always blame me instead of
Android…because you can make a toy app that doesn’t exhibit the
problem. (And then in 2.3.x, a very popular Android device had a bug
that crashed apps that used SoundPool. And now I’ve seen a few cases
that have high latency with SoundPool.)

OpenSL ES is the only C API, and really is the future of Android
audio. But it’s still not there yet. I’m not willing to claim AP is
the problem though. The real problem at least right now is much
deeper. The problems go as deep as the kernel and the scheduler. There
was actually a very good talk given by their audio guy at Google I/O.
He went pretty hard core and built his own measurement equipment
(using photodiode and oscilloscope) to measure latency. They did make
some good progress in 4.1. However, they still have long ways to go.

P.S. I have non-latency gripes about their OpenSL ES API extensions
for loading/decoding. You must read from a file or URI. They don’t
have a read from memory system or anything that is fopen/fread/fseek
signature compatible, so it is impossible to interface with SDL_RWops.

-EricOn 4/3/14, David Olofson wrote:

On Thu, Apr 3, 2014 at 6:52 AM, LBandy wrote:
[…]

I’m using SDL_Mixer (the newest available on Mercurial). On my PC the
sound is okay, but when running the game on Android, the sound lags around
half a second.
[…]

Android is notorious for awful audio latency… Haven’t investigated
it deeply, but by the looks of it, the API is fundamentally broken by
design. (It doesn’t technically need to be callback based - though all
serious realtime audio APIs are - but it simply has to play by the
same rules as a callback based API, to avoid intermediate buffering
behind the scenes.)

You might want to try some simple test using the SDL audio API
directly, but I wouldn’t think SDL_Mixer is part of this problem. Only
the sample rate and buffer size should matter. (Higher sample rates
and smaller buffers decrease latency.)

Are you getting better latencies with apps that use the Android audio
API directly? (Possible if they use other APIs, as those might bypass
the problematic API that SDL is using. I suspect that would be of no
use to SDL, as the APIs I’m aware of are not streaming APIs - but if
you really have to, you might be able to use that instead of
SDL_Mixer…)


//David Olofson - Consultant, Developer, Artist, Open Source Advocate


Beginning iPhone Games Development
http://playcontrol.net/iphonegamebook/

2014-04-03 13:28 GMT+02:00 Eric Wing :> On 4/3/14, David Olofson wrote:

On Thu, Apr 3, 2014 at 6:52 AM, LBandy wrote:
[…]

I’m using SDL_Mixer (the newest available on Mercurial). On my PC the
sound is okay, but when running the game on Android, the sound lags
around

half a second.
[…]

Android is notorious for awful audio latency… Haven’t investigated
it deeply, but by the looks of it, the API is fundamentally broken by
design. (It doesn’t technically need to be callback based - though all
serious realtime audio APIs are - but it simply has to play by the
same rules as a callback based API, to avoid intermediate buffering
behind the scenes.)

You might want to try some simple test using the SDL audio API
directly, but I wouldn’t think SDL_Mixer is part of this problem. Only
the sample rate and buffer size should matter. (Higher sample rates
and smaller buffers decrease latency.)

Are you getting better latencies with apps that use the Android audio
API directly? (Possible if they use other APIs, as those might bypass
the problematic API that SDL is using. I suspect that would be of no
use to SDL, as the APIs I’m aware of are not streaming APIs - but if
you really have to, you might be able to use that instead of
SDL_Mixer…)


//David Olofson - Consultant, Developer, Artist, Open Source Advocate

I agree with David that a simple test should be done. But David is
also correct that Android audio is famously terrible. I usually have a
list of links I refer people to, but I misplaced it. I have been
fighting Android audio for years now and I always get blamed, but it’s
completely Android’s fault.

Yes, their APIs are terrible. They have like 4. SoundPool, AudioTrack,
MediaPlayer, and OpenSL ES. The first 3 are Java, the last one is C.
SoundPool is a massive hack that historically had significantly lower
latency, but the API is so nerfed, it wasn’t really usable for any
real world cases. But this is why people always blame me instead of
Android…because you can make a toy app that doesn’t exhibit the
problem. (And then in 2.3.x, a very popular Android device had a bug
that crashed apps that used SoundPool. And now I’ve seen a few cases
that have high latency with SoundPool.)

OpenSL ES is the only C API, and really is the future of Android
audio. But it’s still not there yet. I’m not willing to claim AP is
the problem though. The real problem at least right now is much
deeper. The problems go as deep as the kernel and the scheduler. There
was actually a very good talk given by their audio guy at Google I/O.
He went pretty hard core and built his own measurement equipment
(using photodiode and oscilloscope) to measure latency. They did make
some good progress in 4.1. However, they still have long ways to go.

P.S. I have non-latency gripes about their OpenSL ES API extensions
for loading/decoding. You must read from a file or URI. They don’t
have a read from memory system or anything that is fopen/fread/fseek
signature compatible, so it is impossible to interface with SDL_RWops.

This comes to mind: https://www.youtube.com/watch?v=d3kfEeMZ65c
not sure how helpful it is to SDL though…

Thanks for the input guys!
I’m involved in various projects using different sound playback libraries.
In another game where I’m using fmod, it gives a much better result, as the lag is nearly unnoticable. So I’m just curious, if there is a known method using the SDL library to play sounds in a way that could be acceptable, or should I use external libraries for this problem? Should I try SDL_Audio and expect some progress?------------------------
Runic Girls - a hexa match3 game with girls created using SDL 2.0:
http://www.facebook.com/RunicGirls

Gameplay trailer:

Unless there’s a bug in SDL_Sound, I don’t think bypassing it is going
to help. It’s certainly possible there’s a problem, but it doesn’t
seem likely to me, based on what I know about how it’s implemented.
(No time stamps or “sub-buffer” timing logic that can break, AFAIK.)

Unfortunately, I suspect bypassing the SDL audio API isn’t going to
help either, as the problem is in Android itself.

As I suspected, it might be possible to get lower latency for basic
stuff with another Android API - but as Eric pointed out, that instead
has even more serious issues on certain Android versions.

So, in short, do try other options, but don’t hope for miracles… :-/On Thu, Apr 3, 2014 at 10:01 PM, LBandy wrote:

Thanks for the input guys!
I’m involved in various projects using different sound playback libraries.
In another game where I’m using fmod, it gives a much better result, as the
lag is nearly unnoticable. So I’m just curious, if there is a known method
using the SDL library to play sounds in a way that could be acceptable, or
should I use external libraries for this problem? Should I try SDL_Audio and
expect some progress?


//David Olofson - Consultant, Developer, Artist, Open Source Advocate

.— Games, examples, libraries, scripting, sound, music, graphics —.
| http://consulting.olofson.net http://olofsonarcade.com |
’---------------------------------------------------------------------’

I’m a little skeptical that you will see any major difference between
SDL_Audio and SDL_mixer since the latter is built on the former. But
if you try it, I would like to know your results.

I know early on, fmod was affected by the same Android latency
problems. Theoretically, you can only optimize around the areas that
are not Android, so it could be they’ve streamlined their own data
pipeline more than SDL_Audio.

Apportable’s OpenAL soft fork attempted to do just this, and made an
integer/fixed-point modification to OpenAL (as well as implement an
OpenSL ES backend). (The hardware floating point situation on Android
is also a mes.)

Apportable’s fork is what I use (and tweak). It did improve things
quite a bit from where they were with OpenAL Soft. And Android 4.1
helped improve things a lot more. But I still perceive the latency
problem, though normal people complain a lot less than they used to.

I would suggest trying ALmixer (my library). It is an SDL_mixer
inspired API, but using OpenAL as the backend. I have special hooks to
make it work better on Android and with OpenAL Soft.

Here is a hello world program which you can experiment with the
latency. (I really recommend Android 4.1+)

Since the API is very close to SDL_mixer, if it works better for you,
then it should be pretty easy to port to ALmixer.

Thanks
EricOn 4/3/14, LBandy wrote:

Thanks for the input guys!
I’m involved in various projects using different sound playback libraries.
In another game where I’m using fmod, it gives a much better result, as the
lag is nearly unnoticable. So I’m just curious, if there is a known method
using the SDL library to play sounds in a way that could be acceptable, or
should I use external libraries for this problem? Should I try SDL_Audio and
expect some progress?


Beginning iPhone Games Development
http://playcontrol.net/iphonegamebook/

OP said that the sound lags 0.5 seconds. That’s way more than just Android
being awkward. I’m not getting that in my apps, but I don’t think I can
offer any suggestions right now beside the 1024 byte buffer size I use.

Jonny D

Pffft. 0.5 seconds is nothing on Android. I’ve seen Angry Birds have
3-5 seconds latency on Android devices.

Believe me on this. Android audio is terrible.

-EricOn 4/3/14, Jonathan Dearborn wrote:

OP said that the sound lags 0.5 seconds. That’s way more than just Android
being awkward. I’m not getting that in my apps, but I don’t think I can
offer any suggestions right now beside the 1024 byte buffer size I use.

Jonny D


Beginning iPhone Games Development
http://playcontrol.net/iphonegamebook/

That’s ridiculous. And funny.

Jonny DOn Thu, Apr 3, 2014 at 6:41 PM, Eric Wing wrote:

On 4/3/14, Jonathan Dearborn <@Jonathan_Dearborn> wrote:

OP said that the sound lags 0.5 seconds. That’s way more than just
Android
being awkward. I’m not getting that in my apps, but I don’t think I can
offer any suggestions right now beside the 1024 byte buffer size I use.

Jonny D

Pffft. 0.5 seconds is nothing on Android. I’ve seen Angry Birds have
3-5 seconds latency on Android devices.

Believe me on this. Android audio is terrible.

-Eric

Beginning iPhone Games Development
http://playcontrol.net/iphonegamebook/


SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Message-ID:
<CA+Q62MBRSLaTc9pJYtGHdS2raViUhc2yo6u031EdPb8WV2nJFA at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

P.S. I have non-latency gripes about their OpenSL ES API extensions
for loading/decoding. You must read from a file or URI. They don’t
have a read from memory system or anything that is fopen/fread/fseek
signature compatible, so it is impossible to interface with SDL_RWops.

-Eric

And even if we were willing to use FUSE,

  1. I’m going to guess that Android doesn’t normally support it (at
    least for apps), and,
  2. It would provide enough latency itself,
    right?> Date: Thu, 3 Apr 2014 04:28:23 -0700

From: Eric Wing
To: SDL Development List
Subject: Re: [SDL] Mix_PlayChannel Android lag

And even if we were willing to use FUSE,

  1. I’m going to guess that Android doesn’t normally support it (at
    least for apps), and,

Android doesn’t support it as far as I know. I don’t know enough about
FUSE’s permission requirements about whether it is compatible with
Android’s security model. But anything needs to be in userland and
live in Android’s app sandbox. My hunch is that FUSE won’t work
without rooting your device.

But I don’t think FUSE helps here. There are really 2 different
problems which got entangled.

The first problem is that all assets inside an apk are basically
inside a zip, and the standard C ways of accessing a file are useless.
You must use their special APIs to read anything (AssetManager). (And
I have nightmare performance problem stories about this API too.)

But the other problem is that you really want access to Android’s
native decoder system in my very strong opinion. You get multiple
benefits if you use Android’s decoder system:

  • Optimized for both battery life and performance on the platform
  • Protected under Google’s patent umbrella protection for formats like AAC
  • Don’t bloat your apk size with 3rd party codecs (and remember you
    need multiple versions for each architecture you support and some of
    these have complicated build systems that you have to port to
    Android…also remember that Android has a pathetic apk max size limit
    of like 50MB).
  • Also, so codecs are really hard to find commercial friendly, free
    3rd party implementations for (AAC)

Since I deal with both iOS and Android a lot, it turns out AAC is the
best format for shipping. iOS doesn’t support OGG. While you could
bundle a Ogg Vorbis or Tremor decoder with iOS, Apple has optimized
the heck out of Core Audio for both battery and performance (and even
has on-board hardware decoders) and bundling Ogg yourself just doesn’t
reach that. And I saw my binary sizes go up. The next obvious format
is MP3, but there are royalty obligations if you the app/game maker
distribute an MP3. So that’s out. AAC is the sequel to MP3 and doesn’t
make distributors pay royalties, so when looking at the end user
experience of game makers and end-players, AAC looks like the best of
bad options. (My world mostly deals with people who want to ship
games, not fight the war against software patents.)

Anyway, the only way to acces Android’s native decoders is through
their file or URI API. I think they did this partly because they
conflated the AssetManager problem with the decoder API. (The
abstraction leaked.)

  1. It would provide enough latency itself,
    right?

This one I think you might be actually mitigate if you want to do the
extra hoop jumping by prefetching data into memory if it was an issue.

-Eric–
Beginning iPhone Games Development
http://playcontrol.net/iphonegamebook/

Message-ID:
<CA+Q62MA2U4j7tH7+=e=+uK7nQm0WRn545btsz+62ZMtjCR4uww at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

And even if we were willing to use FUSE,

  1. I’m going to guess that Android doesn’t normally support it (at
    least for apps), and,

Android doesn’t support it as far as I know.

While I half-recall something about Android being ported to run on BSD
or something (not that it matters, since FUSE has versions for more
than just Linux), Android mostly runs on Linux, and Linux supports
FUSE, so Android SUPPORTS FUSE… but FUSE requires a kernel module
which Google apparently doesn’t require, so…

Vendor-dependent, in essence. Regardless, a brief search suggested
that there are FUSE compiles for Android, so that part isn’t the
issue, the apparent lack of a requirement for it is the issue.

I don’t know enough about
FUSE’s permission requirements about whether it is compatible with
Android’s security model. But anything needs to be in userland and
live in Android’s app sandbox. My hunch is that FUSE won’t work
without rooting your device.

For at least some devices, yes. I suppose that we could try to figure
out a test to see if FUSE is present, but would that really be worth
the effort without a fall-back?

But I don’t think FUSE helps here. There are really 2 different
problems which got entangled.

The first problem is that all assets inside an apk are basically
inside a zip, and the standard C ways of accessing a file are useless.
You must use their special APIs to read anything (AssetManager). (And
I have nightmare performance problem stories about this API too.)

Wait, wait, wait… are you saying that the “best” API is only
supported for resource files that you’ve stuffed into your package?
Please tell me that I misread what you were trying to imply.

If ordinary files ARE supported, and the API doesn’t do something
weird, then FUSE would almost guaranteed work IF you could run it…
but if only packaged files are supported then it would be futile
regardless.

But the other problem is that you really want access to Android’s
native decoder system in my very strong opinion. You get multiple
benefits if you use Android’s decoder system:

I don’t properly recall the email that I originally replied to, but
from you mentioning this I’m guessing: the good-latency API doesn’t
interact with the Android audio decoder that you’re talking about,
does it?

also remember that Android has a pathetic apk max size limit
of like 50MB).

Actually I think that sorta/kinda makes sense, what with cell-modem
data-plan charges, but I’ll concede that if I were implementing a
packaging system I’d allow packages to provide files & folders to
other packages, to be hard or soft linked into their “virtual
package”, so it also sorta doesn’t make sense to me…

Anyway, the only way to acces Android’s native decoders is through
their file or URI API. I think they did this partly because they
conflated the AssetManager problem with the decoder API. (The
abstraction leaked.)

Unfortunately I think that this gets at the real fix that’s required:
someone needs to fix the Android implementation (from my perspective,
preferable to a trivially pipeline-able memory-mapping-based system).
That way we could get sensible latency on plenty of handsets in, oh, a
few years. I’m thinking it’s not going to happen from just us.> Date: Fri, 4 Apr 2014 01:29:22 -0700

From: Eric Wing
To: SDL Development List
Subject: Re: [SDL] Mix_PlayChannel Android lag