Sdl audio jittering

Florian_Hufsky · November 12, 2005, 9:13pm

Hi,

Due to the lack of an easy to use compressed audio capable sdl audio
library (ever tried sdl_mixer with mp3/ogg sounds?) I’m working on an
Audiere version that solely uses SDL.

Everything’s fine until now, as Audiere does everything to load and mix
the music. It gives me a steady stream of 16Bit PCM samples that only
need to be shoved into the sdl audio buffer.

The whole thing works great already. Unless the CPU gets a bit busy.
Then things start to jitter. However running Audiere with the Windows
backend works still fine, so it’s not a performance Problem of audiere.

Ok, my question is if you could help me get rid of that jittering at
higher CPU loads. I guess someone with a more decent knowledge of SDL
audio internals might instantly know the answer.

How the thing works:

I have a

freq = 44100;
channels = 2; //stereo
format = AUDIO_S16;
samples = 2048;
SDL_AudioSpec (verified after initalizin, and it’s the same format that
audiere gives us)

In the audio callback I do the following:

let audiere mix the next streamlen bytes of audio data
put it in the sdl audio stream

That’s exactly the same thing audiere does in the windows backend.

I guess the jittering is because of audiere taking too much time to mix
data (for sdl). However I’m worried because with the windows backend
everything works fine.
But maybe the windows backend is written a way so that the audio
callback can run multiple times at once.

Do you have any idea what could be the problem? Or what causes audio
jittering under SDL in general.

If you want I can publish a patch to the audiere source to make it SDLified.

Huge thanks in advance,
Florian Hufsky

David_Olofson · November 12, 2005, 10:41pm

[…]

Ok, my question is if you could help me get rid of that jittering at
higher CPU loads. I guess someone with a more decent knowledge of
SDL audio internals might instantly know the answer.

What do you mean by jittering, more specifically? Are you getting
audio drop-outs, or is it a matter of varying event->audio output
latency?

How the thing works:

I have a

freq = 44100;
channels = 2; //stereo
format = AUDIO_S16;
samples = 2048;
SDL_AudioSpec (verified after initalizin, and it’s the same format
that audiere gives us)

In the audio callback I do the following:

let audiere mix the next streamlen bytes of audio data

put it in the sdl audio stream

Are you sure Audiere actually mixes the exact number of bytes you ask
for? If not, it may not actually render the same amount of data each
time you call it, which means you’ll start missing deadlines long
before you’re at 100% CPU load.

That’s exactly the same thing audiere does in the windows backend.

Same buffer size too?

A general purpose OS usually has hideous scheduling jitter, even for
so called “real time priority”, which severely restricts the latency
(buffer size) and CPU power available for real time DSP. Say you can
use 50% of the CPU time without problems with a certain buffer size.
Cut the buffer size in half, and you could end up getting drop-outs
even if you use virtually no CPU time att all. It does not scale
towards zero latency. It scales towards the buffer size that
corresponds to the worst case scheduling latency.

I guess the jittering is because of audiere taking too much time to
mix data (for sdl). However I’m worried because with the windows
backend everything works fine.

I can’t swear that SDL isn’t doing something funny, because I’ve been
having trouble with Audiality over the SDL audio backend all the
time, and never been able to figure out what’s going on…

However, this is on Windows only! On Linux, I get excellent
performance whether I use SDL audio or OSS. Even on old Linux kernels
that have worse real time performance than Windows, Audiality peforms
much better than on Windows.

But maybe the windows backend is written a way so that the audio
callback can run multiple times at once.

Not sure what you mean… What would that accomplish?

You need to generate exactly the number of samples the callback needs

and the only way to minimizi the sensitivity to scheduling jitter
is to keep the CPU load as steady as possible.

Do you have any idea what could be the problem? Or what causes audio
jittering under SDL in general.

Well, from what I’ve seen, SDL audio works great on Linux, whereas on
Windows, there are serious latency problems - by musical synthesis
standards, at least. (I’d say anything above 10 ms is unacceptable
for interactive real time synhesis, but you don’t need to get
anywhere near those figures in games. In fact, if you do, you’ll need
to delay the sound effects so they aren’t played before the user can
see the visual events!)

I don’t know if the latency problems are caused by Windows, SDL or the
combination, but I suspect that it’s mostly a Windows issue. To get
anywhere near the “standard” Linux, BeOS and Mac OS latencies on
Windows, you need to use ASIO, EASI or similar “bypass” audio API
instead of DirectSound. (KernelStreams might work, but only if you
disable the internal software mixer.)

One thing you might try is to log and analyze timestamps from various
places, such as when you enter and leave the SDL audio callback. (Use
x86 asm RDTSC, Win32 performance counters or something. You might
get away with ms accuracy when looking for critical latency peaks,
but you’ll get more interesting data with ?s accuracy or better.)

//David Olofson - Programmer, Composer, Open Source Advocate

.- Audiality -----------------------------------------------.
| Free/Open Source audio engine for games and multimedia. |
| MIDI, modular synthesis, real time effects, scripting,… |
`-----------------------------------> http://audiality.org -’
— http://olofson.net — http://www.reologica.se —On Saturday 12 November 2005 22.13, Florian Hufsky wrote:

Gabriel_Gambetta · November 13, 2005, 3:12am

Due to the lack of an easy to use compressed audio capable sdl audio
library (ever tried sdl_mixer with mp3/ogg sounds?)

Yep, all our games use SDL_mixer with ogg soundtrack and sound effects.
What’s so bad about it?

--Gabriel--

Gabriel Gambetta
Mystery Studio - http://www.mysterystudio.com
Gabriel on Graphics - http://gabrielongraphics.blogspot.com

Ricardo_Cruz · November 13, 2005, 3:51pm

SDL_mixer can’t reproduce two OGG/MP3/MOD sounds at once, so they are no good
for sound effects. They can only be played in the music channel.

RicardoEm Domingo 13 Novembro 2005 03:12, o Gabriel escreveu:

Due to the lack of an easy to use compressed audio capable sdl audio
library (ever tried sdl_mixer with mp3/ogg sounds?)

Yep, all our games use SDL_mixer with ogg soundtrack and sound effects.
What’s so bad about it?

–Gabriel

–
Anyone can hold the helm when the sea is calm.
– Publius Syrus

Gerry_Jo_Jellestad · November 13, 2005, 7:22pm

P? Sun, 13 Nov 2005 16:51:30 +0100, skrev Ricardo Cruz :

SDL_mixer can’t reproduce two OGG/MP3/MOD sounds at once, so they are no
good
for sound effects. They can only be played in the music channel.

That’s wrong. OGG and MP3 sound effects work fine in Linux, at least.
Also, MODs are tracked music, so it doesn’t make sense to use those for
sound effects.

Gerry

David_Olofson · November 13, 2005, 9:33pm

P? Sun, 13 Nov 2005 16:51:30 +0100, skrev Ricardo Cruz
:

SDL_mixer can’t reproduce two OGG/MP3/MOD sounds at once, so they
are no good
for sound effects. They can only be played in the music channel.

That’s wrong. OGG and MP3 sound effects work fine in Linux, at
least.

…and if it works there (uhm, “here” - I only use Windows for the
occasional game that refuses to run on Linux), it should work on
other platforms, provided you bring along the libs needed for the
decoding. Decoding audio files is totally platform independent code,
at least in theory.

Also, MODs are tracked music, so it doesn’t make sense to use those
for sound effects.

Actually, why not?

Back in the C64 days, it was pretty common that games used the music
player to play sound effects. (*) Of course, this was mostly because
actively driving a SID or “similar” sound chip (well, other chips at
the time were really rather primitive, compared to the SID) into
playing back “structured audio” in real time was pretty much the only
way to generate serious sound effects. So, the music player was
pretty much the right tool for the job anyway. Either way, this
method did have the advantage that you could generate very long,
complex and dynamic sound effects without storing insane amounts of
data.

It’s a bit sad that these days, you often end up pretending you’re not
hearing the ambience loops repeating, or recognizing the sound
effects when you hear them the millionth time - sometimes even the
first time you’re playing the game.

(*) This was also done on the Amiga to some extent, because the
limited amount of RAM sometimes left the musician with
practically no room for samples. Though, the most common
result of that was probably that the music was “chip tunes”,
while the sound effects were simply cut down to very short,
very low quality samples.

//David Olofson - Programmer, Composer, Open Source Advocate

.- Audiality -----------------------------------------------.
| Free/Open Source audio engine for games and multimedia. |
| MIDI, modular synthesis, real time effects, scripting,… |
`-----------------------------------> http://audiality.org -’
— http://olofson.net — http://www.reologica.se —On Sunday 13 November 2005 20.22, Gerry JJ wrote:

Florian_Hufsky · November 13, 2005, 9:58pm

Due to the lack of an easy to use compressed audio capable sdl audio
library (ever tried sdl_mixer with mp3/ogg sounds?)

Yep, all our games use SDL_mixer with ogg soundtrack and sound effects.
What’s so bad about it?

you can’t play multiple ogg files at once.

Florian_Hufsky · November 13, 2005, 10:07pm

Sorry for taking so long to answer, but I’m currently quite busy.

Ok, my question is if you could help me get rid of that jittering at
higher CPU loads. I guess someone with a more decent knowledge of
SDL audio internals might instantly know the answer.

What do you mean by jittering, more specifically? Are you getting
audio drop-outs, or is it a matter of varying event->audio output
latency?

I can’t tell you for sure, but I guess small audio drop-outs best
describes it.
It sounds like the buffer is written only 7/8 but already played. Could
be something different though.

Haven’t tried event->audio yet. For now I’m just playing a single mp3.

How the thing works:

I have a

freq = 44100;
channels = 2; //stereo
format = AUDIO_S16;
samples = 2048;
SDL_AudioSpec (verified after initalizin, and it’s the same format
that audiere gives us)

In the audio callback I do the following:

let audiere mix the next streamlen bytes of audio data

put it in the sdl audio stream

Are you sure Audiere actually mixes the exact number of bytes you ask
for? If not, it may not actually render the same amount of data each
time you call it, which means you’ll start missing deadlines long
before you’re at 100% CPU load.

I pass audiere the number of chunks to mix and assume it mixes that amount.

That’s exactly the same thing audiere does in the windows backend.

Same buffer size too?

Ah. I’ll take a look at that.

A general purpose OS usually has hideous scheduling jitter, even for
so called “real time priority”, which severely restricts the latency
(buffer size) and CPU power available for real time DSP. Say you can
use 50% of the CPU time without problems with a certain buffer size.
Cut the buffer size in half, and you could end up getting drop-outs
even if you use virtually no CPU time att all. It does not scale
towards zero latency. It scales towards the buffer size that
corresponds to the worst case scheduling latency.

Nice to know, thanks.

I guess the jittering is because of audiere taking too much time to
mix data (for sdl). However I’m worried because with the windows
backend everything works fine.

I can’t swear that SDL isn’t doing something funny, because I’ve been
having trouble with Audiality over the SDL audio backend all the
time, and never been able to figure out what’s going on…

However, this is on Windows only! On Linux, I get excellent
performance whether I use SDL audio or OSS. Even on old Linux kernels
that have worse real time performance than Windows, Audiality peforms
much better than on Windows.

sdl_mixer works just fine so i guess it’s not a SDL specific problem.
Maybe SDL + an audio callback which takes too long to update the buffer.

But maybe the windows backend is written a way so that the audio
callback can run multiple times at once.

Not sure what you mean… What would that accomplish?

Hm… right. … Wait… I think the Windows backend writes data to
the audio buffer all the time unless it’s filled. I guess that’s quite a
difference to SDL’s on-demand filling.
…
I have to check that.

Do you have any idea what could be the problem? Or what causes audio
jittering under SDL in general.

Well, from what I’ve seen, SDL audio works great on Linux, whereas on
Windows, there are serious latency problems - by musical synthesis
standards, at least. (I’d say anything above 10 ms is unacceptable
for interactive real time synhesis, but you don’t need to get
anywhere near those figures in games. In fact, if you do, you’ll need
to delay the sound effects so they aren’t played before the user can
see the visual events!)

I don’t know if the latency problems are caused by Windows, SDL or the
combination, but I suspect that it’s mostly a Windows issue. To get
anywhere near the “standard” Linux, BeOS and Mac OS latencies on
Windows, you need to use ASIO, EASI or similar “bypass” audio API
instead of DirectSound. (KernelStreams might work, but only if you
disable the internal software mixer.)

One thing you might try is to log and analyze timestamps from various
places, such as when you enter and leave the SDL audio callback. (Use
x86 asm RDTSC, Win32 performance counters or something. You might
get away with ms accuracy when looking for critical latency peaks,
but you’ll get more interesting data with ?s accuracy or better.)

Ok. I’m going to check the timing and if audiere really gives me the
right amount of samples.

Thanks for your help so far.

Ricardo_Cruz · November 14, 2005, 12:43am

Mix_Chunk structure is created with either Mix_LoadWAV_RW() or Mix_LoadWAV()
that only accept WAV files as arguments. On the other hand, at least last
time I checked, MixMusic does support a lot of different formats but has an
unique channnel, so you can’t play a lot of those files at once.
This is a reason why a lot of people use SDL_sound or FMOD.

PS: this is vanilla SDL_mixer we’re talking about, right?

Cheers,
RicardoEm Domingo 13 Novembro 2005 19:22, o Gerry JJ escreveu:

P? Sun, 13 Nov 2005 16:51:30 +0100, skrev Ricardo Cruz <@Ricardo_Cruz>:

SDL_mixer can’t reproduce two OGG/MP3/MOD sounds at once, so they are no
good
for sound effects. They can only be played in the music channel.

That’s wrong. OGG and MP3 sound effects work fine in Linux, at least.
Also, MODs are tracked music, so it doesn’t make sense to use those for
sound effects.

Gerry

–
Security check: aaaINTRUDER ALERT!

Gabriel_Gambetta · November 14, 2005, 2:16am

Yep, all our games use SDL_mixer with ogg soundtrack and sound effects.
What’s so bad about it?
you can’t play multiple ogg files at once.

Yes I can… download
http://www.mysterystudio.com/demos/WildWestWendyDEMO.exe for proof

    --Gabriel--

Gabriel Gambetta
Mystery Studio - http://www.mysterystudio.com
Gabriel on Graphics - http://gabrielongraphics.blogspot.com

Gabriel_Gambetta · November 14, 2005, 2:18am

Mix_Chunk structure is created with either Mix_LoadWAV_RW() or Mix_LoadWAV()
that only accept WAV files as arguments.

Not true, at least Mix_LoadWAV_RW() loads ogg just fine.

PS: this is vanilla SDL_mixer we’re talking about, right?

Yep.
–Gabriel–
Gabriel Gambetta
Mystery Studio - http://www.mysterystudio.com
Gabriel on Graphics - http://gabrielongraphics.blogspot.com

Ricardo_Cruz · November 14, 2005, 3:18pm

Thank you very much for the information.

You guys must agree that the function name is misleading and should be
changed. I know a few people that didn’t use SDL_mixer because they thought
it didn’t support samples for anything different than WAV. And so did I.

Cheers,
RicardoEm Segunda 14 Novembro 2005 02:18, o Gabriel escreveu:

Mix_Chunk structure is created with either Mix_LoadWAV_RW() or
Mix_LoadWAV() that only accept WAV files as arguments.

Not true, at least Mix_LoadWAV_RW() loads ogg just fine.

PS: this is vanilla SDL_mixer we’re talking about, right?

Yep.
–Gabriel

–
The light at the end of the tunnel is the headlight of an approaching
train.

Florian_Hufsky · November 14, 2005, 9:19pm

Gabriel schrieb:

Yep, all our games use SDL_mixer with ogg soundtrack and sound effects.
What’s so bad about it?

you can’t play multiple ogg files at once.

Yes I can… download
http://www.mysterystudio.com/demos/WildWestWendyDEMO.exe for proof
   --Gabriel

Ok. You can’t in a clean way.
I guess you decode the .oggs on loading and use them as sounds in
sdl_mixer. Or something equally complicated. I haven’t seen a good way
yet.

Good way as in.

THE_Sound *s = mix_loadafilethatcontainsaudiodata(“bla.mp3”);
THE_Sound *paralell = mix_loadafilethatcontainsaudiodata(“bananarama.ogg”);
mix_play(s);
mix-play(paralell);
//be happy

Gabriel_Gambetta · November 14, 2005, 11:44pm

El lun, 14-11-2005 a las 15:18 +0000, Ricardo Cruz escribi?:

You guys must agree that the function name is misleading and should be
changed. I know a few people that didn’t use SDL_mixer because they thought
it didn’t support samples for anything different than WAV. And so did I.

No doubt the function name is misleading. I guess it only loaded WAV
files a long time ago, then loading other formats was added but the name
was kept to maintain backwards compatibility. At the very least it
should be documented in the header file.

    --Gabriel--

Gabriel Gambetta
Mystery Studio - http://www.mysterystudio.com
Gabriel on Graphics - http://gabrielongraphics.blogspot.com

Gabriel_Gambetta · November 14, 2005, 11:42pm

Ok. You can’t in a clean way.
I guess you decode the .oggs on loading and use them as sounds in
sdl_mixer. Or something equally complicated. I haven’t seen a good way
yet.

Good way as in.

THE_Sound *s = mix_loadafilethatcontainsaudiodata(“bla.mp3”);
THE_Sound *paralell = mix_loadafilethatcontainsaudiodata(“bananarama.ogg”);
mix_play(s);
mix-play(paralell);
//be happy

Actually yes, I do it almost like you say. Ripping my source to show
only the relevant lines,

    Mix_Chunk* m_pData;
    SDL_RWops* pRWOps = SDL_RWFromMem(pResource->getData(),
    pResource->getSize());
    m_pData = Mix_LoadWAV_RW(pRWOps, false);
    [...]
    return Mix_PlayChannel(0, m_pData, 0);

Instead of keeping telling me that the thing I’ve been doing for ages
doesn’t work, why don’t you just try it and see that it works?

    --Gabriel--

Gabriel Gambetta
Mystery Studio - http://www.mysterystudio.com
Gabriel on Graphics - http://gabrielongraphics.blogspot.com

Florian_Hufsky · November 15, 2005, 6:58pm

Gabriel schrieb:

El lun, 14-11-2005 a las 15:18 +0000, Ricardo Cruz escribi?:

You guys must agree that the function name is misleading and should be
changed. I know a few people that didn’t use SDL_mixer because they thought
it didn’t support samples for anything different than WAV. And so did I.

No doubt the function name is misleading. I guess it only loaded WAV
files a long time ago, then loading other formats was added but the name
was kept to maintain backwards compatibility. At the very least it
should be documented in the header file.
   --Gabriel

what? so i can load .ogg and use it as sounds (ie: play multichannel) in
sdl_mixer?!

Florian_Hufsky · November 15, 2005, 7:00pm

Gabriel schrieb:

Ok. You can’t in a clean way.
I guess you decode the .oggs on loading and use them as sounds in
sdl_mixer. Or something equally complicated. I haven’t seen a good way
yet.

Good way as in.

THE_Sound *s = mix_loadafilethatcontainsaudiodata(“bla.mp3”);
THE_Sound *paralell = mix_loadafilethatcontainsaudiodata(“bananarama.ogg”);
mix_play(s);
mix-play(paralell);
//be happy

Actually yes, I do it almost like you say. Ripping my source to show
only the relevant lines,
   Mix_Chunk* m_pData;
   SDL_RWops* pRWOps = SDL_RWFromMem(pResource->getData(),
   pResource->getSize());
   m_pData = Mix_LoadWAV_RW(pRWOps, false);
   [...]
   return Mix_PlayChannel(0, m_pData, 0);
Instead of keeping telling me that the thing I’ve been doing for ages
doesn’t work, why don’t you just try it and see that it works?
   --Gabriel

Ok, so I’ve said it doesn’t work.

Let me specify it: It doesn’t work in an easy and obvious way.
Havin Mix_LoadWAV_RW load other formats too is a bit strange after all.
And if sdl_mixer is capable of that why didn’t they add a generic
load_compressedformatintosound() ?

Gabriel_Gambetta · November 16, 2005, 4:52am

Havin Mix_LoadWAV_RW load other formats too is a bit strange after all.
And if sdl_mixer is capable of that why didn’t they add a generic
load_compressedformatintosound() ?

As I said in the other email, I think there’s little doubt that the name
is indeed misleading. But hey, it works

    --Gabriel--

Gabriel Gambetta
Mystery Studio - http://www.mysterystudio.com
Gabriel on Graphics - http://gabrielongraphics.blogspot.com

Ricardo_Cruz · November 16, 2005, 7:26pm

And it is a documented behavior, so it should work well in all systems:
http://www.libsdl.org/cgi/docwiki.cgi/Mix_5fLoadWAV

I’m really (happily) suprised about this. I really think you should change
the function name. You just need to add a macro like:
#define Mix_LoadSample Mix_LoadWAV
This will not break API or ABI compatibility and it would be not misleading.

Cheers,
RicardoEm Quarta 16 Novembro 2005 04:52, o Gabriel escreveu:

Havin Mix_LoadWAV_RW load other formats too is a bit strange after all.
And if sdl_mixer is capable of that why didn’t they add a generic
load_compressedformatintosound() ?

As I said in the other email, I think there’s little doubt that the name
is indeed misleading. But hey, it works
    --Gabriel

–
Die, v.:
To stop sinning suddenly.
– Elbert Hubbard

Gerry_Jo_Jellestad · November 17, 2005, 7:32pm

P? Sun, 13 Nov 2005 22:33:15 +0100, skrev David Olofson
:

Also, MODs are tracked music, so it doesn’t make sense to use those
for sound effects.

Actually, why not?

Well, okay, saying it doesn’t make sense was maybe a bit harsh. Still,
using a MOD for sound effects wouldn’t have much advantage over using
WAV or other sampled formats unless the sound effects were multichannel
and/or using selective/partial repeating or fancy effects (but those
could just as well be applied to regular waves), would it ?

Back in the C64 days, it was pretty common that games used the music
player to play sound effects. (*) Of course, this was mostly because
actively driving a SID or “similar” sound chip (well, other chips at
the time were really rather primitive, compared to the SID) into
playing back “structured audio” in real time was pretty much the only
way to generate serious sound effects.

Not the only way =). Sampling using the volume click bug was pretty
popular for some time, until Commodore fixed that bug in the 8580
version of the SID. There were still ways to get around the fix, though,
and later on people discovered how to get true 12-bit PCM sound using
the pulse waveform without a very noticable carrier tone. This wasn’t
much used in games, though (mostly because playing digi required obscene
amounts of both memory and rastertime, not to mention the insane timing
requirements of the PCM method …)

So, the music player was
pretty much the right tool for the job anyway. Either way, this
method did have the advantage that you could generate very long,
complex and dynamic sound effects without storing insane amounts of
data.

Yes, but I don’t think this could be done to the same degree with a
regular MOD file. SID audio is pretty much just regular programs
that happens to manipulate audio registers in stead of putting things
on the screen or calculating other stuff, so there’s literally no
restrictions on what they can do (CPU speed willing, of course).

It’s a bit sad that these days, you often end up pretending you’re not
hearing the ambience loops repeating, or recognizing the sound
effects when you hear them the millionth time - sometimes even the
first time you’re playing the game.

Yeah, I completely agree.

Gerry