Working on the projectM music visualizer (cross-platform Milkdrop). The SDL version uses the audio capture feature but I notice significant lag of several frames at least, compared to other versions like the Music.app plugin.
Is there any way to do some “low-latency” audio capture? I want the visuals to react to music as real-time as possible.
The code is here: projectm/pmSDL.cpp at master · projectM-visualizer/projectm · GitHub
What platform are you on? There’s several ways you can tweak your audio setup above SDL to improve things. Then, selecting an audio driver can have an effect. Since most platforms have a couple of audio driver options. But looks like you’re on Windows?
SDL audio input itself can be configured to have a small buffer size. But then sometimes you need a large enough buffer to run audio processing on there for things like onset or pitch detection.
Additionally, in your main loop, make sure you’re not sleeping or using the display - which may limit things to 60HZ or so.
There are some improvements that could be done to the SDL audiodriver input code to make things quicker for visualization or more low latency. I think on windows there’s some newer low latency options (using IAudioClient3 WASAPI APIs instead of IAudioClient).
Here’s some Windows docs on low latency audio:
By default, all applications in Windows 10 will use 10ms buffers to render and capture audio. If an application needs to use small buffers, then it needs to use the new AudioGraph settings or the WASAPI IAudioClient3 interface, in order to do so.
I think to get best performance on Windows WASAPI the SDL driver would need some improvements listed in that article. Not sure 100% though.
I only had a quick look at your code… can you reduce copying data or reduce mixing number of channels down? If you’re only using a single channel (like from a microphone), you could avoid that mix.
Theoretically, Windows WASPI with the right driver can do:
buffer sizes between 128 samples (2.66ms@48kHz) and 480 samples (10ms@48kHz).
I can’t see from your code what samples you’re doing. Assuming 500 (10ms) and your at 60HZ, then note they don’t sink 50% of the time. So for many of your frames you are rendering 20ms ago. If your frames are driven by the audio, then you’ll be able to draw as quickly as possible. Get audio → draw frame.
Did you try 48/22 kHz? Maybe that’s a bit faster. I know on one system 48 is the fastest… I guess because internally it does everything at 48, so the other formats need resampling.
Hope some of this rambling could be helpful. Good luck!
cheers,
That’s very helpful, thanks! It’s cross-platform, I’m mainly testing on macOS.
I cleaned up the code a bit here projectm/src/projectM-sdl at 664369233d3be36ab938e9d9b5dd33afa757fddc · projectM-visualizer/projectm · GitHub
You can see what’s in the main loop and the audio capture setup. I’m experimenting with samples and mono audio, not much luck so far with getting things “tighter.”
I tried moving drawing/sleeping to another thread and OpenGL/macOS were really not happy about creating an OpenGL context not on the main thread, or drawing to it from another thread, so I’m kinda unclear on how to separate that stuff out into another thread.