Raspberry Pi: No sound without pulseaudio

Sound doesn’t work when using the kms driver (fkms works) without pulseaudio running. I’m not sure if this is a bug or a feature and if it’s a bug if its SDLs fault or the Raspberry Pi OS, but it seems unreasonable to me that you must have pulseaudio to get it to work. Setting SDL_AUDIODRIVER=alsa does not work either. The funny thing is that ‘speaker-test’ from alsa-utils gives sound without pulseaudio. Have I wrongly configured SDL?