Could do a “make check”, was thinking more of separating it like the
test section though, although make check would also be interesting and
more accessible.
Basically, what I like best is a single command that you can run as
easily as possible (as little previous setup as possible), and gives
you a single pass/fail overall result. You can have more details in
the output, of course, but the very last end of the output should be
either PASS or FAIL (yes, in capital letters!). 
Putting it in the makefile is handy, because you can simply make the
rule that runs the test depend on the test binary, so you can’t screw
up with “one last change, run the test, it passes (but I forgot to
recompile before running it, it would have failed, but a bunch of
other people will get to find out the painful way when they svn
update)”. The tests themselves can be all in one big binary, or in a
suite of binaries run in succession, but you can set it up so that the
overall “make check” will fail if any of them fails, for example, and
you also gain a few freebies like being able to do “make check -j3” to
actually run multiple tests at a time on a multi-core machine (tests
are never fast enough!). If the tests are in multiple binaries, you
can also run just one very easily.
The key idea is that testing before commit saves a lot of time between
all the different people who are impacted by a bad check-in, and the
only way people will do testing is if it’s super-easy. One command to
run before doing the commit is more likely to be used. At work, our
revision control system allows per-directory pre-commit hooks that run
on the client (instead of on the server), so it’s one step further:
it’s IMPOSSIBLE to even commit code that doesn’t pass the tests in
those directories!
The idea is to try to do as many platform neutral test cases as
possible, comparing API specification vs results. ?Some simple examples
would be to test blitting and alpha. ?The idea being that doing platform
neutral test cases will benefit the largest amount of users.
Sounds good!
Eventually I guess I’ll have no choice but to make platform-specific
tests, although I’ll try to avoid that as much as possible.
You might be able to go a long way doing mocking, providing fake
implementations of platform-specific APIs that just check that they
are called in the right order and with the right parameters, returning
canned output (so you can make them appear to “fail” at will to test
error-handling, say). It’s best to do as little mocking as possible,
because it opens the possibility of baking your assumptions in (in the
worst case, you can make a test that verifies that the code is
buggy, by accident!), but when dealing with the kind of APIs that
SDL uses, they seem like a good idea.
If you go to the extreme of mocking, you could actually do things like
test the Windows backend code on Linux (or any other platform), say,
which is pretty awesome for those changes that involve systems where
you do not have access (see the recent WinCE changes, for example, Sam
doesn’t have what’s needed to test this at all, he has to commit on
good faith).
My intention is to wrap it all up in a set of automated bash scripts
that will keep a log and indicate errors, warnings and just general
logging messages. ? This should then be able to be used to identify the
problems which could be either an issue with user set up or a problem
with SDL itself. ?Finally the output could be used for finding and
fixing regressions.
If anyone has advice or possible ideas for automated testing, feel free
to spread them here. ?My experience with automated testing isn’t as good
as I’d like it to be.
My first pick would probably be googletest
(http://code.google.com/p/googletest/), since I use that all the time
and I know it works really well, but it’s written in the One True
Language (also known as the Most Evil Language, depending), so it
might not be acceptable (it can test C code, though, and seriously,
who the heck has a C compiler but no C++ compiler?!?). I also commonly
use Boost.Test, but it’s in the same boat of excellent/evilness…
There’s cmockery (http://code.google.com/p/cmockery/), which is pretty
simple, in C, and unlike most others, provides explicit support for
mocking, which might be very handy here, as I mentioned.
CuTest (http://cutest.sourceforge.net/) is all right, although rather
basic, but has the great quality of being tiny and super-easy to drop
right into SDL, so that there wouldn’t be any dependencies to install
for testing (removing hurdles to testing, which is good!).
Check (http://check.sourceforge.net/) is well-used, and most
importantly, widely packaged on Linux distributions, so it’s often
just an “aptitude” or “yum” command away. In the same line of thought,
CPlusTest is bundled with Mac OS X (well, Xcode), but it’s in C++, not
much of a fan of it either.
CUnit (http://cunit.sourceforge.net/) is classic, but I personally
think it’s a bit crazy and too much (a curses UI?!? whoa).
Automated tests are SUPER addictive. Once you have them, it feels
crippling to work on a project that doesn’t (for an example, it’s my
main excuse for why my “efficient waiting for event” patch is not done
yet!). ;-)On Thu, May 7, 2009 at 4:50 PM, Edgar Simo wrote:
–
http://pphaneuf.livejournal.com/
In Tux Paint we ended up creating our own UI (well, rather,