[GSoC] Automated Test Suite Monthly Report

SDL Development
1

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

As you may know, I’m doing the GSoC project of creating an automatic
test suite to find and detect regressions in SDL automatically 1. This
will be a benefit in the future as regressions and bugs will be easier
and faster to detect.

As you may know I’ve had final exams until this Monday (included), which
have prevented me from coding as much as I would have liked to. That
however is now over so I can code freely until the completion of the
project.

Some information of the design, after looking at the libraries that
Pierre Phaneuf suggested 2, I’ve decided to write up a very simple one
similar to CuTest 3. I’ve called the simple testsuite maker SDL_AT for
SDL Autoomatic Testing, it currently consists of a few functions and
basically divides tests into what I call suites and cases. You create
multiple testsuites each with multiple testcases and it’ll report what
goes wrong/right. Currently it’s very simple, but if needed it can be
easily expanded to write success/fails into a database or create output
already formatted for posting to the bug tracker 4.

The code is currently located in my branch 5 specifically in
SDL/test/automated 6. It contains only a few tests for:

  1. RWops
  2. Surface - Only primitives and blending primitves at the moment.

Things on rendering tests. As per Sam’s indication, I am considering
that all implementations are going to draw exactly the same pixel per
pixel and am testing against prerendered images considering that the
current Linux implementation does indeed work well (please inform me if
it doesn’t). To avoid possible disturbances caused by loading external
files, the images are stored in C structs so the values can be accessed
directly with no need of parsing nor loading.

Following systems I will add (most likely in this order):

  1. Surface - Finish blitting and others.
  2. Rect - Collision testing.
  3. Pixel Formatting - Conversions and the like.
  4. Video - Rendering tests (not sure if it makes sense since it’s most
    likely OpenGL and OpenGL isn’t pixel perfect)

The problem which was mentioned before is that there’s a lot of stuff
I’d like to be able to write test suites for, but I’m not entirely sure
if it’s possible to do without writing platform specific tests and sort
of duplicated the code in SDL for that platform. An example would be the
entire event system (would need user interaction most likely). I’m not
really sure what the best way to approach this would be.

Edgar

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpMcnoACgkQolm4VNX3QTyURgCfefaeGBGVCprrLFwCl1DfZc54
Hq4An0Qo5jVi8cINq9q4snApuAEnEfly
=2DzO
-----END PGP SIGNATURE-----

2

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

As you may know, I’m doing the GSoC project of creating an automatic
test suite to find and detect regressions in SDL automatically [1]. This
will be a benefit in the future as regressions and bugs will be easier
and faster to detect.

As you may know I’ve had final exams until this Monday (included), which
have prevented me from coding as much as I would have liked to. That
however is now over so I can code freely until the completion of the
project.

sweet!

Some information of the design, after looking at the libraries that
Pierre Phaneuf suggested [2], I’ve decided to write up a very simple one
similar to CuTest [3]. I’ve called the simple testsuite maker SDL_AT for
SDL Autoomatic Testing, it currently consists of a few functions and
basically divides tests into what I call suites and cases. You create
multiple testsuites each with multiple testcases and it’ll report what
goes wrong/right. Currently it’s very simple, but if needed it can be
easily expanded to write success/fails into a database or create output
already formatted for posting to the bug tracker [4].

Cool. Using a test suite with a build bot is very useful for
writing cross platform code, and having it tested automatically. Then
people writing code on linux can have there changes tested by the
build bot automatically. (NOTE: this also encourages people to
write tests for their code… so they can have the buildbot test it
for them).

The code is currently located in my branch [5] specifically in
SDL/test/automated [6]. It contains only a few tests for:

  1. RWops
  2. Surface - Only primitives and blending primitves at the moment.

Things on rendering tests. As per Sam’s indication, I am considering
that all implementations are going to draw exactly the same pixel per
pixel and am testing against prerendered images considering that the
current Linux implementation does indeed work well (please inform me if
it doesn’t).

Note, that SDL can draw differently depending on the video mode set.
ie, 24bit, or 32bit. So it won’t render exactly the same in all
situations because of this.

To avoid possible disturbances caused by loading external
files, the images are stored in C structs so the values can be accessed
directly with no need of parsing nor loading.

Consider for some tests reusing the data available in the current examples?

Following systems I will add (most likely in this order):

  1. Surface - Finish blitting and others.
  2. Rect - Collision testing.
  3. Pixel Formatting - Conversions and the like.
  4. Video - Rendering tests (not sure if it makes sense since it’s most
    likely OpenGL and OpenGL isn’t pixel perfect)

The problem which was mentioned before is that there’s a lot of stuff
I’d like to be able to write test suites for, but I’m not entirely sure
if it’s possible to do without writing platform specific tests and sort
of duplicated the code in SDL for that platform. An example would be the
entire event system (would need user interaction most likely). I’m not
really sure what the best way to approach this would be.

For pygame(python bindings for SDL) we found that it is impossible to
test some subsystems automatically. Since they require input/output.

Interactive tests is what we called a set of tests that require human
interaction. They just take a Y/N answers to questions so people can
run through them very quickly. By default the ‘interactive’ tests
aren’t run of course - and we try to make the interactive tests only
when it’s impossible to test otherwise.

Also note, that SDL wasn’t designed to be testable(but is fairly
testable)… however now is the chance for you to say ‘this particular
API is not testable… please consider these changes so it’s possible
for me to automatically test it’.

You can ‘mock’ out some parts, and test most of the API (the 1.2 API anyway).

You might find the python tests useful for you in some cases, to see
how we tested particular parts. Or if you have trouble testing one
part, please feel free to ask how we did it - we might be able to give
some insight.

cheers,On Thu, Jul 2, 2009 at 6:40 PM, Edgar Simo wrote:

3

Den Thu, 02 Jul 2009 10:40:26 +0200
skrev Edgar Simo :

  1. Video - Rendering tests (not sure if it makes sense since it’s most
    likely OpenGL and OpenGL isn’t pixel perfect)

As long as you keep coordinates at whole integers and the textures
aren’t scaled or stretched in any way (except flipping), and the texture
and framebuffer format has enough bits per component to hold the
requested colors, OpenGL should be pixel perfect for textured
axis-aligned rectangles. That’s what I’d expect SDL to do when doing an
unscaled render, so I think such a test would be useful. Pixel
perfection is pretty important for games with pixel art, and on eg
Linux opengl is the only backend you’ve got if you want hardware
acceleration or vsync.

  • Gerry
4

The problem which was mentioned before is that there’s a lot of stuff
I’d like to be able to write test suites for, but I’m not entirely sure
if it’s possible to do without writing platform specific tests and sort
of duplicated the code in SDL for that platform. An example would be the
entire event system (would need user interaction most likely). I’m not
really sure what the best way to approach this would be.

One way of testing the upper layer of SDL, as well as the software
renderer, would be to make a mock driver. From what has been seen
recently on this mailing list, there seems to be some problems with
it, on occasion, and there’s no reason why the software renderer would
not be well tested. It can then be used as a reference driver, maybe
there would then be a way of using it as part of testing for the other
drivers.On Thu, Jul 2, 2009 at 4:40 AM, Edgar Simo wrote:

–

5

Things on rendering tests. As per Sam’s indication, I am considering
that all implementations are going to draw exactly the same pixel per
pixel and am testing against prerendered images considering that the
current Linux implementation does indeed work well (please inform me if
it doesn’t).

Note, that SDL can draw differently depending on the video mode set.
ie, 24bit, or 32bit. ?So it won’t render exactly the same in all
situations because of this.

That is quite true, but I think that many of the functions can be
tested with a single pixel, or at least, very few pixels, which should
be easy to programmatically generate, instead of having all the
versions “canned”. There might be a great number of pixel format
combination.

For pygame(python bindings for SDL) we found that it is impossible to
test some subsystems automatically. ?Since they require input/output.

One can go a long way without requiring human interaction. At the
extreme, one could write a fake X server that could be controlled by
the tests. Somewhat less ambitiously, one could run Xvnc and use
libVNCclient to simulate the input and get the state of the screen
from. One could also pull back some more, and make a mock libX11 that
provides just the subset of functions used by the X11 driver in SDL
(could be made easier by the dynamic support for libX11), but the more
you go back, the more you leave the opportunity for bugs to be in the
mock. You might just enshrine SDL bugs in the mock itself, faithfully
ensuring that the bug remains. But human interaction? Eurgh…On Thu, Jul 2, 2009 at 8:57 PM, Ren? Dudfield wrote:

–

6

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

Ren? Dudfield wrote:
|> The code is currently located in my branch [5] specifically in
|> SDL/test/automated [6]. It contains only a few tests for:
|>
|> 1) RWops
|> 2) Surface - Only primitives and blending primitves at the moment.
|>
|> Things on rendering tests. As per Sam’s indication, I am considering
|> that all implementations are going to draw exactly the same pixel per
|> pixel and am testing against prerendered images considering that the
|> current Linux implementation does indeed work well (please inform me if
|> it doesn’t).
|
| Note, that SDL can draw differently depending on the video mode set.
| ie, 24bit, or 32bit. So it won’t render exactly the same in all
| situations because of this.

Yes, I still have to handle different depths, only done 32 bits at the
moment. I would like to support 16 and paletted (which will be a bit
more complicated).

|
|> To avoid possible disturbances caused by loading external
|> files, the images are stored in C structs so the values can be accessed
|> directly with no need of parsing nor loading.
|>
|
| Consider for some tests reusing the data available in the current
examples?

Well I want to avoid stuff like surface loading in case for some reason
that’s what’s broken. A bit of paranoia perhaps. I am working with small
resolution (80x60) to not bloat the code, it’s about 35 kB per image and
I’ll only need perhaps 6 tops.

| For pygame(python bindings for SDL) we found that it is impossible to
| test some subsystems automatically. Since they require input/output.
|
| Interactive tests is what we called a set of tests that require human
| interaction. They just take a Y/N answers to questions so people can
| run through them very quickly. By default the ‘interactive’ tests
| aren’t run of course - and we try to make the interactive tests only
| when it’s impossible to test otherwise.

Yeah, I probably will have no choice but to do something like that. What
I don’t like about it is that it introduces “human error”, and it may
generate fake bug reports.

|
| Also note, that SDL wasn’t designed to be testable(but is fairly
| testable)… however now is the chance for you to say ‘this particular
| API is not testable… please consider these changes so it’s possible
| for me to automatically test it’.
|
| You can ‘mock’ out some parts, and test most of the API (the 1.2 API
anyway).
|
| You might find the python tests useful for you in some cases, to see
| how we tested particular parts. Or if you have trouble testing one
| part, please feel free to ask how we did it - we might be able to give
| some insight.
|

Where are the tests you guys have located?

Edgar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpN0/QACgkQolm4VNX3QTy6vACg7LI3VBN9gHnlZuSKuKnrgWXp
mXsAnA0tn1L9c0HgcdNay9o3qb9a7jDL
=QrEg
-----END PGP SIGNATURE-----

7

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

Pierre Phaneuf wrote:
| That is quite true, but I think that many of the functions can be
| tested with a single pixel, or at least, very few pixels, which should
| be easy to programmatically generate, instead of having all the
| versions “canned”. There might be a great number of pixel format
| combination.
|
|> For pygame(python bindings for SDL) we found that it is impossible to
|> test some subsystems automatically. Since they require input/output.
|
| One can go a long way without requiring human interaction. At the
| extreme, one could write a fake X server that could be controlled by
| the tests. Somewhat less ambitiously, one could run Xvnc and use
| libVNCclient to simulate the input and get the state of the screen
| from. One could also pull back some more, and make a mock libX11 that
| provides just the subset of functions used by the X11 driver in SDL
| (could be made easier by the dynamic support for libX11), but the more
| you go back, the more you leave the opportunity for bugs to be in the
| mock. You might just enshrine SDL bugs in the mock itself, faithfully
| ensuring that the bug remains. But human interaction? Eurgh…
|

I want to stay away from writing non-portable code. The idea is to try
to write code that everyone can use, even future implementations. When
balancing “human error” vs “non-portability”, I think the "human error"
will have the edge. However interactive tests won’t be on by default if
they are done as Ren? mentioned.

Edgar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpN1JIACgkQolm4VNX3QTwajwCg1IO6NxyrdS5ErZTIUI8B3bIg
dkQAoNrwX1I1pnmbuT7ozSPIaP2Q/7TC
=jSWm
-----END PGP SIGNATURE-----

8

This is great stuff, Edgar.

As far as testing input, I’m perfectly fine with a test suite that requires
human input and isn’t automatically run. Testing the IME or mouse button
support is a perfect example of this.

The tests can detect subsystem driver and platform and do exercise platform
specific functionality, but they should all use the SDL APIs. Please let me
know if you run across something that requires platform dependent code to
test properly.

Thanks!On Thu, Jul 2, 2009 at 1:40 AM, Edgar Simo wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

As you may know, I’m doing the GSoC project of creating an automatic
test suite to find and detect regressions in SDL automatically 1. This
will be a benefit in the future as regressions and bugs will be easier
and faster to detect.

As you may know I’ve had final exams until this Monday (included), which
have prevented me from coding as much as I would have liked to. That
however is now over so I can code freely until the completion of the
project.

Some information of the design, after looking at the libraries that
Pierre Phaneuf suggested 2, I’ve decided to write up a very simple one
similar to CuTest 3. I’ve called the simple testsuite maker SDL_AT for
SDL Autoomatic Testing, it currently consists of a few functions and
basically divides tests into what I call suites and cases. You create
multiple testsuites each with multiple testcases and it’ll report what
goes wrong/right. Currently it’s very simple, but if needed it can be
easily expanded to write success/fails into a database or create output
already formatted for posting to the bug tracker 4.

The code is currently located in my branch 5 specifically in
SDL/test/automated [6]. It contains only a few tests for:

  1. RWops
  2. Surface - Only primitives and blending primitves at the moment.

Things on rendering tests. As per Sam’s indication, I am considering
that all implementations are going to draw exactly the same pixel per
pixel and am testing against prerendered images considering that the
current Linux implementation does indeed work well (please inform me if
it doesn’t). To avoid possible disturbances caused by loading external
files, the images are stored in C structs so the values can be accessed
directly with no need of parsing nor loading.

Following systems I will add (most likely in this order):

  1. Surface - Finish blitting and others.
  2. Rect - Collision testing.
  3. Pixel Formatting - Conversions and the like.
  4. Video - Rendering tests (not sure if it makes sense since it’s most
    likely OpenGL and OpenGL isn’t pixel perfect)

The problem which was mentioned before is that there’s a lot of stuff
I’d like to be able to write test suites for, but I’m not entirely sure
if it’s possible to do without writing platform specific tests and sort
of duplicated the code in SDL for that platform. An example would be the
entire event system (would need user interaction most likely). I’m not
really sure what the best way to approach this would be.

Edgar

[6]:

http://www.libsdl.org/cgi/viewvc.cgi/branches/gsoc2009_unit_tests/SDL/test/automated/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpMcnoACgkQolm4VNX3QTyURgCfefaeGBGVCprrLFwCl1DfZc54
Hq4An0Qo5jVi8cINq9q4snApuAEnEfly
=2DzO
-----END PGP SIGNATURE-----

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

–
-Sam Lantinga, Founder and President, Galaxy Gameworks LLC

9

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Edgar Simo wrote:

The problem which was mentioned before is that there’s a lot of stuff
I’d like to be able to write test suites for, but I’m not entirely sure
if it’s possible to do without writing platform specific tests and sort
of duplicated the code in SDL for that platform. An example would be the
entire event system (would need user interaction most likely). I’m not
really sure what the best way to approach this would be.

Hello Edgar,

Could you run a VM (like Virtualbox or Qemu) and let a script do mouse
movements and keyboard input and then take screenshots that you run through a
comparison program?

Best regards,
Chris

  • -- Christopher C. Eineke -- Email: chris at chriseineke.com -*-
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

iEYEAREKAAYFAkpOWzoACgkQPOmqd0kEEbukWQCeN5ybcK4SrbnspK6NYyYQliUo
2M4AoMDAgehyq0SrKkCrkBMFt2ducjg1
=7jni
-----END PGP SIGNATURE-----

10

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

Christopher Eineke wrote:
| Edgar Simo wrote:
|> The problem which was mentioned before is that there’s a lot of stuff
|> I’d like to be able to write test suites for, but I’m not entirely sure
|> if it’s possible to do without writing platform specific tests and sort
|> of duplicated the code in SDL for that platform. An example would be the
|> entire event system (would need user interaction most likely). I’m not
|> really sure what the best way to approach this would be.
|
| Hello Edgar,
|
| Could you run a VM (like Virtualbox or Qemu) and let a script do mouse
| movements and keyboard input and then take screenshots that you run
through a
| comparison program?

That sounds awfully overcomplicated, and shoots up the “dependencies"
for the test suite by a lot. Currently I want to keep dependencies at
"just SDL”.

Edgar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpPgnYACgkQolm4VNX3QTzpNQCggWyZS7ZQDcaX/rH8iCgSpAr6
8uwAoNwc0jmLq3LoFN2fbOnwH98E7TcW
=shd4
-----END PGP SIGNATURE-----

11

Things on rendering tests. As per Sam’s indication, I am considering
that all implementations are going to draw exactly the same pixel per
pixel and am testing against prerendered images considering that the
current Linux implementation does indeed work well (please inform me if
it doesn’t).

Note, that SDL can draw differently depending on the video mode set.
ie, 24bit, or 32bit. So it won’t render exactly the same in all
situations because of this.

That is quite true, but I think that many of the functions can be
tested with a single pixel, or at least, very few pixels, which should
be easy to programmatically generate, instead of having all the
versions “canned”. There might be a great number of pixel format
combination.

For pygame(python bindings for SDL) we found that it is impossible to
test some subsystems automatically. Since they require input/output.

One can go a long way without requiring human interaction. At the
extreme, one could write a fake X server that could be controlled by
the tests. Somewhat less ambitiously, one could run Xvnc and use
libVNCclient to simulate the input and get the state of the screen
from. One could also pull back some more, and make a mock libX11 that
provides just the subset of functions used by the X11 driver in SDL
(could be made easier by the dynamic support for libX11), but the more
you go back, the more you leave the opportunity for bugs to be in the
mock. You might just enshrine SDL bugs in the mock itself, faithfully
ensuring that the bug remains. But human interaction? Eurgh…

Yes it’s not optimal, but it is impossible to test SDL properly without
human interaction.

Also you want to be able to test specific real hardware and drivers. Mocks
don’t help with that in all cases. So rather than pretend the non-automated
cases don’t exist, it’s easy enough to just encode specific tests as
interactive tests.

Automated tests are definitely much more useful, and making a test
interactive should be a last resort.On Fri, Jul 3, 2009 at 2:40 PM, Pierre Phaneuf wrote:

On Thu, Jul 2, 2009 at 8:57 PM, Ren? Dudfield<@Rene_Dudfield> wrote:

12

| You might find the python tests useful for you in some cases, to see
| how we tested particular parts. Or if you have trouble testing one
| part, please feel free to ask how we did it - we might be able to give
| some insight.
|

Where are the tests you guys have located?

http://pygame.org/wiki/svn

In the tests/ directory.

cheers,On Fri, Jul 3, 2009 at 7:48 PM, Edgar Simo wrote:

13

Yes it’s not optimal, but it is impossible to test SDL properly without
human interaction.

Also you want to be able to test specific real hardware and drivers.? Mocks
don’t help with that in all cases.? So rather than pretend the non-automated
cases don’t exist, it’s easy enough to just encode specific tests as
interactive tests.

People tests kernels without human interactions. Vendors (like
Intel, ATI and nVidia) themselves use automated tests suites and
diagnostics. They tend to go all the way and attach monitoring
hardware to the DVI and VGA connectors, but that’s kind of overboard,
we’re not debugging the OpenGL hardware here, we just have to make
sure that what SDL does is sensible. We can also have a few raw
OpenGL tests (which do not use SDL at all, but use whatever SDL uses)
to ensure that the platform calls are doing what we expect them to do,
but that’s not testing SDL itself so much as testing our assumptions
(so that someone having problems with SDL can see if we think the
platform is misbehaving).

In terms of code coverage of the platform-independent parts, I think
you could reach 100% of coverage with fully automatic tests, and
that’d be pretty good, in itself. The X11 driver that is broken right
now (with testsprite2), for example, I bet you could make an automated
test for that which would be able to run even on platforms without
X11. That one, right there, would be quite useful.

At work, every time I say something couldn’t be tested in an automated
way, someone always manage to prove me wrong, no matter how crazy
the damn thing is (and there’s OpenGL/Direct3D work in there:
http://src.chromium.org/svn/trunk/src/o3d/tests/)… ;-)On Sun, Jul 5, 2009 at 10:43 PM, Ren? Dudfield wrote:

–

14

Yes it’s not optimal, but it is impossible to test SDL properly without
human interaction.

Also you want to be able to test specific real hardware and drivers.
Mocks
don’t help with that in all cases. So rather than pretend the
non-automated
cases don’t exist, it’s easy enough to just encode specific tests as
interactive tests.

People tests kernels without human interactions. Vendors (like
Intel, ATI and nVidia) themselves use automated tests suites and
diagnostics. They tend to go all the way and attach monitoring
hardware to the DVI and VGA connectors, but that’s kind of overboard,
we’re not debugging the OpenGL hardware here, we just have to make
sure that what SDL does is sensible. We can also have a few raw
OpenGL tests (which do not use SDL at all, but use whatever SDL uses)
to ensure that the platform calls are doing what we expect them to do,
but that’s not testing SDL itself so much as testing our assumptions
(so that someone having problems with SDL can see if we think the
platform is misbehaving).

In terms of code coverage of the platform-independent parts, I think
you could reach 100% of coverage with fully automatic tests, and
that’d be pretty good, in itself. The X11 driver that is broken right
now (with testsprite2), for example, I bet you could make an automated
test for that which would be able to run even on platforms without
X11. That one, right there, would be quite useful.

At work, every time I say something couldn’t be tested in an automated
way, someone always manage to prove me wrong, no matter how crazy
the damn thing is (and there’s OpenGL/Direct3D work in there:
http://src.chromium.org/svn/trunk/src/o3d/tests/).http://src.chromium.org/svn/trunk/src/o3d/tests/).…
:wink:

–

I think we’re agreeing here.

Just using public SDL APIs to test 100% of SDL is impossible. Something
like 97% of the pygame tests are non-interactive. The non-interactive tests
are way more useful than the interactive ones.

here’s some tests you can’t automate:

  • prove that the sound coming out of SDL is not crackling, and coming out
    of the correct speakers.
  • prove that the CD has actually ejected.
  • prove that the left mouse button actually generated a left mouse button
    event.
  • prove that the K_q key button actually generated a K_q keyboard event.
  • etc.

Those things can be tested on some platforms with emulation, or by using
some non-SDL APIs… but not all.

Since a lot of SDL is interacting with real world devices, it’s impossible
to test that stuff completely without human interaction - or some non-SDL
APIs, or computer vision/listening techniques.

However, as a reminder… we don’t need 100% automated testing, automated
testing is the most useful testing.

cheers,On Mon, Jul 6, 2009 at 1:59 PM, Pierre Phaneuf wrote:

On Sun, Jul 5, 2009 at 10:43 PM, Ren? Dudfield<@Rene_Dudfield> wrote: