Full crashreporter in OpenLieroX

Heya,

I thought that maybe interesting for some other devs here:

We just finished a full crashreporter system, based on Google Breakpad. It doesn’t use the same architecture though - the whole report is generated on the client side. I.e., if it crashes, it will generate a nicely readable stacktrace from all threads and some more stuff, add the users configfile (stuff like passwords are removed automatically) and also adds the full stdout+stderr output from the crashed process and sends that to some mail.

Technically:

For stdout+stderr, for Linux/Unix/MacOSX, I coded something similar to the tee-tool but somehow more handy - this is then forked from the process, then dup2 on 0 and 1. For Windows, I am just doing an freopen. This teeStdout-handler forwards the data both to stdout and to a file (or at the very beginning into a buffer before it has a handy logfilename). In all cases, it even writes all the output as long as the application is running, no matter what happens (so absolutly no problems when it crashes).

When it crashes, Breakpads client part will generate a minidump and then start a new instance of our game with that minidumpfile and the logfile as a parameter.

That new instance will generate all needed Breakpad Symbolfiles on the fly and parse the minidumpfile and generates the crashreport. Then it collects the whole environment, the data from logfile and the configfile. It has a small SMTP client builtin and uses this to post everything to our mailinglist (later on perhaps directly into some bugtracker).

Breakpad needs some extensions to work that way. A lot of fixes for Windows were needed to make all the non-client part working (like the processor and that stuff). Also, for Linux, DWARF support was missing, only Stabs was supported (whereby DWARF is the common standard). Some people worked already on that but that was somehow incomplete and didn’t worked for me (see http://code.google.com/p/google-breakpad/issues/detail?id=130 for more details). But I fixed that (at least that it works on all systems where I have tested it).

So, we finally have a really simple crash reporting tool which works for Windows, MacOSX and Linux and we don’t really have to do any additional work to keep that system running.

I just made the first public release today, so we’ll how it behaves in the wild. :slight_smile:

If you want to try the end result: It’s OpenLieroX 0.58 beta5: http://openlierox.net , https://sourceforge.net/projects/openlierox/

If you want to see the code:
Breakpad stuff:
http://openlierox.git.sourceforge.net/git/gitweb.cgi?p=openlierox/openlierox;a=tree;f=src/breakpad;hb=refs/heads/0.58
TeeStdout stuff: (search for teeStdoutInit)
http://openlierox.git.sourceforge.net/git/gitweb.cgi?p=openlierox/openlierox;a=blob;f=src/main.cpp;hb=refs/heads/0.58

  • Albert

Here three examples, always the same crashing dummy function (Cmd_crash::exec):

MacOSX:
https://sourceforge.net/mailarchive/forum.php?thread_name=20091020033053.7814F50DEDC%40mail.az2000.de&forum_name=openlierox-devel

Windows:
https://sourceforge.net/mailarchive/forum.php?thread_name=20091022040628.6EBFC56D19A%40mail.az2000.de&forum_name=openlierox-devel

Linux:
https://sourceforge.net/mailarchive/forum.php?thread_name=20091022060304.97D7250DE46%40mail.az2000.de&forum_name=openlierox-devel

Note also the added console output after the crash report.

Very cool! :)On Thu, Oct 22, 2009 at 12:03 AM, albert wrote:

Heya,

I thought that maybe interesting for some other devs here:

We just finished a full crashreporter system, based on Google Breakpad. It
doesn’t use the same architecture though - the whole report is generated on
the client side. I.e., if it crashes, it will generate a nicely readable
stacktrace from all threads and some more stuff, add the users configfile
(stuff like passwords are removed automatically) and also adds the full
stdout+stderr output from the crashed process and sends that to some mail.

Technically:

For stdout+stderr, for Linux/Unix/MacOSX, I coded something similar to the
tee-tool but somehow more handy - this is then forked from the process, then
dup2 on 0 and 1. For Windows, I am just doing an freopen. This
teeStdout-handler forwards the data both to stdout and to a file (or at the
very beginning into a buffer before it has a handy logfilename). In all
cases, it even writes all the output as long as the application is running,
no matter what happens (so absolutly no problems when it crashes).

When it crashes, Breakpads client part will generate a minidump and then
start a new instance of our game with that minidumpfile and the logfile as a
parameter.

That new instance will generate all needed Breakpad Symbolfiles on the fly
and parse the minidumpfile and generates the crashreport. Then it collects
the whole environment, the data from logfile and the configfile. It has a
small SMTP client builtin and uses this to post everything to our
mailinglist (later on perhaps directly into some bugtracker).

Breakpad needs some extensions to work that way. A lot of fixes for Windows
were needed to make all the non-client part working (like the processor and
that stuff). Also, for Linux, DWARF support was missing, only Stabs was
supported (whereby DWARF is the common standard). Some people worked already
on that but that was somehow incomplete and didn’t worked for me (see
http://code.google.com/p/google-breakpad/issues/detail?id=130 for more
details). But I fixed that (at least that it works on all systems where I
have tested it).

So, we finally have a really simple crash reporting tool which works for
Windows, MacOSX and Linux and we don’t really have to do any additional work
to keep that system running.

I just made the first public release today, so we’ll how it behaves in the
wild. [image: Smile]

If you want to try the end result: It’s OpenLieroX 0.58 beta5:
http://openlierox.net , https://sourceforge.net/projects/openlierox/

If you want to see the code:
Breakpad stuff:

http://openlierox.git.sourceforge.net/git/gitweb.cgi?p=openlierox/openlierox;a=tree;f=src/breakpad;hb=refs/heads/0.58
TeeStdout stuff: (search for teeStdoutInit)

http://openlierox.git.sourceforge.net/git/gitweb.cgi?p=openlierox/openlierox;a=blob;f=src/main.cpp;hb=refs/heads/0.58

  • Albert

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org


-Sam Lantinga, Founder and President, Galaxy Gameworks LLC