Parallel build?

SDL Development
1

With the growing number of multicore machines out there it would be nice
if the SDL make system could fire off multiple compiles at the same
time. Has anyone looked into doing that?

Bob Pendleton--

±-------------------------------------+

2

you might know about make -j X, or you might not. The cool thing is
that it’s not SDL-specific. I’ve made it an alias in my .bashrc long
ago.

StephaneOn 1/17/08, Bob Pendleton wrote:

With the growing number of multicore machines out there it would be nice
if the SDL make system could fire off multiple compiles at the same
time. Has anyone looked into doing that?

3

With the growing number of multicore machines out there it would be nice
if the SDL make system could fire off multiple compiles at the same
time. Has anyone looked into doing that?

Doesn’t it just use GNU make?

make -j 5

GregoryOn Thu, 17 Jan 2008, Bob Pendleton wrote:

4

-j [jobs], --jobs[=jobs]
Specifies the number of jobs (commands) to run simultaneously.
If there is more than one -j option, the last one is
effective. If the -j option is given without an argument,
make will not limit the number of jobs that can run
simultaneously.On Thu, 17 Jan 2008 13:41:45 -0600 Bob wrote:

With the growing number of multicore machines out there it would be
nice if the SDL make system could fire off multiple compiles at the
same time. Has anyone looked into doing that?

5

Think this is what you’re looking for.

http://www.gnu.org/software/make/manual/make.html#ParallelOn 1/17/08, Bob Pendleton wrote:

With the growing number of multicore machines out there it would be nice
if the SDL make system could fire off multiple compiles at the same
time. Has anyone looked into doing that?

   Bob Pendleton

–
±-------------------------------------+

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

6

With the growing number of multicore machines out there it would be nice
if the SDL make system could fire off multiple compiles at the same
time. Has anyone looked into doing that?

you might know about make -j X, or you might not. The cool thing is
that it’s not SDL-specific. I’ve made it an alias in my .bashrc long
ago.

Thank you, I thought it was a much more difficult problem than it really
is. That happens sometimes.

Bob PendletonOn Thu, 2008-01-17 at 21:10 +0100, Stephane Marchesin wrote:

On 1/17/08, Bob Pendleton <@Bob_Pendleton> wrote:

Stephane

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

–
±-------------------------------------+

7

Well just sticking -j X after the make doesn’t guarantee parallel
building. It just allow X concurrent makes to exist. If the structure
of the Makefile is serial then parallel building don’t do anything.
Well except just a little bit more memory consumption. In reality
though Makefiles are allways at least partially parallel. One can of
course tune the Makefile to allow more concurrent builds.

So the rule is combine files/modules as late as you can and avoid
unnecessary cross referencing. Ie the more the build process resembles
a wide and shallow tree the better it will parallelize.On Thursday 17 January 2008, Bob Pendleton wrote:

On Thu, 2008-01-17 at 21:10 +0100, Stephane Marchesin wrote:

On 1/17/08, Bob Pendleton wrote:

With the growing number of multicore machines out there it would
be nice if the SDL make system could fire off multiple compiles
at the same time. Has anyone looked into doing that?

you might know about make -j X, or you might not. The cool thing is
that it’s not SDL-specific. I’ve made it an alias in my .bashrc
long ago.

Thank you, I thought it was a much more difficult problem than it
really is. That happens sometimes.

8

With the growing number of multicore machines out there it would
be nice if the SDL make system could fire off multiple compiles
at the same time. Has anyone looked into doing that?

you might know about make -j X, or you might not. The cool thing is
that it’s not SDL-specific. I’ve made it an alias in my .bashrc
long ago.

Thank you, I thought it was a much more difficult problem than it
really is. That happens sometimes.

Well just sticking -j X after the make doesn’t guarantee parallel
building. It just allow X concurrent makes to exist. If the structure
of the Makefile is serial then parallel building don’t do anything.
Well except just a little bit more memory consumption. In reality
though Makefiles are allways at least partially parallel. One can of
course tune the Makefile to allow more concurrent builds.

So the rule is combine files/modules as late as you can and avoid
unnecessary cross referencing. Ie the more the build process resembles
a wide and shallow tree the better it will parallelize.

I added -j 10 to the calls to make in my build and rebuild scripts and
cut the build time by a factor of 2 on a dual CPU computer. I was happy
with that.

	Bob PendletonOn Sat, 2008-01-19 at 01:00 +0200, Sami N??t?nen wrote:

On Thursday 17 January 2008, Bob Pendleton wrote:

On Thu, 2008-01-17 at 21:10 +0100, Stephane Marchesin wrote:

On 1/17/08, Bob Pendleton <@Bob_Pendleton> wrote:

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

–
±-------------------------------------+

9

With the growing number of multicore machines out there it
would be nice if the SDL make system could fire off multiple
compiles at the same time. Has anyone looked into doing that?

you might know about make -j X, or you might not. The cool
thing is that it’s not SDL-specific. I’ve made it an alias in
my .bashrc long ago.

Thank you, I thought it was a much more difficult problem than it
really is. That happens sometimes.

Well just sticking -j X after the make doesn’t guarantee parallel
building. It just allow X concurrent makes to exist. If the
structure of the Makefile is serial then parallel building don’t do
anything. Well except just a little bit more memory consumption. In
reality though Makefiles are allways at least partially parallel.
One can of course tune the Makefile to allow more concurrent
builds.

So the rule is combine files/modules as late as you can and avoid
unnecessary cross referencing. Ie the more the build process
resembles a wide and shallow tree the better it will parallelize.

I added -j 10 to the calls to make in my build and rebuild scripts
and cut the build time by a factor of 2 on a dual CPU computer. I was
happy with that.

Most Makefiles are constructed automatically according to the source
files. Modular source tree is the easiest way to develop and thus this
will almost always give speed increase. But for handmade Makefiles the
story is totally different. There are Makefiles that will brake either
the build process or the executable if parallel building is used.

In your case you can get better overall performance with something
like -j3 or -j4, because the extra makes are only using unnecessary
resources. As a rule of the thumb number of CPUs + 1 or 2 will be
enough. One or two processes waiting for IO operations and the rest
doing hard work. :wink:

I use -j5 for my quad core CPU.On Saturday 19 January 2008, Bob Pendleton wrote:

On Sat, 2008-01-19 at 01:00 +0200, Sami N??t?nen wrote:

On Thursday 17 January 2008, Bob Pendleton wrote:

On Thu, 2008-01-17 at 21:10 +0100, Stephane Marchesin wrote:

On 1/17/08, Bob Pendleton wrote:

  Bob Pendleton

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

10

Bob Pendleton wrote:> On Sat, 2008-01-19 at 01:00 +0200, Sami N??t?nen wrote:

On Thursday 17 January 2008, Bob Pendleton wrote:

On Thu, 2008-01-17 at 21:10 +0100, Stephane Marchesin wrote:

On 1/17/08, Bob Pendleton wrote:

With the growing number of multicore machines out there it would
be nice if the SDL make system could fire off multiple compiles
at the same time. Has anyone looked into doing that?
you might know about make -j X, or you might not. The cool thing is
that it’s not SDL-specific. I’ve made it an alias in my .bashrc
long ago.
Thank you, I thought it was a much more difficult problem than it
really is. That happens sometimes.
Well just sticking -j X after the make doesn’t guarantee parallel
building. It just allow X concurrent makes to exist. If the structure
of the Makefile is serial then parallel building don’t do anything.
Well except just a little bit more memory consumption. In reality
though Makefiles are allways at least partially parallel. One can of
course tune the Makefile to allow more concurrent builds.

So the rule is combine files/modules as late as you can and avoid
unnecessary cross referencing. Ie the more the build process resembles
a wide and shallow tree the better it will parallelize.

I added -j 10 to the calls to make in my build and rebuild scripts and
cut the build time by a factor of 2 on a dual CPU computer. I was happy
with that.

If you want to accelerate successive builds I would recommend ccache and if you
have more than one machine sitting idle (i.e. lab, coworker’s/wife’s machine)
you can combine with distcc with very good results.

bill
11

Bob Pendleton wrote:

With the growing number of multicore machines out there it
would be nice if the SDL make system could fire off multiple
compiles at the same time. Has anyone looked into doing that?

you might know about make -j X, or you might not. The cool thing
is that it’s not SDL-specific. I’ve made it an alias in my
.bashrc long ago.

Thank you, I thought it was a much more difficult problem than it
really is. That happens sometimes.

Well just sticking -j X after the make doesn’t guarantee parallel
building. It just allow X concurrent makes to exist. If the
structure of the Makefile is serial then parallel building don’t
do anything. Well except just a little bit more memory
consumption. In reality though Makefiles are allways at least
partially parallel. One can of course tune the Makefile to allow
more concurrent builds.

So the rule is combine files/modules as late as you can and avoid
unnecessary cross referencing. Ie the more the build process
resembles a wide and shallow tree the better it will parallelize.

I added -j 10 to the calls to make in my build and rebuild scripts
and cut the build time by a factor of 2 on a dual CPU computer. I
was happy with that.

If you want to accelerate successive builds I would recommend ccache
and if you have more than one machine sitting idle (i.e. lab,
coworker’s/wife’s machine) you can combine with distcc with very good
results.

The order which distcc and ccache are used is also important.

so adding distcc first in front of the path and then ccache again to the
front of the path is the correct way. Ie all compilations are cached
locally.

Few notes about distcc:

Make sure gcc is the exact same version in all the distcc hosts.

For slower computers using distcc, without local compilation is faster
if you got at least 100MB LAN connection between the hosts. Especially
so if your faster machine has more memory. For example my 2.4GHz P4 is
compiling everything through distcc in my quad core 2.4GHz machine. If
I add one local thread the speed goes down a litle bit even though
there is one thread more compiling. This besides that my P4 has 1GB of
memory. Ie this system is so hevily IO bound that it is faster to not
compile anything locally.On Saturday 19 January 2008, Vassilis Virvilis wrote:

On Sat, 2008-01-19 at 01:00 +0200, Sami N??t?nen wrote:

On Thursday 17 January 2008, Bob Pendleton wrote:

On Thu, 2008-01-17 at 21:10 +0100, Stephane Marchesin wrote:

On 1/17/08, Bob Pendleton wrote: