Strange SDL_net problem - byte-swapped integers

James_Haley · October 17, 2007, 5:40am

Hi,

I’m adding network support to my Doom port, and everything works fine as long as the two clients are running the same OS and compiled version of the executable. But as soon as we try to mix and match, things go crazy. We’re trying to have Linux and Win32 machines talk to each other, but here’s an example of what happens:

1982972966 01110110 00110001 11000100 00100110

Received by Windows:
650391926 00100110 11000100 00110001 01110110

As you can see, the data has been byte-swapped. This is despite the number in question having been transmitted using SDLNet_Write32, and received using SDLNet_Read32. Why is an extra byte-swap occurring to the data only when Linux is involved, and how do I fix it? AFAIK the Linux machine is little-endian so that should have nothing to do with it.

JamesSent by Linux:

Help yourself to FREE treats served up daily at the Messenger Caf?. Stop by today.
http://www.cafemessenger.com/info/info_sweetstuff2.html?ocid=TXT_TAGLM_OctWLtagline

James_Haley · October 17, 2007, 6:18am

Hello,

I hate to reply to my own message right off, but upon carefully examining SDL_net.h, it seems this is a glaringly obvious bug in the library. Observe:

#define SDLNet_Write32(value, areap)
do
{
Uint8 *area = (Uint8 *)(areap);
area[3] = (value>> 24) & 0xFF;
area[2] = (value>> 16) & 0xFF;
area[1] = (value>> 8) & 0xFF;
area[0] = value & 0xFF;
} while ( 0 )

This is the definition of SDLNet_Write32 used for little-endian hosts. It would be fantastic, except that it’s writing the bytes into the array in the completely wrong order. Let us say that value is 0xAABBCCDD. On a little-endian host this is stored as DDCCBBAA in consecutive memory bytes, however, bit shifts are NOT relative to host endianness – they are endian independent. This means that, for example, the byte (value>> 24) & 0xff is AA, not DD.

In order for this integer to be properly translated to big-endian, the array indices should be reversed. (value>> 24) & 0xff should be going to the first memory location, so that it ends up as AABBCCDD. This in fact makes it exactly the same as the definition of this macro for big-endian hosts. In case that seems unintuitive, let me again iterate that bit shifts are not endian dependent. This means that the only thing that matters is the relation of the shifts to the array indices, and for a fixed output format (in this case big-endian), the relationship will remain constant.

As for why this is differing in behavior between Windows and Linux, I cannot explain. There must be something different about the compilation that’s causing the macros to not be used…

James> From: @James_Haley

To: sdl at libsdl.org
Date: Wed, 17 Oct 2007 00:40:34 -0500
Subject: [SDL] Strange SDL_net problem - byte-swapped integers

Hi,

I’m adding network support to my Doom port, and everything works fine as long as the two clients are running the same OS and compiled version of the executable. But as soon as we try to mix and match, things go crazy. We’re trying to have Linux and Win32 machines talk to each other, but here’s an example of what happens:

Sent by Linux:
1982972966 01110110 00110001 11000100 00100110

Received by Windows:
650391926 00100110 11000100 00110001 01110110

As you can see, the data has been byte-swapped. This is despite the number in question having been transmitted using SDLNet_Write32, and received using SDLNet_Read32. Why is an extra byte-swap occurring to the data only when Linux is involved, and how do I fix it? AFAIK the Linux machine is little-endian so that should have nothing to do with it.

James

Help yourself to FREE treats served up daily at the Messenger Caf?. Stop by today.
http://www.cafemessenger.com/info/info_sweetstuff2.html?ocid=TXT_TAGLM_OctWLtagline

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Boo!?Scare away worms, viruses and so much more! Try Windows Live OneCare!
http://onecare.live.com/standard/en-us/purchase/trial.aspx?s_cid=wl_hotmailnews

Matthias_Weigand · October 17, 2007, 8:31am

Hi James,

i think, you’re right. Actually i don’t understand, how to use the
SDLNet_Write and -Read macros.
I handled this network byte order stuff by using SDL_SwapBE32, which is
defined in SDL_endian.h. These macros swap the byte order, if you are on
a little endian machine. This definitely does the job correctly.
Furthermore the SDL_SwapBE macros don’t do anything, if your host
machine’s architecture is already big endian.

Matthias

James Haley wrote:> Hello,

I hate to reply to my own message right off, but upon carefully examining SDL_net.h, it seems this is a glaringly obvious bug in the library. Observe:

#define SDLNet_Write32(value, areap)
do
{
Uint8 *area = (Uint8 *)(areap);
area[3] = (value>> 24) & 0xFF;
area[2] = (value>> 16) & 0xFF;
area[1] = (value>> 8) & 0xFF;
area[0] = value & 0xFF;
} while ( 0 )

This is the definition of SDLNet_Write32 used for little-endian hosts. It would be fantastic, except that it’s writing the bytes into the array in the completely wrong order. Let us say that value is 0xAABBCCDD. On a little-endian host this is stored as DDCCBBAA in consecutive memory bytes, however, bit shifts are NOT relative to host endianness – they are endian independent. This means that, for example, the byte (value>> 24) & 0xff is AA, not DD.

In order for this integer to be properly translated to big-endian, the array indices should be reversed. (value>> 24) & 0xff should be going to the first memory location, so that it ends up as AABBCCDD. This in fact makes it exactly the same as the definition of this macro for big-endian hosts. In case that seems unintuitive, let me again iterate that bit shifts are not endian dependent. This means that the only thing that matters is the relation of the shifts to the array indices, and for a fixed output format (in this case big-endian), the relationship will remain constant.

As for why this is differing in behavior between Windows and Linux, I cannot explain. There must be something different about the compilation that’s causing the macros to not be used…

James

Gabriele_Greco · October 17, 2007, 10:00am

James Haley wrote:

#define SDLNet_Write32(value, areap)
do
{
Uint8 *area = (Uint8 *)(areap);
area[3] = (value>> 24) & 0xFF;
area[2] = (value>> 16) & 0xFF;
area[1] = (value>> 8) & 0xFF;
area[0] = value & 0xFF;
} while ( 0 )

I’ve just looked at SDL_net.h (1.2.7) since I use it in a project and
your post frightened me and this code (that’s surely buggy, at least
in a little endian machine) is only used if SDL_DATA_ALIGNED != 0

Looking in the head of header we can see:

#if defined(sparc) || defined(mips)
#define SDL_DATA_ALIGNED 1
#endif
#ifndef SDL_DATA_ALIGNED
#define SDL_DATA_ALIGNED 0
#endif

So the code you quote is NEVER used cause both mips and sparc are big
endian machines. In the (usual) case when SDL_DATA_ALIGNED == 0 the code
of SDLNet_Write32 is simply:

*(Uint32 *)(areap) = SDL_SwapBE32(value);

So I fear that you define SDL_DATA_ALIGNED somewhere else and this is
causing your problems.–
Bye,
Gabry

cocobear · October 17, 2007, 10:07am

? Wed, 17 Oct 2007 01:18:26 -0500
James Haley ??:

Hello,

I hate to reply to my own message right off, but upon carefully
examining SDL_net.h, it seems this is a glaringly obvious bug in the
library. Observe:

#define SDLNet_Write32(value, areap)
do
{
Uint8 *area = (Uint8 *)(areap);
area[3] = (value>> 24) & 0xFF;
area[2] = (value>> 16) & 0xFF;
area[1] = (value>> 8) & 0xFF;
area[0] = value & 0xFF;
} while ( 0 )

This is the definition of SDLNet_Write32 used for little-endian
hosts. It would be fantastic, except that it’s writing the bytes into
the array in the completely wrong order. Let us say that value is
0xAABBCCDD. On a little-endian host this is stored as DDCCBBAA in
consecutive memory bytes, however, bit shifts are NOT relative to
host endianness – they are endian independent. This means that, for
example, the byte (value>> 24) & 0xff is AA, not DD.

In order for this integer to be properly translated to big-endian,
the array indices should be reversed. (value>> 24) & 0xff should be
going to the first memory location, so that it ends up as AABBCCDD.
This in fact makes it exactly the same as the definition of this
macro for big-endian hosts. In case that seems unintuitive, let me
again iterate that bit shifts are not endian dependent. This means
that the only thing that matters is the relation of the shifts to the
array indices, and for a fixed output format (in this case
big-endian), the relationship will remain constant.

It seemed that it do NONTHING with little-endian.

As for why this is differing in behavior between Windows and Linux, I
cannot explain. There must be something different about the
compilation that’s causing the macros to not be used…

James

#if defined(sparc) || defined(mips)
#define SDL_DATA_ALIGNED 1
#endif
#ifndef SDL_DATA_ALIGNED
#define SDL_DATA_ALIGNED 0
#endif

Here SDL_DATA_ALIGNED is defined.

#if !SDL_DATA_ALIGNED
#define SDLNet_Write32(value, areap)
*(Uint32 *)(areap) = SDL_SwapBE32(value);
#else
…
#define SDLNet_Write32(value, areap)

if the macro is not used, probably SDL_DATA_ALIGNED is defined as 1 in
your system.

Just a guess.I am new to SDL.> > From: haleyjd at hotmail.com

To: sdl at libsdl.org
Date: Wed, 17 Oct 2007 00:40:34 -0500
Subject: [SDL] Strange SDL_net problem - byte-swapped integers

Hi,

I’m adding network support to my Doom port, and everything works
fine as long as the two clients are running the same OS and
compiled version of the executable. But as soon as we try to mix
and match, things go crazy. We’re trying to have Linux and Win32
machines talk to each other, but here’s an example of what happens:

Sent by Linux:
1982972966 01110110 00110001 11000100 00100110

Received by Windows:
650391926 00100110 11000100 00110001 01110110

As you can see, the data has been byte-swapped. This is despite the
number in question having been transmitted using SDLNet_Write32,
and received using SDLNet_Read32. Why is an extra byte-swap
occurring to the data only when Linux is involved, and how do I fix
it? AFAIK the Linux machine is little-endian so that should have
nothing to do with it.

James

Help yourself to FREE treats served up daily at the Messenger Caf?.
Stop by today.
http://www.cafemessenger.com/info/info_sweetstuff2.html?ocid=TXT_TAGLM_OctWLtagline
_______________________________________________ SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Boo!?Scare away worms, viruses and so much more! Try Windows Live
OneCare!
http://onecare.live.com/standard/en-us/purchase/trial.aspx?s_cid=wl_hotmailnews
_______________________________________________ SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Regards!

/*
*Welcome to cocobear’s home!
*http://cocobear.cn
*/

cocobear · October 17, 2007, 10:14am

? Wed, 17 Oct 2007 12:00:06 +0200
Gabriele Greco <gabriele.greco at darts.it> ??:

James Haley wrote:

#define SDLNet_Write32(value, areap)
do
{
Uint8 *area = (Uint8 *)(areap);
area[3] = (value>> 24) & 0xFF;
area[2] = (value>> 16) & 0xFF;
area[1] = (value>> 8) & 0xFF;
area[0] = value & 0xFF;
} while ( 0 )

I’ve just looked at SDL_net.h (1.2.7) since I use it in a project and
your post frightened me and this code (that’s surely buggy, at
least in a little endian machine) is only used if SDL_DATA_ALIGNED !=
0

Looking in the head of header we can see:

#if defined(sparc) || defined(mips)
#define SDL_DATA_ALIGNED 1
#endif
#ifndef SDL_DATA_ALIGNED
#define SDL_DATA_ALIGNED 0
#endif

So the code you quote is NEVER used cause both mips and sparc are big
endian machines. In the (usual) case when SDL_DATA_ALIGNED == 0 the
code of SDLNet_Write32 is simply:
*(Uint32 *)(areap) = SDL_SwapBE32(value);
So I fear that you define SDL_DATA_ALIGNED somewhere else and this is
causing your problems.

Could it be the SDL_net that recognize his system running on “sparc” or
"mips".

–
Bye,
Gabry

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Regards!

/*
*Welcome to cocobear’s home!
*http://cocobear.cn
*/

James_Haley · October 17, 2007, 3:44pm

Now I’m not sure what’s going on because you’re definitely right. That macro shouldn’t be getting defined as 1 on my friend’s system, which is a Celeron M. But I find it highly coincidental that this incorrect code would be there (which btw would be wrong on both big and little endian machines), and I would be experiencing this sort of problem.

We’ll continue to investigate. At least I did catch something that’s actually wrong, even if my individual problem turns out to be unrelated

James> Date: Wed, 17 Oct 2007 18:14:40 +0800

To: sdl at lists.libsdl.org
From: cocobear.cn at gmail.com
Subject: Re: [SDL] Strange SDL_net problem - byte-swapped integers

? Wed, 17 Oct 2007 12:00:06 +0200
Gabriele Greco <gabriele.greco at darts.it> ??:
James Haley wrote:

#define SDLNet_Write32(value, areap)
do
{
Uint8 *area = (Uint8 *)(areap);
area[3] = (value>> 24) & 0xFF;
area[2] = (value>> 16) & 0xFF;
area[1] = (value>> 8) & 0xFF;
area[0] = value & 0xFF;
} while ( 0 )

I’ve just looked at SDL_net.h (1.2.7) since I use it in a project and
your post frightened me and this code (that’s surely buggy, at
least in a little endian machine) is only used if SDL_DATA_ALIGNED !=
0

Looking in the head of header we can see:

#if defined(sparc) || defined(mips)
#define SDL_DATA_ALIGNED 1
#endif
#ifndef SDL_DATA_ALIGNED
#define SDL_DATA_ALIGNED 0
#endif

So the code you quote is NEVER used cause both mips and sparc are big
endian machines. In the (usual) case when SDL_DATA_ALIGNED == 0 the
code of SDLNet_Write32 is simply:
*(Uint32 *)(areap) = SDL_SwapBE32(value);
So I fear that you define SDL_DATA_ALIGNED somewhere else and this is
causing your problems.
Could it be the SDL_net that recognize his system running on “sparc” or
"mips".

–
Bye,
Gabry

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Regards!

/*
*Welcome to cocobear’s home!
*http://cocobear.cn
*/

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Windows Live Hotmail and Microsoft Office Outlook ? together at last. ?Get it now.
http://office.microsoft.com/en-us/outlook/HA102225181033.aspx?pid=CL100626971033

Evan_Rinehart · October 17, 2007, 10:05pm

I don’t know if you guys know this but…

Whenever you write integers to a socket, make sure you put them in “network
byte order” first. This is a convention followed by all networked programs.
If a program gets an integer from the network, it assumes its in network
byte order and then swaps if this is opposite of the local hosts native
order. In C you would use ntoh or hton functions to convert from local to
network order, which swaps if necessary. If you wish to do this with SDL,
use SDL_SwapBE32 and the like. Obviously using the SDL functions means you
need to know that network byte order is big endian. Hope this helps.

EvanROn 10/17/07, James Haley wrote:

Now I’m not sure what’s going on because you’re definitely right. That
macro shouldn’t be getting defined as 1 on my friend’s system, which is a
Celeron M. But I find it highly coincidental that this incorrect code would
be there (which btw would be wrong on both big and little endian machines),
and I would be experiencing this sort of problem.

We’ll continue to investigate. At least I did catch something that’s
actually wrong, even if my individual problem turns out to be unrelated

James

Date: Wed, 17 Oct 2007 18:14:40 +0800
To: sdl at lists.libsdl.org
From: cocobear.cn at gmail.com
Subject: Re: [SDL] Strange SDL_net problem - byte-swapped integers

?? Wed, 17 Oct 2007 12:00:06 +0200
Gabriele Greco <gabriele.greco at darts.it> ???:

James Haley wrote:

#define SDLNet_Write32(value, areap)
do
{
Uint8 *area = (Uint8 *)(areap);
area[3] = (value>> 24) & 0xFF;
area[2] = (value>> 16) & 0xFF;
area[1] = (value>> 8) & 0xFF;
area[0] = value & 0xFF;
} while ( 0 )

I’ve just looked at SDL_net.h (1.2.7) since I use it in a project and
your post frightened me and this code (that’s surely buggy, at
least in a little endian machine) is only used if SDL_DATA_ALIGNED !=
0

Looking in the head of header we can see:

#if defined(sparc) || defined(mips)
#define SDL_DATA_ALIGNED 1
#endif
#ifndef SDL_DATA_ALIGNED
#define SDL_DATA_ALIGNED 0
#endif

So the code you quote is NEVER used cause both mips and sparc are big
endian machines. In the (usual) case when SDL_DATA_ALIGNED == 0 the
code of SDLNet_Write32 is simply:

*(Uint32 *)(areap) = SDL_SwapBE32(value);

So I fear that you define SDL_DATA_ALIGNED somewhere else and this is
causing your problems.

Could it be the SDL_net that recognize his system running on “sparc” or
"mips".

–
Bye,
Gabry

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Regards!

/*
*Welcome to cocobear’s home!
*http://cocobear.cn
*/

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Windows Live Hotmail and Microsoft Office Outlook ?C together at last. Get
it now!http://office.microsoft.com/en-us/outlook/HA102225181033.aspx?pid=CL100626971033

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

cocobear · October 18, 2007, 12:56am

? Wed, 17 Oct 2007 17:05:33 -0500
"Evan Rinehart" ??:

I don’t know if you guys know this but…

Whenever you write integers to a socket, make sure you put them in
"network byte order" first. This is a convention followed by all
networked programs. If a program gets an integer from the network, it
assumes its in network byte order and then swaps if this is opposite
of the local hosts native order. In C you would use ntoh or hton
functions to convert from local to network order, which swaps if
necessary. If you wish to do this with SDL, use SDL_SwapBE32 and the
like. Obviously using the SDL functions means you need to know that
network byte order is big endian. Hope this helps.

EvanR

It’s sure, I think everybody know it.> On 10/17/07, James Haley wrote:

Now I’m not sure what’s going on because you’re definitely right.
That macro shouldn’t be getting defined as 1 on my friend’s
system, which is a Celeron M. But I find it highly coincidental
that this incorrect code would be there (which btw would be wrong
on both big and little endian machines), and I would be
experiencing this sort of problem.

We’ll continue to investigate. At least I did catch something that’s
actually wrong, even if my individual problem turns out to be
unrelated

James

Date: Wed, 17 Oct 2007 18:14:40 +0800
To: sdl at lists.libsdl.org
From: @cocobear
Subject: Re: [SDL] Strange SDL_net problem - byte-swapped integers

? Wed, 17 Oct 2007 12:00:06 +0200
Gabriele Greco <gabriele.greco at darts.it> ??:

James Haley wrote:

#define SDLNet_Write32(value, areap)
do
{
Uint8 *area = (Uint8 *)(areap);
area[3] = (value>> 24) & 0xFF;
area[2] = (value>> 16) & 0xFF;
area[1] = (value>> 8) & 0xFF;
area[0] = value & 0xFF;
} while ( 0 )

I’ve just looked at SDL_net.h (1.2.7) since I use it in a
project and your post frightened me and this code (that’s
surely buggy, at least in a little endian machine) is only used
if SDL_DATA_ALIGNED != 0

Looking in the head of header we can see:

#if defined(sparc) || defined(mips)
#define SDL_DATA_ALIGNED 1
#endif
#ifndef SDL_DATA_ALIGNED
#define SDL_DATA_ALIGNED 0
#endif

So the code you quote is NEVER used cause both mips and sparc
are big endian machines. In the (usual) case when
SDL_DATA_ALIGNED == 0 the code of SDLNet_Write32 is simply:

*(Uint32 *)(areap) = SDL_SwapBE32(value);

So I fear that you define SDL_DATA_ALIGNED somewhere else and
this is causing your problems.

Could it be the SDL_net that recognize his system running on
"sparc" or “mips”.

–
Bye,
Gabry

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Regards!

/*
*Welcome to cocobear’s home!
*http://cocobear.cn
*/

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Windows Live Hotmail and Microsoft Office Outlook _C together at
last. Get it
now!http://office.microsoft.com/en-us/outlook/HA102225181033.aspx?pid=CL100626971033

SDL mailing list
SDL at lists.libsdl.org
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org

Regards!

/*
*Welcome to cocobear’s home!
*http://cocobear.cn
*/