Patch: Implements ToUnicode() on 9x/ME/2K/XP for windib and directx backends

John_Popplewell · January 14, 2006, 12:34am

Hi,

I’ve been working on the Windows port for Tux Paint.

It was pointed out recently that simple keyboard composition doesn’t
work properly using SDL on the Windows or X11 back-ends.

Simple composition is where a ‘dead’ key like an acute/grave accent is
followed by an e,a,o etc. resulting in a single character.

This means that users of languages other than English have problems with
general text entry in applications that use SDL e.g. Tux Paint and
probably most of the GUIs.

Rather than trying to implement a complete solution using IM/IME (which
would allow support of Chinese, Korean, Japanese and other languages), I
thought it would be useful to fix what are effectively bugs in the
current ‘windib’ and ‘directx’ back-ends.

Many of you will be familiar with the SDL_keysym ‘unicode’ field which
is supposed to contain a 16-bit Unicode character, but on the current
Windows back-ends is actually a code-page relative 8-bit number.

This patch fixes that and shouldn’t affect existing applications too
much as they mostly use values < 128, which are unchanged.

The patch also fixes a couple of ‘windib’ problems: identifying the left
and right shift keys on Windows 9x/ME, and a problem with a missing
less-than/greater-than key found on international keyboards.

The patch is against current CVS, but would be easy to apply to the
current SDL-1.2.9.

I’ve tested it on a couple of Win98 systems (I don’t have 95 or ME), 2K
and XP. I’ve tried with UK, US, Spanish, German, and Polish keyboard
layouts.

I’m hoping that it’s possible for this to be applied to CVS,

best regards,
John Popplewell.
P.S. I’m looking at a patch for X11 and have something that nearly works

-------------- next part --------------
Index: wincommon/SDL_lowvideo.h===================================================================

RCS file: /home/sdlweb/libsdl.org/cvs/SDL12/src/video/wincommon/SDL_lowvideo.h,v

retrieving revision 1.14

diff -c -r1.14 SDL_lowvideo.h

*** wincommon/SDL_lowvideo.h 16 Feb 2004 21:09:23 -0000 1.14

— wincommon/SDL_lowvideo.h 13 Jan 2006 21:57:37 -0000

*** 108,111 ****

— 108,114 ----

 GDL_CreateWindow as well */

LONG CALLBACK WinMessage(HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam);

/* JFP: Implementation of ToUnicode() that works on 9x/ME/2K/XP */
extern int SDL_ToUnicode(UINT vkey, UINT scancode, BYTE *keystate, Uint16 *wchars, int wsize, UINT flags);

#endif /* SDL_lowvideo_h */

Index: wincommon/SDL_sysevents.c

===================================================================

RCS file: /home/sdlweb/libsdl.org/cvs/SDL12/src/video/wincommon/SDL_sysevents.c,v

retrieving revision 1.28

diff -c -r1.28 SDL_sysevents.c

*** wincommon/SDL_sysevents.c 29 Sep 2005 09:43:00 -0000 1.28

— wincommon/SDL_sysevents.c 13 Jan 2006 21:57:37 -0000

*** 710,712 ****

— 710,753 ----

app_registered = 0;

}

/* JFP: Implementation of ToUnicode() that works on 9x/ME/2K/XP */
static int Is9xME()
{
OSVERSIONINFO info;
memset(&info, 0, sizeof(info));
info.dwOSVersionInfoSize = sizeof(info);
if (!GetVersionEx(&info)) {
```
  return 0;
```
}
return (info.dwPlatformId == VER_PLATFORM_WIN32_WINDOWS);
}
static int GetCodePage()
{
char buff[8];
int lcid = MAKELCID(LOWORD(GetKeyboardLayout(0)), SORT_DEFAULT);
int cp = GetACP();
if (GetLocaleInfo(lcid, LOCALE_IDEFAULTANSICODEPAGE, buff, sizeof(buff))) {
```
  cp = atoi(buff);
```
}
return cp;
}
int SDL_ToUnicode(UINT vkey, UINT scancode, BYTE *keystate, Uint16 *wchars, int wsize, UINT flags)
{
if (Is9xME()) {
```
  BYTE	chars[2];
```

  if (ToAsciiEx(vkey, scancode, keystate, (WORD*)chars, 0, GetKeyboardLayout(0)) == 1) {

  	return MultiByteToWideChar(GetCodePage(), 0, chars, 1, wchars, wsize);

```
  }
```
}
else {

  return ToUnicode(vkey, scancode, keystate, wchars, wsize, flags);

}
return 0;
}

Index: windib/SDL_dibevents.c

===================================================================

RCS file: /home/sdlweb/libsdl.org/cvs/SDL12/src/video/windib/SDL_dibevents.c,v

retrieving revision 1.25

diff -c -r1.25 SDL_dibevents.c

*** windib/SDL_dibevents.c 29 Sep 2005 09:43:00 -0000 1.25

— windib/SDL_dibevents.c 13 Jan 2006 21:57:38 -0000

*** 55,60 ****

— 55,64 ----

#define REPEATED_KEYMASK (1<<30)

#define EXTENDED_KEYMASK (1<<24)

/* Scancodes for LSHIFT and RSHIFT keys */
#define SC_LSHIFT 42
#define SC_RSHIFT 54

/* DJM: If the user setup the window for us, we want to save his window proc,

 and give him a chance to handle some messages. */

static WNDPROC userWindowProc = NULL;

*** 90,96 ****

					wParam = VK_RSHIFT;

					prev_shiftstates[1] = TRUE;

				} else {

! /* Huh? */

				}

				break;

			case VK_MENU:

— 94,110 ----

					wParam = VK_RSHIFT;

					prev_shiftstates[1] = TRUE;

				} else {

! /* On Win98 GetKeyState() doesn’t distinguish shift-keys,

! * so try using scancode */

! Uint8 scancode = HIWORD(lParam) & 0xFF;

!

! if (!prev_shiftstates[0] && scancode == SC_LSHIFT) {

! wParam = VK_LSHIFT;

! prev_shiftstates[0] = TRUE;

! } else if (!prev_shiftstates[1] && scancode == SC_RSHIFT) {

! wParam = VK_RSHIFT;

! prev_shiftstates[1] = TRUE;

! }

				}

				break;

			case VK_MENU:

*** 143,149 ****

					wParam = VK_RSHIFT;

					prev_shiftstates[1] = FALSE;

				} else {

! /* Huh? */

				}

				break;

			case VK_MENU:

— 157,173 ----

					wParam = VK_RSHIFT;

					prev_shiftstates[1] = FALSE;

				} else {

! /* On Win98 GetKeyState() doesn’t distinguish shift-keys,

! * so try using scancode */

! Uint8 scancode = HIWORD(lParam) & 0xFF;

!

! if (prev_shiftstates[0] && scancode == SC_LSHIFT) {

! wParam = VK_LSHIFT;

! prev_shiftstates[0] = FALSE;

! } else if (prev_shiftstates[1] && scancode == SC_RSHIFT) {

! wParam = VK_RSHIFT;

! prev_shiftstates[1] = FALSE;

! }

				}

				break;

			case VK_MENU:

*** 237,242 ****

— 261,267 ----

VK_keymap[VK_EQUALS] = SDLK_EQUALS;

VK_keymap[VK_LBRACKET] = SDLK_LEFTBRACKET;

VK_keymap[VK_BACKSLASH] = SDLK_BACKSLASH;

VK_keymap[VK_OEM_102] = SDLK_LESS;

VK_keymap[VK_RBRACKET] = SDLK_RIGHTBRACKET;

VK_keymap[VK_GRAVE] = SDLK_BACKQUOTE;

VK_keymap[VK_BACKTICK] = SDLK_BACKQUOTE;

*** 341,357 ****

keysym->sym = VK_keymap[vkey];

keysym->mod = KMOD_NONE;

keysym->unicode = 0;

! if ( pressed && SDL_TranslateUNICODE ) { /* Someday use ToUnicode() */

#ifdef NO_GETKEYBOARDSTATE

	/* Uh oh, better hope the vkey is close enough.. */

	keysym->unicode = vkey;

#else

! BYTE keystate[256];

! BYTE chars[2];

	GetKeyboardState(keystate);

! if ( ToAscii(vkey,scancode,keystate,(WORD *)chars,0) == 1 ) {

! keysym->unicode = chars[0];

#endif /* NO_GETKEYBOARDSTATE */

— 366,383 ----

keysym->sym = VK_keymap[vkey];

keysym->mod = KMOD_NONE;

keysym->unicode = 0;

! if ( pressed && SDL_TranslateUNICODE ) {

#ifdef NO_GETKEYBOARDSTATE

	/* Uh oh, better hope the vkey is close enough.. */

	keysym->unicode = vkey;

#else

! BYTE keystate[256];

! Uint16 wchars[2];

	GetKeyboardState(keystate);

! if (SDL_ToUnicode(vkey, scancode, keystate, wchars, sizeof(wchars)/sizeof(wchars[0]), 0) == 1)

! {

! keysym->unicode = wchars[0];

#endif /* NO_GETKEYBOARDSTATE */

Index: windib/SDL_vkeys.h

===================================================================

RCS file: /home/sdlweb/libsdl.org/cvs/SDL12/src/video/windib/SDL_vkeys.h,v

retrieving revision 1.6

diff -c -r1.6 SDL_vkeys.h

*** windib/SDL_vkeys.h 4 Jan 2004 16:49:27 -0000 1.6

— windib/SDL_vkeys.h 13 Jan 2006 21:57:38 -0000

*** 77,79 ****

— 77,80 ----

#define VK_RBRACKET 0xDD

#define VK_APOSTROPHE 0xDE

#define VK_BACKTICK 0xDF

#define VK_OEM_102 0xE2

Index: windx5/SDL_dx5events.c

===================================================================

RCS file: /home/sdlweb/libsdl.org/cvs/SDL12/src/video/windx5/SDL_dx5events.c,v

retrieving revision 1.25

diff -c -r1.25 SDL_dx5events.c

*** windx5/SDL_dx5events.c 11 Aug 2005 05:08:28 -0000 1.25

— windx5/SDL_dx5events.c 13 Jan 2006 21:57:39 -0000

*** 824,846 ****

keysym->sym = DIK_keymap[scancode];

keysym->mod = KMOD_NONE;

keysym->unicode = 0;

! if ( pressed && SDL_TranslateUNICODE ) { /* Someday use ToUnicode() */

	UINT vkey;

#ifndef NO_GETKEYBOARDSTATE
```
  BYTE keystate[256];
```
```
  BYTE chars[2];
```

#endif

  vkey = MapVirtualKey(scancode, 1);

#ifdef NO_GETKEYBOARDSTATE

  /* Uh oh, better hope the vkey is close enough.. */

  keysym->unicode = vkey;

#else

  GetKeyboardState(keystate);

! if ( ToAscii(vkey,scancode,keystate,(WORD *)chars,0) == 1 ) {

! keysym->unicode = chars[0];

! #endif

}

return(keysym);

}

— 824,846 ----

keysym->sym = DIK_keymap[scancode];

keysym->mod = KMOD_NONE;

keysym->unicode = 0;

! if ( pressed && SDL_TranslateUNICODE ) {

	UINT vkey;



	vkey = MapVirtualKey(scancode, 1);

#ifdef NO_GETKEYBOARDSTATE

	/* Uh oh, better hope the vkey is close enough.. */

	keysym->unicode = vkey;

#else

```
  BYTE	keystate[256];
```
```
  Uint16	wchars[2];
```

	GetKeyboardState(keystate);

! if (SDL_ToUnicode(vkey, scancode, keystate, wchars, sizeof(wchars)/sizeof(wchars[0]), 0) == 1)

! {

! keysym->unicode = wchars[0];

! #endif /* NO_GETKEYBOARDSTATE */

}

return(keysym);

}

John_Popplewell · January 14, 2006, 2:16am

Ooops! Here is the batch without the extra line ending, and zipped to
hide those ugly tab characters.

I test applied it using:

$ cd cvs/SDL12/src/video/
$ patch -p0 <~/win32-unicode-fix.diff

Sorry about that,
cheers,
John.

-------------- next part --------------
A non-text attachment was scrubbed…
Name: win32-unicode-fix.diff.zip
Type: application/x-zip-compressed
Size: 2258 bytes
Desc: not available
URL: http://lists.libsdl.org/pipermail/sdl-libsdl.org/attachments/20060114/a2453642/attachment.binOn Sat, Jan 14, 2006 at 12:34:36AM +0000, John Popplewell wrote:

<snip!>

The patch is against current CVS, but would be easy to apply to the
current SDL-1.2.9.

<snip!>

I’m hoping that it’s possible for this to be applied to CVS,

best regards,
John Popplewell.
P.S. I’m looking at a patch for X11 and have something that nearly works

icculus · January 14, 2006, 7:00am

I’m hoping that it’s possible for this to be applied to CVS,

Can you take a look at the patch here…
https://bugzilla.libsdl.org/show_bug.cgi?id=39

…and see if your’s is better. I’d like to get one of these into CVS.
Alex, please chime in, too.

P.S. I’m looking at a patch for X11 and have something that nearly
works

There’s Unicode input support for X11 in CVS already; is this something
else?

–ryan.

John_Popplewell · January 14, 2006, 11:30am

I’m hoping that it’s possible for this to be applied to CVS,

Can you take a look at the patch here…
https://bugzilla.libsdl.org/show_bug.cgi?id=39

…and see if your’s is better. I’d like to get one of these into CVS.
Alex, please chime in, too.

I hadn’t seen that They are very similar, but mine returns Unicode
characters correctly on win9x as well as on NT platforms. I like the
one-off platform detection, and it would be possible to change my patch
to detect keyboard mapping changes and thereby do minimal code-page
determinations using WM_INPUTLANGCHANGE.

The patch also includes a couple of unrelated win9x windib fixes.

P.S. I’m looking at a patch for X11 and have something that nearly
works

There’s Unicode input support for X11 in CVS already; is this something
else?

The existing unicode support doesn’t seem to support composed
characters, or at least it doesn’t work on XFree86 4.3.0.1 or Xorg 6.8.2
that I have available here,

cheers,
John.On Sat, Jan 14, 2006 at 02:00:27AM -0500, Ryan C. Gordon wrote:

–ryan.

SDL mailing list
SDL at libsdl.org
http://www.libsdl.org/mailman/listinfo/sdl

slouken · January 19, 2006, 9:17am

The final version of your patch is in CVS, and you can preview it here:
http://www.libsdl.org/cvs/SDL.dll

Thanks!
-Sam Lantinga, Senior Software Engineer, Blizzard Entertainment