RenderGeometryRaw producing different results in Metal vs OpenGL

I’m trying to get Nuklear working with the SDL2_Renderer as its backend. There’s a demo of this in their repo at demo/sdl_renderer which I managed to compile and run. However, the window border and title doesn’t show up with Metal as the render driver, but it does with OpenGL as the render driver.

Here’s an image showing the difference, OpenGL on the left and Metal on the right:

I did an error check on the return from SDL_RenderGeometryRaw but the result was 0 at every call. I’m not sure how to debug this issue; it’s strange to me that some of the geometry (the widgets, in particular) does show up but the rest doesn’t (consistently so).

Some details of my setup: macOS Monterey 12.2.1, ARM64 (M1), Nuklear 4.9.6, SDL2 2.0.20 from Homebrew, clang 12.0.5 (target arm64-apple-darwin21.3.0)

I’m testing some hypotheses:

  1. Hypothesis: the first draw command each frame gets ignored for whatever reason. So, I threw in a synthetic RenderGeometry call with a single degenerate triangle before any Nuklear render calls. No change.

Next, I stepped through the nk_draw_for_each loop and rendered each iteration in a separate frame. What I found is that three calls were made each frame, one to draw the window background, the next to draw the widgets, and then finally one to draw the window border and that triangle at the bottom-right corner. Again, with OpenGL, the window and its border show up, but with Metal, only the widgets show up. This is still odd, because the widget geometry is more complex. Rounded corners = good, simple rectangles = bad.

  1. Hypothesis: the triangles are wound the wrong way. So I iterated through the element indices, swapping every first and third. No change. Triangle winding is probably not an issue for 2D rendering.

  2. Hypothesis: the renderer wasn’t being flushed often enough. So I manually flushed it. No change.

  3. Hypothesis: one bad triangle was messing up the whole bunch. So I sent every triangle to RenderGeometryRaw separately. No change. Fused this with hypothesis 3 and flushed after every triangle also. No change.

@kbolino
looks strange …
if you compile SDL2 yourself, can you hard-code some
constant color in
src/render/metal/SDL_render_metal.m
METAL_QueueGeometry

eg, something like 0x10101010
( around: *((SDL_Color *)verts++) = color; )

if you’re not compiling SDL2 yourself,
try to call some SDL_RenderFillRect with various color maybe …

1 Like

Great suggestion!

Thanks to it, I found the problem: the clipping rectangle. The window background/border draw commands set a cliprect of {-8192, -8192, 16384, 16384} which worked fine in OGL and D3D but apparently confuses Metal. I clamped the cliprect to the viewport and it draws properly differently in Metal now.

This clamping might be causing other problems though… I didn’t think through the min and max. Anyway, this worked:

r.x = NK_CLAMP(viewport.x, (int)(cmd->clip_rect.x), viewport.w);
r.y = NK_CLAMP(viewport.y, (int)(cmd->clip_rect.y), viewport.h);
r.w = NK_CLAMP(0, (int)(cmd->clip_rect.w), viewport.w);
r.h = NK_CLAMP(0, (int)(cmd->clip_rect.h), viewport.h);

Results:

Addendum: it was not necessary to clamp w or h, only x and y had to be clamped. Also, I wouldn’t trust my choice of bounds above, since the cliprect is relative to the viewport. This is the better clipping choice:

r.x = NK_MAX(0, (int)(cmd->clip_rect.x));
r.y = NK_MAX(0, (int)(cmd->clip_rect.y));
r.w = cmd->clip_rect.w;
r.h = cmd->clip_rect.h;

@kbolino
Just find some message about a “render pass height” max value for scissor commands …

So it would be better to patch / fix SDL2 :

add traces there, to see what is really set.

change {x, y, w, h}
to something ?
if (x < 0) { w += x; x = 0}
if (y < 0) { h += y; y = 0}

The width and height change you suggest makes sense. It’s not necessary in this case (the cliprect being set here seems to mean “the entire viewport”) but would in other cases.

I haven’t built SDL from source, and I’ve never written any Objective-C, but I might be able to dig into it later.

Yeah, Metal doesn’t like it when the scissor rectangle is outside the viewport bounds IIRC.

The interesting thing is that, on macOS and iOS, OpenGL is now implemented as a layer on top of Metal. So the OpenGL layer must be validating the scissor rect, where Metal just tries to use it as-is.

If you try launching your app from Xcode, you can turn on Metal API validation and see what that says (up top, in the center status bar thing, where it says “YourApp > My Mac”, click on that, pick Edit Scheme, then under the Diagnostics tab turn on Metal API validation if it isn’t already)

This is what I’m seeing with the original code:

-[MTLDebugRenderCommandEncoder setScissorRect:]:3635: failed assertion `Set Scissor Rect Validation
(rect.x(18446744073709543424) + rect.width(16384))(8192) must be <= render pass width(2400)
(rect.y(18446744073709543424) + rect.height(16384))(8192) must be <= render pass height(1600)

If I just ensure the x and y coords are not negative, I get:

-[MTLDebugRenderCommandEncoder setScissorRect:]:3635: failed assertion `Set Scissor Rect Validation
(rect.x(0) + rect.width(16384))(16384) must be <= render pass width(2400)
(rect.y(0) + rect.height(16384))(16384) must be <= render pass height(1600)

If I also apply the adjustment to the width and height suggest by @Sylvain_Becker, I get:

-[MTLDebugRenderCommandEncoder setScissorRect:]:3635: failed assertion `Set Scissor Rect Validation
(rect.x(0) + rect.width(8192))(8192) must be <= render pass width(2400)
(rect.y(0) + rect.height(8192))(8192) must be <= render pass height(1600)

Finally, if I incorporate knowledge of the viewport and clamp the width and height of the clip rect, the failed assertions go away. This is the adjustment code I had to apply in total:

if (r.x < 0) {
    r.w += r.x;
    r.x = 0;
}
if (r.y < 0) {
    r.h += r.y;
    r.y = 0;
}
if (r.w > viewport.w) {
    r.w = viewport.w;
}
if (r.h > viewport.h) {
    r.h = viewport.h;
}

I’m not 100% sure the viewport is the right choice here; it’s unchanged from the default so happens to be identical to the renderer output.

I might peak under the hood tomorrow and see how viewports and clip rects are handled. It might also be necessary to clamp the viewport coordinates and dimensions.

Yeah, 8192 x 8192 is definitely outside the viewport bounds :stuck_out_tongue_winking_eye:

And yeah, the viewport rect should probably be clamped to match or fit inside the drawable size.

edit: it’s also kind of weird that Nuklear wants to set the scissor rect to such huge values.

I think this is fixed with : METAL: clip rect w/h must be <= render pass · libsdl-org/SDL@3bebdac · GitHub

2 Likes

Looks like that did it.

I pulled main on SDL and reset Nuklear to before I patched the example and it rendered properly with no Metal validation errors showing up.