Pixel-art sprites in a 3D world: upright cards, cut-outs and depth

Hydro started as a 2D engine. Its test village was a flat XY plane: ground, roads, trees and houses all drawn as quads, sorted by a SortOrder and then by Y. The goal for Eonlore, the game I'm building on it, was a look that mixes 2D and 3D: pixel-art characters standing in lit 3D scenes, seen from a tilted camera.

The tempting route was a 3D rewrite. I took a smaller one: keep the world on the ground plane, and let each sprite decide how it stands up.

One field: facing

SpriteRendererComponent got a Facing:

• Flat lies on the ground, as before. Ground, roads, decals, water.
• Upright stands on the middle of its bottom edge and turns to face the camera: yaw and lean, never roll. Trees, houses, characters.
• Billboard faces the camera fully about its centre. Particles, glows, icons.

The renderer turns the flat quad about the point it stands on. A top-down camera returns the quad unchanged, so existing 2D scenes look exactly as before.

glm::mat4 SceneSpriteRenderer::ApplyFacing(const glm::mat4& quad, SpriteFacing facing,
                                           const SceneCameraBasis& camera)
{
    if (facing == SpriteFacing::Flat || camera.IsTopDown())
    {
        return quad;
    }

    // Turn the quad about where it stands: its bottom edge (Upright) or its centre (Billboard).
    const bool upright = facing == SpriteFacing::Upright;
    const glm::vec3 pivot = glm::vec3(quad * (upright ? glm::vec4(0.0f, -0.5f, 0.0f, 1.0f)
                                                      : glm::vec4(0.0f, 0.0f, 0.0f, 1.0f)));
    const glm::mat3& rotation = upright ? camera.Upright : camera.Rotation;
    return glm::translate(glm::mat4(1.0f), pivot) * glm::mat4(rotation)
         * glm::translate(glm::mat4(1.0f), -pivot) * quad;
}

Box2D physics, the lights and the blob shadows kept working unchanged, because they already lived on the ground plane. A script marked every sprite in the village as flat or upright, and pressing V eases the camera between top-down and a 50° tilt.

Sorting stops being enough

With a tilted camera, Y-sorting becomes sorting by distance along the camera's view direction. That gets the order right per sprite, but not per pixel: a character walking through a tree's branches is either fully in front or fully behind.

The fix is a depth buffer, which meant a decision I had been putting off: require Vulkan 1.3 for dynamic rendering. Pixel art has hard alpha edges, so sprites can write depth like opaque geometry if the transparent pixels are thrown away:

// Cut-out sprites: depth test and write, no blending. A pixel either covers
// what is behind it or is discarded before it writes anything.
void main()
{
    vec4 color = SampleTexture(int(texIndex), fragTexCoord) * fragColor;
    if (color.a < 0.5)
    {
        discard;
    }
    outColor = vec4(color.rgb + color.rgb * fragEmission, 1.0);
    outEntityID = entityID;
}

Trees and the player now cut each other per pixel, with no split tree sprites and no sorting tricks. Only genuinely translucent things (water, particles, glows) still need back-to-front sorting.

The bug that set the frame order

The frame now draws meshes, then decals, then cut-out sprites, then translucent quads. Decals come before the cut-outs for a reason I found the hard way: in a top-down view, standing sprites lie on the ground plane at the same depth as a decal. Drawn later, a decal passes the LESS_OR_EQUAL depth test and paints over the characters standing on it. Decals also get a small depth bias towards the camera, so they sit on ground geometry of the same height without flickering.

What I'd tell myself at the start

• Change the data model, not the world. One enum on a component did more than a 3D rewrite would have, and every old scene kept working.
• Make the old behaviour a special case of the new. Top-down is just "the camera isn't tilted", so there is one code path, not two.
• Pixel art is a gift for depth. Hard alpha means cut-outs, and cut-outs mean you rarely have to sort.

Next in this series: shadows from the same cut-out cards, and the HDR, bloom and depth-of-field stack that gives the tilt-shift look.