After submitting my RPG project Harmony Tree as CyberVGA raycaster test, I have been refactoring engine's rendering pipeline to replace the temporary raycaster with my own texture sampled wall and floor renderer. The new system uses a pinhole camera model with Q16.16 fixed-point arithmetic, since the entire project targets real MS-DOS hardware and must run efficiently on 486-class CPUs.

Writing this fixed-point library introduced two major challenges:

1. Implementing correct Q16.16 multiplication with 64-bit intermediates, and
2. Designing a reliable saturating integer division routine for perspective projection.

During this transition I encountered a subtle but highly disruptive failure mode: projected wall vertices occasionally exploded to extreme values, wrapped around the screen, or collapsed into inverted configurations. At first this looked like a coordinate-system bug, but world-space and camera-space transforms were fully correct.

To isolate the problem, I wrote a new fx_div_q16_sat routine — my saturating fixed-point divider — and examined the projection of a simple test wall. Sampling vertex B (the one that consistently produced unstable results) revealed the failure clearly.

What Actually Happened?

My projection pipeline performs the usual transformation:

screen_x = (f * x_cam / z_cam) + center_x
screen_y = center_y - (f * y_cam / z_cam)

where f is the focal length expressed in fixed-point pixels. When f ≈ 240 (for a comfortable ~75° FOV), the intermediate term:

fx_x = x_cam * f

reaches into the billions in Q16.16. That part is fine; the multiply routine uses 64-bit intermediates. However, before performing the actual (fx_x << 16) / z division, my saturating division tried to detect overflow by checking:

if (|num| >= |den| << 15) 
    overflow…

In Q16.16, den (the reciprocal depth) is commonly around 0.5 to 16. Shifting it by 15 bits overflows the 32-bit register and wraps to zero. Once that happens, the condition |num| >= 0 always evaluates true.

So the divider always reported “overflow” for valid values and returned the saturated output (INT_MIN or INT_MAX). After converting back to pixels, I was getting results like:

x ≈ -32609, y ≈ -32648

which manifested as disappearing geometry, clipping inversion, and walls jumping to the opposite side of the screen.

How It Was Fixed?

The correct overflow check was the one I already used my saturated reciprocal integer divider:

if ( (|num| >> 15) >= |den| ) 
    overflow...

which avoids shifting the denominator upward and instead reduces the numerator downward. After patching fx_div_q16_sat to use the proper test, all projections immediately stabilized. Take a look at the final result of wall transformation and projection:

... And Surprise!

Uh oh, there seems to be another problem! Even though my divisions looked correct, my test pixels were a bit misplaced than I've expected. While calculating video ram offsets for the according pixel I saw that I've mistakenly written this formula:

int vid_offset = vid_backbuffer_address + Ay * SCREEN_WIDTH + Ax

Which is almost completely wrong because I'm working on ModeX. This means VGA memory is planar and offsets should've been calculated as follows:

int vid_offset = vid_backbuffer_address + Ay * MODEX_PITCH_BYTES + (Ax/4)

where MODEX_PITCH_BYTES is 80 for the resolution of 320x240 and current x coordinate represented as Ax should've been also divided by 4 because of the VGA planar memory.

After hours of debugging and countless efforts to find multiple culprits in a pipeline, here's the final screenshot:

Final Words

With these corrections:

• Perspective projection now produces stable, precise screen coordinates.
• No more wraparound or inversion artifacts.
• Mode X pixel writes are fully accurate.
• The engine is ready for the next stage: full column rasterization with texture sampling.

This bug was deceptively simple but had far-reaching consequences. Debugging it forced me to revisit and re-validate every assumption about fixed-point math, my coordinate spaces and transformations between them, perspective correctness, and VGA Mode X memory layout. It has been a meaningful and instructive step forward.

I will continue documenting internal details as CyberVGA’s renderer evolves. Looking forward to putting this engine to real use in my game projects.