Projection#

This page explains how a SplatRenderer turns splats into what it draws: the methods, their costs, and the code. All of it is internal to render/. Apps choose a method and params on a renderer, its views, and its objects (Rendering, Params).

Code: src/render/SplatProjection.ts (one object per method), projections.tsl.ts (the four methods), projection.tsl.ts (their shared parts), splatCamera.ts (the camera as uniforms), splatTransform.tsl.ts (each object's transforms), splatParams.ts, SplatProjector.ts (the programs and the dispatch), splatDraw.ts (the draw material), and gpu/sortKey.tsl.ts (GpuSortKeys, in gpu/ because wgpu/ reads the keys too). All of it is TSL, so it runs on both backends.

The projection turns each splat into a few uint32 words, holding what the draw needs to place and shade a quad for every eye, plus one sort key. It is a writer body like any other, so GpuArrayWriter (a fragment pass, both backends) and WgpuArrayWriter (compute, WebGPU) can both run it, with the same bits. SplatProjector builds one program per generator; the program generates each splat, transforms it to head space (applying the modifier), and projects it.

// What SplatRenderer does on each render (internal functions)

// Its settings, with the render's view's over them
const settings = splatRenderer.settingsOf(view);

// The head plus 1 eye, or 2 eyes from an XR ArrayCamera
SplatCamera.set(renderer, camera);

const projection = SplatProjection.get(settings.method);
const arrays = new GpuArrays({
  renderer,
  bundles: projection.outputs(SplatCamera.eyes),
  storage: true,
});

projectCollection({
  renderer,
  settings,
  collection: settings.collection,
  arrays,
  pass: "full",
  cullKeys: true,
});

Generate and project in one pass#

A generator is a splat reader or a TSL function that computes splat index in the shader (Generators and modifiers). Generation and projection run in one kernel, with no intermediate splat arrays:

const ring: SplatGenerator = {
  generateSplat: (i) =>
    GpuSplats.Splat(
      centerOf(i),
      vec3(0.01),
      vec4(0, 0, 0, 1),
      colorOf(i),
      float(1),
      uint(i),
    ),
};

splatRenderer.collection = new SplatCollection([
  { generator: ring, count, matrixWorld: ringObject.matrixWorld },
]);

Each method reads one splat form (projection.input, "cov" or "splat"): ewa reads a covariance, and the others read scales and a rotation. When the generator's form differs, the program converts (GpuSplats.toCov / fromCov), so a projection only sees one kind of splat node.

Methods#

• The ray methods (3dgut, htgs, raygs) use their outline only to place the quad. Each pixel then finds where its ray passes closest to the splat's center in 3D, so the shading stays exact even where the outline is approximate.
• In the splat's whitened frame the ray is o + t d, and the density uses |o × d|² / |d|², with o × d interpolated across the quad. The mathematically equal |o|² - (o·d)² / |d|² loses all precision to float32 rounding once a splat is thousands of times its own size away.
• Inside a scene, a ray-method splat whose ellipsoid contains or crosses the camera correctly covers the whole screen. In the benchmarks that meant 800 to 4,500 fragments per pixel, against ewa's 30 to 250.

Each method has four steps: project (the words of one splat), look (rgb, opacity, and whether it is culled), vertex (one quad corner from the words), and density (per fragment). They share the LoD opacity stretch, depth of field, culling, boundFromCov, the anti-aliasing filter, and screen clipping. ewa projects covariances directly; the ray methods need a rotation and scales, so they diagonalize each cov splat once.

SplatProjection: one object per method#

SplatProjection.get(method) returns the object for a method (SplatProjection.METHODS lists them):

• input, headWords, eyeWords, and words(eyes).
• bundle(eyes) / outputs(eyes): the projected bundle (w0, w1, ...) and a projection writer's outputs (projected and sortKeys).
• project(splatNode, eyes): the TSL that projects one head-space splat; returns its words, whether any eye shows it, whether it is invisible regardless of the view (zero opacity or size), and its distance from the head.
• write(splat, eyes): the writer body that projects splat(index) and writes its sort key.
• SplatProjection.writeKeys(splat, form): a body that writes only the sort key, for any method (the CPU sort's pass). SplatProjection.setKeys(bits, cull): the key width and culling uniforms.
• For the draw: reader(eyes), loadWords(reader, eyes, index), look, vertex, and density.

The camera: head and eyes#

Every splat is read once and projected for all eyes; the loop over the eyes unrolls in JavaScript when the shader is built.

• The head is the camera in mono and the XR rig's pose in stereo. Splats reach the projection in head space. Words the eyes share are computed once, in head space: the center, color, opacity, rotation and scales, and the filter inflation.
• Each eye computes only what differs, through fromHead (a uniform per eye; the identity in mono): its 2D bound or, for raygs, its quad frame.
• The draw rebuilds eye-space data from the head words with the same fromHead. In the draw's shaders, SplatCamera.viewIndex is the eye being drawn.
• SplatCamera.set(renderer, camera) (or setViews({ head, eyes, size, ndcZMin })) sets the uniforms. The size comes from an ArrayCamera's viewport, the render target, or the canvas. It never modifies the camera: when an eye's projection uses the other clip-space convention (a camera made for WebGL used on WebGPU), the uniforms hold a converted projection.
• The ray methods inflate splats by the head's pixel footprint, exact in mono and nearly exact for each eye in stereo.

Words#

The words of one splat (the head words, then each eye's) fill the uint32x4 channels w0, w1, and w2 of the bundle projected; the last channel is uint32x2 when only 1 or 2 words remain. In the table, semicolons separate words:

• center is an octahedral direction of 20 + 20 bits (the top 16 + 16 in the first head word, the low 4 + 4 in bits 24 to 31 of the last head word) and a float32 distance from the head. The direction is within 4e-6 rad, 0.3 px at 4K through a 0.5° lens. The distance is the only depth value, and it is precise enough for ray evaluation and the sort.
• ewa's depth20 is the view depth's float32 bits above 2^-14, rounded to 20: 5 exponent bits (2^-14 to 2^18 units) and 15 mantissa bits, so steps of 1.5e-5 to 3e-5 of the distance (1.5 to 3 cm at 1 km). Its angle12 is the bound's angle on [0, π) in steps of π / 4096. The depth only places the quad for the depth test: the sort keys come from the unquantized distance.
• Quaternions in words are QuatOct32 codes.
• The ray methods' bounds are rounded outward, since their quads only need to cover the splat.
• An eye is culled when its opacity byte (ewa) or its second word (the others) is zero.
• The sort key adds 4 B per splat: 20 / 36 B (mono / stereo) for ewa and 36 / 44 B for the ray methods. Every layout fits both writers' limits of 4 channels and 64 B.

Sort keys#

The bundle sortKeys (SortKeys.KEYS) holds one RGBA8 texel per splat: a back-to-front key from the splat's distance to the head (GpuSortKeys.compute, then pack). The width (16, 24, or 32 bits) is a uniform. cullKeys (a uniform) chooses which splats get the invalid key. For the GPU sort, which compacts invalid keys away, these are the splats culled in every eye. For the CPU sort, whose order arrives late, they are only invisible splats and splats outside sortGuard, and the draw culls per eye. More: Sorting.

The draw#

• The draw has no vertex attributes, only an index buffer of 1024 quads (quadIndices: 4q + [0, 1, 2, 2, 1, 3]). It is a single draw call, and each instance draws 1024 splats (WgpuSplatSorter.QUADS_PER_INSTANCE): splat instanceIndex × 1024 + vertexIndex / 4, with the corner in bits 0 and 1 of vertexIndex. GPUs run vertex shaders in batches, and an instance per quad left most of each batch idle (10-15% of the draw).
• Because the draw is indexed, each quad's 2 shared corners are shaded once: 4 vertex shader invocations per splat, not 6.
• The order comes from the GPU sorter's values (an indirect draw) or from the applied CPU order's index list. The vertex stage loads the splat's words, rebuilds its eye-space data, and places the corner; the varyings each method's density reads are written explicitly, so the fragment stage repeats none of the vertex work.
• There is one material, built against real (empty) projected arrays from the start, and rebuilt in place when the method, the eye count, or the order source changes.
• Splats blend premultiplied "over" (one, one minus source alpha), back to front, in sRGB-encoded color as they were trained; they test depth but do not write it, so opaque meshes drawn first hide them.

Params#

SplatParams holds the defaults and the uniform nodes every program and draw reads (SplatParams.nodes, set before each dispatch and draw). SplatViewParams (maxStdDev, minAlpha, falloff) are read by the draw too, so there is one set per render. SplatObjectParams (lodInflate, blurAmount, preBlurAmount, clipXY, minPixelRadius, maxPixelRadius, minContribution, focalDistance, apertureTan, and sortGuard) are set per object. Renderers, views, and objects hold plain values. Every param and its default: Params.

With lodInflate, the projection grows a stretched splat (an opacity above 1) and stores opacity 1; without it, the stretched opacity is stored as is, and the draw treats any stored opacity above 1 as a stretch.

Source: docs/projection.md