Multi-splat design#

Radix draws many splat objects in one sort through collections, a renderer with per-viewpoint views, versioned orders, and object releases. What each part does is described in Rendering, Views and CPU orders, Lifetimes, and Projection. This page explains why the parts exist: what each one is for, which constraint requires it, and which simpler designs were tried first and dropped. multi-splat.html covers the same material step by step, with interactive diagrams.

The model (generators, immutable collections, a renderer with per-viewpoint views, versioned orders, and object releases) has more parts than drawing splats might seem to need. Each part addresses a constraint that a simpler design does not meet:

An app that targets only WebGPU needs little of this: it creates a SplatRenderer of a collection, adds it to the scene, and renders from any camera. The sort runs on the GPU in the same frame, so nothing depends on the camera between renders and no views are needed.

const collection = new SplatCollection([
  SplatCollection.arrays(splats, { matrixWorld: model.matrixWorld }),
]);
const splatRenderer = new SplatRenderer({ renderer, collection });
scene.add(splatRenderer);

renderer.render(scene, camera);

The constraints#

1. Splats must be sorted together#

Gaussian splats are blended back to front. Two objects drawn in separate draw calls, each sorted separately, blend incorrectly wherever they overlap: one is drawn entirely over the other, even where their splats interleave in depth (for example a hand inside a cloud, or two scans of the same room). Therefore every object a renderer draws is projected into one shared set of projected splats, sorted as one list, and drawn in one draw call.

A single draw call requires some settings to be the same for every splat in it:

2. Each object has its own transform and its own generator#

An object can be a file read from arrays, a subset of a file drawn through an index list, or a TSL function that generates splats in the shader (a ring, a particle system, or an effect). A typical subset is a Level-of-Detail (LoD) subset: the splats of a LoD tree chosen for the current camera. Radix does not build LoD trees itself: the app picks the subset and passes its index list (SplatGenerator.indices). Each object has its own world transform (its matrixWorld: object to world, such as an Object3D's matrixWorld), which can change every frame.

One shared set is possible because the projection's output never depends on an object's world transform. Every splat is first transformed into head space (the camera in mono, the XR rig's pose in stereo), and the words store head-relative data: a direction and distance from the head, a rotation and scales in head space, and 2D bounds per eye. The draw reads only the head-to-eye matrices. The sort key is the distance from the head. Therefore any mix of objects with any transforms projects into the same set and sorts correctly, for all four methods.

The per-object transform is applied once per splat, during projection:

3. Shader compilation is slow, and the same objects are drawn every frame#

Compiling a pipeline stalls the frame in which it happens, and a projection program is large (a generator, transforms, a modifier, and a method). A design that compiles every frame, or for every object every frame, is not usable. Therefore:

4. Batching across objects is limited#

Each object costs one dispatch. Running many objects in one program would reduce dispatches, but WebGPU's baseline allows 16 sampled textures per shader stage, and one stored splat file already uses 2 or more. Batching therefore has to happen in the data, not in the program: a splat atlas stores many objects together in one large GpuArrays (like a texture atlas), with a table of per-object matrices, read by a single generator. This fits the model as another generator. It is not built yet; one dispatch per object is sufficient for tens of objects.

5. Compute runs before the frame's draws#

three.js submits each compute dispatch immediately, but submits a render's draws only when render() finishes. Two SplatRenderers in one scene therefore both project and both sort before either one draws. If they shared a projected buffer or a sorter, both would draw the second renderer's results. Therefore each SplatRenderer owns its projected splats and its WgpuSplatSorter. Only things that are not written per draw are shared: the program cache, the uniforms, and the quad index buffer. Rendering one renderer from several cameras in separate render() calls works: each call submits its compute and then its draws, in order.

6. WebGL2 has no compute shaders, so its sort arrives late#

On WebGPU the sort runs on the GPU in the same frame as the projection, so any collection is drawn immediately. On WebGL2 there is no compute shader to sort with. The fallback computes the sort keys on the GPU (in a fragment pass), reads them back asynchronously, and sorts them on a worker. Sorting never runs on the main thread, because a large sort would drop frames. The resulting order arrives several frames later.

An order is a list of indices into the projected set: index 5,000 means "the 5,000th splat of this exact collection of objects, in this order, with these counts". If an object is added or removed, or changes its count or its LoD subset, the same indices refer to different splats, and drawing with the old order draws the wrong splats. A camera move is different: the order becomes slightly out of date, but every index still refers to the same splat, so a late order is still usable.

The rest of the model follows from this constraint.

7. Several viewpoints#

Two cameras (split screen, a mirror, a thumbnail, or an offline still) need two orders on WebGL2, because an order is sorted for one camera position. They can also draw different things: their own LoD subsets of the same file, or a different modifier. Most of what a renderer draws is the same for every viewpoint, though, and on WebGPU only the camera differs. Therefore a viewpoint holds only what is its own: its order, and any settings it changes. One XR stereo pair is a single viewpoint, because sort keys are distances from the head and an order is valid for every direction at one position.

8. Effects that apply to every splat#

Effects such as depth coloring, normal coloring, and a wave in a gizmo's frame should apply to everything a renderer draws, like three.js's scene.overrideMaterial, without editing every generator. Switching between them each frame (RGB, then a depth thumbnail) should compile nothing after the first time.

How the model follows from the constraints#

SplatCollection: the collection an order is sorted for#

Constraint 6 requires the collection of splats to be a value that can be stored, so that an order can record which collection it belongs to and a viewpoint can keep drawing that collection. A SplatCollection is an immutable collection of objects: it makes each object's generator, data, and count read-only on the object itself, without copying, and computes offsets once, so the order of its objects matters.

SplatRenderer: the settings every viewpoint shares#

The scene mesh holds what is the same for every viewpoint: the collection, the method and draw params (constraint 1), per-object param defaults, the modifier (constraint 8), the sort mode, autoSort, and its own projected arrays and sorter (constraint 5). In each render it projects its collection for that render's camera, sorts it on the GPU or uses the viewpoint's CPU order, and draws. On WebGPU an app needs nothing more: one renderer can be rendered from any number of cameras.

SplatView: what one viewpoint changes#

A view is one viewpoint of a renderer, made on first use by splatRenderer.viewFor(key), where the key is a camera or any value of the app's. It holds the viewpoint's CPU order (constraints 6 and 7) and any settings it changes; unset ones follow the renderer, field by field, and an object's params override both. A render draws the explicit splatRenderer.view if the app set one, else its camera's view, else the renderer's own settings.

SplatOrder and versions: using every finished sort#

In CPU mode a view draws view.order.collection, not the current collection. Setting splatRenderer.collection (or a view's collection) specifies what to draw next; the old collection is still drawn until an order for a newer one is applied. The first frames, before any order exists, draw nothing.

// The applied order's collection is still drawn
splatRenderer.collection = next;

// Sort keys of `next` computed on the GPU now, sorted on a worker
const view = splatRenderer.viewFor(camera);
const order = await view.sort(camera);

// Drawn from now on, unless a later sort's order is shown
view.apply(order);

Several rules for choosing which order to show were considered:

Sorts run on one worker pool shared by every renderer and view, SplatRenderer.sortPool, so views sort in parallel up to its size (SplatRenderer.sortWorkers, 4 by default), the number of parallel sorts stays bounded however many renderers and views a page has, and sorts never wait behind file loads in WorkerPool.default. It is created on the first CPU sort and makes workers as sorts need them. Alternatively, sortWorker points a renderer at a pool the app owns.

On-demand rendering#

An app that renders only when something changed needs to know when Radix has something new to show. With the GPU sort nothing happens between renders, so only the app's own changes matter, and the app knows about those. With the CPU sort an order lands between renders, and the next sort can only start in a render (it needs the render's camera). onInvalidate(view?) is called whenever Radix did or was asked to do something that needs a render: an order applied, or a sort requested. The loop then stops by itself: the order lands, the app renders, that render starts another sort only if the camera moved meanwhile, and rendering stops when nothing changes.

Object releases: freeing what an old collection still uses#

Because a viewpoint can keep drawing an old collection for several frames, and sorts can be in flight for collections the app has already replaced, the app cannot know when it is safe to free an index list or a file's arrays. Freeing something that an applied order still draws causes a crash or incorrect output on screen.

A renderer's live collections are:

Therefore an object can have a release callback, and Radix calls it once that object is in no live collection of any renderer:

const subset = SplatCollection.object({
  generator: listed,
  data: { splats, indices },
  count: chosen.length,
  matrixWorld,
  // Return the LoD subset's index list to the pool; the splats stay
  release: () => pool.push(indices),
});
splatRenderer.collection = new SplatCollection([subset]);

Radix determines only when to release, which is the part the app can't see. What to free, and how, is up to the app: Radix never looks inside data and frees nothing itself.

The first design was a sweep that tracked the generators and data in live collections and freed whatever the app had released once nothing referenced it. Finding what an object uses required looking inside its data, which works only for data shapes Radix knows, and freeing shared resources for the app made Radix responsible for lifetimes it has no information about. Reference counts have the same problem, and the counts can also be wrong.

Modifiers and their spaces#

A renderer's modifier(splat) (or covModifier(cov); a view can set its own, or null for none) is a TSL function applied to every splat it draws, before projection, so the draw material never changes. It is part of the program cache key, so define it once; switching between a few stored modifiers compiles nothing after the first frame of each.

Dropped designs#

Each of these designs was considered while developing the model, and each fails one of the constraints above or asks more of the app:

What this asks of an app#

Source: docs/multi-splat.md