How Radix projects, sorts, and draws several splat objects together, and why the API uses collections, a renderer with per-viewpoint views, orders, and object releases, on WebGPU and on the WebGL2 fallback.
Each part builds on the previous one. For reference: the design notes, and the API in rendering, views, and lifetimes.
A scene can contain several splat objects: loaded files, Level-of-Detail (LoD) subsets of files (the splats of a LoD tree chosen for the current camera, drawn through an index list), and splats generated in a shader. Each object has its own transform, and some move every frame. Splats are semi-transparent and must be blended back to front. Where two objects overlap, their splats have to be sorted as one list.
If each object is sorted and drawn separately, the object drawn second covers the first everywhere, including where it is behind it. If all splats are sorted together, every splat is blended in the correct order.
Requirement: all objects drawn by a renderer are projected into one shared collection, sorted once, and drawn with one draw call. The rest of the design follows from this requirement and from one limitation of WebGL2: it can't sort on the GPU (part 5).
On WebGPU the sort runs on the GPU in the same frame as the projection and the draw. Each render draws the renderer's current collection, from whichever camera it is given.
const collection = new SplatCollection([
// a loaded file
SplatCollection.arrays(scan, { matrixWorld: scanObject.matrixWorld }),
// splats generated in a shader
{ generator: sparkles, count: 5000, matrixWorld: emitter.matrixWorld },
]);
const splatRenderer = new SplatRenderer({ renderer, collection });
scene.add(splatRenderer);
renderer.render(scene, camera);
// projected and sorted again
renderer.render(scene, otherCamera);
// drawn on the next frame
splatRenderer.collection = new SplatCollection([/* other objects */]);
A WebGPU-only app does not need to handle:
release runs as soon as a new collection no longer contains it.Two classes: SplatCollection holds what to draw, and SplatRenderer draws it with the settings every viewpoint shares. The following parts explain what else is needed when the same app also runs on WebGL2: orders, and a SplatView per camera to hold them.
See also: SplatRenderer Quickstart
A SplatCollection holds objects. Each object has a generator (the shader code that produces splat i), the generator's data, a splat count, and a matrixWorld: the object's world transform (object to world, for example an Object3D's matrixWorld). Each render, the renderer runs one GPU pass per object and writes its output at the object's offset in one shared projected collection.
Before projection, each splat is transformed into camera space (head space in XR). The projected output is stored relative to the camera: a direction, a distance, and a screen-space shape. It does not depend on the object's world transform. Objects with different world transforms can therefore share one collection, and the sort key, the distance from the camera, is comparable across all objects.
Settings that the draw reads must be the same for the whole collection: the projection method, the number of sort key bits (sortBits: 16, 24, or 32), maxStdDev, minAlpha, and falloff. These are set on the renderer (a view can set its own params for its renders; the method stays the renderer's). Settings used only during projection can differ per object: blur, depth of field, and sortGuard. The renderer's objectParams holds their defaults, a view's can override them, and object.params overrides both.
Transforms: rotations with uniform scale use a fast path. Non-uniform scale, shear, and mirroring are applied to the splat's covariance. A uniform flag selects the path, so changing a world transform never rebuilds a program.
See also: SplatCollection Transforms
Each object's pass is a GPU program that generates the splat, transforms it, optionally applies a modifier, and projects it. Building a program can take long enough to drop frames, so Radix caches programs and reuses them. Values that change often are passed to the program as bound data or uniforms.
fromSplats (files) and sparkles.Programs are cached by generator, modifier, projection method, eye count, and backend. Only changes to these build new programs. Loading another file, using a new index list, moving an object, or changing blur, depth of field, or the drawn SH degree does not.
Before each object's pass the renderer sets that object's uniforms: its transform, its params, and the generator context (the view origin and the object's transforms in its own space), which every generator receives as an argument. Three.js writes each pass's uniforms when the pass is encoded, so one program draws any number of objects, each with its own values: this is how spherical harmonics are seen from the right place for every object and every camera.
Objects are not combined into one pass because each object binds its own textures, and a shader stage can bind at most 16. Combining many objects into one pass would require storing their data together in shared arrays (a splat atlas, like a texture atlas), which a generator could read in the future.
Create generators once. A generator is a program; its data is bound before each pass. A single fromSplats({ remap: "indices" }) generator can draw every file of a layout through any index list. A new LoD subset is new data, not a new program.
See also: The program cache Generators
WebGL2 has no compute shaders, so the sort can't run on the GPU. Radix computes the sort keys on the GPU, reads them back, and sorts them on a worker thread. This takes several frames.
Until the new order arrives, the renderer draws with the previous order. Splats are still projected every frame, so their positions are current; only the draw order is a few frames old, which is rarely visible.
An order is a list of positions in the projected collection, for example: draw position 7, then 2, then 4. Position 7 refers to a splat in one particular collection of objects.
Orders and collections: each order records the collection it was made for, and Radix keeps drawing that collection until a newer order is available. Parts 6 and 7 describe how.
See also: The CPU sort
SplatCollection is the collection of objects an order belongs to. A collection can't be modified after it is created, so checking whether an order belongs to a collection is an object comparison. To draw a different set of objects, create a new collection.
Versioning: the collection object is its own version. No separate counter is needed.
See also: SplatCollection
On WebGL2 each camera gets a view (splatRenderer.viewFor(camera), made for every camera the renderer renders) that holds its order. view.sort(camera) returns an order for the collection that was current when the sort started, and view.apply(order) draws with it. autoSort does both automatically: one sort at a time per view, and a new sort whenever the camera, an object, or the collection has changed, or when the app calls requestSort().
const view = splatRenderer.viewFor(camera); // order.collection: the collection it was made for const order = await view.sort(camera); // draws order.collection with this order from now on view.apply(order); // sort again at the next render (also with autoSort off) view.requestSort();
The chart follows one view for 120 frames with autoSort, and each color is one collection. The app sets a new collection at a fixed interval (top row), for example because it recomputes a LoD subset as the camera moves. Each bar in the middle is one sort, in the color of the collection it sorts: it starts with the latest collection and takes a number of frames. When it finishes, its order is applied and the bottom row, the collection actually drawn, switches to that color (the thin line). The bottom row is the top row delayed by the sorts; collections that change while a sort runs are skipped.
order.version counts the view's sorts in the order they start.view.sort() calls on a pool of several workers), sorts can finish out of order. An order that finishes after the order of a later sort was applied is discarded. At most one finished order waits to be applied.autoSort keeps one sort in flight. Each new sort uses the latest collection and camera, so no worker time goes to orders that would be discarded.requestSort() (on a view, or on the renderer for every view) asks for a new sort when something changed that autoSort can't detect, such as splats produced by a generator moving or a generator's uniform changing. With autoSort: false it is the only trigger. If a sort is in flight, the new sort starts when it finishes, because the running sort may have read its sort keys before the change.onInvalidate: it is called when an order is applied (and on requestSort()), so the app renders the frame that shows it.Delay: a camera's first renders draw nothing until its first order arrives. After that, the display is at most one sort behind the latest change.
On WebGL2 an old collection can still be drawn or sorted after the app has replaced it. The app can't determine on its own when an index list or a file is no longer in use. Freeing it too early causes incorrect rendering or errors.
On WebGPU, a renderer's only live collections are its current collection and its views' own collections. On WebGL2, its live collections also include each view's collection currently drawn and any collections being sorted. An object's release callback runs once no live collection of any renderer contains that object.
// Once: a generator that reads stored splats through an index list,
// and a pool of index lists
const listed = SplatGenerator.fromSplats({ remap: "indices" });
const pool = [];
// Each time the app picks a new LoD subset: `chosen` holds the indices
// of the splats to draw, written into a pooled index list if there is one
const indices = SplatGenerator.indices(renderer, chosen, pool.pop());
const subset = SplatCollection.object({
generator: listed,
// the file's splat arrays (kept) and this subset's index list
data: { splats, indices },
count: chosen.length,
matrixWorld: model.matrixWorld,
// once nothing draws or sorts it: back to the pool
release: () => pool.push(indices),
});
splatRenderer.collection = new SplatCollection([subset]);
dispose(), a warning is logged and its objects are released.Backend difference: on WebGPU only the current collections are live, so a release runs as soon as a collection is replaced. On WebGL2 the same code releases a few frames later, and a pool needs a few more index lists.
See also: Object releases
Most of what a renderer draws is the same from every camera: the collection, the method, the params, and the modifier. These are the SplatRenderer's. A WebGL2 order is only valid for one camera position, though, and two cameras may draw different LoD subsets of the same data. That per-viewpoint part is a SplatView: splatRenderer.viewFor(camera) makes one on first use, holding the camera's order and any settings it changes.
// the renderer's settings
renderer.render(scene, leftCamera);
// the camera's view: its collection and modifier over the rest
splatRenderer.viewFor(rightCamera, { collection: rightSubset, modifier: tint });
renderer.render(scene, rightCamera);
// an explicit view: the same camera, drawn differently
splatRenderer.view = splatRenderer.viewFor("depth", { modifier: depthColor });
renderer.render(scene, leftCamera);
// back to choosing by camera
splatRenderer.view = undefined;
splatRenderer.view if the app set one, otherwise the view of its camera, otherwise the renderer's own settings. With the CPU sort the renderer makes a view for every camera it renders, because each camera needs its own order; with the GPU sort it makes none.splatRenderer.viewKey = (camera) => key maps cameras to view keys (one key for all of them, or a stable id per viewpoint), and setting splatRenderer.view draws one view for every render. The renderer warns once renders have made more than 32 views.SplatRenderers drawn in one render() each have their own projected collection and sorter, because three.js runs all compute work of a frame before its draws.A renderer's modifier (or a view's own, or null for none) is a TSL function applied to every splat it draws, similar to three.js's overrideMaterial. Examples are depth coloring, normal visualization, and wave displacement. The modifier runs in a selectable space. Changing the space only changes matrices, not the program.
modifier, covModifier, or both; the program converts between forms only where necessary.See also: Modifiers and their spaces
The same app code runs on both backends. The difference is how quickly changes appear. In this example, the app adds and removes splat objects over time (streaming in parts of a scene, say), so each change is a new collection, while the camera orbits.
| WebGPU | WebGL2 | |
|---|---|---|
| objects added or removed (a new collection) | drawn or gone on the next frame | when the sort started after the change finishes, a few frames later |
| a collection changed every frame | every change drawn | one change drawn per sort; the ones in between are skipped |
| first frames of a new camera | drawn immediately | empty until its first order arrives |
| views | only to draw a viewpoint differently | one per camera (made by the renderer) |
| camera movement | exact order every frame | positions current, order a few frames old |
| index lists a pool needs (part 8) | 2: the one in use, and the next one, filled before the old one is released | a few more: the drawn one and the ones being sorted are in use too |
| extra threads | none | a pool per renderer, a worker per sort in flight |
// WebGPU, else WebGL2
const renderer = await createRenderer();
// sortMode: "gpu" on WebGPU, "cpu" on WebGL2
const splatRenderer = new SplatRenderer({
renderer,
collection,
onInvalidate,
});
scene.add(splatRenderer);
// autoSort is on by default
renderer.render(scene, camera);
One code path: write the app with collections and object releases as if WebGL2 support were required, and let the renderer make the views it needs. On WebGPU this adds no cost, and on WebGL2 the same code works with a delay of a few frames.
See also: One code path for both backends
Each of these designs was considered and rejected because it violates one of the requirements above.
See also: Dropped designs
SplatCollection only when the set of objects changes.count on every object. Free data an object owns in its release callback, not when replacing the collection.viewKey (or view) so they share views.dispose() on renderers that are no longer needed, and on views of cameras or keys that are gone.autoSort handles sorting automatically, and onInvalidate tells an on-demand app when to render.See also: What this asks of an app Lifetimes