ArrayData#

ArrayData is the table every other part of Radix stores data in: splats, LoD trees, sort keys, draw orders, and any data of an app's own. Rows are elements and columns are typed-array channels, grouped into named bundles. A table can be sent to workers without copying, saved to RDX files, and uploaded to the GPU (GpuArrays) row for row.

Code: src/core/ (formats.ts, bundles.ts, ArrayData.ts, layout.ts), in @sparkjsdev/radix/core.

const POINTS = defineBundle({ position: "float32x4", color: "unorm8x4" });

const data = new ArrayData({ bundles: { points: POINTS }, count: 1000 });
const { position, color } = data.bundles.points.channels;

// position.data is a Float32Array of capacity * 4 floats, and color.data a
// Uint8Array, where offset 4 is element 1
position.data[0] = 1.5;
color.data.set([255, 128, 0, 255], 4);

Formats#

A format is a scalar type, optionally with a component count: "uint32", "uint32x4", "unorm8x4", "float16x2".

• Formats with 1, 2, or 4 components can be stored on the GPU. Other counts ("uint32x5", "float32x3") stay on the CPU.
• ArrayFormats.info(format) returns a FormatInfo: scalar, components, scalarBytes, bytes, array (the typed array class), and gpu (whether the GPU can store it).
• ArrayFormats.make(scalar, components) builds a format from its parts.
• Types: ArrayFormat, ScalarType, TypedArray, TypedArrayConstructor, and ArrayOf<F> (the typed array of format F).

Bundles#

A bundle is a named set of channels that are used together: a codec, a reader, or a file section binds to all of them at once. Its definition is plain data.

// The definition of Splats.EXT
const EXT = defineBundle({ extA: "uint32x4", extB: "uint32x4" });

// 2 elements per GPU texel
const PAIRS = defineBundle({ pair: "uint32x2" }, { packing: 2 });

// Rebuilt after loading, so not saved to RDX
const DERIVED = defineBundle({ data: "uint32x5" }, { serialize: false });

• The definition's type keeps the channel names and formats, so data.bundles.extSplats.channels.extA.data is typed as a Uint32Array.
• ArrayData.INDICES ({ index: "uint32" }, packing 4) is the standard bundle for lists of element indices: draw orders, and subsets such as LoD subsets.
• Types: BundleDef, BundleChannels, and Schema (bundle names to definitions).

ArrayData#

const data = new ArrayData({
  bundles: { extSplats: Splats.EXT },
  count: 1000,
}).addBundle("pairs", PAIRS);

// A Uint32Array of capacity * 4
data.bundles.extSplats.channels.extA.data;

// Grows every bundle, keeping the contents
data.ensureCapacity(5000);

Every bundle of one table has the same rows. A LoD tree with one node per splat is a bundle next to the splats; data with a different row count, such as a draw order, goes in a table of its own.

Types: ArrayBundle<D>, ArrayBundles<S>, ArrayChannel<F>, and TransferableArrayData.

Sending to workers#

// A plain object holding the same arrays
const t = data.toTransferable();

// Moves the buffers rather than copying them
worker.postMessage(t, ArrayData.transferables(t));

// In the worker
const received = ArrayData.fromTransferable(t);

• After sending, data holds detached arrays and must not be used until its arrays come back.
• data.setFromTransferable(back) puts the returned arrays back into data in place. Bundles and channels with the same names and formats keep their objects, so references held elsewhere stay valid.
• ArrayData.transferables(value) collects the ArrayBuffers inside any value, for postMessage's transfer list. WorkerPool uses it for every call (Workers).

Double buffering#

a.swap(b) exchanges the contents of two tables. An app can read a front table while it fills the back one, then swap them; GpuArrays.swap does the same on the GPU.

TextureLayout#

TextureLayout maps elements to texels of a 2D array texture: 1024 texels per row, a power-of-two height up to 4096, and 2 to 512 layers. The CPU side uses it to size capacities, and the GPU side for uploads, readbacks, copies, and shader addressing.

The rule for the shape and the addressing formula are in GPU arrays.

Gotchas#

• Capacities are multiples of 4096. A table of 10 rows allocates 4096.
• Arrays passed to a worker are detached until they come back. Hold on to the channel object (bundle.channels.extA) rather than its data array: setFromTransferable keeps channel objects of the same format but puts the returned arrays in them.
• Bundles with packing 2 or 4 need starts that are multiples of the packing when uploaded, copied, or read on the GPU.

Source: docs/arraydata.md