Docs 2.4.4 · 4 min read

Number formats: f64, f32, f16, bf16, FP8 and int8

Apsi tensors can hold six floating-point formats and the usual integer ones. All of them work in ordinary code, in @native machine code and in @gpu kernels, with one rule that keeps results trustworthy: small formats are a way to store numbers, not to compute with them. Values are read into 32-bit registers, all arithmetic happens in f32 (or f64), and results are rounded once, correctly, when they are stored.

Making tensors

let w = zeros(4096, 4096, dtype=bf16)          # 32 MB instead of 128 MB in f64
let h = randn(1024, 1024).astype("float16")
let q = zeros(1000, dtype=fp8_e4m3)
var acc: f32 = 0.0                              # a scalar kept in f32 (also f16, bf16, f64)

bf16 and the FP8 formats come from the ml_dtypes package (the format library JAX and TensorFlow use), which is installed with Apsi. They behave like any NumPy dtype: slicing, astype, saving and loading all work.

In @native and @gpu code

@native
fn scale(x, y, a: Float):              # x, y may be f64, f32, f16, bf16, FP8 or int8 tensors
    for i in 0..len(x):
        y[i] = x[i] * a

Why this design:

• Accuracy. Adding 100,000 copies of 0.1 in f16 arithmetic stops at 256, because 256 + 0.1 rounds back to 256. With the data in f16 and the sum in f32 the total is 9,998.4; with a Float (f64) accumulator it is 9,997.55859375 — exactly 100,000 × the f16 value of 0.1. Products of two f16 or bf16 numbers are exact in f32.
• Speed. Most loops are limited by memory bandwidth, not arithmetic. Reading 2 bytes instead of 8 per element moves 4× less data; FP8 moves 8× less.
• The same answer everywhere. The conversions are written with integer operations, so every CPU (with or without F16C) and every GPU produces the same bits. They are checked against NumPy and ml_dtypes for every one of the 65,536 16-bit patterns and all 256 FP8 patterns.

Rounding details: conversions to a smaller format round to the nearest value, ties to even. NaN stays NaN. A value too large for f16, bf16 or FP8 E5M2 becomes infinity; FP8 E4M3 has no infinity, so it becomes NaN (the OCP FP8 standard and ml_dtypes do the same). Use clip(x, -448, 448) first if you want saturation instead.

Matrix products and Tensor Cores

A matrix product written as loops (see Performance) uses the fastest kernel available for its format:

cuBLAS comes with the CUDA Toolkit, or pip install nvidia-cublas-cu12 (no toolkit needed). apsi native info says whether Apsi found it.

Smaller still: FP6, FP4, int4 and others

Formats below 8 bits (fp6_e3m2, fp6_e2m3, fp4_e2m1, int4, nf4 …) are available for quantisation: quantize(x, fp4_e2m1) packs a tensor, dequantize unpacks it, @precision(fp4_e2m1) runs a function as if its values had that format, and load_safetensors(path, dtype=int4) loads weights into it. See AI and use precision.

Choosing a format

• Results you publish or bill for: f64. It costs little on a CPU.
• Large arrays where 7 digits are plenty: f32 — half the memory of f64, and far faster on consumer GPUs (an RTX 3060 does f64 at 1/64 of its f32 speed).
• Model weights and activations: bf16 for training, f16 or bf16 for inference, FP8 E4M3 when memory is tight.
• Accumulators: always f32 or f64 (var acc: f32 = 0.0), whatever the data format.

Found a mistake? Report it · Markdown source