Docs 2.4.4 · 3 min read

Tables (use dataframe)

A columnar table engine built into Apsi: typed columns, whole-column kernels, no per-cell objects. It reads CSV, JSON and JSON Lines, filters, groups, joins, sorts and writes, with an API close to Polars.

use dataframe as df

let sales = df.read_csv("sales.csv")                 # needs --allow-read for the file
let eu = sales.filter((df.col("amount") > 90) & (df.col("region") == "EU"))
let by = sales.group_by("customer").agg(total=df.col("amount").sum(), n=df.count(), avg=df.col("amount").mean())
let top = by.sort("total", descending=true).head(3)
let full = top.join(df.read_csv("customers.csv"), on="customer", how="left")
print(full)
full.write_csv("top.csv")                            # needs --allow-write

shape: (3, 5)
┌──────────┬───────┬─────┬────────┬──────┐
│ customer │ total │   n │    avg │ tier │
│ str      │ float │ int │  float │  str │
├──────────┼───────┼─────┼────────┼──────┤
│ cid      │   300 │   1 │    300 │ gold │
│ ann      │ 219.5 │   2 │ 109.75 │ gold │
│ dan      │   150 │   1 │    150 │ null │
└──────────┴───────┴─────┴────────┴──────┘

Building tables

Expressions

df.col("name") refers to a column; df.lit(v) is a constant; df.count() counts rows in a group. Expressions combine with + - * / % ** == != < <= > >= & | ~ and are evaluated column-wise, without Python loops.

• numbers: abs sqrt log exp round(n) cast("float")
• aggregates: sum mean min max std var median count first last n_unique
• windows: cumsum shift(k) rolling_mean(w)
• nulls and sets: is_null fill_null(v) is_in([…])
• strings: str_lower str_upper str_strip str_len str_contains str_startswith str_endswith str_replace(a, b)
• names: alias("new"), or keyword arguments: with_columns(double=df.col("x") * 2)
• anything else: map(fn) (runs per value — slower)

Table operations

select, with_columns, drop, rename, filter, head, tail, sample(n, seed=), sort(by, descending=), unique(subset=), group_by(…).agg(…), join(other, on= | left_on= right_on=, how="inner" | "left" | "outer", suffix="_right"), value_counts(column), describe(), concat.

Reading back: height, width, shape, columns, dtypes, row(i), rows(), to_dicts(), to_dict(), to_numpy(), to_pandas(), memory_bytes(); writing: write_csv, write_jsonl, write_json, to_json.

How it is fast

• Columns are contiguous typed arrays (int64, float64, bool, datetime64). A million numbers take 8 MB.
• Strings are dictionary-encoded: an int32 code per row into a list of unique values. Comparing, grouping and joining on strings compares integers.
• Group-by factorises the keys once and reduces with bincount / reduceat. Joins match the factorised keys by direct addressing. Filters are boolean masks.

Measured against pandas 2 on 2 million rows (AMD EPYC 7763 GitHub Actions runner (4 vCPU), best of 3, results checked equal):

Reproduce with benchmarks/dataframe/run.py and benchmarks/dataframe/csv.py.

Found a mistake? Report it · Markdown source