Docs 2.4.4 · 8 min read

Hugging Face models without Python (use hub)

use hub downloads models from the Hugging Face Hub, tokenizes text and runs the models natively — embeddings, classification, named entities, fill-mask and text generation — with Apsi's own code. You do not need Python's transformers, torch, huggingface_hub or tokenizers packages, and you do not need --allow-python.

use hub

let sentiment = hub.pipeline("sentiment-analysis", "distilbert/distilbert-base-uncased-finetuned-sst-2-english")
print(sentiment("Apsi is a joy to use"))                 # {"label": "POSITIVE", "score": 0.99…}

let st = hub.model("sentence-transformers/all-MiniLM-L6-v2")
print(st.similarity("A cat sits on the mat", "A kitten is on the rug"))
let vectors = st.embed(["first text", "second text"])     # [2, 384], normalised like sentence-transformers

let lm = hub.model("HuggingFaceTB/SmolLM2-135M")
print(lm.generate("The three laws of motion are", max_new_tokens=40))
lm.generate("Once upon a time", max_new_tokens=60, temperature=0.8, top_p=0.95, stream=fn(t) => print(t, end=""))

use hub

print(hub.search("sentence similarity", task="sentence-similarity", limit=5))
print(hub.files("sentence-transformers/all-MiniLM-L6-v2"))

let cfg = hub.config("sentence-transformers/all-MiniLM-L6-v2")              # config.json as a dict
let w = hub.load("sentence-transformers/all-MiniLM-L6-v2")                  # {name: tensor} from safetensors
print(w["embeddings.word_embeddings.weight"].shape)

let tok = hub.tokenizer("sentence-transformers/all-MiniLM-L6-v2")           # tokenizer.json, natively
let e = tok.encode("Apsi reads like Python.")
print(e.tokens, e.ids, e.attention_mask)
print(tok.decode(e.ids))

Run it with apsi run app.apsi --allow-net=huggingface.co. Downloads of large files are redirected by the Hub to its storage hosts; the permission for huggingface.co covers them.

Functions

Tokens, private and gated models

use env
hub.login(env.secret("HF_TOKEN"))              # or set HF_TOKEN in the environment, or pass token= to a call
let w = hub.load("my-org/private-model")

• The token is sent only to the Hub's own address, never to the storage host a download is redirected to.
• Gated models (Llama, Gemma …) need you to accept their terms on the model page once; after that the token works.
• HF_ENDPOINT points the client at a mirror or a self-hosted Hub.

Cache, resuming and checking

• Files are kept in ~/.apsi/hub (or APSI_HUB_CACHE), one folder per commit, so a model that changes on the Hub never mixes old and new files.
• An interrupted download continues where it stopped the next time you ask for the file.
• Large files are checked against the SHA-256 the Hub publishes; a file that does not match is deleted, never used.
• hub.offline(true) (or APSI_HUB_OFFLINE=1) answers from the cache only — for machines without internet.

Tokenizers

tokenizer.json files of the Hugging Face tokenizers format are read natively:

encode(text, pair=none, add_special_tokens=true) returns ids, tokens and attention_mask; special tokens are added by the tokenizer's own post-processor template. decode(ids, skip_special_tokens=true) turns ids back into text.

Pre-tokenizer patterns written with \p{L} / \p{N} (Llama 3, Qwen) are translated to Python's regular expressions; letters of the common scripts (Latin, Greek, Cyrillic, Arabic, Hebrew, Devanagari, Thai, Hangul, kana, CJK) are covered, so very rare scripts can split differently from the Rust tokenizers library.

Running models natively

hub.model(repo) downloads config.json, the safetensors weights (also sharded; F32, F16 and BF16), the tokenizer and, for sentence-transformers models, their pooling settings, then runs the network with NumPy.

• embed(texts, normalize=) — sentence vectors with the model's own pooling (mean, CLS or max) and, when the model says so, L2 normalisation, as sentence-transformers does. similarity(a, b) — cosine similarity.
• classify(texts, top_k=1) — labels from the checkpoint's own head and id2label (softmax, or sigmoid for multi-label models).
• tokens(text) — one label per token (named-entity models). fill_mask(text, top_k=5) — the best words for [MASK] / <mask>.
• generate(prompt, max_new_tokens=64, temperature=0, top_k=0, top_p=1, repetition_penalty=1, seed=, stream=) — text generation with a key/value cache. temperature=0 is greedy. It stops at the model's end-of-sequence token (from generation_config.json or config.json). stream receives each new piece of text as it appears.
• hidden(texts) — the last hidden states, for your own heads.

How it is checked: tiny checkpoints written in the exact Hugging Face file layout are run in the test suite. The encoder is compared with an independent per-head reference implementation, padding is checked not to change results, and cached generation is compared with recomputing the whole sequence at every step. On a machine with transformers and torch, APSI_HF_COMPARE=1 python -m unittest tests.test_hub_models also compares against the reference library.

Methods and what they return

Texts longer than the model's max_position_embeddings are cut at that length. Lists are padded to the longest text and processed as one batch, which is several times faster than one call per text.

generate options:

Recipes

Semantic search over your documents

use hub
let st = hub.model("sentence-transformers/all-MiniLM-L6-v2")
let docs = ["How to reset a password", "Shipping times to Europe", "Refunds for damaged items"]
let index = st.embed(docs)                              # [3, 384], normalised
let q = st.embed("my parcel arrived broken")
let scores = index @ q                                   # cosine similarity, because both are normalised
let best = argmax(scores)
print(docs[best], scores[best])                          # Refunds for damaged items

For many documents, embed them in lists of 32–128 and keep the matrix (save_npy, or use memory for more than fits in RAM).

Classifying a batch

let clf = hub.pipeline("sentiment-analysis", "distilbert/distilbert-base-uncased-finetuned-sst-2-english")
for r in clf(["great product", "arrived late and broken"]):
    print(r["label"], round(r["score"], 3))

Named entities, joined into words

let ner = hub.model("dslim/bert-base-NER")
var words = []
for t in ner.tokens("Ada Lovelace was born in London"):
    if t["token"].startswith("##") and len(words) > 0:
        words[-1]["token"] += t["token"][2:]
    elif t["label"] != "O":
        words.append(t)
print(words)          # Ada (B-PER), Lovelace (I-PER), London (B-LOC)

Streaming text as it is generated

let lm = hub.model("Qwen/Qwen2.5-0.5B-Instruct")
let prompt = "<|im_start|>user\nName three uses of lasers.<|im_end|>\n<|im_start|>assistant\n"
lm.generate(prompt, max_new_tokens=120, temperature=0.7, top_p=0.9, seed=1, stream=fn(t) => print(t, end=""))

Chat models expect their own prompt format (above: Qwen's). The format is in the model card and in tokenizer_config.json under chat_template.

A model you already have on disk (downloaded elsewhere, or your own fine-tune saved by transformers)

let m = hub.model_from_folder("models/my-classifier")   # config.json, model.safetensors, tokenizer.json

Errors you may see

Speed and size

Everything runs on the CPU in float32 with NumPy's multi-threaded BLAS. Encoders such as MiniLM and BERT-base handle sentences in milliseconds; decoders up to about 1–2 billion parameters generate at interactive speed on a modern CPU (a 1.1B model needs about 4.4 GB of RAM in float32). For bigger models, use GGUF with llama.cpp, or PyTorch through hf_pipeline.

Not native yet

• Gemma 2 and 3 (extra norms and soft-capping), Phi, Falcon, MPT and mixture-of-experts models, T5 and BART (encoder–decoder), vision and audio models.
• Chat templates (Jinja in tokenizer_config.json): write the prompt in the model's format yourself.
• Quantised GGUF formats other than Q8_0 and Q4_0 for hub.load.

For those, hf_pipeline(task, model) in use models runs Python's transformers (needs --allow-python). PyTorch .bin checkpoints are pickles and can run code when loaded; hub.load refuses them. Download them with hub.download and open them with torch_load under --allow-python only if you trust the publisher.

Found a mistake? Report it · Markdown source