core

A fastllm-style Chat over litert_lm - message helpers, an ordered callback system, human-in-the-loop tool approval, sync streaming, and token_count-based usage tracking.

Messages

litert speaks a small message schema (Message, Contents, and Content subtypes). These helpers build them from ordinary Python values.

mk_content maps one value to a litert Content: a str becomes text, bytes become an image or audio (sniffed with detect_mime), and a Path becomes an image or audio file reference (by MIME). An existing Content passes through.
mk_msg wraps content into a Message (default role user), and also accepts an existing Message or a {'role','content'} dict.
mk_msgs normalises a mixed list into canonical litert message dicts, used to seed a conversation’s history.

source

mk_msgs

def mk_msgs(
    msgs
):

Normalize a list of messages to litert message dicts.

source

mk_msg

def mk_msg(
    content, role:str='user'
):

Create a litert Message from str/bytes/list/dict/Message.

source

mk_content

def mk_content(
    o
):

Convert o to a litert Content, sniffing bytes/files for image vs audio.

from fastcore.test import test_eq

test_eq(mk_msg("hello").to_json(), {"role": "user", "content": [{"type": "text", "text": "hello"}]})
test_eq(mk_msg("hi", role="model").to_json()["role"], "model")
test_eq([x["role"] for x in mk_msgs(["a", mk_msg("b", role="model")])], ["user", "model"])
assert isinstance(mk_content("x"), Text)
# bytes are sniffed: image vs audio
assert isinstance(mk_content(b'\x89PNG\r\n\x1a\n' + b'\x00'*16), ImageBytes)
assert isinstance(mk_content(b'RIFF\x00\x00\x00\x00WAVE' + b'\x00'*16), AudioBytes)

Usage tracking

litert doesn’t return per-response token counts, but conv.token_count exposes the running KV-cache size (prefill plus decode). UsageStats records one turn’s prompt, completion, and total tokens, derived from the count delta around the turn, and adds across turns. Compare chat.token_count (or chat.pct_full) against ctx_limit to decide when to compress the conversation.

source

UsageStats

def UsageStats(
    prompt_tokens:int=0, completion_tokens:int=0, total_tokens:int=0, n:int=0
):

Token usage for a chat turn, fed by conv.token_count diffs.

a = UsageStats(prompt_tokens=10, completion_tokens=5, total_tokens=15, n=1)
b = UsageStats(prompt_tokens=3, completion_tokens=2, total_tokens=20, n=1)
c = a + b
test_eq((c.prompt_tokens, c.completion_tokens, c.n), (13, 7, 2))
assert "in=13" in repr(c) and "out=7" in repr(c)
test_eq((a + None).prompt_tokens, 10)

Callbacks

A small, ordered callback system inspired by fastllm. A ChatCallback subclass hooks named events (after_msgs, before_send, after_response, before_tool_calls, after_tool_calls) and reads live turn state off the chat via GetAttr, so self.turn_msg is chat.turn_msg. run_cbs dispatches one event to every enabled callback in order, forwarding anything a callback yields into the output stream.

source

run_cbs

def run_cbs(
    chat, event
):

Dispatch event to enabled callbacks in order; forward any yielded stream items.

source

ChatCallback

def ChatCallback(
    *args, **kwargs
):

Base chat callback; reads chat state via GetAttr (self.turn_msg -> chat.turn_msg).

class _Dummy: pass
class _A(ChatCallback):
    order = 20
    def before_send(self): self.chat.log.append('A')
class _B(ChatCallback):
    order = 10
    def before_send(self):
        self.chat.log.append('B')
        yield {'text': 'from-B'}
d = _Dummy(); d.log = []
a, b = _A(), _B(); a.chat = d; b.chat = d
d.cbs = L(a, b)
out = list(run_cbs(d, 'before_send'))
test_eq(d.log, ['B', 'A'])
test_eq(out, [{'text': 'from-B'}])
test_eq(repr(a), '_A')

Streaming display

StreamFormatter turns a litert response stream into markdown as it arrives: text passes through, and tool calls render as a compact ⏳ name(args) line. display_stream consumes a markdown-chunk stream (what chat(msg, stream=True) yields) and renders it live in a notebook via IPython. mk_tr_details formats a completed tool call as a collapsible JSON block.

source

display_stream

def display_stream(
    chunks
):

Progressively render a markdown-chunk stream (e.g. chat(msg, stream=True)) live in a notebook; returns the full markdown.

source

StreamFormatter

def StreamFormatter(
    mx:int=2000, showthink:bool=True
):

Format a litert response stream to markdown; thinking streams as a blockquote.

source

mk_tr_details

def mk_tr_details(
    name, args, result, mx:int=2000
):

<details> JSON block for a completed tool call.

source

Resp

def Resp(
    *args, **kwargs
):

A litert response dict that renders as markdown (thinking + text + tool calls/responses) in notebooks.

source

thought

def thought(
    resp
):

The model’s thinking (channels.thought) for a litert response/chunk, or ’’.

source

resp_text

def resp_text(
    resp
):

Join text parts of a litert response/chunk dict.

Built-in callbacks

Three callbacks make up _dflt_cbs and run on every chat unless you pass default_cbs=False. HistoryCallback records each outgoing message and reply into chat.hist. UsageCallback folds the turn’s token counts into chat.use, read from a conv.token_count delta. ToolReminderCallback appends a short reminder to outgoing messages, but only when the chat has tools, nudging the model to summarise tool results in prose before it continues.

Order matters. HistoryCallback and UsageCallback sit at the front (low order) so a turn is recorded before feature callbacks like PyFenceCallback react to it and feed messages back. A callback reads live turn state off the chat through GetAttr, so self.turn_res is chat.turn_res.

source

ToolReminderCallback

def ToolReminderCallback(
    tool_reminder:str='\n<system-reminder>After every tool call result, briefly summarise in prose what you found before continuing or calling another tool.</system-reminder>'
):

Inject a tool-summary reminder into the outgoing message when tools are registered.

source

TruncationCallback

def TruncationCallback(
    max_tokens
):

Flag turn_res['truncated'] when the reply reaches max_tokens output tokens (best-effort).

source

truncated

def truncated(
    resp
):

Whether resp was flagged as cut off at the token cap.

source

UsageCallback

def UsageCallback(
    *args, **kwargs
):

Fold each response’s token usage into chat.use from a token_count diff.

source

HistoryCallback

def HistoryCallback(
    *args, **kwargs
):

Record the outgoing message and the response into chat.hist.

Tool calling, approval, and history

litert runs the tool-call loop inside the engine. ChatToolHandler bridges that loop back to Python: for each call it records the request and result into chat.hist (built with litert’s own Message and ToolResponse), fires the before_tool_calls and after_tool_calls callbacks, and consults chat.approve(tool_call) before executing. Returning False blocks the tool and feeds a “Denied by human operator” response back to the model. That is the hook for human-in-the-loop gating, shown below.

source

ChatToolHandler

def ChatToolHandler(
    chat
):

Bridge litert’s in-engine tool loop to Chat callbacks, HITL approval, and history.

Loading models & Chat

Default litert-community Gemma repos live at huggingface.co/litert-community. get_model resolves a .litertlm file with a cache-first ladder: an explicit model_path, then the local HuggingFace cache (scan_cache_dir, no network), then a download. It prefers the native build over -web variants, which omit the CPU/GPU decode graph.

Chat ties this together into a callable. Build it (it constructs or accepts an Engine), then call it like a function: one turn per call, updating history and usage. create_engine is a patchable classmethod that builds the engine and creates cache_dir if given. Pass a prebuilt engine= to share one model across several chats. Engine and conversation are entered on an ExitStack, so close() releases them in the right order and never closes an engine you supplied.

source

Chat

def Chat(
    engine:Engine=None, # Litert engine, or None to build one
    model_id:str='litert-community/gemma-4-E2B-it-litert-lm', # huggingface model id; see https://huggingface.co/litert-community
    model_path:PathLike=None, # local litertlm model path
    backend:Backend=CPU(thread_count=None), # backend for the engine
    multimodal:bool=True, # multimodal model
    cache_dir:PathLike=None, # cache dir for the engine
    enable_speculative_decoding:NoneType=None, eng_kw:NoneType=None, # extra litert Engine kwargs
    sp:str='', # system prompt
    messages:NoneType=None, # message history to prefill the conversation
    tools:NoneType=None, # tools to register with the engine
    ctx_limit:NoneType=None, # context window, for pct_full
    approve:NoneType=None, # approval function for tool calls
    tool_max_len:NoneType=None, # truncate string tool results longer than this (protects context)
    think:bool=False, # enable the model's thinking channel (if supported)
    filter_think:bool=True, # keep thinking out of the KV cache (saves context)
    temp:NoneType=None, top_k:NoneType=None, top_p:NoneType=None,
    seed:NoneType=None, # sampler knobs (build a SamplerConfig)
    sampler_config:NoneType=None, # or pass a full SamplerConfig (overrides the knobs)
    max_output_tokens:NoneType=None, conv_kw:NoneType=None, # extra create_conversation kwargs
    cbs:NoneType=None, # callbacks to register, checkout default callbacks liek HistoryCallback, UsageCallback, ToolReminderCallback
    default_cbs:bool=True, # add default callbacks.
):

Sync chat over a local litert_lm engine. Callbacks record history/usage; _send drives one message.

source

get_model

def get_model(
    model_id, model_path:NoneType=None
):

Return a local .litertlm path: model_path, else HF cache, else download.

Managing callbacks

add_cb registers a callback and returns the instance; add_cbs takes a list and returns the instances. remove_cb drops one by instance, or by class to remove every callback of that type, and remove_cbs handles several at once. To run a callback for a single turn, pass cbs= to the call: chat(msg, cbs=[PyFenceCallback(...)]) registers them before the turn and removes them after, so nothing leaks into later turns.

set_min_log_severity(3)
get_model(gemma4_12b)

'/Users/71293/.cache/huggingface/hub/models--litert-community--gemma-4-12B-it-litert-lm/snapshots/44cf85a326f79b814fa86a60af414c042755b43a/gemma-4-12B-it.litertlm'

Human-in-the-loop tool approval

Any approve(tool_call) -> bool passed to Chat(approve=...) is consulted by ChatToolHandler before each tool runs. hitl_policy builds one from a per-tool policy: 'approved' (auto-run), 'dont_run' (always block), or 'check' (ask, via _ask_console by default). Or supply your own function for custom logic such as logging, rate-limiting, or prompting a UI. See the worked example below.

source

hitl_policy

def hitl_policy(
    modes, ask:function=_ask_console
):

Build an approve(tool_call) from per-tool modes: ‘approved’ | ‘check’ | ‘dont_run’.

Using Chat

A short tour of the main features. These cells build a real model, so they are set to run manually rather than in the test suite.

chat=Chat(backend=Backend.GPU(), cache_dir='.cache/litertlm', think=True)
# chat_12 = Chat(Chat.create_engine(gemma4_12b, multimodal=False, cache_dir='.cache/litertlm',be=Backend.GPU()))

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784178935.259078 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

# set_min_log_severity(2)
r = chat("Reply with exactly: pong")
assert 'pong' in resp_text(r).lower()
assert chat.hist[-1] is r and chat.hist[0]['role'] == 'user'
assert chat.use.total_tokens > 0 and chat.token_count > 0
print(chat.use); chat.print_hist()

W0000 00:00:1784178936.197760 66965901 sampler_factory.cc:450] WebGPU sampler not available, falling back to statically linked C API: UNAVAILABLE: ERROR: [third_party/odml/litert_lm/runtime/components/sampler_factory.cc:294]
└ Could not load shared library libLiteRtTopKWebGpuSampler.dylib: dlopen(libLiteRtTopKWebGpuSampler.dylib, 0x0005): tried: 'libLiteRtTopKWebGpuSampler.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibLiteRtTopKWebGpuSampler.dylib' (no such file), '/usr/lib/libLiteRtTopKWebGpuSampler.dylib' (no such file, not in dyld cache), 'libLiteRtTopKWebGpuSampler.dylib' (no such file).
=== Source Location Trace: === 
./third_party/odml/litert/litert/cc/litert_macros.h:573

total=104|in=103|out=1|turns=1

user

Reply with exactly: pong

assistant

🧠 Thinking

Thinking Process:

Analyze the Request: The user has instructed me to reply with exactly the word “pong”.

Determine the Constraint: The constraint is strict: “Reply with exactly: pong”.

Formulate the Response: The response must be the string “pong”.

Final Output Generation: pong

pong

chat(['what is pong','do you know?'])

🧠 Thinking

Thinking Process:

Analyze the Request: The user is asking “what is pongdo you know?”

Analyze the Context (Implicit): The previous interaction was a very short, specific instruction: “Reply with exactly: pong”. The model responded with “pong”. The user is now asking about “pongdo”.

Identify Potential Interpretations of “pongdo”:

Typo/Misspelling: Is it a misspelling of “Pong”?

Specific Term/Acronym: Is it a term in a niche field (programming, gaming, science)?

Combination: Is it a combination of “Pong” and another word?

Contextual Link: Does it relate to the previous interaction (the word “pong”)?

Determine the Most Likely Meaning: Given the previous interaction, the most direct link is the game Pong. “Pongdo” is not a standard, widely recognized term.

Formulate a Strategy: Since “pongdo” is ambiguous, the best approach is to address the most probable interpretation (Pong) and then ask for clarification if the user meant something else.

Draft the Response (Self-Correction/Refinement):

Initial thought: I don’t know what pongdo is.

Refinement: Be helpful. Assume they mean Pong.

Final Output Generation. (Since the user asked “what is pongdo you know?”, I should provide the most relevant information.)

I am not familiar with the term “pongdo.”

Could you please provide more context? It might be:

A misspelling of Pong (the classic arcade game).
A term from a specific game, programming language, or niche topic.
A specific name or acronym.

If you can give me more information, I would be happy to tell you what I know!

Images and audio

Some Gemma builds accept image and audio content alongside text.

from PIL import Image

im=Image.open('images.jpeg');im

chat(['explain this image', img_bytes(im)])
# you can also use ImageFile and ImageBytes

🧠 Thinking

Here’s a thinking process to arrive at the suggested response:

Analyze the Request: The user has provided an image and asked me to “explain this image.”

Analyze the Image:

Subject: The central focus is a dog.

Breed/Appearance: The dog has reddish-brown/tan fur, erect ears, and a strong build. It strongly resembles a German Shepherd or a similar working breed.

Action/Pose: The dog is looking slightly off-camera (to the right), mouth is open (tongue is visible), and appears happy or excited (panting/smiling).

Setting: The background is outdoors, consisting of a dirt path or trail, with blurred green foliage/trees on either side. The lighting suggests daytime.

Mood: Happy, alert, friendly, outdoorsy.

Determine the Explanation Strategy: A good explanation should cover the main elements (who, what, where, how) and the overall feeling.

Draft the Explanation (Iterative Refinement):

Initial thought: It’s a dog on a path. It looks happy.

Adding detail (Focusing on the dog): The dog looks like a German Shepherd. It has brown fur and is panting.

Adding detail (Focusing on the setting): It’s outdoors on a dirt path with greenery.

Synthesizing and Polishing: Combine these points into a coherent, descriptive paragraph.

Final Review against the Goal: Does this explain the image accurately? Yes. (Self-Correction: Ensure the tone is descriptive and engaging.)

Generate the Output. (This leads to the final provided answer.)

This is a photograph of a beautiful, happy-looking dog outdoors.

Here is a detailed breakdown of the image:

Subject: The main focus is a medium-to-large dog, likely a German Shepherd or a similar working breed, characterized by its rich, reddish-brown fur and erect ears.
Action/Expression: The dog appears alert and happy. Its mouth is open, and its tongue is hanging out, suggesting it might be panting slightly from activity or excitement. It is looking off to the side, giving it an engaged and friendly expression.
Setting: The dog is standing on a dirt or gravel path. The background is soft and blurred (shallow depth of field), showing natural greenery, trees, and foliage, indicating an outdoor, natural environment, perhaps a park or a wooded trail.
Mood: The overall mood of the photo is warm, joyful, and adventurous, capturing a moment of a dog enjoying time outdoors.

Example: gating a tool behind approval

A custom approve gate that auto-allows a safe tool and blocks a destructive one. ChatToolHandler routes every tool call through it; a denied call is recorded in history as Denied by human operator and reported back to the model, which continues without it.

def add(a: int, b: int) -> int:
    'Add two integers.\n\nArgs:\n    a: first addend\n    b: second addend'
    return a + b

def delete_files(path: str) -> str:
    'Delete everything under a path.\n\nArgs:\n    path: directory to wipe'
    return f"wiped {path}"

def approve_gate(tc):
    "chat.approve: allow any tool except destructive ones; log every decision."
    name = tc['function']['name']
    ok = name != 'delete_files'
    print(f"[approval] {_tc_summary(name, tc['function'].get('arguments', {}))} -> {'ALLOW' if ok else 'DENY'}")
    return ok

chat = Chat(tools=[add, delete_files], approve=approve_gate, sp="Use the tools to satisfy the request.")
r = chat("Add 2 and 3, then delete everything under /tmp/data.")
print('\n', resp_text(r))
chat.print_hist()   # the blocked call is logged as a 'Denied by human operator' tool response
chat.close()

# Declarative equivalent (interactive 'check' prompts on the console via _ask_console):
# Chat(tools=[add, delete_files], approve=hitl_policy({'add': 'approved', 'delete_files': 'dont_run'}))

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784178958.305192 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

[approval] <code>add(a=2.0, b=3.0)</code> -> ALLOW
[approval] <code>delete_files(path=&#x27;/tmp/data&#x27;)</code> -> DENY

 I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under `/tmp/data`.

user

Add 2 and 3, then delete everything under /tmp/data.

model

🔧 add({‘a’: 2.0, ‘b’: 3.0})

tool

↩︎ add: 5.0

model

🔧 delete_files({‘path’: ‘/tmp/data’})

tool

↩︎ delete_files: Denied by human operator

assistant

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under /tmp/data.

🔧 delete_files({‘path’: ‘/tmp/data’})

WARNING: Cache file is stale. Setting stale flag.

Policy-driven approval

You can also drive approvals declaratively with hitl_policy instead of writing your own function.

chat=Chat(tools=[add, delete_files], approve=hitl_policy({'add': 'approved'}))

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784178965.189187 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

r = chat("Add 2 and 3, then delete everything under /tmp/data.")
print('\n', resp_text(r))
chat.print_hist()   # the blocked call is logged as a 'Denied by human operator' tool response
chat.close()


 I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under `/tmp/data`.

user

Add 2 and 3, then delete everything under /tmp/data.

model

🔧 add({‘a’: 2.0, ‘b’: 3.0})

tool

↩︎ add: 5.0

model

🔧 delete_files({‘path’: ‘/tmp/data’})

tool

↩︎ delete_files: Denied by human operator

assistant

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under /tmp/data.

🔧 delete_files({‘path’: ‘/tmp/data’})

WARNING: Cache file is stale. Setting stale flag.

Streaming turns

Pass stream=True and iterate the result: Chat returns a generator of markdown chunks (formatted by StreamFormatter) instead of a response dict. History and usage are finalised once the stream is exhausted.

ch = Chat(backend=Backend.GPU())

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784178987.905959 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

# stream chunks to a terminal:
for c in ch("Count: one two three", stream=True): print(c, end='', flush=True)

That's a simple count!

If you'd like me to do

W0000 00:00:1784178989.277368 66968557 sampler_factory.cc:450] WebGPU sampler not available, falling back to statically linked C API: UNAVAILABLE: ERROR: [third_party/odml/litert_lm/runtime/components/sampler_factory.cc:294]
└ Could not load shared library libLiteRtTopKWebGpuSampler.dylib: dlopen(libLiteRtTopKWebGpuSampler.dylib, 0x0005): tried: 'libLiteRtTopKWebGpuSampler.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibLiteRtTopKWebGpuSampler.dylib' (no such file), '/usr/lib/libLiteRtTopKWebGpuSampler.dylib' (no such file, not in dyld cache), 'libLiteRtTopKWebGpuSampler.dylib' (no such file).
=== Source Location Trace: === 
./third_party/odml/litert/litert/cc/litert_macros.h:573

 something with that count, please let me know. For example, would you like me to:

* **Continue counting?** (e.g., "four, five, six...")
* **Count something else?**
* **Do a math problem?**
* **Something else entirely?**

# or render live in a notebook:
display_stream(ch("count fibonacci for 10 places", stream=True))

Here are the first 10 numbers in the Fibonacci sequence:

The Fibonacci sequence starts with 0 and 1, and each subsequent number is the sum of the two preceding ones.

0
1
1 (0 + 1)
2 (1 + 1)
3 (1 + 2)
5 (2 + 3)
8 (3 + 5)
13 (5 + 8)
21 (8 + 13)
34 (13 + 21)

The first 10 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34

'Here are the first 10 numbers in the Fibonacci sequence:\n\nThe Fibonacci sequence starts with 0 and 1, and each subsequent number is the sum of the two preceding ones.\n\n1. **0**\n2. **1**\n3. **1** (0 + 1)\n4. **2** (1 + 1)\n5. **3** (1 + 2)\n6. **5** (2 + 3)\n7. **8** (3 + 5)\n8. **13** (5 + 8)\n9. **21** (8 + 13)\n10. **34** (13 + 21)\n\n**The first 10 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34**'

assert chat.hist[-1] == chat.turn_res and chat.use.completion_tokens > 0

fired = []
class Log(ChatCallback):
    def before_tool_calls(self): fired.append('before')
    def after_tool_calls(self):  fired.append('after')

chat = Chat(tools=[add], sp="Use the add tool for arithmetic.")
chat.add_cb(Log)
r = chat("What is 21 + 21? Use the tool.")
print(fired)
assert 'before' in fired and 'after' in fired
assert any(m.get('role') == 'tool' for m in chat.hist)
print(resp_text(r)); chat.print_hist()

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784179010.555272 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

['before', 'after']
The result of adding 21 and 21 is 42.

user

What is 21 + 21? Use the tool.

model

🔧 add({‘a’: 21.0, ‘b’: 21.0})

tool

↩︎ add: 42.0

assistant

The result of adding 21 and 21 is 42.

Running python from replies

PyFenceCallback turns the chat into a code interpreter. After a reply it finds the last ```python fence, runs it through safepyrun (sandboxed, so imports like socket and importlib are blocked) in a namespace that persists across the conversation, feeds the output back as a ```result block, and re-queries, up to max_rounds times. Enable it with Chat(cbs=[PyFenceCallback]). Execution goes through the same approve hook as a synthetic python tool, so hitl_policy({'python': 'check'}) prompts before running. chat.run_py(code) runs a snippet directly (ban_defs=False allows def and class).

source

PyFenceCallback

def PyFenceCallback(
    max_rounds:int=5, done:NoneType=None
):

Run ``python fences, feed results back, loop untildone(chat)(default: no fence).max_rounds` is a safety cap.

source

output_matches

def output_matches(
    expected
):

done policy: stop once the last code output contains str(expected) (e.g. the answer to match against).

source

task_complete

def task_complete(
    chat
):

done policy: judge (via classify, isolated) whether the latest result completes the request.

source

Chat.run_py

def run_py(
    code, ban_defs:bool=False, g:NoneType=None
):

Run code in this chat’s sandboxed, persistent namespace; return stdout + last-expr repr.

source

run_coro

def run_coro(
    coro
):

Run an awaitable to completion from sync code, even inside a running event loop.

source

mk_result_fence

def mk_result_fence(
    out
):

Feed a code result back, prompting a prose answer or more code.

source

extract_fence

def extract_fence(
    text, tag:str='answer'
):

Contents of the last ``<tag> fence intext`, else the whole stripped text.

source

extract_code

def extract_code(
    text
):

Code of the last ``python fence intext`, else None.

Stopping the loop with `done`

Pass a done function to PyFenceCallback to end the loop early. output_matches stops once the code output contains an expected string.

chat = Chat(sp="When asked to compute something, reply with a ```python fence that prints or evaluates the answer.")
r = chat("Use python to compute 2**5.", cbs=[PyFenceCallback(done=output_matches(32))])
chat.print_hist()   # you'll see the ```python turn, a ```result turn, then the final answer
chat.close()

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784179491.903529 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

user

Use python to compute 2**5.

assistant

print(2**5)

user

WARNING: Cache file is stale. Setting stale flag.

Letting the model decide

Or leave termination to the model: with no done, the loop stops as soon as a reply has no code fence.

chat = Chat(cbs=[PyFenceCallback], sp="When asked to compute something, reply with a ```python fence that prints or evaluates the answer.")
r = chat("Use python to compute 2**5.")
chat.print_hist()
chat.close()

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784179784.097886 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

user

Use python to compute 2**5.

assistant

print(2**5)

user

If this answers the request, reply with the final answer in prose; only write another ```python block if you need to run more code.

assistant

WARNING: Cache file is stale. Setting stale flag.

Utilities: classify, structured, benchmark

classify and structured each run one-shot in a throwaway conversation on the shared engine, isolated from the live chat, so they leave its history and KV cache untouched. classify generates a label and matches it against your options. structured forces a tool call and returns schema(**arguments). bench reports init time, time to first token, and prefill and decode tokens per second.

litert’s run_text_scoring log-likelihood scoring is not available on this runtime, so classify generates and label-matches rather than scoring.

source

bench

def bench(
    model_id:str='litert-community/gemma-4-E2B-it-litert-lm', model_path:NoneType=None,
    backend:CPU=CPU(thread_count=None), prefill_tokens:int=64, decode_tokens:int=64, **kw
):

Benchmark init time, TTFT, and prefill/decode tokens-per-sec via litert’s Benchmark.

source

Chat.structured

def structured(
    prompt, schema, sp:str='Call the tool to answer.'
):

One-shot structured output: the model calls schema (a function/class); returns schema(**arguments).

source

Chat.classify

def classify(
    text, labels, sp:str='Reply with only the single best label and nothing else.'
):

One-shot label for text, run in a throwaway conversation (isolated from this chat).

# classify + structured on a real model
sm = Chat(cache_dir='.cache/litertlm')
test_eq(sm.classify("I absolutely loved this film!", ['positive', 'negative']), 'positive')

from dataclasses import dataclass
@dataclass
class Person: name:str; age:int
p = sm.structured("Extract the person: John Smith is 30 years old.", Person)
print(p); assert isinstance(p, Person) and 'John' in p.name
sm.close()

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784180085.450553 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180085.451727 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180085.965230 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

Person(name='John Smith', age=30.0)

Grading answers

chat.check turns the model into a graded question-answerer. It asks question in an isolated conversation (like classify/structured, so nothing touches chat.hist), pulls the model’s answer out of a ``answer fence with [extract_fence](https://vedicreader.github.io/rishi/core.html#extract_fence) (falling back to the whole reply if the model skips the fence), and grades it againstexpected. Grading isgrade_fn(answer, expected) -> bool, defaulting to [_matches](https://vedicreader.github.io/rishi/core.html#_matches) (the answer must containexpected, or any value in anexpectedlist). Pass your owngrade_fnfor custom logic, or setllm_judge=Trueto grade with the model instead (chat.grades, built onclassify); passjudge=a second [Chat`](https://vedicreader.github.io/rishi/core.html#chat) to have a stronger model do the grading.

source

Chat.check

def check(
    question, expected, grade_fn:function=_matches, llm_judge:bool=False, judge:NoneType=None, tag:str='answer',
    sp:str='Answer the question, then put your final answer inside a ```answer fence.'
):

Ask question in a throwaway conversation, extract the ``<tag> answer, and grade it againstexpected`.

source

Chat.grades

def grades(
    question, expected, actual
):

LLM-as-judge (on this chat’s engine): is actual a correct answer to question given reference expected?

# Default grade is a deterministic match (no judge needed):
qa = Chat(cache_dir='.cache/litertlm')
print(qa.check("What is the capital of France?", "Paris").ok)
# Custom grader: any `answer, expected -> bool`:
print(qa.check("What is 2 + 2?", "4", grade_fn=lambda a, e: e in a).ok)

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784180090.782984 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180090.783925 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

True

W0000 00:00:1784180091.531015 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

True

# LLM-as-judge, optionally with a bigger model doing the grading:
judge = Chat(model_id=gemma4_12b, backend=Backend.GPU(),multimodal=False, cache_dir='.cache/litertlm')

W0000 00:00:1784180096.290555 66921307 litert_lm_loader.h:158] TFLite model type: TF_LITE_VISION_ENCODER not found for backend constraints. Skipping.
W0000 00:00:1784180096.290570 66921307 litert_lm_loader.h:174] TFLite model type: TF_LITE_VISION_ENCODER not found for prefer activation type. Use system's default backend activation type. System's default activation type for Text decoder is fp16. Vision encoder and audio encoder default is fp32.
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784180109.242849 66921307 litert_lm_loader.cc:289] Section not found: 
W0000 00:00:1784180109.242932 66921307 litert_lm_loader.h:129] TFLite model for type: TF_LITE_END_OF_VISION not found. Skipping.
W0000 00:00:1784180109.271313 66921307 litert_lm_loader.cc:289] Section not found: 
W0000 00:00:1784180109.271323 66921307 litert_lm_loader.h:129] TFLite model for type: TF_LITE_PER_LAYER_EMBEDDER not found. Skipping.
W0000 00:00:1784180109.271470 66921307 litert_lm_loader.h:158] TFLite model type: TF_LITE_VISION_ENCODER not found for backend constraints. Skipping.
W0000 00:00:1784180109.271475 66921307 litert_lm_loader.h:174] TFLite model type: TF_LITE_VISION_ENCODER not found for prefer activation type. Use system's default backend activation type. System's default activation type for Text decoder is fp16. Vision encoder and audio encoder default is fp32.
W0000 00:00:1784180109.279493 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

r = qa.check("Name a primary colour.", "red, blue, or yellow", judge=judge)   # judge -> llm_judge
print(r.answer, '->', r.ok)
judge.close(); qa.close()

W0000 00:00:1784180112.863596 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180113.297144 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180113.351198 66992751 sampler_factory.cc:450] WebGPU sampler not available, falling back to statically linked C API: UNAVAILABLE: ERROR: [third_party/odml/litert_lm/runtime/components/sampler_factory.cc:294]
└ Could not load shared library libLiteRtTopKWebGpuSampler.dylib: dlopen(libLiteRtTopKWebGpuSampler.dylib, 0x0005): tried: 'libLiteRtTopKWebGpuSampler.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibLiteRtTopKWebGpuSampler.dylib' (no such file), '/usr/lib/libLiteRtTopKWebGpuSampler.dylib' (no such file, not in dyld cache), 'libLiteRtTopKWebGpuSampler.dylib' (no such file).
=== Source Location Trace: === 
./third_party/odml/litert/litert/cc/litert_macros.h:573

Red -> True

# grades (LLM judge) + check on a real model
sm = Chat(cache_dir='.cache/litertlm')
assert sm.grades("What is the capital of France?", "Paris", "Paris") is True
assert sm.grades("What is the capital of France?", "Paris", "London") is False
r = sm.check("What is the capital of France?", "Paris")            # deterministic default
print(r.answer, '->', r.ok); assert r.ok
assert sm.check("What is 2 + 2?", "4", llm_judge=True).ok is True  # self as judge
sm.close()

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784180138.089785 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180138.090966 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180139.074466 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180139.374913 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

Paris -> True

W0000 00:00:1784180140.049391 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180140.676127 66921307 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

Installing the skill

skill.md ships with the package. A harness can drop it into the standard skill directories with mv_skill_md, which writes SKILL.md under .claude/skills/rishi/ and .agents/skills/rishi/ at the git root. It is a dry run by default and only prints the targets; pass dry_run=False to write, or dir= to install somewhere else.

source

mv_skill_md

def mv_skill_md(
    dry_run:bool=True, dir:NoneType=None
):

Copy the bundled skill.md into .claude and .agents skill dirs so a harness can load the rishi skill.

source

repo_root

def repo_root()->Path:

Root of the current git repository, or None if not in one.

Messages

mk_msgs

mk_msg

mk_content

Usage tracking

UsageStats

Callbacks

run_cbs

ChatCallback

Streaming display

display_stream

StreamFormatter

mk_tr_details

Resp

thought

resp_text

Built-in callbacks

ToolReminderCallback

TruncationCallback

truncated

UsageCallback

HistoryCallback

Tool calling, approval, and history

ChatToolHandler

Loading models & Chat

Chat

get_model

Managing callbacks

Human-in-the-loop tool approval

hitl_policy

Using Chat

Images and audio

Example: gating a tool behind approval

Policy-driven approval

Streaming turns

Running python from replies

PyFenceCallback

output_matches

task_complete

Chat.run_py

run_coro

mk_result_fence

extract_fence

extract_code

Stopping the loop with done

Letting the model decide

Utilities: classify, structured, benchmark

bench

Chat.structured

Chat.classify

Grading answers

Chat.check

Chat.grades

Installing the skill

mv_skill_md

repo_root

Stopping the loop with `done`