# litert
## Messages
litert speaks a small message schema (`Message`, `Contents`, and
`Content` subtypes). These helpers build them from ordinary Python
values.
- `mk_content` maps one value to a litert `Content`: a `str` becomes
text, `bytes` become an image or audio (sniffed with `detect_mime`),
and a `Path` becomes an image or audio file reference (by MIME). An
existing `Content` passes through.
- `mk_msg` wraps content into a `Message` (default role `user`), and
also accepts an existing `Message` or a `{'role','content'}` dict.
- `mk_msgs` normalises a mixed list into canonical litert message dicts,
used to seed a conversation’s history.
``` python
_mk_content("hello\n").text
```
'hello\n'
``` python
from fastcore.test import test_eq
```
``` python
test_eq(_mk_msg("hello").to_json(), {"role": "user", "content": [{"type": "text", "text": "hello"}]})
test_eq(_mk_msg("hi", role="model").to_json()["role"], "model")
test_eq([x["role"] for x in _mk_msgs(["a", _mk_msg("b", role="model")])], ["user", "model"])
assert isinstance(_mk_content("x"), Text)
# bytes are sniffed: image vs audio
assert isinstance(_mk_content(b'\x89PNG\r\n\x1a\n' + b'\x00'*16), ImageBytes)
assert isinstance(_mk_content(b'RIFF\x00\x00\x00\x00WAVE' + b'\x00'*16), AudioBytes)
```
## Portable history
`chat.hist` is kept in one backend-agnostic **canonical** shape
(OpenAI-style dicts: `assistant`/`user`/`system`/`tool` roles, string
content, `channels.thought`, and `tool_calls`/`tool_call_id` for tool
rounds) so a conversation can be handed from one backend to another.
litert’s own wire format differs (`model` role, list content,
`image`/`audio` blobs, `tool_response` parts, id-less tool calls), so
two converters bridge it:
- `fmt2hist` — litert-native messages (or a
`Message`/[`Resp`](https://vedicreader.github.io/rishi/core.html#resp))
→ canonical history dicts; used to record turns into `chat.hist`.
Past-turn media collapses to an `[image]`/`[audio]` placeholder.
- `hist2fmt` — canonical history dicts → litert `Message`s the engine
can ingest; used to seed a conversation from a (possibly ported)
history.
Because llama’s history is already canonical,
`Chat('litert/…', messages=llama_chat.hist)` and
`Chat('llama/…', messages=litert_chat.hist)` both work — see the port
example in the [overview](index.html).
## Usage tracking
litert doesn’t return per-response token counts, but `conv.token_count`
exposes the running KV-cache size (prefill plus decode).
[`UsageStats`](https://vedicreader.github.io/rishi/core.html#usagestats)
records one turn’s `prompt`, `completion`, and `total` tokens, derived
from the count delta around the turn, and adds across turns. Compare
`chat.token_count` (or `chat.pct_full`) against `ctx_limit` to decide
when to compress the conversation.
## Callbacks
A small, ordered callback system inspired by fastllm. A
[`ChatCallback`](https://vedicreader.github.io/rishi/core.html#chatcallback)
subclass hooks named events (`after_msgs`, `before_send`,
`after_response`, `before_tool_calls`, `after_tool_calls`) and reads
live turn state off the chat via `GetAttr`, so `self.turn_msg` is
`chat.turn_msg`.
[`run_cbs`](https://vedicreader.github.io/rishi/core.html#run_cbs)
dispatches one event to every enabled callback in `order`, forwarding
anything a callback yields into the output stream.
## Streaming display
[`StreamFormatter`](https://vedicreader.github.io/rishi/core.html#streamformatter)
turns a litert response stream into markdown as it arrives: text passes
through, and tool calls render as a compact `⏳ name(args)` line.
[`display_stream`](https://vedicreader.github.io/rishi/core.html#display_stream)
consumes a markdown-chunk stream (what `chat(msg, stream=True)` yields)
and renders it live in a notebook via IPython.
[`mk_tr_details`](https://vedicreader.github.io/rishi/core.html#mk_tr_details)
formats a completed tool call as a collapsible JSON block.
## Built-in callbacks
Three callbacks make up `_dflt_cbs` and run on every chat unless you
pass `default_cbs=False`. `HistoryCallback` records each outgoing
message and reply into `chat.hist`. `UsageCallback` folds the turn’s
token counts into `chat.use`, read from a `conv.token_count` delta.
`ToolReminderCallback` appends a short reminder to outgoing messages,
but only when the chat has tools, nudging the model to summarise tool
results in prose before it continues.
Order matters. `HistoryCallback` and `UsageCallback` sit at the front
(low `order`) so a turn is recorded before feature callbacks like
[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback)
react to it and feed messages back. A callback reads live turn state off
the chat through `GetAttr`, so `self.turn_res` is `chat.turn_res`.
## Tool calling, approval, and history
litert runs the tool-call loop inside the engine.
[`ChatToolHandler`](https://vedicreader.github.io/rishi/litert.html#chattoolhandler)
bridges that loop back to Python: for each call it records the request
and result into `chat.hist` in canonical form (a paired assistant
`tool_calls` entry and a `tool` result, so the round survives a hand-off
to another backend), fires the `before_tool_calls` and
`after_tool_calls` callbacks, and consults `chat.approve(tool_call)`
before executing. Returning `False` blocks the tool and feeds a “Denied
by human operator” response back to the model. That is the hook for
human-in-the-loop gating, shown below.
------------------------------------------------------------------------
source
### ChatToolHandler
``` python
def ChatToolHandler(
chat
):
```
*Bridge litert’s in-engine tool loop to Chat callbacks, HITL approval,
and history.*
## Loading models & Chat
Default litert-community Gemma repos live at
[huggingface.co/litert-community](https://huggingface.co/litert-community).
`get_model` resolves a `.litertlm` file with a cache-first ladder: an
explicit `model_path`, then the local HuggingFace cache
(`scan_cache_dir`, no network), then a download. It prefers the native
build over `-web` variants, which omit the CPU/GPU decode graph.
[`Chat`](https://vedicreader.github.io/rishi/core.html#chat) ties this
together into a callable. Build it (it constructs or accepts an
`Engine`), then call it like a function: one turn per call, updating
history and usage. `create_engine` is a patchable classmethod that
builds the engine and creates `cache_dir` if given. Pass a prebuilt
`engine=` to share one model across several chats. Engine and
conversation are entered on an `ExitStack`, so `close()` releases them
in the right order and never closes an engine you supplied.
------------------------------------------------------------------------
source
### LitertChat
``` python
def LitertChat(
model:NoneType=None, runtime:NoneType=None, model_path:NoneType=None, engine:NoneType=None,
backend:CPU=CPU(thread_count=None), multimodal:bool=True, cache_dir:NoneType=None,
enable_speculative_decoding:NoneType=None, eng_kw:NoneType=None, sp:str='', messages:NoneType=None,
tools:NoneType=None, ctx_limit:NoneType=None, approve:NoneType=None, tool_max_len:NoneType=None,
max_steps:int=10, think:bool=False, filter_think:bool=True, temp:NoneType=None, top_k:NoneType=None,
top_p:NoneType=None, seed:NoneType=None, sampler_config:NoneType=None, max_output_tokens:NoneType=None,
conv_kw:NoneType=None, cbs:NoneType=None, default_cbs:bool=True
):
```
*Sync chat over a local litert_lm engine.*
``` python
# fmt2hist / hist2fmt round-trip (model-free)
_nat = [
'hi',
Message.model(Contents.of('hello')).to_json(),
{'role': 'model', 'tool_calls': [{'type': 'function', 'function': {'name': 'add', 'arguments': {'a': 2, 'b': 3}}}]},
{'role': 'tool', 'content': [{'type': 'tool_response', 'name': 'add', 'response': 5}]},
]
_canon = LitertChat.fmt2hist(_nat)
test_eq(_canon[0], {'role': 'user', 'content': 'hi'})
test_eq(_canon[1], {'role': 'assistant', 'content': 'hello'})
test_eq(_canon[2]['role'], 'assistant')
test_eq(_canon[2]['tool_calls'][0]['function'], {'name': 'add', 'arguments': {'a': 2, 'b': 3}})
assert _canon[2]['tool_calls'][0]['id'] # a synthetic id was assigned
test_eq(_canon[3], {'role': 'tool', 'name': 'add', 'content': '5'})
# canonical -> litert Messages ready for the engine
_ms = LitertChat.hist2fmt(_canon)
test_eq([m.role.value for m in _ms], ['user', 'model', 'model', 'tool'])
test_eq(str(_ms[1]), 'hello')
test_eq(_ms[2].tool_calls[0].name, 'add')
test_eq(_ms[3].contents.to_json(), [{'type': 'tool_response', 'name': 'add', 'response': '5'}])
# fmt2hist is idempotent on an already-canonical (llama-origin) tool round
_llama = [{'role': 'assistant', 'content': '', 'tool_calls': [{'id': 'call_x', 'type': 'function',
'function': {'name': 'add', 'arguments': {'a': 1}}}]},
{'role': 'tool', 'tool_call_id': 'call_x', 'name': 'add', 'content': '1'}]
test_eq(LitertChat.fmt2hist(_llama), _llama)
```
### Managing callbacks
`add_cb` registers a callback and returns the instance; `add_cbs` takes
a list and returns the instances. `remove_cb` drops one by instance, or
by class to remove every callback of that type, and `remove_cbs` handles
several at once. To run a callback for a single turn, pass `cbs=` to the
call: `chat(msg, cbs=[PyFenceCallback(...)])` registers them before the
turn and removes them after, so nothing leaks into later turns.
``` python
set_min_log_severity(5)
_get_model(gemma4_12b)
```
'/Users/71293/.cache/huggingface/hub/models--litert-community--gemma-4-12B-it-litert-lm/snapshots/44cf85a326f79b814fa86a60af414c042755b43a/gemma-4-12B-it.litertlm'
## Human-in-the-loop tool approval
Any `approve(tool_call) -> bool` passed to `Chat(approve=...)` is
consulted by
[`ChatToolHandler`](https://vedicreader.github.io/rishi/litert.html#chattoolhandler)
before each tool runs.
[`hitl_policy`](https://vedicreader.github.io/rishi/core.html#hitl_policy)
builds one from a per-tool policy: `'approved'` (auto-run), `'dont_run'`
(always block), or `'check'` (ask, via
[`_ask_console`](https://vedicreader.github.io/rishi/core.html#_ask_console)
by default). Or supply your own function for custom logic such as
logging, rate-limiting, or prompting a UI. See the worked example below.
## Using Chat
A short tour of the main features. These cells build a real model, so
they are set to run manually rather than in the test suite.
``` python
chat=Chat(backend=Backend.GPU(), cache_dir='.cache/litertlm', think=True)
# chat_12 = LitertChat(engine=LitertChat.create_engine(gemma4_12b, multimodal=False, cache_dir='.cache/litertlm', be=Backend.GPU()))
```
``` python
# set_min_log_severity(2)
r = chat("Reply with exactly: pong")
assert 'pong' in resp_text(r).lower()
assert resp_text(chat.hist[-1]) == resp_text(r) and chat.hist[0]['role'] == 'user'
assert chat.use.total_tokens > 0 and chat.token_count > 0
print(chat.use); chat.print_hist()
```
total=104|in=103|out=1|turns=1
**user**
Reply with exactly: pong
------------------------------------------------------------------------
**assistant**
> **🧠 Thinking**
>
> Thinking Process:
>
> 1. **Analyze the Request:** The user has instructed me to reply with
> *exactly* the word “pong”.
> 2. **Determine the Constraint:** The constraint is strict: “Reply
> with exactly: pong”.
> 3. **Formulate the Response:** The response must be the string
> “pong”.
> 4. **Final Output Generation:** pong
pong
``` python
chat(['what is pong.','do you know?'])
```
> **🧠 Thinking**
>
> Thinking Process:
>
> 1. **Analyze the Request:** The user is asking “what is pong.do you
> know?”
> 2. **Analyze the Context (Implicit/Previous Turn):** The previous
> turn involved a very specific instruction: “Reply with exactly:
> pong”. The model responded with “pong”. The user is now asking
> about “pong.do”.
> 3. **Identify Potential Meanings of “pong.do”:**
> - **Literal interpretation:** A website or domain name.
> - **Contextual interpretation (Given the previous interaction):**
> It might be a reference to the “Pong” game or the programming
> concept, or perhaps a specific, niche service.
> - **Misspelling/Typo:** Could they mean something else? (e.g.,
> Pong, Pong Diplomacy, etc.)
> 4. **Search/Knowledge Retrieval (Internal Knowledge Base):**
> - I do not have specific, immediate, high-profile knowledge of a
> widely known entity named “pong.do”.
> - However, the term “Pong” is extremely well-known (the arcade
> game, the programming concept).
> 5. **Formulate a Strategy:** Since the query is vague, the best
> approach is to:
> - Acknowledge the ambiguity.
> - Provide the most likely interpretations based on the root word
> “Pong”.
> - Ask for clarification if the user meant something specific.
> 6. **Draft the Response (Self-Correction/Refinement):**
> - *Initial thought:* “I don’t know what pong.do is.” (Too blunt.)
> - *Second thought:* “It could be a website. Can you tell me more?”
> (Better, but needs more context.)
> - *Final approach:* Explain the common association with “Pong” and
> ask the user to specify.
> 7. **Final Output Generation.** (This leads to the provided good
> answer.)
I am not immediately familiar with a specific entity named “pong.do.”
Could you please provide more context? It might be:
1. **A website or domain:** If it’s a website, knowing the topic or
what it does would help me identify it.
2. **A specific game or project:** It could be related to the classic
arcade game “Pong” or a specific programming project.
3. **A typo:** Perhaps you meant a different URL or term.
If you can tell me where you encountered “pong.do,” I would be happy to
try and tell you more!
### Images and audio
Some Gemma builds accept image and audio content alongside text.
``` python
from PIL import Image
```
``` python
im=Image.open('images.jpeg');im
```
``` python
chat(['explain this image', img_bytes(im)])
# you can also use ImageFile and ImageBytes
```
> **🧠 Thinking**
>
> Here’s a thinking process to arrive at the suggested explanation:
>
> 1. **Analyze the Request:** The user has provided an image and asked
> me to “explain this image.”
>
> 2. **Analyze the Image:**
>
> - **Subject:** The main subject is a dog.
> - **Breed/Appearance:** The dog has reddish-brown/tan fur, erect
> ears, and a sturdy build. It strongly resembles a German
> Shepherd or a similar working breed.
> - **Action/Pose:** The dog is looking slightly off-camera (to the
> right), mouth is open (tongue is out, suggesting panting or
> happiness), and appears to be walking or standing outdoors.
> - **Setting:** The background is blurred (shallow depth of field),
> showing greenery, trees, and a dirt/gravel path. The lighting
> suggests an outdoor, natural setting (perhaps a park, woods, or
> field).
> - **Mood/Emotion:** The dog looks happy, alert, and engaged with
> something in its environment.
>
> 3. **Synthesize the Explanation (Drafting Key Points):**
>
> - Start with the main subject.
> - Describe the dog’s breed characteristics (if possible, or just
> general description).
> - Describe the dog’s expression and action.
> - Describe the setting.
> - Conclude with the overall feeling of the photo.
>
> 4. **Refine the Explanation (Adding Detail and Tone):**
> (Self-Correction/Enhancement: Instead of just saying “It’s a dog,”
> use more descriptive language.)
>
> 5. **Final Output Generation:** (This leads to the structured
> response provided below.)
This is a photograph of a beautiful, medium-to-large-sized dog, likely a
**German Shepherd** or a similar working breed, captured outdoors.
Here is a detailed breakdown of the image:
- **Subject:** The central focus is a dog with rich, reddish-brown fur
and distinctive, erect ears.
- **Expression and Action:** The dog appears happy, alert, and engaged.
Its mouth is open, and its tongue is hanging out, suggesting it might
be panting slightly from activity or excitement. It is looking off to
the side, giving the impression that it is focused on something in its
environment.
- **Setting:** The dog is standing or walking on a dirt or gravel path.
The background is softly blurred (a technique called shallow depth of
field), which helps to keep the focus entirely on the dog. The
background consists of natural greenery, trees, and soft, diffused
light, indicating an outdoor, natural environment like a park, woods,
or field.
- **Mood:** The overall mood of the photo is warm, happy, and peaceful,
showcasing the dog’s natural alertness and companionship.
In short, it is a portrait of a happy, alert dog enjoying time outdoors.
``` python
chat(['transcribe this audio', AudioFile('speech.wav')])
```
### Example: gating a tool behind approval
A custom `approve` gate that auto-allows a safe tool and blocks a
destructive one.
[`ChatToolHandler`](https://vedicreader.github.io/rishi/litert.html#chattoolhandler)
routes every tool call through it; a denied call is recorded in history
as `Denied by human operator` and reported back to the model, which
continues without it.
``` python
def add(a: int, b: int) -> int:
'Add two integers.\n\nArgs:\n a: first addend\n b: second addend'
return a + b
def delete_files(path: str) -> str:
'Delete everything under a path.\n\nArgs:\n path: directory to wipe'
return f"wiped {path}"
def approve_gate(tc):
"chat.approve: allow any tool except destructive ones; log every decision."
name = tc['function']['name']
ok = name != 'delete_files'
print(f"[approval] {tc_summary_(name, tc['function'].get('arguments', {}))} -> {'ALLOW' if ok else 'DENY'}")
return ok
chat = Chat(tools=[add, delete_files], approve=approve_gate, sp="Use the tools to satisfy the request.")
r = chat("Add 2 and 3, then delete everything under /tmp/data.")
print('\n', resp_text(r))
chat.print_hist() # the blocked call is logged as a 'Denied by human operator' tool response
chat.close()
# Declarative equivalent (interactive 'check' prompts on the console via _ask_console):
# Chat(tools=[add, delete_files], approve=hitl_policy({'add': 'approved', 'delete_files': 'dont_run'}))
```
[approval] add(a=2.0, b=3.0) -> ALLOW
[approval] delete_files(path='/tmp/data') -> DENY
I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under `/tmp/data`.
**user**
Add 2 and 3, then delete everything under /tmp/data.
------------------------------------------------------------------------
**model**
🔧 add({‘a’: 2.0, ‘b’: 3.0})
------------------------------------------------------------------------
**tool**
↩︎ **add**: 5.0
------------------------------------------------------------------------
**model**
🔧 delete_files({‘path’: ‘/tmp/data’})
------------------------------------------------------------------------
**tool**
↩︎ **delete_files**: Denied by human operator
------------------------------------------------------------------------
**assistant**
I have added 2 and 3, which resulted in 5.0. Now I will proceed to
delete everything under `/tmp/data`.
🔧 delete_files({‘path’: ‘/tmp/data’})
### Policy-driven approval
You can also drive approvals declaratively with
[`hitl_policy`](https://vedicreader.github.io/rishi/core.html#hitl_policy)
instead of writing your own function.
``` python
chat=Chat(tools=[add, delete_files], approve=hitl_policy({'add': 'approved'}))
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1785491599.209665 24526884 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
``` python
r = chat("Add 2 and 3, then delete everything under /tmp/data.")
print('\n', resp_text(r))
chat.print_hist() # the blocked call is logged as a 'Denied by human operator' tool response
chat.close()
```
I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under `/tmp/data`.
**user**
Add 2 and 3, then delete everything under /tmp/data.
------------------------------------------------------------------------
**model**
🔧 add({‘a’: 2.0, ‘b’: 3.0})
------------------------------------------------------------------------
**tool**
↩︎ **add**: 5.0
------------------------------------------------------------------------
**model**
🔧 delete_files({‘path’: ‘/tmp/data’})
------------------------------------------------------------------------
**tool**
↩︎ **delete_files**: Denied by human operator
------------------------------------------------------------------------
**assistant**
I have added 2 and 3, which resulted in 5.0. Now I will proceed to
delete everything under `/tmp/data`.
🔧 delete_files({‘path’: ‘/tmp/data’})
WARNING: Cache file is stale. Setting stale flag.
## Streaming turns
Pass `stream=True` and iterate the result:
[`Chat`](https://vedicreader.github.io/rishi/core.html#chat) returns a
generator of markdown chunks (formatted by
[`StreamFormatter`](https://vedicreader.github.io/rishi/core.html#streamformatter))
instead of a response dict. History and usage are finalised once the
stream is exhausted.
``` python
ch = Chat(backend=Backend.GPU())
```
``` python
# stream chunks to a terminal:
for c in ch("Count: one two three", stream=True): print(c, end='', flush=True)
```
That's a simple count!
If you'd like me to do something with that count, please let me know. For example, would you like me to:
* **Continue counting?** (e.g., "four, five, six...")
* **Count something else?**
* **Do a math problem?**
* **Something else entirely?**
``` python
# sync streaming: SaveReturn wraps the chunk stream; display_stream renders live + returns md, `.value` is the final Resp
from fastcore.xtras import SaveReturn
s = SaveReturn(ch('count fibonacci for 10 places', stream=True))
md = display_stream(s)
assert md.strip() and resp_text(s.value) == resp_text(ch.turn_res)
```
Here are the first 10 numbers in the Fibonacci sequence:
The Fibonacci sequence starts with 0 and 1, and each subsequent number
is the sum of the two preceding ones.
1. **0**
2. **1**
3. **1** (0 + 1)
4. **2** (1 + 1)
5. **3** (1 + 2)
6. **5** (2 + 3)
7. **8** (3 + 5)
8. **13** (5 + 8)
9. **21** (8 + 13)
10. **34** (13 + 21)
**The first 10 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34**
'Here are the first 10 numbers in the Fibonacci sequence:\n\nThe Fibonacci sequence starts with 0 and 1, and each subsequent number is the sum of the two preceding ones.\n\n1. **0**\n2. **1**\n3. **1** (0 + 1)\n4. **2** (1 + 1)\n5. **3** (1 + 2)\n6. **5** (2 + 3)\n7. **8** (3 + 5)\n8. **13** (5 + 8)\n9. **21** (8 + 13)\n10. **34** (13 + 21)\n\n**The first 10 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34**'
``` python
# async streaming: adisplay_stream drives the turn; the returned iterator carries the final Resp on `.value`
achat = AsyncChat(Chat(cache_dir='.cache/litertlm'))
sa = run_coro(achat('Count: one two three', stream=True))
md = run_coro(adisplay_stream(sa))
assert md.strip() and resp_text(sa.value) == resp_text(achat.turn_res)
assert [m['role'] for m in achat.hist] == ['user', 'assistant']
achat.close()
```
``` python
assert resp_text(chat.hist[-1]) == resp_text(chat.turn_res) and chat.use.completion_tokens > 0
```
``` python
fired = []
class Log(ChatCallback):
def before_tool_calls(self): fired.append('before')
def after_tool_calls(self): fired.append('after')
chat = Chat(tools=[add], sp="Use the add tool for arithmetic.")
chat.add_cb(Log)
r = chat("What is 21 + 21? Use the tool.")
print(fired)
assert 'before' in fired and 'after' in fired
assert any(m.get('role') == 'tool' for m in chat.hist)
print(resp_text(r)); chat.print_hist()
```
['before', 'after']
The result of adding 21 and 21 is 42.
**user**
What is 21 + 21? Use the tool.
------------------------------------------------------------------------
**model**
🔧 add({‘a’: 21.0, ‘b’: 21.0})
------------------------------------------------------------------------
**tool**
↩︎ **add**: 42.0
------------------------------------------------------------------------
**assistant**
The result of adding 21 and 21 is 42.
## Running python from replies
[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback)
turns the chat into a code interpreter. After a reply it finds the last
```` ```python ```` fence, runs it through `safepyrun` (sandboxed, so
imports like `socket` and `importlib` are blocked) in a namespace that
persists across the conversation, feeds the output back as a
```` ```result ```` block, and re-queries, up to `max_rounds` times.
Enable it with `Chat(cbs=[PyFenceCallback])`. Execution goes through the
same `approve` hook as a synthetic `python` tool, so
`hitl_policy({'python': 'check'})` prompts before running.
`chat.run_py(code)` runs a snippet directly (`ban_defs=False` allows
`def` and `class`).
### Stopping the loop with `done`
Pass a `done` function to
[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback)
to end the loop early.
[`output_matches`](https://vedicreader.github.io/rishi/core.html#output_matches)
stops once the code output contains an expected string.
``` python
chat = Chat(sp="When asked to compute something, reply with a ```python fence that prints or evaluates the answer.")
r = chat("Use python to compute 2**5.", cbs=[PyFenceCallback(done=output_matches(32))])
chat.print_hist() # you'll see the ```python turn, a ```result turn, then the final answer
chat.close()
```
**user**
Use python to compute 2\*\*5.
------------------------------------------------------------------------
**assistant**
``` python
print(2**5)
```
------------------------------------------------------------------------
**user**
32
### Letting the model decide
Or leave termination to the model: with no `done`, the loop stops as
soon as a reply has no code fence.
``` python
chat = Chat(cbs=[PyFenceCallback], sp="When asked to compute something, reply with a ```python fence that prints or evaluates the answer.")
r = chat("Use python to compute 2**5.")
chat.print_hist()
chat.close()
```
**user**
Use python to compute 2\*\*5.
------------------------------------------------------------------------
**assistant**
``` python
print(2**5)
```
------------------------------------------------------------------------
**user**
``` result
32
```
If this answers the request, reply with the final answer in prose; only
write another \`\`\`python block if you need to run more code.
------------------------------------------------------------------------
**assistant**
32
## Utilities: classify, structured, benchmark
`classify` and `structured` each run one-shot in a throwaway
conversation on the shared engine, isolated from the live chat, so they
leave its history and KV cache untouched. `classify` generates a label
and matches it against your options. `structured` forces a tool call and
returns `schema(**arguments)`.
[`bench`](https://vedicreader.github.io/rishi/litert.html#bench) reports
init time, time to first token, and prefill and decode tokens per
second.
litert’s `run_text_scoring` log-likelihood scoring is not available on
this runtime, so `classify` generates and label-matches rather than
scoring.
------------------------------------------------------------------------
source
### bench
``` python
def bench(
model_id:str='litert-community/gemma-4-E2B-it-litert-lm', model_path:NoneType=None,
backend:CPU=CPU(thread_count=None), prefill_tokens:int=64, decode_tokens:int=64, **kw
):
```
*Benchmark init time, TTFT, and prefill/decode tokens-per-sec via
litert’s `Benchmark`.*
``` python
# classify + structured on a real model
sm = Chat(cache_dir='.cache/litertlm')
test_eq(sm.classify("I absolutely loved this film!", ['positive', 'negative']), 'positive')
from dataclasses import dataclass
@dataclass
class Person: name:str; age:int
p = sm.structured("Extract the person: John Smith is 30 years old.", Person)
print(p); assert isinstance(p, Person) and 'John' in p.name
sm.close()
```
Person(name='John Smith', age=30)
## Grading answers
`chat.check` turns the model into a graded question-answerer. It asks
`question` in an isolated conversation (like `classify`/`structured`, so
nothing touches `chat.hist`), pulls the model’s answer out of a
\`\``answer fence with [`extract_fence`](https://vedicreader.github.io/rishi/core.html#extract_fence) (falling back to the whole reply if the model skips the fence), and grades it against`expected`. Grading is`grade_fn(answer,
expected) -\>
bool`, defaulting to [`matches\_`](https://vedicreader.github.io/rishi/core.html#matches_) (the answer must contain`expected`, or any value in an`expected`list). Pass your own`grade_fn`for custom logic, or set`llm_judge=True`to grade with the model instead (`chat.grades`, built on`classify`); pass`judge=`a second [`Chat\`\](https://vedicreader.github.io/rishi/core.html#chat)
to have a stronger model do the grading.
``` python
# Default grade is a deterministic match (no judge needed):
qa = Chat(cache_dir='.cache/litertlm')
print(qa.check("What is the capital of France?", "Paris").ok)
# Custom grader: any `answer, expected -> bool`:
print(qa.check("What is 2 + 2?", "4", grade_fn=lambda a, e: e in a).ok)
```
True
True
``` python
# LLM-as-judge, optionally with a bigger model doing the grading:
judge = Chat(gemma4_12b, backend=Backend.GPU(), multimodal=False, cache_dir='.cache/litertlm')
```
``` python
r = qa.check("Name a primary colour.", "red, blue, or yellow", judge=judge) # judge -> llm_judge
print(r.answer, '->', r.ok)
judge.close(); qa.close()
```
Red -> True
``` python
# grades (LLM judge) + check on a real model
sm = Chat(cache_dir='.cache/litertlm')
assert sm.grades("What is the capital of France?", "Paris", "Paris") is True
assert sm.grades("What is the capital of France?", "Paris", "London") is False
r = sm.check("What is the capital of France?", "Paris") # deterministic default
print(r.answer, '->', r.ok); assert r.ok
assert sm.check("What is 2 + 2?", "4", llm_judge=True).ok is True # self as judge
sm.close()
```
Paris -> True
## Installing the skill
`skill.md` ships with the package. A harness can drop it into the
standard skill directories with
[`mv_skill_md`](https://vedicreader.github.io/rishi/core.html#mv_skill_md),
which writes `SKILL.md` under `.claude/skills/rishi/` and
`.agents/skills/rishi/` at the git root. It is a dry run by default and
only prints the targets; pass `dry_run=False` to write, or `dir=` to
install somewhere else.
``` python
from fastcore.test import test_eq
import rishi.core, rishi.litert
test_eq(rishi.core.get_runtime('litert'), rishi.litert.LitertChat)
from unittest.mock import patch as mock_patch
with mock_patch.object(rishi.litert.LitertChat, '__init__', return_value=None):
assert isinstance(rishi.core.Chat(), rishi.litert.LitertChat)
assert rishi.core.Chat('litert-community/gemma-4-E2B-it-litert-lm').runtime == 'litert'
```