# core
## Messages
litert speaks a small message schema (`Message`, `Contents`, and
`Content` subtypes). These helpers build them from ordinary Python
values.
- [`mk_content`](https://vedicreader.github.io/rishi/core.html#mk_content)
maps one value to a litert `Content`: a `str` becomes text, `bytes`
become an image or audio (sniffed with `detect_mime`), and a `Path`
becomes an image or audio file reference (by MIME). An existing
`Content` passes through.
- [`mk_msg`](https://vedicreader.github.io/rishi/core.html#mk_msg) wraps
content into a `Message` (default role `user`), and also accepts an
existing `Message` or a `{'role','content'}` dict.
- [`mk_msgs`](https://vedicreader.github.io/rishi/core.html#mk_msgs)
normalises a mixed list into canonical litert message dicts, used to
seed a conversation’s history.
------------------------------------------------------------------------
source
### mk_msgs
``` python
def mk_msgs(
msgs
):
```
*Normalize a list of messages to litert message dicts.*
------------------------------------------------------------------------
source
### mk_msg
``` python
def mk_msg(
content, role:str='user'
):
```
*Create a litert `Message` from str/bytes/list/dict/Message.*
------------------------------------------------------------------------
source
### mk_content
``` python
def mk_content(
o
):
```
*Convert `o` to a litert `Content`, sniffing bytes/files for image vs
audio.*
``` python
from fastcore.test import test_eq
```
``` python
test_eq(mk_msg("hello").to_json(), {"role": "user", "content": [{"type": "text", "text": "hello"}]})
test_eq(mk_msg("hi", role="model").to_json()["role"], "model")
test_eq([x["role"] for x in mk_msgs(["a", mk_msg("b", role="model")])], ["user", "model"])
assert isinstance(mk_content("x"), Text)
# bytes are sniffed: image vs audio
assert isinstance(mk_content(b'\x89PNG\r\n\x1a\n' + b'\x00'*16), ImageBytes)
assert isinstance(mk_content(b'RIFF\x00\x00\x00\x00WAVE' + b'\x00'*16), AudioBytes)
```
## Usage tracking
litert doesn’t return per-response token counts, but `conv.token_count`
exposes the running KV-cache size (prefill plus decode).
[`UsageStats`](https://vedicreader.github.io/rishi/core.html#usagestats)
records one turn’s `prompt`, `completion`, and `total` tokens, derived
from the count delta around the turn, and adds across turns. Compare
`chat.token_count` (or `chat.pct_full`) against `ctx_limit` to decide
when to compress the conversation.
------------------------------------------------------------------------
source
### UsageStats
``` python
def UsageStats(
prompt_tokens:int=0, completion_tokens:int=0, total_tokens:int=0, n:int=0
):
```
*Token usage for a chat turn, fed by `conv.token_count` diffs.*
``` python
a = UsageStats(prompt_tokens=10, completion_tokens=5, total_tokens=15, n=1)
b = UsageStats(prompt_tokens=3, completion_tokens=2, total_tokens=20, n=1)
c = a + b
test_eq((c.prompt_tokens, c.completion_tokens, c.n), (13, 7, 2))
assert "in=13" in repr(c) and "out=7" in repr(c)
test_eq((a + None).prompt_tokens, 10)
```
## Callbacks
A small, ordered callback system inspired by fastllm. A
[`ChatCallback`](https://vedicreader.github.io/rishi/core.html#chatcallback)
subclass hooks named events (`after_msgs`, `before_send`,
`after_response`, `before_tool_calls`, `after_tool_calls`) and reads
live turn state off the chat via `GetAttr`, so `self.turn_msg` is
`chat.turn_msg`.
[`run_cbs`](https://vedicreader.github.io/rishi/core.html#run_cbs)
dispatches one event to every enabled callback in `order`, forwarding
anything a callback yields into the output stream.
------------------------------------------------------------------------
source
### run_cbs
``` python
def run_cbs(
chat, event
):
```
*Dispatch `event` to enabled callbacks in `order`; forward any yielded
stream items.*
------------------------------------------------------------------------
source
### ChatCallback
``` python
def ChatCallback(
*args, **kwargs
):
```
*Base chat callback; reads chat state via `GetAttr` (`self.turn_msg` -\>
`chat.turn_msg`).*
``` python
class _Dummy: pass
class _A(ChatCallback):
order = 20
def before_send(self): self.chat.log.append('A')
class _B(ChatCallback):
order = 10
def before_send(self):
self.chat.log.append('B')
yield {'text': 'from-B'}
d = _Dummy(); d.log = []
a, b = _A(), _B(); a.chat = d; b.chat = d
d.cbs = L(a, b)
out = list(run_cbs(d, 'before_send'))
test_eq(d.log, ['B', 'A'])
test_eq(out, [{'text': 'from-B'}])
test_eq(repr(a), '_A')
```
## Streaming display
[`StreamFormatter`](https://vedicreader.github.io/rishi/core.html#streamformatter)
turns a litert response stream into markdown as it arrives: text passes
through, and tool calls render as a compact `⏳ name(args)` line.
[`display_stream`](https://vedicreader.github.io/rishi/core.html#display_stream)
consumes a markdown-chunk stream (what `chat(msg, stream=True)` yields)
and renders it live in a notebook via IPython.
[`mk_tr_details`](https://vedicreader.github.io/rishi/core.html#mk_tr_details)
formats a completed tool call as a collapsible JSON block.
------------------------------------------------------------------------
source
### display_stream
``` python
def display_stream(
chunks
):
```
*Progressively render a markdown-chunk stream
(e.g. `chat(msg, stream=True)`) live in a notebook; returns the full
markdown.*
------------------------------------------------------------------------
source
### StreamFormatter
``` python
def StreamFormatter(
mx:int=2000, showthink:bool=True
):
```
*Format a litert response stream to markdown; thinking streams as a
blockquote.*
------------------------------------------------------------------------
source
### mk_tr_details
``` python
def mk_tr_details(
name, args, result, mx:int=2000
):
```
*`` JSON block for a completed tool call.*
------------------------------------------------------------------------
source
### Resp
``` python
def Resp(
*args, **kwargs
):
```
*A litert response dict that renders as markdown (thinking + text + tool
calls/responses) in notebooks.*
------------------------------------------------------------------------
source
### thought
``` python
def thought(
resp
):
```
*The model’s thinking (`channels.thought`) for a litert response/chunk,
or ’’.*
------------------------------------------------------------------------
source
### resp_text
``` python
def resp_text(
resp
):
```
*Join text parts of a litert response/chunk dict.*
## Built-in callbacks
Three callbacks make up `_dflt_cbs` and run on every chat unless you
pass `default_cbs=False`.
[`HistoryCallback`](https://vedicreader.github.io/rishi/core.html#historycallback)
records each outgoing message and reply into `chat.hist`.
[`UsageCallback`](https://vedicreader.github.io/rishi/core.html#usagecallback)
folds the turn’s token counts into `chat.use`, read from a
`conv.token_count` delta.
[`ToolReminderCallback`](https://vedicreader.github.io/rishi/core.html#toolremindercallback)
appends a short reminder to outgoing messages, but only when the chat
has tools, nudging the model to summarise tool results in prose before
it continues.
Order matters.
[`HistoryCallback`](https://vedicreader.github.io/rishi/core.html#historycallback)
and
[`UsageCallback`](https://vedicreader.github.io/rishi/core.html#usagecallback)
sit at the front (low `order`) so a turn is recorded before feature
callbacks like
[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback)
react to it and feed messages back. A callback reads live turn state off
the chat through `GetAttr`, so `self.turn_res` is `chat.turn_res`.
------------------------------------------------------------------------
source
### ToolReminderCallback
``` python
def ToolReminderCallback(
tool_reminder:str='\nAfter every tool call result, briefly summarise in prose what you found before continuing or calling another tool.'
):
```
*Inject a tool-summary reminder into the outgoing message when tools are
registered.*
------------------------------------------------------------------------
source
### TruncationCallback
``` python
def TruncationCallback(
max_tokens
):
```
*Flag `turn_res['truncated']` when the reply reaches `max_tokens` output
tokens (best-effort).*
------------------------------------------------------------------------
source
### truncated
``` python
def truncated(
resp
):
```
*Whether `resp` was flagged as cut off at the token cap.*
------------------------------------------------------------------------
source
### UsageCallback
``` python
def UsageCallback(
*args, **kwargs
):
```
*Fold each response’s token usage into `chat.use` from a `token_count`
diff.*
------------------------------------------------------------------------
source
### HistoryCallback
``` python
def HistoryCallback(
*args, **kwargs
):
```
*Record the outgoing message and the response into `chat.hist`.*
## Tool calling, approval, and history
litert runs the tool-call loop inside the engine.
[`ChatToolHandler`](https://vedicreader.github.io/rishi/core.html#chattoolhandler)
bridges that loop back to Python: for each call it records the request
and result into `chat.hist` (built with litert’s own `Message` and
`ToolResponse`), fires the `before_tool_calls` and `after_tool_calls`
callbacks, and consults `chat.approve(tool_call)` before executing.
Returning `False` blocks the tool and feeds a “Denied by human operator”
response back to the model. That is the hook for human-in-the-loop
gating, shown below.
------------------------------------------------------------------------
source
### ChatToolHandler
``` python
def ChatToolHandler(
chat
):
```
*Bridge litert’s in-engine tool loop to Chat callbacks, HITL approval,
and history.*
## Loading models & Chat
Default litert-community Gemma repos live at
[huggingface.co/litert-community](https://huggingface.co/litert-community).
[`get_model`](https://vedicreader.github.io/rishi/core.html#get_model)
resolves a `.litertlm` file with a cache-first ladder: an explicit
`model_path`, then the local HuggingFace cache (`scan_cache_dir`, no
network), then a download. It prefers the native build over `-web`
variants, which omit the CPU/GPU decode graph.
[`Chat`](https://vedicreader.github.io/rishi/core.html#chat) ties this
together into a callable. Build it (it constructs or accepts an
`Engine`), then call it like a function: one turn per call, updating
history and usage. `create_engine` is a patchable classmethod that
builds the engine and creates `cache_dir` if given. Pass a prebuilt
`engine=` to share one model across several chats. Engine and
conversation are entered on an `ExitStack`, so `close()` releases them
in the right order and never closes an engine you supplied.
------------------------------------------------------------------------
source
### Chat
``` python
def Chat(
engine:Engine=None, # Litert engine, or None to build one
model_id:str='litert-community/gemma-4-E2B-it-litert-lm', # huggingface model id; see https://huggingface.co/litert-community
model_path:PathLike=None, # local litertlm model path
backend:Backend=CPU(thread_count=None), # backend for the engine
multimodal:bool=True, # multimodal model
cache_dir:PathLike=None, # cache dir for the engine
enable_speculative_decoding:NoneType=None, eng_kw:NoneType=None, # extra litert Engine kwargs
sp:str='', # system prompt
messages:NoneType=None, # message history to prefill the conversation
tools:NoneType=None, # tools to register with the engine
ctx_limit:NoneType=None, # context window, for pct_full
approve:NoneType=None, # approval function for tool calls
tool_max_len:NoneType=None, # truncate string tool results longer than this (protects context)
think:bool=False, # enable the model's thinking channel (if supported)
filter_think:bool=True, # keep thinking out of the KV cache (saves context)
temp:NoneType=None, top_k:NoneType=None, top_p:NoneType=None,
seed:NoneType=None, # sampler knobs (build a SamplerConfig)
sampler_config:NoneType=None, # or pass a full SamplerConfig (overrides the knobs)
max_output_tokens:NoneType=None, conv_kw:NoneType=None, # extra create_conversation kwargs
cbs:NoneType=None, # callbacks to register, checkout default callbacks liek HistoryCallback, UsageCallback, ToolReminderCallback
default_cbs:bool=True, # add default callbacks.
):
```
*Sync chat over a local litert_lm engine. Callbacks record
history/usage; `_send` drives one message.*
------------------------------------------------------------------------
source
### get_model
``` python
def get_model(
model_id, model_path:NoneType=None
):
```
*Return a local `.litertlm` path: `model_path`, else HF cache, else
download.*
### Managing callbacks
`add_cb` registers a callback and returns the instance; `add_cbs` takes
a list and returns the instances. `remove_cb` drops one by instance, or
by class to remove every callback of that type, and `remove_cbs` handles
several at once. To run a callback for a single turn, pass `cbs=` to the
call: `chat(msg, cbs=[PyFenceCallback(...)])` registers them before the
turn and removes them after, so nothing leaks into later turns.
``` python
set_min_log_severity(3)
get_model(gemma4_12b)
```
'/Users/71293/.cache/huggingface/hub/models--litert-community--gemma-4-12B-it-litert-lm/snapshots/44cf85a326f79b814fa86a60af414c042755b43a/gemma-4-12B-it.litertlm'
## Human-in-the-loop tool approval
Any `approve(tool_call) -> bool` passed to `Chat(approve=...)` is
consulted by
[`ChatToolHandler`](https://vedicreader.github.io/rishi/core.html#chattoolhandler)
before each tool runs.
[`hitl_policy`](https://vedicreader.github.io/rishi/core.html#hitl_policy)
builds one from a per-tool policy: `'approved'` (auto-run), `'dont_run'`
(always block), or `'check'` (ask, via
[`_ask_console`](https://vedicreader.github.io/rishi/core.html#_ask_console)
by default). Or supply your own function for custom logic such as
logging, rate-limiting, or prompting a UI. See the worked example below.
------------------------------------------------------------------------
source
### hitl_policy
``` python
def hitl_policy(
modes, ask:function=_ask_console
):
```
*Build an `approve(tool_call)` from per-tool modes: ‘approved’ | ‘check’
| ‘dont_run’.*
## Using Chat
A short tour of the main features. These cells build a real model, so
they are set to run manually rather than in the test suite.
``` python
chat=Chat(backend=Backend.GPU(), cache_dir='.cache/litertlm', think=True)
# chat_12 = Chat(Chat.create_engine(gemma4_12b, multimodal=False, cache_dir='.cache/litertlm',be=Backend.GPU()))
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784178935.259078 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
``` python
# set_min_log_severity(2)
r = chat("Reply with exactly: pong")
assert 'pong' in resp_text(r).lower()
assert chat.hist[-1] is r and chat.hist[0]['role'] == 'user'
assert chat.use.total_tokens > 0 and chat.token_count > 0
print(chat.use); chat.print_hist()
```
W0000 00:00:1784178936.197760 66965901 sampler_factory.cc:450] WebGPU sampler not available, falling back to statically linked C API: UNAVAILABLE: ERROR: [third_party/odml/litert_lm/runtime/components/sampler_factory.cc:294]
└ Could not load shared library libLiteRtTopKWebGpuSampler.dylib: dlopen(libLiteRtTopKWebGpuSampler.dylib, 0x0005): tried: 'libLiteRtTopKWebGpuSampler.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibLiteRtTopKWebGpuSampler.dylib' (no such file), '/usr/lib/libLiteRtTopKWebGpuSampler.dylib' (no such file, not in dyld cache), 'libLiteRtTopKWebGpuSampler.dylib' (no such file).
=== Source Location Trace: ===
./third_party/odml/litert/litert/cc/litert_macros.h:573
total=104|in=103|out=1|turns=1
**user**
Reply with exactly: pong
------------------------------------------------------------------------
**assistant**
> **🧠 Thinking**
>
> Thinking Process:
>
> 1. **Analyze the Request:** The user has instructed me to reply with
> *exactly* the word “pong”.
> 2. **Determine the Constraint:** The constraint is strict: “Reply
> with exactly: pong”.
> 3. **Formulate the Response:** The response must be the string
> “pong”.
> 4. **Final Output Generation:** pong
pong
``` python
chat(['what is pong','do you know?'])
```
> **🧠 Thinking**
>
> Thinking Process:
>
> 1. **Analyze the Request:** The user is asking “what is pongdo you
> know?”
>
> 2. **Analyze the Context (Implicit):** The previous interaction was a
> very short, specific instruction: “Reply with exactly: pong”. The
> model responded with “pong”. The user is now asking about
> “pongdo”.
>
> 3. **Identify Potential Interpretations of “pongdo”:**
>
> - **Typo/Misspelling:** Is it a misspelling of “Pong”?
> - **Specific Term/Acronym:** Is it a term in a niche field
> (programming, gaming, science)?
> - **Combination:** Is it a combination of “Pong” and another word?
> - **Contextual Link:** Does it relate to the previous interaction
> (the word “pong”)?
>
> 4. **Determine the Most Likely Meaning:** Given the previous
> interaction, the most direct link is the game *Pong*. “Pongdo” is
> not a standard, widely recognized term.
>
> 5. **Formulate a Strategy:** Since “pongdo” is ambiguous, the best
> approach is to address the most probable interpretation (Pong) and
> then ask for clarification if the user meant something else.
>
> 6. **Draft the Response (Self-Correction/Refinement):**
>
> - *Initial thought:* I don’t know what pongdo is.
> - *Refinement:* Be helpful. Assume they mean Pong.
>
> 7. **Final Output Generation.** (Since the user asked “what is pongdo
> you know?”, I should provide the most relevant information.)
I am not familiar with the term “pongdo.”
Could you please provide more context? It might be:
1. A misspelling of **Pong** (the classic arcade game).
2. A term from a specific game, programming language, or niche topic.
3. A specific name or acronym.
If you can give me more information, I would be happy to tell you what I
know!
### Images and audio
Some Gemma builds accept image and audio content alongside text.
``` python
from PIL import Image
```
``` python
im=Image.open('images.jpeg');im
```
``` python
chat(['explain this image', img_bytes(im)])
# you can also use ImageFile and ImageBytes
```
> **🧠 Thinking**
>
> Here’s a thinking process to arrive at the suggested response:
>
> 1. **Analyze the Request:** The user has provided an image and asked
> me to “explain this image.”
>
> 2. **Analyze the Image:**
>
> - **Subject:** The central focus is a dog.
> - **Breed/Appearance:** The dog has reddish-brown/tan fur, erect
> ears, and a strong build. It strongly resembles a German
> Shepherd or a similar working breed.
> - **Action/Pose:** The dog is looking slightly off-camera (to the
> right), mouth is open (tongue is visible), and appears happy or
> excited (panting/smiling).
> - **Setting:** The background is outdoors, consisting of a dirt
> path or trail, with blurred green foliage/trees on either side.
> The lighting suggests daytime.
> - **Mood:** Happy, alert, friendly, outdoorsy.
>
> 3. **Determine the Explanation Strategy:** A good explanation should
> cover the main elements (who, what, where, how) and the overall
> feeling.
>
> 4. **Draft the Explanation (Iterative Refinement):**
>
> - *Initial thought:* It’s a dog on a path. It looks happy.
> - *Adding detail (Focusing on the dog):* The dog looks like a
> German Shepherd. It has brown fur and is panting.
> - *Adding detail (Focusing on the setting):* It’s outdoors on a
> dirt path with greenery.
> - *Synthesizing and Polishing:* Combine these points into a
> coherent, descriptive paragraph.
>
> 5. **Final Review against the Goal:** Does this explain the image
> accurately? Yes. (Self-Correction: Ensure the tone is descriptive
> and engaging.)
>
> 6. **Generate the Output.** (This leads to the final provided
> answer.)
This is a photograph of a beautiful, happy-looking dog outdoors.
Here is a detailed breakdown of the image:
- **Subject:** The main focus is a medium-to-large dog, likely a German
Shepherd or a similar working breed, characterized by its rich,
reddish-brown fur and erect ears.
- **Action/Expression:** The dog appears alert and happy. Its mouth is
open, and its tongue is hanging out, suggesting it might be panting
slightly from activity or excitement. It is looking off to the side,
giving it an engaged and friendly expression.
- **Setting:** The dog is standing on a dirt or gravel path. The
background is soft and blurred (shallow depth of field), showing
natural greenery, trees, and foliage, indicating an outdoor, natural
environment, perhaps a park or a wooded trail.
- **Mood:** The overall mood of the photo is warm, joyful, and
adventurous, capturing a moment of a dog enjoying time outdoors.
### Example: gating a tool behind approval
A custom `approve` gate that auto-allows a safe tool and blocks a
destructive one.
[`ChatToolHandler`](https://vedicreader.github.io/rishi/core.html#chattoolhandler)
routes every tool call through it; a denied call is recorded in history
as `Denied by human operator` and reported back to the model, which
continues without it.
``` python
def add(a: int, b: int) -> int:
'Add two integers.\n\nArgs:\n a: first addend\n b: second addend'
return a + b
def delete_files(path: str) -> str:
'Delete everything under a path.\n\nArgs:\n path: directory to wipe'
return f"wiped {path}"
def approve_gate(tc):
"chat.approve: allow any tool except destructive ones; log every decision."
name = tc['function']['name']
ok = name != 'delete_files'
print(f"[approval] {_tc_summary(name, tc['function'].get('arguments', {}))} -> {'ALLOW' if ok else 'DENY'}")
return ok
chat = Chat(tools=[add, delete_files], approve=approve_gate, sp="Use the tools to satisfy the request.")
r = chat("Add 2 and 3, then delete everything under /tmp/data.")
print('\n', resp_text(r))
chat.print_hist() # the blocked call is logged as a 'Denied by human operator' tool response
chat.close()
# Declarative equivalent (interactive 'check' prompts on the console via _ask_console):
# Chat(tools=[add, delete_files], approve=hitl_policy({'add': 'approved', 'delete_files': 'dont_run'}))
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784178958.305192 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
[approval] add(a=2.0, b=3.0) -> ALLOW
[approval] delete_files(path='/tmp/data') -> DENY
I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under `/tmp/data`.
**user**
Add 2 and 3, then delete everything under /tmp/data.
------------------------------------------------------------------------
**model**
🔧 add({‘a’: 2.0, ‘b’: 3.0})
------------------------------------------------------------------------
**tool**
↩︎ **add**: 5.0
------------------------------------------------------------------------
**model**
🔧 delete_files({‘path’: ‘/tmp/data’})
------------------------------------------------------------------------
**tool**
↩︎ **delete_files**: Denied by human operator
------------------------------------------------------------------------
**assistant**
I have added 2 and 3, which resulted in 5.0. Now I will proceed to
delete everything under `/tmp/data`.
🔧 delete_files({‘path’: ‘/tmp/data’})
WARNING: Cache file is stale. Setting stale flag.
### Policy-driven approval
You can also drive approvals declaratively with
[`hitl_policy`](https://vedicreader.github.io/rishi/core.html#hitl_policy)
instead of writing your own function.
``` python
chat=Chat(tools=[add, delete_files], approve=hitl_policy({'add': 'approved'}))
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784178965.189187 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
``` python
r = chat("Add 2 and 3, then delete everything under /tmp/data.")
print('\n', resp_text(r))
chat.print_hist() # the blocked call is logged as a 'Denied by human operator' tool response
chat.close()
```
I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under `/tmp/data`.
**user**
Add 2 and 3, then delete everything under /tmp/data.
------------------------------------------------------------------------
**model**
🔧 add({‘a’: 2.0, ‘b’: 3.0})
------------------------------------------------------------------------
**tool**
↩︎ **add**: 5.0
------------------------------------------------------------------------
**model**
🔧 delete_files({‘path’: ‘/tmp/data’})
------------------------------------------------------------------------
**tool**
↩︎ **delete_files**: Denied by human operator
------------------------------------------------------------------------
**assistant**
I have added 2 and 3, which resulted in 5.0. Now I will proceed to
delete everything under `/tmp/data`.
🔧 delete_files({‘path’: ‘/tmp/data’})
WARNING: Cache file is stale. Setting stale flag.
## Streaming turns
Pass `stream=True` and iterate the result:
[`Chat`](https://vedicreader.github.io/rishi/core.html#chat) returns a
generator of markdown chunks (formatted by
[`StreamFormatter`](https://vedicreader.github.io/rishi/core.html#streamformatter))
instead of a response dict. History and usage are finalised once the
stream is exhausted.
``` python
ch = Chat(backend=Backend.GPU())
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784178987.905959 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
``` python
# stream chunks to a terminal:
for c in ch("Count: one two three", stream=True): print(c, end='', flush=True)
```
That's a simple count!
If you'd like me to do
W0000 00:00:1784178989.277368 66968557 sampler_factory.cc:450] WebGPU sampler not available, falling back to statically linked C API: UNAVAILABLE: ERROR: [third_party/odml/litert_lm/runtime/components/sampler_factory.cc:294]
└ Could not load shared library libLiteRtTopKWebGpuSampler.dylib: dlopen(libLiteRtTopKWebGpuSampler.dylib, 0x0005): tried: 'libLiteRtTopKWebGpuSampler.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibLiteRtTopKWebGpuSampler.dylib' (no such file), '/usr/lib/libLiteRtTopKWebGpuSampler.dylib' (no such file, not in dyld cache), 'libLiteRtTopKWebGpuSampler.dylib' (no such file).
=== Source Location Trace: ===
./third_party/odml/litert/litert/cc/litert_macros.h:573
something with that count, please let me know. For example, would you like me to:
* **Continue counting?** (e.g., "four, five, six...")
* **Count something else?**
* **Do a math problem?**
* **Something else entirely?**
``` python
# or render live in a notebook:
display_stream(ch("count fibonacci for 10 places", stream=True))
```
Here are the first 10 numbers in the Fibonacci sequence:
The Fibonacci sequence starts with 0 and 1, and each subsequent number
is the sum of the two preceding ones.
1. **0**
2. **1**
3. **1** (0 + 1)
4. **2** (1 + 1)
5. **3** (1 + 2)
6. **5** (2 + 3)
7. **8** (3 + 5)
8. **13** (5 + 8)
9. **21** (8 + 13)
10. **34** (13 + 21)
**The first 10 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34**
'Here are the first 10 numbers in the Fibonacci sequence:\n\nThe Fibonacci sequence starts with 0 and 1, and each subsequent number is the sum of the two preceding ones.\n\n1. **0**\n2. **1**\n3. **1** (0 + 1)\n4. **2** (1 + 1)\n5. **3** (1 + 2)\n6. **5** (2 + 3)\n7. **8** (3 + 5)\n8. **13** (5 + 8)\n9. **21** (8 + 13)\n10. **34** (13 + 21)\n\n**The first 10 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34**'
``` python
assert chat.hist[-1] == chat.turn_res and chat.use.completion_tokens > 0
```
``` python
fired = []
class Log(ChatCallback):
def before_tool_calls(self): fired.append('before')
def after_tool_calls(self): fired.append('after')
chat = Chat(tools=[add], sp="Use the add tool for arithmetic.")
chat.add_cb(Log)
r = chat("What is 21 + 21? Use the tool.")
print(fired)
assert 'before' in fired and 'after' in fired
assert any(m.get('role') == 'tool' for m in chat.hist)
print(resp_text(r)); chat.print_hist()
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784179010.555272 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
['before', 'after']
The result of adding 21 and 21 is 42.
**user**
What is 21 + 21? Use the tool.
------------------------------------------------------------------------
**model**
🔧 add({‘a’: 21.0, ‘b’: 21.0})
------------------------------------------------------------------------
**tool**
↩︎ **add**: 42.0
------------------------------------------------------------------------
**assistant**
The result of adding 21 and 21 is 42.
## Running python from replies
[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback)
turns the chat into a code interpreter. After a reply it finds the last
```` ```python ```` fence, runs it through `safepyrun` (sandboxed, so
imports like `socket` and `importlib` are blocked) in a namespace that
persists across the conversation, feeds the output back as a
```` ```result ```` block, and re-queries, up to `max_rounds` times.
Enable it with `Chat(cbs=[PyFenceCallback])`. Execution goes through the
same `approve` hook as a synthetic `python` tool, so
`hitl_policy({'python': 'check'})` prompts before running.
`chat.run_py(code)` runs a snippet directly (`ban_defs=False` allows
`def` and `class`).
------------------------------------------------------------------------
source
### PyFenceCallback
``` python
def PyFenceCallback(
max_rounds:int=5, done:NoneType=None
):
```
*Run
\`\``python fences, feed results back, loop until`done(chat)`(default: no fence).`max_rounds\`
is a safety cap.*
------------------------------------------------------------------------
source
### output_matches
``` python
def output_matches(
expected
):
```
*`done` policy: stop once the last code output contains `str(expected)`
(e.g. the answer to match against).*
------------------------------------------------------------------------
source
### task_complete
``` python
def task_complete(
chat
):
```
*`done` policy: judge (via `classify`, isolated) whether the latest
result completes the request.*
------------------------------------------------------------------------
source
### Chat.run_py
``` python
def run_py(
code, ban_defs:bool=False, g:NoneType=None
):
```
*Run `code` in this chat’s sandboxed, persistent namespace; return
stdout + last-expr repr.*
------------------------------------------------------------------------
source
### run_coro
``` python
def run_coro(
coro
):
```
*Run an awaitable to completion from sync code, even inside a running
event loop.*
------------------------------------------------------------------------
source
### mk_result_fence
``` python
def mk_result_fence(
out
):
```
*Feed a code result back, prompting a prose answer or more code.*
------------------------------------------------------------------------
source
### extract_fence
``` python
def extract_fence(
text, tag:str='answer'
):
```
*Contents of the last \`\`` fence in`text\`, else the whole
stripped text.*
------------------------------------------------------------------------
source
### extract_code
``` python
def extract_code(
text
):
```
*Code of the last \`\``python fence in`text\`, else None.*
### Stopping the loop with `done`
Pass a `done` function to
[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback)
to end the loop early.
[`output_matches`](https://vedicreader.github.io/rishi/core.html#output_matches)
stops once the code output contains an expected string.
``` python
chat = Chat(sp="When asked to compute something, reply with a ```python fence that prints or evaluates the answer.")
r = chat("Use python to compute 2**5.", cbs=[PyFenceCallback(done=output_matches(32))])
chat.print_hist() # you'll see the ```python turn, a ```result turn, then the final answer
chat.close()
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784179491.903529 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
**user**
Use python to compute 2\*\*5.
------------------------------------------------------------------------
**assistant**
``` python
print(2**5)
```
------------------------------------------------------------------------
**user**
32
WARNING: Cache file is stale. Setting stale flag.
### Letting the model decide
Or leave termination to the model: with no `done`, the loop stops as
soon as a reply has no code fence.
``` python
chat = Chat(cbs=[PyFenceCallback], sp="When asked to compute something, reply with a ```python fence that prints or evaluates the answer.")
r = chat("Use python to compute 2**5.")
chat.print_hist()
chat.close()
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784179784.097886 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
**user**
Use python to compute 2\*\*5.
------------------------------------------------------------------------
**assistant**
``` python
print(2**5)
```
------------------------------------------------------------------------
**user**
``` result
32
```
If this answers the request, reply with the final answer in prose; only
write another \`\`\`python block if you need to run more code.
------------------------------------------------------------------------
**assistant**
32
WARNING: Cache file is stale. Setting stale flag.
## Utilities: classify, structured, benchmark
`classify` and `structured` each run one-shot in a throwaway
conversation on the shared engine, isolated from the live chat, so they
leave its history and KV cache untouched. `classify` generates a label
and matches it against your options. `structured` forces a tool call and
returns `schema(**arguments)`.
[`bench`](https://vedicreader.github.io/rishi/core.html#bench) reports
init time, time to first token, and prefill and decode tokens per
second.
litert’s `run_text_scoring` log-likelihood scoring is not available on
this runtime, so `classify` generates and label-matches rather than
scoring.
------------------------------------------------------------------------
source
### bench
``` python
def bench(
model_id:str='litert-community/gemma-4-E2B-it-litert-lm', model_path:NoneType=None,
backend:CPU=CPU(thread_count=None), prefill_tokens:int=64, decode_tokens:int=64, **kw
):
```
*Benchmark init time, TTFT, and prefill/decode tokens-per-sec via
litert’s `Benchmark`.*
------------------------------------------------------------------------
source
### Chat.structured
``` python
def structured(
prompt, schema, sp:str='Call the tool to answer.'
):
```
*One-shot structured output: the model calls `schema` (a
function/class); returns `schema(**arguments)`.*
------------------------------------------------------------------------
source
### Chat.classify
``` python
def classify(
text, labels, sp:str='Reply with only the single best label and nothing else.'
):
```
*One-shot label for `text`, run in a throwaway conversation (isolated
from this chat).*
``` python
# classify + structured on a real model
sm = Chat(cache_dir='.cache/litertlm')
test_eq(sm.classify("I absolutely loved this film!", ['positive', 'negative']), 'positive')
from dataclasses import dataclass
@dataclass
class Person: name:str; age:int
p = sm.structured("Extract the person: John Smith is 30 years old.", Person)
print(p); assert isinstance(p, Person) and 'John' in p.name
sm.close()
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784180085.450553 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180085.451727 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180085.965230 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
Person(name='John Smith', age=30.0)
## Grading answers
`chat.check` turns the model into a graded question-answerer. It asks
`question` in an isolated conversation (like `classify`/`structured`, so
nothing touches `chat.hist`), pulls the model’s answer out of a
\`\``answer fence with [`extract_fence`](https://vedicreader.github.io/rishi/core.html#extract_fence) (falling back to the whole reply if the model skips the fence), and grades it against`expected`. Grading is`grade_fn(answer,
expected) -\>
bool`, defaulting to [`\_matches`](https://vedicreader.github.io/rishi/core.html#_matches) (the answer must contain`expected`, or any value in an`expected`list). Pass your own`grade_fn`for custom logic, or set`llm_judge=True`to grade with the model instead (`chat.grades`, built on`classify`); pass`judge=`a second [`Chat\`\](https://vedicreader.github.io/rishi/core.html#chat)
to have a stronger model do the grading.
------------------------------------------------------------------------
source
### Chat.check
``` python
def check(
question, expected, grade_fn:function=_matches, llm_judge:bool=False, judge:NoneType=None, tag:str='answer',
sp:str='Answer the question, then put your final answer inside a ```answer fence.'
):
```
*Ask `question` in a throwaway conversation, extract the
\`\`` answer, and grade it against`expected\`.*
------------------------------------------------------------------------
source
### Chat.grades
``` python
def grades(
question, expected, actual
):
```
*LLM-as-judge (on this chat’s engine): is `actual` a correct answer to
`question` given reference `expected`?*
``` python
# Default grade is a deterministic match (no judge needed):
qa = Chat(cache_dir='.cache/litertlm')
print(qa.check("What is the capital of France?", "Paris").ok)
# Custom grader: any `answer, expected -> bool`:
print(qa.check("What is 2 + 2?", "4", grade_fn=lambda a, e: e in a).ok)
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784180090.782984 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180090.783925 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
True
W0000 00:00:1784180091.531015 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
True
``` python
# LLM-as-judge, optionally with a bigger model doing the grading:
judge = Chat(model_id=gemma4_12b, backend=Backend.GPU(),multimodal=False, cache_dir='.cache/litertlm')
```
W0000 00:00:1784180096.290555 66921307 litert_lm_loader.h:158] TFLite model type: TF_LITE_VISION_ENCODER not found for backend constraints. Skipping.
W0000 00:00:1784180096.290570 66921307 litert_lm_loader.h:174] TFLite model type: TF_LITE_VISION_ENCODER not found for prefer activation type. Use system's default backend activation type. System's default activation type for Text decoder is fp16. Vision encoder and audio encoder default is fp32.
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784180109.242849 66921307 litert_lm_loader.cc:289] Section not found:
W0000 00:00:1784180109.242932 66921307 litert_lm_loader.h:129] TFLite model for type: TF_LITE_END_OF_VISION not found. Skipping.
W0000 00:00:1784180109.271313 66921307 litert_lm_loader.cc:289] Section not found:
W0000 00:00:1784180109.271323 66921307 litert_lm_loader.h:129] TFLite model for type: TF_LITE_PER_LAYER_EMBEDDER not found. Skipping.
W0000 00:00:1784180109.271470 66921307 litert_lm_loader.h:158] TFLite model type: TF_LITE_VISION_ENCODER not found for backend constraints. Skipping.
W0000 00:00:1784180109.271475 66921307 litert_lm_loader.h:174] TFLite model type: TF_LITE_VISION_ENCODER not found for prefer activation type. Use system's default backend activation type. System's default activation type for Text decoder is fp16. Vision encoder and audio encoder default is fp32.
W0000 00:00:1784180109.279493 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
``` python
r = qa.check("Name a primary colour.", "red, blue, or yellow", judge=judge) # judge -> llm_judge
print(r.answer, '->', r.ok)
judge.close(); qa.close()
```
W0000 00:00:1784180112.863596 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180113.297144 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180113.351198 66992751 sampler_factory.cc:450] WebGPU sampler not available, falling back to statically linked C API: UNAVAILABLE: ERROR: [third_party/odml/litert_lm/runtime/components/sampler_factory.cc:294]
└ Could not load shared library libLiteRtTopKWebGpuSampler.dylib: dlopen(libLiteRtTopKWebGpuSampler.dylib, 0x0005): tried: 'libLiteRtTopKWebGpuSampler.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibLiteRtTopKWebGpuSampler.dylib' (no such file), '/usr/lib/libLiteRtTopKWebGpuSampler.dylib' (no such file, not in dyld cache), 'libLiteRtTopKWebGpuSampler.dylib' (no such file).
=== Source Location Trace: ===
./third_party/odml/litert/litert/cc/litert_macros.h:573
Red -> True
``` python
# grades (LLM judge) + check on a real model
sm = Chat(cache_dir='.cache/litertlm')
assert sm.grades("What is the capital of France?", "Paris", "Paris") is True
assert sm.grades("What is the capital of France?", "Paris", "London") is False
r = sm.check("What is the capital of France?", "Paris") # deterministic default
print(r.answer, '->', r.ok); assert r.ok
assert sm.check("What is 2 + 2?", "4", llm_judge=True).ok is True # self as judge
sm.close()
```
WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1784180138.089785 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180138.090966 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180139.074466 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180139.374913 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
Paris -> True
W0000 00:00:1784180140.049391 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
W0000 00:00:1784180140.676127 66921307 mel_filterbank.cc:137] Missing 10 bands starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000
## Installing the skill
`skill.md` ships with the package. A harness can drop it into the
standard skill directories with
[`mv_skill_md`](https://vedicreader.github.io/rishi/core.html#mv_skill_md),
which writes `SKILL.md` under `.claude/skills/rishi/` and
`.agents/skills/rishi/` at the git root. It is a dry run by default and
only prints the targets; pass `dry_run=False` to write, or `dir=` to
install somewhere else.
------------------------------------------------------------------------
source
### mv_skill_md
``` python
def mv_skill_md(
dry_run:bool=True, dir:NoneType=None
):
```
*Copy the bundled `skill.md` into `.claude` and `.agents` skill dirs so
a harness can load the rishi skill.*
------------------------------------------------------------------------
source
### repo_root
``` python
def repo_root()->Path:
```
*Root of the current git repository, or None if not in one.*