litert

A fastllm-style Chat over litert_lm - message helpers, an ordered callback system, human-in-the-loop tool approval, sync streaming, and token_count-based usage tracking.

Messages

litert speaks a small message schema (Message, Contents, and Content subtypes). These helpers build them from ordinary Python values.

mk_content maps one value to a litert Content: a str becomes text, bytes become an image or audio (sniffed with detect_mime), and a Path becomes an image or audio file reference (by MIME). An existing Content passes through.
mk_msg wraps content into a Message (default role user), and also accepts an existing Message or a {'role','content'} dict.
mk_msgs normalises a mixed list into canonical litert message dicts, used to seed a conversation’s history.

_mk_content("hello\n").text

'hello\n'

from fastcore.test import test_eq

test_eq(_mk_msg("hello").to_json(), {"role": "user", "content": [{"type": "text", "text": "hello"}]})
test_eq(_mk_msg("hi", role="model").to_json()["role"], "model")
test_eq([x["role"] for x in _mk_msgs(["a", _mk_msg("b", role="model")])], ["user", "model"])
assert isinstance(_mk_content("x"), Text)
# bytes are sniffed: image vs audio
assert isinstance(_mk_content(b'\x89PNG\r\n\x1a\n' + b'\x00'*16), ImageBytes)
assert isinstance(_mk_content(b'RIFF\x00\x00\x00\x00WAVE' + b'\x00'*16), AudioBytes)

Portable history

chat.hist is kept in one backend-agnostic canonical shape (OpenAI-style dicts: assistant/user/system/tool roles, string content, channels.thought, and tool_calls/tool_call_id for tool rounds) so a conversation can be handed from one backend to another. litert’s own wire format differs (model role, list content, image/audio blobs, tool_response parts, id-less tool calls), so two converters bridge it:

fmt2hist — litert-native messages (or a Message/Resp) → canonical history dicts; used to record turns into chat.hist. Past-turn media collapses to an [image]/[audio] placeholder.
hist2fmt — canonical history dicts → litert Messages the engine can ingest; used to seed a conversation from a (possibly ported) history.

Because llama’s history is already canonical, Chat('litert/…', messages=llama_chat.hist) and Chat('llama/…', messages=litert_chat.hist) both work — see the port example in the overview.

Usage tracking

litert doesn’t return per-response token counts, but conv.token_count exposes the running KV-cache size (prefill plus decode). UsageStats records one turn’s prompt, completion, and total tokens, derived from the count delta around the turn, and adds across turns. Compare chat.token_count (or chat.pct_full) against ctx_limit to decide when to compress the conversation.

Callbacks

A small, ordered callback system inspired by fastllm. A ChatCallback subclass hooks named events (after_msgs, before_send, after_response, before_tool_calls, after_tool_calls) and reads live turn state off the chat via GetAttr, so self.turn_msg is chat.turn_msg. run_cbs dispatches one event to every enabled callback in order, forwarding anything a callback yields into the output stream.

Streaming display

StreamFormatter turns a litert response stream into markdown as it arrives: text passes through, and tool calls render as a compact ⏳ name(args) line. display_stream consumes a markdown-chunk stream (what chat(msg, stream=True) yields) and renders it live in a notebook via IPython. mk_tr_details formats a completed tool call as a collapsible JSON block.

Built-in callbacks

Three callbacks make up _dflt_cbs and run on every chat unless you pass default_cbs=False. HistoryCallback records each outgoing message and reply into chat.hist. UsageCallback folds the turn’s token counts into chat.use, read from a conv.token_count delta. ToolReminderCallback appends a short reminder to outgoing messages, but only when the chat has tools, nudging the model to summarise tool results in prose before it continues.

Order matters. HistoryCallback and UsageCallback sit at the front (low order) so a turn is recorded before feature callbacks like PyFenceCallback react to it and feed messages back. A callback reads live turn state off the chat through GetAttr, so self.turn_res is chat.turn_res.

Tool calling, approval, and history

litert runs the tool-call loop inside the engine. ChatToolHandler bridges that loop back to Python: for each call it records the request and result into chat.hist in canonical form (a paired assistant tool_calls entry and a tool result, so the round survives a hand-off to another backend), fires the before_tool_calls and after_tool_calls callbacks, and consults chat.approve(tool_call) before executing. Returning False blocks the tool and feeds a “Denied by human operator” response back to the model. That is the hook for human-in-the-loop gating, shown below.

source

ChatToolHandler

def ChatToolHandler(
    chat
):

Bridge litert’s in-engine tool loop to Chat callbacks, HITL approval, and history.

Loading models & Chat

Default litert-community Gemma repos live at huggingface.co/litert-community. get_model resolves a .litertlm file with a cache-first ladder: an explicit model_path, then the local HuggingFace cache (scan_cache_dir, no network), then a download. It prefers the native build over -web variants, which omit the CPU/GPU decode graph.

Chat ties this together into a callable. Build it (it constructs or accepts an Engine), then call it like a function: one turn per call, updating history and usage. create_engine is a patchable classmethod that builds the engine and creates cache_dir if given. Pass a prebuilt engine= to share one model across several chats. Engine and conversation are entered on an ExitStack, so close() releases them in the right order and never closes an engine you supplied.

source

LitertChat

def LitertChat(
    model:NoneType=None, runtime:NoneType=None, model_path:NoneType=None, engine:NoneType=None,
    backend:CPU=CPU(thread_count=None), multimodal:bool=True, cache_dir:NoneType=None,
    enable_speculative_decoding:NoneType=None, eng_kw:NoneType=None, sp:str='', messages:NoneType=None,
    tools:NoneType=None, ctx_limit:NoneType=None, approve:NoneType=None, tool_max_len:NoneType=None,
    max_steps:int=10, think:bool=False, filter_think:bool=True, temp:NoneType=None, top_k:NoneType=None,
    top_p:NoneType=None, seed:NoneType=None, sampler_config:NoneType=None, max_output_tokens:NoneType=None,
    conv_kw:NoneType=None, cbs:NoneType=None, default_cbs:bool=True
):

Sync chat over a local litert_lm engine.

# fmt2hist / hist2fmt round-trip (model-free)
_nat = [
    'hi',
    Message.model(Contents.of('hello')).to_json(),
    {'role': 'model', 'tool_calls': [{'type': 'function', 'function': {'name': 'add', 'arguments': {'a': 2, 'b': 3}}}]},
    {'role': 'tool', 'content': [{'type': 'tool_response', 'name': 'add', 'response': 5}]},
]
_canon = LitertChat.fmt2hist(_nat)
test_eq(_canon[0], {'role': 'user', 'content': 'hi'})
test_eq(_canon[1], {'role': 'assistant', 'content': 'hello'})
test_eq(_canon[2]['role'], 'assistant')
test_eq(_canon[2]['tool_calls'][0]['function'], {'name': 'add', 'arguments': {'a': 2, 'b': 3}})
assert _canon[2]['tool_calls'][0]['id']                       # a synthetic id was assigned
test_eq(_canon[3], {'role': 'tool', 'name': 'add', 'content': '5'})

# canonical -> litert Messages ready for the engine
_ms = LitertChat.hist2fmt(_canon)
test_eq([m.role.value for m in _ms], ['user', 'model', 'model', 'tool'])
test_eq(str(_ms[1]), 'hello')
test_eq(_ms[2].tool_calls[0].name, 'add')
test_eq(_ms[3].contents.to_json(), [{'type': 'tool_response', 'name': 'add', 'response': '5'}])

# fmt2hist is idempotent on an already-canonical (llama-origin) tool round
_llama = [{'role': 'assistant', 'content': '', 'tool_calls': [{'id': 'call_x', 'type': 'function',
                                                               'function': {'name': 'add', 'arguments': {'a': 1}}}]},
          {'role': 'tool', 'tool_call_id': 'call_x', 'name': 'add', 'content': '1'}]
test_eq(LitertChat.fmt2hist(_llama), _llama)

Managing callbacks

add_cb registers a callback and returns the instance; add_cbs takes a list and returns the instances. remove_cb drops one by instance, or by class to remove every callback of that type, and remove_cbs handles several at once. To run a callback for a single turn, pass cbs= to the call: chat(msg, cbs=[PyFenceCallback(...)]) registers them before the turn and removes them after, so nothing leaks into later turns.

set_min_log_severity(5)
_get_model(gemma4_12b)

'/Users/71293/.cache/huggingface/hub/models--litert-community--gemma-4-12B-it-litert-lm/snapshots/44cf85a326f79b814fa86a60af414c042755b43a/gemma-4-12B-it.litertlm'

Human-in-the-loop tool approval

Any approve(tool_call) -> bool passed to Chat(approve=...) is consulted by ChatToolHandler before each tool runs. hitl_policy builds one from a per-tool policy: 'approved' (auto-run), 'dont_run' (always block), or 'check' (ask, via _ask_console by default). Or supply your own function for custom logic such as logging, rate-limiting, or prompting a UI. See the worked example below.

Using Chat

A short tour of the main features. These cells build a real model, so they are set to run manually rather than in the test suite.

chat=Chat(backend=Backend.GPU(), cache_dir='.cache/litertlm', think=True)
# chat_12 = LitertChat(engine=LitertChat.create_engine(gemma4_12b, multimodal=False, cache_dir='.cache/litertlm', be=Backend.GPU()))

# set_min_log_severity(2)
r = chat("Reply with exactly: pong")
assert 'pong' in resp_text(r).lower()
assert resp_text(chat.hist[-1]) == resp_text(r) and chat.hist[0]['role'] == 'user'
assert chat.use.total_tokens > 0 and chat.token_count > 0
print(chat.use); chat.print_hist()

total=104|in=103|out=1|turns=1

user

Reply with exactly: pong

assistant

🧠 Thinking

Thinking Process:

Analyze the Request: The user has instructed me to reply with exactly the word “pong”.

Determine the Constraint: The constraint is strict: “Reply with exactly: pong”.

Formulate the Response: The response must be the string “pong”.

Final Output Generation: pong

pong

chat(['what is pong.','do you know?'])

🧠 Thinking

Thinking Process:

Analyze the Request: The user is asking “what is pong.do you know?”

Analyze the Context (Implicit/Previous Turn): The previous turn involved a very specific instruction: “Reply with exactly: pong”. The model responded with “pong”. The user is now asking about “pong.do”.

Identify Potential Meanings of “pong.do”:

Literal interpretation: A website or domain name.

Contextual interpretation (Given the previous interaction): It might be a reference to the “Pong” game or the programming concept, or perhaps a specific, niche service.

Misspelling/Typo: Could they mean something else? (e.g., Pong, Pong Diplomacy, etc.)

Search/Knowledge Retrieval (Internal Knowledge Base):

I do not have specific, immediate, high-profile knowledge of a widely known entity named “pong.do”.

However, the term “Pong” is extremely well-known (the arcade game, the programming concept).

Formulate a Strategy: Since the query is vague, the best approach is to:

Acknowledge the ambiguity.

Provide the most likely interpretations based on the root word “Pong”.

Ask for clarification if the user meant something specific.

Draft the Response (Self-Correction/Refinement):

Initial thought: “I don’t know what pong.do is.” (Too blunt.)

Second thought: “It could be a website. Can you tell me more?” (Better, but needs more context.)

Final approach: Explain the common association with “Pong” and ask the user to specify.

Final Output Generation. (This leads to the provided good answer.)

I am not immediately familiar with a specific entity named “pong.do.”

Could you please provide more context? It might be:

A website or domain: If it’s a website, knowing the topic or what it does would help me identify it.
A specific game or project: It could be related to the classic arcade game “Pong” or a specific programming project.
A typo: Perhaps you meant a different URL or term.

If you can tell me where you encountered “pong.do,” I would be happy to try and tell you more!

Images and audio

Some Gemma builds accept image and audio content alongside text.

from PIL import Image

im=Image.open('images.jpeg');im

chat(['explain this image', img_bytes(im)])
# you can also use ImageFile and ImageBytes

🧠 Thinking

Here’s a thinking process to arrive at the suggested explanation:

Analyze the Request: The user has provided an image and asked me to “explain this image.”

Analyze the Image:

Subject: The main subject is a dog.

Breed/Appearance: The dog has reddish-brown/tan fur, erect ears, and a sturdy build. It strongly resembles a German Shepherd or a similar working breed.

Action/Pose: The dog is looking slightly off-camera (to the right), mouth is open (tongue is out, suggesting panting or happiness), and appears to be walking or standing outdoors.

Setting: The background is blurred (shallow depth of field), showing greenery, trees, and a dirt/gravel path. The lighting suggests an outdoor, natural setting (perhaps a park, woods, or field).

Mood/Emotion: The dog looks happy, alert, and engaged with something in its environment.

Synthesize the Explanation (Drafting Key Points):

Start with the main subject.

Describe the dog’s breed characteristics (if possible, or just general description).

Describe the dog’s expression and action.

Describe the setting.

Conclude with the overall feeling of the photo.

Refine the Explanation (Adding Detail and Tone): (Self-Correction/Enhancement: Instead of just saying “It’s a dog,” use more descriptive language.)

Final Output Generation: (This leads to the structured response provided below.)

This is a photograph of a beautiful, medium-to-large-sized dog, likely a German Shepherd or a similar working breed, captured outdoors.

Here is a detailed breakdown of the image:

Subject: The central focus is a dog with rich, reddish-brown fur and distinctive, erect ears.
Expression and Action: The dog appears happy, alert, and engaged. Its mouth is open, and its tongue is hanging out, suggesting it might be panting slightly from activity or excitement. It is looking off to the side, giving the impression that it is focused on something in its environment.
Setting: The dog is standing or walking on a dirt or gravel path. The background is softly blurred (a technique called shallow depth of field), which helps to keep the focus entirely on the dog. The background consists of natural greenery, trees, and soft, diffused light, indicating an outdoor, natural environment like a park, woods, or field.
Mood: The overall mood of the photo is warm, happy, and peaceful, showcasing the dog’s natural alertness and companionship.

In short, it is a portrait of a happy, alert dog enjoying time outdoors.

chat(['transcribe this audio', AudioFile('speech.wav')])

Example: gating a tool behind approval

A custom approve gate that auto-allows a safe tool and blocks a destructive one. ChatToolHandler routes every tool call through it; a denied call is recorded in history as Denied by human operator and reported back to the model, which continues without it.

def add(a: int, b: int) -> int:
    'Add two integers.\n\nArgs:\n    a: first addend\n    b: second addend'
    return a + b

def delete_files(path: str) -> str:
    'Delete everything under a path.\n\nArgs:\n    path: directory to wipe'
    return f"wiped {path}"

def approve_gate(tc):
    "chat.approve: allow any tool except destructive ones; log every decision."
    name = tc['function']['name']
    ok = name != 'delete_files'
    print(f"[approval] {tc_summary_(name, tc['function'].get('arguments', {}))} -> {'ALLOW' if ok else 'DENY'}")
    return ok

chat = Chat(tools=[add, delete_files], approve=approve_gate, sp="Use the tools to satisfy the request.")
r = chat("Add 2 and 3, then delete everything under /tmp/data.")
print('\n', resp_text(r))
chat.print_hist()   # the blocked call is logged as a 'Denied by human operator' tool response
chat.close()

# Declarative equivalent (interactive 'check' prompts on the console via _ask_console):
# Chat(tools=[add, delete_files], approve=hitl_policy({'add': 'approved', 'delete_files': 'dont_run'}))

[approval] <code>add(a=2.0, b=3.0)</code> -> ALLOW
[approval] <code>delete_files(path=&#x27;/tmp/data&#x27;)</code> -> DENY

 I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under `/tmp/data`.

user

Add 2 and 3, then delete everything under /tmp/data.

model

🔧 add({‘a’: 2.0, ‘b’: 3.0})

tool

↩︎ add: 5.0

model

🔧 delete_files({‘path’: ‘/tmp/data’})

tool

↩︎ delete_files: Denied by human operator

assistant

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under /tmp/data.

🔧 delete_files({‘path’: ‘/tmp/data’})

Policy-driven approval

You can also drive approvals declaratively with hitl_policy instead of writing your own function.

chat=Chat(tools=[add, delete_files], approve=hitl_policy({'add': 'approved'}))

WARNING: [npu_registry.cc:34] NPU accelerator could not be loaded and registered: kLiteRtStatusErrorInvalidArgument.
W0000 00:00:1785491599.209665 24526884 mel_filterbank.cc:137] Missing 10 bands  starting at 0 in mel-frequency design. Perhaps too many channels or not enough frequency resolution in spectrum. (fft_length: 257 sample_rate: 16000 mel_channel_count: 128 lower_frequency_limit: 0 upper_frequency_limit: 8000

r = chat("Add 2 and 3, then delete everything under /tmp/data.")
print('\n', resp_text(r))
chat.print_hist()   # the blocked call is logged as a 'Denied by human operator' tool response
chat.close()


 I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under `/tmp/data`.

user

Add 2 and 3, then delete everything under /tmp/data.

model

🔧 add({‘a’: 2.0, ‘b’: 3.0})

tool

↩︎ add: 5.0

model

🔧 delete_files({‘path’: ‘/tmp/data’})

tool

↩︎ delete_files: Denied by human operator

assistant

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete everything under /tmp/data.

🔧 delete_files({‘path’: ‘/tmp/data’})

WARNING: Cache file is stale. Setting stale flag.

Streaming turns

Pass stream=True and iterate the result: Chat returns a generator of markdown chunks (formatted by StreamFormatter) instead of a response dict. History and usage are finalised once the stream is exhausted.

ch = Chat(backend=Backend.GPU())

# stream chunks to a terminal:
for c in ch("Count: one two three", stream=True): print(c, end='', flush=True)

That's a simple count!

If you'd like me to do something with that count, please let me know. For example, would you like me to:

* **Continue counting?** (e.g., "four, five, six...")
* **Count something else?**
* **Do a math problem?**
* **Something else entirely?**

# sync streaming: SaveReturn wraps the chunk stream; display_stream renders live + returns md, `.value` is the final Resp
from fastcore.xtras import SaveReturn
s = SaveReturn(ch('count fibonacci for 10 places', stream=True))
md = display_stream(s)
assert md.strip() and resp_text(s.value) == resp_text(ch.turn_res)

Here are the first 10 numbers in the Fibonacci sequence:

The Fibonacci sequence starts with 0 and 1, and each subsequent number is the sum of the two preceding ones.

0
1
1 (0 + 1)
2 (1 + 1)
3 (1 + 2)
5 (2 + 3)
8 (3 + 5)
13 (5 + 8)
21 (8 + 13)
34 (13 + 21)

The first 10 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34

'Here are the first 10 numbers in the Fibonacci sequence:\n\nThe Fibonacci sequence starts with 0 and 1, and each subsequent number is the sum of the two preceding ones.\n\n1. **0**\n2. **1**\n3. **1** (0 + 1)\n4. **2** (1 + 1)\n5. **3** (1 + 2)\n6. **5** (2 + 3)\n7. **8** (3 + 5)\n8. **13** (5 + 8)\n9. **21** (8 + 13)\n10. **34** (13 + 21)\n\n**The first 10 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34**'

# async streaming: adisplay_stream drives the turn; the returned iterator carries the final Resp on `.value`
achat = AsyncChat(Chat(cache_dir='.cache/litertlm'))
sa = run_coro(achat('Count: one two three', stream=True))
md = run_coro(adisplay_stream(sa))
assert md.strip() and resp_text(sa.value) == resp_text(achat.turn_res)
assert [m['role'] for m in achat.hist] == ['user', 'assistant']
achat.close()

assert resp_text(chat.hist[-1]) == resp_text(chat.turn_res) and chat.use.completion_tokens > 0

fired = []
class Log(ChatCallback):
    def before_tool_calls(self): fired.append('before')
    def after_tool_calls(self):  fired.append('after')

chat = Chat(tools=[add], sp="Use the add tool for arithmetic.")
chat.add_cb(Log)
r = chat("What is 21 + 21? Use the tool.")
print(fired)
assert 'before' in fired and 'after' in fired
assert any(m.get('role') == 'tool' for m in chat.hist)
print(resp_text(r)); chat.print_hist()

['before', 'after']
The result of adding 21 and 21 is 42.

user

What is 21 + 21? Use the tool.

model

🔧 add({‘a’: 21.0, ‘b’: 21.0})

tool

↩︎ add: 42.0

assistant

The result of adding 21 and 21 is 42.

Running python from replies

PyFenceCallback turns the chat into a code interpreter. After a reply it finds the last ```python fence, runs it through safepyrun (sandboxed, so imports like socket and importlib are blocked) in a namespace that persists across the conversation, feeds the output back as a ```result block, and re-queries, up to max_rounds times. Enable it with Chat(cbs=[PyFenceCallback]). Execution goes through the same approve hook as a synthetic python tool, so hitl_policy({'python': 'check'}) prompts before running. chat.run_py(code) runs a snippet directly (ban_defs=False allows def and class).

Stopping the loop with `done`

Pass a done function to PyFenceCallback to end the loop early. output_matches stops once the code output contains an expected string.

chat = Chat(sp="When asked to compute something, reply with a ```python fence that prints or evaluates the answer.")
r = chat("Use python to compute 2**5.", cbs=[PyFenceCallback(done=output_matches(32))])
chat.print_hist()   # you'll see the ```python turn, a ```result turn, then the final answer
chat.close()

user

Use python to compute 2**5.

assistant

print(2**5)

user

Letting the model decide

Or leave termination to the model: with no done, the loop stops as soon as a reply has no code fence.

chat = Chat(cbs=[PyFenceCallback], sp="When asked to compute something, reply with a ```python fence that prints or evaluates the answer.")
r = chat("Use python to compute 2**5.")
chat.print_hist()
chat.close()

user

Use python to compute 2**5.

assistant

print(2**5)

user

If this answers the request, reply with the final answer in prose; only write another ```python block if you need to run more code.

assistant

Utilities: classify, structured, benchmark

classify and structured each run one-shot in a throwaway conversation on the shared engine, isolated from the live chat, so they leave its history and KV cache untouched. classify generates a label and matches it against your options. structured forces a tool call and returns schema(**arguments). bench reports init time, time to first token, and prefill and decode tokens per second.

litert’s run_text_scoring log-likelihood scoring is not available on this runtime, so classify generates and label-matches rather than scoring.

source

bench

def bench(
    model_id:str='litert-community/gemma-4-E2B-it-litert-lm', model_path:NoneType=None,
    backend:CPU=CPU(thread_count=None), prefill_tokens:int=64, decode_tokens:int=64, **kw
):

Benchmark init time, TTFT, and prefill/decode tokens-per-sec via litert’s Benchmark.

# classify + structured on a real model
sm = Chat(cache_dir='.cache/litertlm')
test_eq(sm.classify("I absolutely loved this film!", ['positive', 'negative']), 'positive')

from dataclasses import dataclass
@dataclass
class Person: name:str; age:int
p = sm.structured("Extract the person: John Smith is 30 years old.", Person)
print(p); assert isinstance(p, Person) and 'John' in p.name
sm.close()

Person(name='John Smith', age=30)

Grading answers

chat.check turns the model into a graded question-answerer. It asks question in an isolated conversation (like classify/structured, so nothing touches chat.hist), pulls the model’s answer out of a ``answer fence with [extract_fence](https://vedicreader.github.io/rishi/core.html#extract_fence) (falling back to the whole reply if the model skips the fence), and grades it againstexpected. Grading isgrade_fn(answer, expected) -> bool, defaulting to [matches_](https://vedicreader.github.io/rishi/core.html#matches_) (the answer must containexpected, or any value in anexpectedlist). Pass your owngrade_fnfor custom logic, or setllm_judge=Trueto grade with the model instead (chat.grades, built onclassify); passjudge=a second [Chat`](https://vedicreader.github.io/rishi/core.html#chat) to have a stronger model do the grading.

# Default grade is a deterministic match (no judge needed):
qa = Chat(cache_dir='.cache/litertlm')
print(qa.check("What is the capital of France?", "Paris").ok)
# Custom grader: any `answer, expected -> bool`:
print(qa.check("What is 2 + 2?", "4", grade_fn=lambda a, e: e in a).ok)

True
True

# LLM-as-judge, optionally with a bigger model doing the grading:
judge = Chat(gemma4_12b, backend=Backend.GPU(), multimodal=False, cache_dir='.cache/litertlm')

r = qa.check("Name a primary colour.", "red, blue, or yellow", judge=judge)   # judge -> llm_judge
print(r.answer, '->', r.ok)
judge.close(); qa.close()

Red -> True

# grades (LLM judge) + check on a real model
sm = Chat(cache_dir='.cache/litertlm')
assert sm.grades("What is the capital of France?", "Paris", "Paris") is True
assert sm.grades("What is the capital of France?", "Paris", "London") is False
r = sm.check("What is the capital of France?", "Paris")            # deterministic default
print(r.answer, '->', r.ok); assert r.ok
assert sm.check("What is 2 + 2?", "4", llm_judge=True).ok is True  # self as judge
sm.close()

Paris -> True

Installing the skill

skill.md ships with the package. A harness can drop it into the standard skill directories with mv_skill_md, which writes SKILL.md under .claude/skills/rishi/ and .agents/skills/rishi/ at the git root. It is a dry run by default and only prints the targets; pass dry_run=False to write, or dir= to install somewhere else.

from fastcore.test import test_eq
import rishi.core, rishi.litert
test_eq(rishi.core.get_runtime('litert'), rishi.litert.LitertChat)
from unittest.mock import patch as mock_patch
with mock_patch.object(rishi.litert.LitertChat, '__init__', return_value=None):
    assert isinstance(rishi.core.Chat(), rishi.litert.LitertChat)
    assert rishi.core.Chat('litert-community/gemma-4-E2B-it-litert-lm').runtime == 'litert'

Messages

Portable history

Usage tracking

Callbacks

Streaming display

Built-in callbacks

Tool calling, approval, and history

ChatToolHandler

Loading models & Chat

LitertChat

Managing callbacks

Human-in-the-loop tool approval

Using Chat

Images and audio

Example: gating a tool behind approval

Policy-driven approval

Streaming turns

Running python from replies

Stopping the loop with done

Letting the model decide

Utilities: classify, structured, benchmark

bench

Grading answers

Installing the skill

Stopping the loop with `done`