# llama
## Messages
llama.cpp speaks the OpenAI chat schema, so messages are plain dicts.
These helpers mirror `rishi.core.mk_msg`/`mk_msgs` but build
OpenAI-style messages instead of litert `Message`s.
- `mk_content` maps one value to an OpenAI content part: a `str` becomes
text; `bytes` or a `Path` become a base64 `image_url` or `input_audio`
part, sniffed by MIME. Both need an `mmproj` projector on the engine -
see [Images and audio](#images-and-audio).
- `mk_msg` wraps content into a message dict (default role `user`); a
dict passes through. All-text parts collapse to a plain string for
maximum template compatibility.
- `mk_msgs` normalises a mixed list, used to seed `Chat(messages=...)`.
``` python
from fastcore.test import test_eq, test_fail
```
``` python
test_eq(_mk_msg("hello"), {'role': 'user', 'content': 'hello'})
test_eq(_mk_msg(["a", "b"]), {'role': 'user', 'content': 'a\nb'})
test_eq(_mk_msg({'role': 'assistant', 'content': 'hi'}), {'role': 'assistant', 'content': 'hi'})
test_eq(_mk_msgs(['hi', {'role': 'assistant', 'content': 'hello'}]),
[{'role': 'user', 'content': 'hi'}, {'role': 'assistant', 'content': 'hello'}])
png = b'\x89PNG\r\n\x1a\n' + b'0' * 8
assert _mk_msg([png, 'what is this?'])['content'][0]['type'] == 'image_url'
test_eq(_mk_content(b'RIFF0000WAVE')['type'], 'input_audio') # audio is sniffed too
test_fail(lambda: _mk_content(b'%PDF-1.4 junk'), contains='text, image, and audio')
```
## Tool schemas
[`mk_toolspec`](https://vedicreader.github.io/rishi/llama.html#mk_toolspec)
turns a python callable into an OpenAI/llama.cpp tool spec using
toolslm’s `get_schema` (docstrings and
[docments](https://fastcore.fast.ai/docments.html)-style comments become
descriptions). Spec dicts pass through, so you can hand-write schemas
too. `get_schema` adds a stray `title` key for classes and dataclasses,
which we drop: it would otherwise be rendered into the prompt’s tool
listing and handed to llama.cpp’s JSON-schema grammar builder by
`structured`.
(Note the direction: toolslm also exports an `mk_tool`, which does the
reverse - it builds a callable *from* a JSON schema.)
------------------------------------------------------------------------
source
### mk_toolspec
``` python
def mk_toolspec(
f
):
```
*OpenAI/llama.cpp tool spec for callable `f` (via toolslm `get_schema`);
spec dicts pass through.*
``` python
def _add(
a:int, # first addend
b:int=0 # second addend
):
'Add two integers.'
return a + b
s = mk_toolspec(_add)
test_eq(s['type'], 'function')
test_eq(s['function']['name'], '_add')
test_eq(s['function']['parameters']['required'], ['a'])
test_eq(mk_toolspec(s), s) # spec dicts pass through
# `get_schema` adds a stray `title` for classes/dataclasses (the `structured` path); it must not reach
# the rendered tool listing or llama.cpp's JSON-schema grammar builder
from dataclasses import dataclass
@dataclass
class _Point:
'A 2d point'
x: int # the x coord
y: int # the y coord
p = mk_toolspec(_Point)
test_eq(p['function']['name'], '_Point')
test_eq(p['function']['parameters']['required'], ['x', 'y'])
assert 'title' not in p['function']['parameters'], p['function']['parameters']
```
## Think tags and tool-call tags
Reasoning GGUF models (Qwen3, DeepSeek-R1 distills, …) put their
thinking inside `...`, and Hermes-format models emit tool
calls as `{json}` text. llama.cpp’s generic chat
handler passes both through verbatim, so we parse them ourselves:
- [`split_think`](https://vedicreader.github.io/rishi/llama.html#split_think)
moves think blocks into a separate thought string (which
[`Chat`](https://vedicreader.github.io/rishi/core.html#chat) reports
via `channels.thought` and keeps out of the context on later turns,
like litert’s `filter_think`).
- [`parse_tool_tags`](https://vedicreader.github.io/rishi/llama.html#parse_tool_tags)
extracts tool-call blocks into OpenAI-style `tool_calls` entries.
------------------------------------------------------------------------
source
### parse_tool_tags
``` python
def parse_tool_tags(
text
):
```
*Parse Hermes/Qwen-style `{json}` blocks; returns
`(clean_text, tool_calls)`.*
------------------------------------------------------------------------
source
### split_think
``` python
def split_think(
text
):
```
*Split `...` blocks out of `text`; returns
`(clean_text, thought)`.*
``` python
test_eq(split_think('hmm\n\nhi'), ('hi', 'hmm'))
test_eq(split_think('no tags'), ('no tags', ''))
test_eq(split_think('cut off mid-'), ('', 'cut off mid-'))
txt, tcs = parse_tool_tags('calling now\n{"name": "add", "arguments": {"a": 1}}')
test_eq(txt, 'calling now')
test_eq(tcs[0]['function'], {'name': 'add', 'arguments': {'a': 1}})
test_eq(parse_tool_tags('not json')[1], [])
```
## Response normalization
[`norm_resp`](https://vedicreader.github.io/rishi/llama.html#norm_resp)
converts a llama.cpp chat completion into the same shape `rishi.core`
responses use - an assistant dict with `content`, `channels.thought`,
parsed `tool_calls` (JSON-string arguments become dicts; ``
tags are folded in), a
[`truncated`](https://vedicreader.github.io/rishi/core.html#truncated)
flag on `finish_reason == 'length'`, and the `usage` block - wrapped in
[`Resp`](https://vedicreader.github.io/rishi/core.html#resp) so it
renders as markdown in notebooks.
[`_oai_msg`](https://vedicreader.github.io/rishi/llama.html#_oai_msg) is
the reverse projection used when re-sending history: rishi-only keys
(`channels`, `usage`, …) are dropped - so thinking never re-enters the
context - and tool-call arguments are re-encoded as JSON strings.
------------------------------------------------------------------------
source
### norm_resp
``` python
def norm_resp(
r
):
```
*Normalize a llama.cpp chat completion to a rishi-style
[`Resp`](https://vedicreader.github.io/rishi/core.html#resp) dict.*
``` python
raw = {'choices': [{'message': {'role': 'assistant',
'content': '2+3The answer is 5.\n{"name": "add", "arguments": {"a": 2, "b": 3}}',
'tool_calls': [{'id': 'x1', 'type': 'function',
'function': {'name': 'mul', 'arguments': '{"a": 4}'}}]},
'finish_reason': 'stop'}],
'usage': {'prompt_tokens': 10, 'completion_tokens': 5, 'total_tokens': 15}}
r = norm_resp(raw)
test_eq(resp_text(r), 'The answer is 5.')
test_eq(thought(r), '2+3')
test_eq([tc['function']['name'] for tc in r['tool_calls']], ['mul', 'add'])
test_eq(r['tool_calls'][0]['function']['arguments'], {'a': 4})
assert not truncated(r)
o = _oai_msg(r)
assert 'channels' not in o and 'usage' not in o
test_eq(o['tool_calls'][0]['function']['arguments'], '{"a": 4}')
test_eq(_oai_msg({'role': 'tool', 'tool_call_id': 't1', 'name': 'add', 'content': '5'}),
{'role': 'tool', 'tool_call_id': 't1', 'name': 'add', 'content': '5'})
test_eq(_sum_usage([{'prompt_tokens': 1, 'completion_tokens': 2, 'total_tokens': 3}, None,
{'prompt_tokens': 10, 'completion_tokens': 20, 'total_tokens': 30}]),
{'prompt_tokens': 11, 'completion_tokens': 22, 'total_tokens': 33})
```
## Streaming
[`StreamSplit`](https://vedicreader.github.io/rishi/llama.html#streamsplit)
is a stateful splitter for streamed text: it routes `...`
content to the thought channel and holds back `` blocks (so
raw JSON never hits the display), emitting litert-style chunk dicts that
[`rishi.core.StreamFormatter`](https://vedicreader.github.io/rishi/core.html#streamformatter)
already knows how to render. Tags split across chunk boundaries are
handled by holding back any suffix that could still become a tag.
[`_acc_tc`](https://vedicreader.github.io/rishi/llama.html#_acc_tc)
accumulates structured `tool_calls` deltas (from chat handlers with
native tool support).
------------------------------------------------------------------------
source
### StreamSplit
``` python
def StreamSplit():
```
*Stateful splitter for streamed text: `` -\> thought chunks,
`` blocks held back and parsed.*
``` python
def _run_split(deltas):
sp, chunks = StreamSplit(), []
for d in deltas: chunks += list(sp.feed(d))
chunks += list(sp.finish())
return sp, chunks
# tags split across chunk boundaries
sp, chunks = _run_split(['let me', ' see\n\nans', 'wer: 5 {"name": "add", "arguments": {"a": 1}}'])
test_eq(sp.thought, 'let me see')
test_eq(sp.text, 'answer: 5 ')
test_eq(sp.tool_calls[0]['function'], {'name': 'add', 'arguments': {'a': 1}})
assert all(('channels' in c) or ('content' in c) for c in chunks)
# plain text passes through; unterminated think flushes to thought
sp, _ = _run_split(['just ', 'text'])
test_eq((sp.text, sp.thought, sp.tool_calls), ('just text', '', []))
sp, _ = _run_split(['never closed'])
test_eq(sp.thought, 'never closed')
# a lone '<' that never becomes a tag is still emitted
sp, _ = _run_split(['a < b', ' and c'])
test_eq(sp.text, 'a < b and c')
acc = []
_acc_tc(acc, [{'index': 0, 'id': 'c1', 'function': {'name': 'add', 'arguments': ''}}])
_acc_tc(acc, [{'index': 0, 'function': {'arguments': '{"a"'}}])
_acc_tc(acc, [{'index': 0, 'function': {'arguments': ': 2}'}}])
test_eq(acc, [{'id': 'c1', 'type': 'function', 'function': {'name': 'add', 'arguments': '{"a": 2}'}}])
```
## Built-in callbacks
The callback system is shared with `rishi.core`
([`ChatCallback`](https://vedicreader.github.io/rishi/core.html#chatcallback),
[`run_cbs`](https://vedicreader.github.io/rishi/core.html#run_cbs), and
the same event names), so callbacks written for one backend generally
work on the other. llama.cpp is stateless per call, though, so here
`chat.hist` *is* the conversation state and
[`Chat`](https://vedicreader.github.io/rishi/core.html#chat) maintains
it directly - there is no `HistoryCallback`. `UsageCallback` folds the
turn’s `usage` block (summed across tool rounds) into `chat.use`, and
`ToolReminderCallback` injects the same reminder text as the litert
backend.
## Loading GGUF models
Any GGUF repo on the HuggingFace Hub works; the Qwen3 and Gemma 3 ids
below are handy defaults (Qwen3 gives you thinking *and* Hermes-style
tool calls out of the box). `get_model` resolves a `.gguf` file with the
same cache-first ladder as `rishi.core.get_model`: explicit
`model_path`, then the local HF cache (no network), then a download -
preferring the requested `quant` and skipping `mmproj` projector files.
[`get_mmproj`](https://vedicreader.github.io/rishi/llama.html#get_mmproj)
is the complement: it resolves the projector itself the same way, for
vision models.
------------------------------------------------------------------------
source
### get_mmproj
``` python
def get_mmproj(
model_id, mmproj_path:NoneType=None
):
```
*Return a local `mmproj` projector path: `mmproj_path`, else HF cache,
else download.*
``` python
fs = ['README.md', 'model-Q8_0.gguf', 'model-Q4_K_M.gguf', 'mmproj-model.gguf']
test_eq(_gguf(fs), 'model-Q4_K_M.gguf')
test_eq(_gguf(fs, 'Q8_0'), 'model-Q8_0.gguf')
test_eq(_gguf(fs, 'IQ2_XS'), 'model-Q8_0.gguf') # falls back to first non-mmproj gguf
test_eq(_gguf(['README.md']), None)
test_eq(_mmproj(fs), 'mmproj-model.gguf') # the complement of _gguf
test_eq(_mmproj(['model-Q4_K_M.gguf']), None)
test_eq(_mmproj(['mmproj-dir/model.gguf']), None) # matches the filename, not the directory
```
------------------------------------------------------------------------
source
### gpu_offload_supported
``` python
def gpu_offload_supported():
```
*True if the installed llama-cpp-python build has GPU (Metal/CUDA)
offload compiled in.*
``` python
# build-level GPU check: True on the Metal (macOS/arm64) or CUDA wheel, False on a CPU wheel
res = gpu_offload_supported()
assert isinstance(res, bool)
print('gpu offload supported:', res)
```
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.028 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 30150.67 MB
gpu offload supported: True
## Audio
llama.cpp itself has full audio support: the shipped `libmtmd` exports
`mtmd_bitmap_init_from_audio`, `mtmd_support_audio` and preprocessors
for whisper, Qwen3-A, Gemma-4a, conformer and granite-speech.
llama-cpp-python’s *chat* layer just never reaches it - `llama_types`
declares only `text` and `image_url` parts, `get_image_urls` collects
images alone, and `_create_bitmap_from_bytes` always takes the stb_image
path.
So rather than fork the handler, we `@patch` the four methods standing
between it and the audio API it already links against. Every patch is a
superset of the original: images take exactly the path they took before,
and anything unrecognised falls through to upstream.
The one thing llama.cpp can’t do for us here is decode the container -
its `mtmd_helper_bitmap_init_from_buf` only decodes images (this wheel
bundles stb_image, not miniaudio).
[`read_audio`](https://vedicreader.github.io/rishi/llama.html#read_audio)
covers that gap with
[`soundfile`](https://python-soundfile.readthedocs.io) (libsndfile), so
WAV/FLAC/OGG/MP3 and the other libsndfile formats all decode - not just
WAV. It returns the mono float32 samples `mtmd_bitmap_init_from_audio`
wants, resampled to whatever rate the projector reports.
One upstream quirk worth knowing: for Voxtral, llama.cpp’s mtmd encodes
a single clip twice. It builds two audio chunks for one `input_audio`
part, so the projector runs its encode/decode pass twice on the same
samples, roughly doubling audio latency and the audio tokens that land
in the context. This is not a rishi bug - rishi hands the handler
exactly one marker and one bitmap in one completion call, and the
doubling reproduces with the plain upstream `llama-mtmd-cli` (build
b10194) with no Python in the path. It is invariant to clip length (a
0.4s and a 6s clip both encode twice), so it is not the encoder slicing
a long clip into windows. The audio path is still flagged experimental
upstream ([discussion
\#13759](https://github.com/ggml-org/llama.cpp/discussions/13759));
until it settles, treat Voxtral audio as slow-but-working and prefer
litert or a cloud model when audio quality and speed matter.
------------------------------------------------------------------------
source
### read_audio
``` python
def read_audio(
o, sr:int=16000
):
```
*Decode audio `bytes`/`Path` (any libsndfile format) to contiguous mono
float32 samples at `sr` Hz, ready for mtmd.*
------------------------------------------------------------------------
source
### MTMDChatHandler.get_image_urls
``` python
def get_image_urls(
messages
):
```
*Media data-URIs in document order - audio as well as images (upstream
collects images only).*
``` python
import io
from fastcore.all import Path
from contextlib import ExitStack
from llama_cpp import mtmd_cpp
import tempfile
```
``` python
# read_audio decodes the real 24s clip and resamples it, preserving duration
wav = Path('speech.wav').read_bytes()
a = read_audio(wav, 16000)
test_eq(a.dtype, np.dtype('float32'))
assert a.flags['C_CONTIGUOUS'] and -1.0 <= a.min() and a.max() <= 1.0
test_eq(round(len(a) / 16000), 24) # 24s in, 24s out
test_eq(round(len(read_audio(wav, 8000)) / 8000), 24) # and at a different target rate
# soundfile handles compressed containers too, not just WAV
sig = (0.2 * np.sin(2 * np.pi * 220 * np.arange(8000) / 8000)).astype(np.float32)
for fmt in ('FLAC', 'OGG', 'MP3'):
b = io.BytesIO(); sf.write(b, sig, 8000, format=fmt)
d = read_audio(b.getvalue(), 8000)
test_eq(d.dtype, np.dtype('float32')); assert d.flags['C_CONTIGUOUS'], fmt
assert abs(len(d) - 8000) <= 2400, (fmt, len(d)) # ~1s back; lossy codecs pad a little
# stereo is downmixed to mono
st = np.stack([sig, np.zeros_like(sig)], axis=1)
b = io.BytesIO(); sf.write(b, st, 8000, format='WAV', subtype='FLOAT')
test_eq(read_audio(b.getvalue(), 8000).ndim, 1)
test_fail(lambda: read_audio(b'not audio at all'), contains='could not decode audio')
```
``` python
# a template that renders `content` directly (Llama/Voxtral-style) must get a STRING, not a list repr:
# _get_template_messages collapses media+text to a clean string with the marker inline
h = object.__new__(MTMDChatHandler)
msg = {'role': 'user', 'content': [{'type': 'text', 'text': 'Transcribe this clip.'},
{'type': 'input_audio', 'input_audio': {'data': 'AAA', 'format': 'wav'}}]}
conv = h._get_template_messages([msg], '')[0]['content']
test_eq(conv, 'Transcribe this clip.\n')
assert isinstance(conv, str) # never a Python list repr
# a plain-string message is passed through untouched
test_eq(h._get_template_messages([{'role': 'user', 'content': 'hi'}], '')[0]['content'], 'hi')
```
``` python
# the patched handler collects image *and* audio media, in document order
with tempfile.NamedTemporaryFile(suffix='.gguf') as f:
h = MTMDChatHandler(f.name, verbose=False)
img = Path('images.jpeg').read_bytes()
msgs = [_mk_msg([img, 'and this?', wav])]
urls = h.get_image_urls(msgs)
test_eq(len(urls), 2)
assert urls[0].startswith('data:image/jpeg;base64,') and urls[1].startswith('data:audio/wav;base64,')
# every media part becomes the marker; text is left alone
# media parts become the marker and the all-text content collapses to a clean string (marker inline)
test_eq(h._get_template_messages(msgs, '<__media__>')[0]['content'], '<__media__>\nand this?\n<__media__>')
# and the data-URI round-trips back to the exact bytes the model will decode
test_eq(h.load_image(urls[1]), wav)
test_eq(h.load_image(urls[0]), img)
```
``` python
# the decoded buffer is accepted by real llama.cpp: `mtmd_bitmap_init_from_audio` needs no
# context or model, so the actual C boundary is exercised here rather than mocked
bm = mtmd_cpp.mtmd_bitmap_init_from_audio(len(a), a.ctypes.data_as(ctypes.POINTER(ctypes.c_float)))
assert bm is not None, 'llama.cpp rejected the audio buffer'
try:
assert mtmd_cpp.mtmd_bitmap_is_audio(bm)
test_eq(mtmd_cpp.mtmd_bitmap_get_n_bytes(bm), len(a) * 4) # float32 per sample
back = np.ctypeslib.as_array(ctypes.cast(mtmd_cpp.mtmd_bitmap_get_data(bm),
ctypes.POINTER(ctypes.c_float)), shape=(len(a),))
assert np.array_equal(back, a), 'samples corrupted crossing the C boundary'
finally: mtmd_cpp.mtmd_bitmap_free(bm)
```
## Chat
Same shape as
[`rishi.core.Chat`](https://vedicreader.github.io/rishi/core.html#chat):
build it (it constructs or accepts a `llama_cpp.Llama`), then call it
like a function - one turn per call, updating history and usage;
`stream=True` yields markdown chunks instead. `create_engine` is a
patchable classmethod; pass a prebuilt `engine=` to share one model
across chats.
The tool loop runs in Python (litert runs it inside the engine): tools
are passed to the chat template *and* described to llama.cpp, responses
are checked for structured `tool_calls` or `` tags, each call
is routed through `approve` (so
[`hitl_policy`](https://vedicreader.github.io/rishi/core.html#hitl_policy)
works unchanged), executed via toolslm’s `call_func`, and fed back as
`role='tool'` messages - up to `max_steps` rounds per turn.
`think` uses the Qwen-style soft switch: `True`/`False` appends `/think`
or `/no_think` to the system prompt; `None` leaves the model’s default.
Thinking is split out of replies into `channels.thought` and never
re-sent, mirroring litert’s `filter_think`.
------------------------------------------------------------------------
source
### LlamaChat
``` python
def LlamaChat(
model:NoneType=None, runtime:NoneType=None, model_path:NoneType=None, engine:NoneType=None, quant:str='Q4_K_M',
n_ctx:int=8192, n_gpu_layers:int=0, mmproj:NoneType=None, eng_kw:NoneType=None, sp:str='',
messages:NoneType=None, tools:NoneType=None, ctx_limit:NoneType=None, approve:NoneType=None,
tool_max_len:NoneType=None, max_steps:int=10, think:NoneType=None, temp:NoneType=None, top_k:NoneType=None,
top_p:NoneType=None, seed:NoneType=None, max_output_tokens:NoneType=None, comp_kw:NoneType=None,
cbs:NoneType=None, default_cbs:bool=True
):
```
*Sync chat over a local llama.cpp model - the
[`rishi.core.Chat`](https://vedicreader.github.io/rishi/core.html#chat)
API with a Python-side tool loop.*
## Utilities: classify, structured, run_py
Same one-shot helpers as the litert backend, run stateless on this
chat’s engine (isolated from the conversation). `structured` forces the
tool call with `tool_choice`, which llama.cpp turns into a JSON-schema
grammar - so the arguments always parse. `run_py` and `grades` are
shared with `rishi.core` directly, so
[`PyFenceCallback`](https://vedicreader.github.io/rishi/core.html#pyfencecallback)
works on this backend too.
## AsyncChat
An async facade over
[`Chat`](https://vedicreader.github.io/rishi/core.html#chat) with the
identical constructor (or pass an existing
[`Chat`](https://vedicreader.github.io/rishi/core.html#chat)): blocking
llama.cpp calls run in a worker thread via `asyncio.to_thread`, so an
event loop stays responsive. `await achat(msg)` for one turn;
`async for c in await achat(msg, stream=True)` to stream. Everything
else (`hist`, `use`, `print_hist`, callbacks, …) is delegated to the
wrapped [`Chat`](https://vedicreader.github.io/rishi/core.html#chat).
### Tests
Pure helpers
([`split_think`](https://vedicreader.github.io/rishi/llama.html#split_think),
[`norm_resp`](https://vedicreader.github.io/rishi/llama.html#norm_resp),
[`StreamSplit`](https://vedicreader.github.io/rishi/llama.html#streamsplit),
[`_strip_media`](https://vedicreader.github.io/rishi/llama.html#_strip_media),
and the real `MTMDChatHandler` media plumbing) are unit-tested above
with literal fixtures and the real `mtmd_cpp` C boundary - no model
needed. The full pipeline (system prompt, think filtering, the tool loop
with approval, streaming + async streaming, usage, structured output,
and multimodal) runs against a real `qwen3_06b` (and a real projector)
in the `#| eval: false` cells below - run them locally with the models
present.
``` python
from fastcore.test import test_eq
# _strip_media collapses past-turn media to a text placeholder; plain strings pass through untouched (pure fn, no model)
test_eq(_strip_media({'role': 'user', 'content': 'plain'}), {'role': 'user', 'content': 'plain'})
past = {'role': 'user', 'content': [{'type': 'input_audio', 'input_audio': {'data': 'x', 'format': 'wav'}},
{'type': 'text', 'text': 'clip 1'}]}
test_eq(_strip_media(past)['content'], '[audio]\nclip 1')
assert _is_media({'type': 'image_url', 'image_url': {'url': 'data:img'}})
assert not _is_media({'type': 'text', 'text': 'hi'})
```
``` python
# hist2fmt / fmt2hist across backends (model-free): a litert-origin canonical tool round feeds llama.cpp
_h = [
{'role': 'user', 'content': 'What is 2+3?'},
{'role': 'assistant', 'content': '', 'tool_calls': [{'id': 'call_1', 'type': 'function',
'function': {'name': 'add', 'arguments': {'a': 2, 'b': 3}}}]},
{'role': 'tool', 'tool_call_id': 'call_1', 'name': 'add', 'content': '5'},
{'role': 'assistant', 'content': 'It is 5.'},
]
_oai = LlamaChat.hist2fmt(_h)
test_eq(_oai[2], {'role': 'tool', 'tool_call_id': 'call_1', 'name': 'add', 'content': '5'})
test_eq(_oai[1]['tool_calls'][0]['function']['arguments'], '{"a": 2, "b": 3}') # args JSON-encoded for llama.cpp
test_eq(LlamaChat.fmt2hist(_h), _h) # no-op on already-canonical dicts
```
``` python
from fastcore.test import test_eq
import rishi.core, rishi.llama
test_eq(rishi.core.get_runtime('llama'), rishi.llama.LlamaChat)
from unittest.mock import patch as mock_patch
with mock_patch.object(rishi.llama.LlamaChat, '__init__', return_value=None):
assert isinstance(rishi.core.Chat('Qwen/Qwen3-0.6B-GGUF'), rishi.llama.LlamaChat)
assert rishi.core.Chat('/models/x.gguf').runtime == 'llama'
```
## Using Chat
A short tour, mirroring the litert one. These cells download and run a
real GGUF model, so they are set to run manually rather than in the test
suite.
``` python
chat = LlamaChat(qwen3_06b, think=False, temp=0.0)
r = chat("Reply with exactly: pong")
assert 'pong' in resp_text(r).lower()
assert chat.hist[-1] is r and chat.hist[0]['role'] == 'user'
assert chat.use.total_tokens > 0 and chat.token_count > 0
print(chat.use); chat.print_hist()
```
llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
total=26|in=21|out=5|turns=1
**user**
Reply with exactly: pong
------------------------------------------------------------------------
**assistant**
pong
### Tools and approval
Tool specs go both into the chat template (Qwen-style templates render
them natively) and through our Hermes-tag parser, so tool calling works
with plain GGUF chat models. `approve` and
[`hitl_policy`](https://vedicreader.github.io/rishi/core.html#hitl_policy)
behave exactly as in the litert backend; a denied call is fed back as
`Denied by human operator`.
``` python
def add(
a: int, # first addend
b: int # second addend
) -> int:
'Add two integers.'
return a + b
chat = LlamaChat(qwen3_06b, tools=[add], think=False, temp=0.0, sp='Use the tools to satisfy the request.')
r = resp_text(chat('What is 12345 plus 67890? Use the add tool.'))
assert '80235' in r or '80,235' in r
chat.print_hist()
```
llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
**user**
What is 12345 plus 67890? Use the add tool. After every
tool call result, briefly summarise in prose what you found before
continuing or calling another tool.
------------------------------------------------------------------------
**assistant**
🔧 add({‘a’: 12345, ‘b’: 67890})
------------------------------------------------------------------------
**tool**
80235
------------------------------------------------------------------------
**assistant**
> **🧠 Thinking**
>
> Okay, the user asked for 12345 plus 67890 using the add tool. I called
> the add function with those numbers. The result came back as 80235.
> Now I need to summarize that in prose. Let me check the calculation
> again to be sure. 12,345 plus 67,890… adding them step by step:
> 12,345 + 60,000 is 72,345, then plus 7,890 makes 80,235. Yep, that’s
> correct. So the final answer is 80,235.
The sum of 12,345 and 67,890 is **80,235**.
``` python
chat = LlamaChat(qwen3_06b, tools=[add], approve=hitl_policy({'add': 'dont_run'}),
think=False, temp=0.0, sp='Use the tools to satisfy the request.')
r = chat('What is 2 plus 3? Use the add tool.')
assert any(m.get('role') == 'tool' and 'Denied' in str(m.get('content')) for m in chat.hist)
chat.print_hist()
```
llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
**user**
What is 2 plus 3? Use the add tool. After every tool
call result, briefly summarise in prose what you found before continuing
or calling another tool.
------------------------------------------------------------------------
**assistant**
🔧 add({‘a’: 2, ‘b’: 3})
------------------------------------------------------------------------
**tool**
Denied by human operator
------------------------------------------------------------------------
**assistant**
> **🧠 Thinking**
>
> Okay, the user asked, “What is 2 plus 3? Use the add tool.” I called
> the add function with a=2 and b=3. The tool response was denied by the
> human operator. Now I need to explain that the result was denied. Let
> me check the tool’s response again. The tool says “Denied by human
> operator.” So the answer is 5, but it was denied. I should inform the
> user that the result was denied and provide the correct sum. Make sure
> to mention that the tool’s response was denied and then state the
> final answer.
The result of 2 plus 3 was denied by the human operator. The correct
answer is 5.
### Streaming and thinking
`stream=True` yields markdown chunks (thinking streams as a blockquote,
tool calls as `⏳` lines) - render live with
[`display_stream`](https://vedicreader.github.io/rishi/core.html#display_stream),
exactly like the litert backend. With a Qwen3 model, `think=True` turns
the thinking channel on.
``` python
ch = LlamaChat(qwen3_06b, think=True)
for c in ch('Count: one two three', stream=True): print(c, end='', flush=True)
print('\n', ch.use)
```
llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
> **🧠 Thinking**
>
>
> Okay, the user wrote "Count: one two three" and I need to respond. Let me think about how to handle this.
>
> First, I should acknowledge the input. Maybe say "Count: one two three" to confirm. Then, I can provide a helpful response. Since the user is asking for a count, perhaps they need to count something. But the message is just a list. Maybe they want to count the numbers in the message. Let me check the numbers: one, two, three. That's three numbers. So the count is three. But the user might be expecting a different answer. Wait, maybe they want to count the numbers in the message, which is three. But maybe they want to know how many numbers are there. So the answer would be three. But I should make sure to respond clearly. Let me structure the response to confirm the count and offer further assistance.
>
Count: one two three
There are **three** numbers in the message. Let me know if you need help with anything else! 😊
total=236|in=23|out=213|turns=1
``` python
# sync streaming: SaveReturn wraps the chunk stream; display_stream renders live + returns md, `.value` is the final Resp
from fastcore.xtras import SaveReturn
s = SaveReturn(ch('And backwards?', stream=True))
md = display_stream(s)
assert md.strip() and resp_text(s.value) == resp_text(ch.turn_res)
```
> **🧠 Thinking**
>
> Okay, the user just asked “And backwards?” after the previous count.
> Let me think about how to respond.
>
> First, I need to check if there’s a specific number of numbers in the
> message. The original count was “one two three”, which is three
> numbers. So, if they want it backwards, it should be “three two one”.
>
> I should confirm that the count is still three, which it is. Then,
> make sure the response is clear and friendly. Maybe add an emoji to
> keep it positive. Also, ask if they need anything else. That way, the
> conversation stays helpful.
Count: three two one\
There are **three** numbers in the message. Let me know if you need help
with anything else! 😊
``` python
# async streaming: adisplay_stream drives the turn; the returned iterator carries the final Resp on `.value`
achat = AsyncChat(qwen3_06b, think=False, temp=0.0)
sa = run_coro(achat('Count to three.', stream=True))
md = run_coro(adisplay_stream(sa))
assert md.strip() and resp_text(sa.value) == resp_text(achat.turn_res)
assert [m['role'] for m in achat.hist] == ['user', 'assistant']
achat.close()
```
llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
> **🧠 Thinking**
Count to three:\
1. **One**\
2. **Two**\
3. **Three**
### One-shot utilities
``` python
chat = LlamaChat(qwen3_06b)
print(chat.classify('I absolutely loved this movie!', ['positive', 'negative']))
```
llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
positive
``` python
@dataclass
class Age: year:int; month:int
@dataclass
class Person:
'A Person record. eg: Rama, born 1995-06'
name: str # the person's name
age: Age # the person's age (year and month of birth)
o = _mk_obj(Person, {'name': 'Alice', 'age': {'year': 1995, 'month': 6}})
test_eq(o.name, 'Alice')
test_eq(o.age, Age(1995, 6))
assert isinstance(o.age, Age)
```
``` python
p = chat.structured('Alice was born in June 1995.', Person)
assert p.name == 'Alice' # `structured` fills required fields (grammar-constrained)
assert isinstance(p.age, Age) # nested dataclasses are rebuilt, not left as dicts
p
```
Person(name='Alice', age=Age(year=1995, month=6))
### Images and audio
A multimodal GGUF is two files: the model and an `mmproj` projector (its
image/audio encoder). Pass `mmproj=True` to resolve the projector from
the same repo through the same cache-first ladder as the model, or give
an explicit path. rishi then builds llama.cpp’s `MTMDChatHandler` -
carrying the audio patches from [above](#audio) - which splices the
media into the model’s own chat template.
Mix images and audio into the message list exactly as on the litert
backend: raw `bytes` or a `Path`. rishi sniffs the MIME type and emits
an `image_url` or `input_audio` content part, and the handler turns each
into an mtmd bitmap in document order.
Audio is decoded from WAV by
[`read_audio`](https://vedicreader.github.io/rishi/llama.html#read_audio)
and resampled to whatever rate the projector reports. Other containers
(mp3, flac) raise: this wheel bundles stb_image but no audio decoder, so
there is nothing to decode them with - convert to WAV first. said that
llama-cpp-python’s audio capability isn’t great. I’d stick to litert or
cloud models for now.
``` python
# downloads the model *and* its mmproj projector once, then loads both from cache
vchat = LlamaChat('unsloth/gemma-4-E2B-it-GGUF', mmproj=True)
r = vchat(['Describe this image.', Path('images.jpeg')]) # raw bytes work too
print(resp_text(r))
vchat.close()
```
llama_context: n_ctx_seq (8192) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_iswa: using full-size SWA cache (ref: https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
llama_kv_cache: the V embeddings have different sizes across layers and FA is not enabled - padding V cache to 512
llama_kv_cache: the V embeddings have different sizes across layers and FA is not enabled - padding V cache to 512
add_text: <|turn>user
Describe this image.
add_text: <|image>
image_tokens->nx = 81
image_tokens->ny = 1
batch_f32 size = 1
add_text:
add_text:
<|turn>model
encoding image slice...
clip_image_batch_encode: copying image 1/1 to input buffer (nx=432, ny=432)
clip_image_batch_encode: output embedding shape [1536, 81, 1]
image slice encoded in 135 ms
decoding image batch 1/1, n_tokens_batch = 81
image decoded (batch 1/1) in 337 ms
This is a photograph of a **German Shepherd dog** outdoors.
Here are some details based on the image:
* **Subject:** The main subject is a medium-to-large German Shepherd.
* **Appearance:** The dog has rich, reddish-brown and black fur. It appears alert and healthy.
* **Pose/Expression:** The dog is looking directly toward the camera with an engaged and focused expression. Its mouth is slightly open, and its tongue is visible, suggesting it might be panting slightly or happy.
* **Setting:** The background is soft and blurred (bokeh), suggesting an outdoor, natural setting, possibly a park, field, or wooded area. The lighting seems soft, indicating it might be an overcast day or shaded.
* **Mood:** The overall mood of the photo is energetic, loyal, and friendly.
In summary, it is a high-quality portrait of a handsome German Shepherd in an outdoor environment.
``` python
achat = LlamaChat('unsloth/gemma-4-E2B-it-GGUF', mmproj=True)
print(resp_text(achat(['Transcribe this clip.', Path('speech.wav')])))
achat.close()
```
llama_context: n_ctx_seq (8192) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_iswa: using full-size SWA cache (ref: https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
llama_kv_cache: the V embeddings have different sizes across layers and FA is not enabled - padding V cache to 512
llama_kv_cache: the V embeddings have different sizes across layers and FA is not enabled - padding V cache to 512
add_text: <|turn>user
Transcribe this clip.
add_text: <|audio>
audio_tokens->n_tokens = 600
add_text:
add_text:
<|turn>model
encoding audio slice...
clip_image_batch_encode: output embedding shape [1536, 600, 1]
audio slice encoded in 159 ms
decoding audio batch 1/2, n_tokens_batch = 512
audio decoded (batch 1/2) in 2195 ms
decoding audio batch 2/2, n_tokens_batch = 88
audio decoded (batch 2/2) in 644 ms
Dancing in the masquerade idol truth and plain sight jaded pop roll click shot who will I be today or not but such a tide as moving seems a sleep too full for sound and foam when that which drew from out the boundless deep turns again home twilight and evening bell and after that
``` python
# audio-only projector recovery: ultravox has no vision encoder, so upstream's `_init_mtmd_context` would
# hard-fail. A clean load proves the patch recovered (ctx built + audio supported -> keep it), and a non-empty
# reply proves the audio bitmap path runs end-to-end. Downloads the ~1B model + its mmproj once, then caches.
uvchat = LlamaChat('ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF', mmproj=True)
assert resp_text(uvchat(['Transcribe this clip.', Path('speech.wav')])).strip()
uvchat.close()
```
llama_context: n_ctx_seq (8192) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
add_text: <|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 02 Aug 2026
<|eot_id|><|start_header_id|>user<|end_header_id|>
Transcribe this clip.
audio_tokens->n_tokens = 187
audio_tokens->n_tokens = 187
add_text: <|eot_id|><|start_header_id|>assistant<|end_header_id|>
encoding audio slice...
clip_image_batch_encode: output embedding shape [2048, 187, 1]
audio slice encoded in 660 ms
decoding audio batch 1/1, n_tokens_batch = 187
audio decoded (batch 1/1) in 3683 ms
encoding audio slice...
clip_image_batch_encode: output embedding shape [2048, 187, 1]
audio slice encoded in 452 ms
decoding audio batch 1/1, n_tokens_batch = 187
audio decoded (batch 1/1) in 822 ms
our image part is byte-for-byte what llama.cpp’s multimodal handler
scans for (`get_image_urls` is an instance method here - the audio patch
above widened it)
``` python
img = Path('images.jpeg').read_bytes()
m = _mk_msg([img, 'what is this?'])
with tempfile.NamedTemporaryFile(suffix='.gguf') as f:
test_eq(MTMDChatHandler(f.name, verbose=False).get_image_urls([m]), [m['content'][0]['image_url']['url']])
test_eq(m['content'][1], {'type': 'text', 'text': 'what is this?'})
test_eq(_mk_content(Path('images.jpeg')), _mk_content(img)) # a Path is read and sniffed like bytes
# audio is no longer refused - it becomes an OpenAI `input_audio` part
p = _mk_content(Path('speech.wav'))
test_eq((p['type'], p['input_audio']['format']), ('input_audio', 'wav'))
```
### AsyncChat
``` python
chat = AsyncChat(qwen3_06b, think=True, temp=0.5, sp='When asked to compute something, reply with a ```python fence that prints the answer.')
run_coro(chat("give me a fibonacci function in python using it's equation"))
```
llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
> **🧠 Thinking**
>
> Okay, I need to create a Fibonacci function in Python using the
> mathematical equation. Let me think about how to approach this.
>
> First, the Fibonacci sequence is defined by the recurrence relation
> F(n) = F(n-1) + F(n-2), with base cases F(0) = 0 and F(1) = 1. So, the
> function should take an integer n and return the nth Fibonacci number.
>
> But wait, the user mentioned using the equation. Hmm, the equation for
> the nth Fibonacci number is F(n) = (phi^n - psi^n)/sqrt(5), where phi
> is (1+sqrt(5))/2 and psi is (1-sqrt(5))/2. That’s the closed-form
> formula. So maybe I can use this formula instead of the iterative
> approach.
>
> But the user might be expecting the iterative method. Let me check
> both approaches. The iterative approach is straightforward and easy to
> implement. It’s commonly used because it avoids floating-point issues
> with large numbers. Also, using the formula directly would be more
> efficient and accurate for very large n.
>
> Wait, but the user said “using it’s equation”. The equation is the
> closed-form formula. So I should use that. Let me make sure I can
> implement that.
>
> So, for the closed-form formula, the function can calculate the nth
> term using the formula. But how to handle the equation? The equation
> is F(n) = (phi^n - psi^n)/sqrt(5). So, for a given n, compute this
> value and return it.
>
> But what about the base cases? For n=0 and n=1, the formula returns 0
> and 1, respectively. So the function can handle those cases.
>
> But how to compute this in Python. Let me think about the steps.
>
> First, compute phi and psi. Then, for each n, compute phi^n and psi^n,
> subtract them, divide by sqrt(5), and return the result.
>
> But wait, for very large n, computing phi^n might be a problem due to
> floating-point precision. However, the formula is exact for integers
> n, so the result should be exact. So even for very large n, the
> formula should work accurately.
>
> So, the function can be written as:
>
> def fibonacci(n): if n \< 0: raise ValueError(“n must be a
> non-negative integer”) if n == 0 or n == 1: return 0 return ( (
> (phi**n - psi**n) / sqrt(5) ) )
>
> But the user might want the function to be efficient. Alternatively,
> using the iterative approach is better for large n. Let me think about
> the iterative approach.
>
> The iterative approach is straightforward:
>
> def fibonacci(n): if n \< 0: raise ValueError(“n must be a
> non-negative integer”) if n == 0 or n == 1: return 0 a, b = 0, 1 for
> \_ in range(n-1): a, b = b, a + b return b
>
> This approach avoids any floating-point calculations and is efficient
> for large n. Since the formula is exact, the iterative method is
> better for large n.
>
> The user might be confused between the two approaches. So I should
> explain which one is better. The iterative method is more efficient
> and accurate for large n, while the formula is exact but may not be as
> efficient in practice.
>
> But the user asked to compute something using the equation, so perhaps
> using the formula is acceptable. However, the iterative method is more
> practical for the purposes of code.
>
> Wait, but the user said “using it’s equation”. So perhaps the answer
> should use the closed-form formula. Let me check the calculation
> again.
>
> Yes, the formula is F(n) = (phi^n - psi^n)/sqrt(5). So, implementing
> that would be the correct approach.
>
> So, the code would be:
>
> def fibonacci(n): if n \< 0: raise ValueError(“n must be a
> non-negative integer”) if n == 0 or n == 1: return 0 return ( ( ( (
> (1 + sqrt(5))/2 ) \*\* n - ( (1 - sqrt(5))/2 ) \*\* n ) / sqrt(5) ) )
>
> But since n can be very large, using floating-point numbers could lead
> to precision issues. However, the formula is exact for integers n, so
> the result will be exact. But for very large n, this might not be the
> most efficient approach.
>
> Alternatively, using the iterative method is better. So, perhaps the
> answer should use the iterative approach.
>
> But the user asked to compute something using the equation. So,
> perhaps using the formula is correct. However, the iterative method is
> more practical.
>
> But the user might not be aware of the formula, so it’s better to
> explain both approaches.
>
> Wait, but the user’s instruction says “give me a fibonacci function in
> python using it’s equation”. So, perhaps the answer should use the
> equation. Let me check.
>
> The equation is F(n) = (phi^n - psi^n)/sqrt(5). So, implementing that
> in Python.
>
> But for very large n, the formula might not be accurate. For example,
> if n is 1000, then the value might be a floating-point number with
> some precision loss. However, for practical purposes, using the
> formula is acceptable.
>
> So, the code would be:
>
> import math
>
> def fibonacci(n): if n \< 0: raise ValueError(“n must be a
> non-negative integer”) if n == 0 or n == 1: return 0 phi = (1 +
> math.sqrt(5)) / 2 psi = (1 - math.sqrt(5)) / 2 return ( (phi**n -
> psi**n) / math.sqrt(5) )
>
> But perhaps the user expects the function to be written with the
> equation. So, using the equation would be the correct approach.
>
> Therefore, the answer would be to write the function using the
> formula.
>
> But since the user might not have access to math.sqrt, perhaps using
> the iterative approach is better. However, the formula is correct.
>
> Alternatively, perhaps the user wants the function to compute the
> Fibonacci number using the formula, so the code would use the formula.
>
> But the problem is that the formula uses floating-point calculations.
> However, the formula is exact for integers n.
>
> So, the code would be as follows.
>
> But for very large n, the formula may not be precise. However, the
> problem statement doesn’t specify the constraints on n, so assuming
> that n is a non-negative integer.
>
> So, the final answer would be the code using the formula.
``` python
import math
def fibonacci(n):
if n < 0:
raise ValueError("n must be a non-negative integer")
if n == 0 or n == 1:
return 0
phi = (1 + math.sqrt(5)) / 2
psi = (1 - math.sqrt(5)) / 2
return (phi**n - psi**n) / math.sqrt(5)
```
### Stopping with an LLM judge
`done` is any `chat -> bool`, so instead of
[`output_matches`](https://vedicreader.github.io/rishi/core.html#output_matches)
(a substring check on the last code output) you can let a model decide
when the loop is finished. The built-in
[`task_complete`](https://vedicreader.github.io/rishi/core.html#task_complete)
judges - via `chat.classify`, isolated - whether the conversation has
satisfied the request. Or write your own `done` that closes over a judge
[`Chat`](https://vedicreader.github.io/rishi/core.html#chat) (pass a
stronger model as the judge if you like), mirroring
[`output_matches`](https://vedicreader.github.io/rishi/core.html#output_matches)’
factory shape. This is backend-agnostic - the same `done` works with the
litert [`Chat`](https://vedicreader.github.io/rishi/core.html#chat).
``` python
def judged_by(judge, task):
"PyFenceCallback `done`: stop once `judge` (any Chat) rules the last code output answers `task`."
return lambda chat: judge.classify(
f"Task: {task}\nProgram output:\n{getattr(chat, 'turn_code_out', '')}\n\n"
"Does the output correctly and completely answer the task?", ['yes', 'no']) == 'yes'
sp = ''
# a custom LLM judge - here the model judges its own output; pass a stronger Chat as `judge` for real use
judge = LlamaChat(qwen3_4b, think=True, temp=0.6, sp=sp)
run_coro(chat('return python code with binet formula for fibonacci series for the first 5 positions',
cbs=[PyFenceCallback(done=judged_by(judge, 'fibonacci series upto 5 elements. so, [0,1,1,2,3]'))]))
chat.hist[-1]
```
llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
> **🧠 Thinking**
>
> Okay, let’s see. The user asked for a Fibonacci function using the
> Binet formula for the first five positions. They tried to run the code
> but got a NameError and then another error about ‘fib_nums’. Now they
> have the output from the code.
>
> The first code block had the binet function, and when they tried to
> use it, they had a NameError. Then, in the next attempt, they included
> the ‘fib_nums’ variable, which was not defined. But the user’s latest
> code now shows the output. So the answer should confirm that the code
> works and provide the result.
>
> I need to make sure the final answer is in prose, not markdown. The
> user probably wants to know that the code now outputs the first five
> Fibonacci numbers correctly. Since the output matches the expected
> result, the answer is correct. No more code blocks needed here.
The code now correctly outputs the first five Fibonacci numbers using
the Binet formula. The result is:
[0.0, 1.0, 1.0, 2.0, 3.0000000000000004]
This matches the expected output for the first five positions of the
Fibonacci series.
``` python
# zero-config alternative: the built-in `task_complete` judges completion via chat.classify (no expected value)
run_coro(chat('Use python to compute 2**10.', cbs=[PyFenceCallback(done=task_complete)]))
```
> **🧠 Thinking**
>
> Okay, the user wants me to compute 2 raised to the 10th power using
> Python. Let me think about how to approach this.
>
> First, I remember that in Python, exponentiation is done with the \*\*
> operator. So 2**10 should give 1024. But wait, I should check if
> there’s any edge case here. For example, if the exponent is 0, but 2^0
> is 1, which is a special case. But the user is asking for 2**10, which
> is straightforward.
>
> Wait, the user might not be familiar with the \*\* operator. Maybe
> they want to confirm that using 2\*\*10 is correct. Also, maybe they
> want to see how the result is calculated. But since the question is
> just to compute it, the code would be simple.
>
> Let me write the code. The user probably expects the result to be
> printed. So the code would be:
>
> print(2 \*\* 10)
>
> But I should make sure that there’s no need for any additional steps.
> The result is 1024, so the code should output that. The user might
> also want to see how the calculation works, but the question is just
> to compute it. So the final answer is 1024.
The calculation of 210 is straightforward using Python’s
exponentiation operator. Here’s the result:
``` python
print(2 ** 10)
```
The output is:
1024
``` python
chat.hist[-1]
```
{'role': 'user', 'content': '1024\n'}
``` python
chat.close()
```