ollama

Run models through a local Ollama daemon, with rishi installing Ollama, pulling models and starting ollama serve for you when needed.

from fastcore.test import test_eq, test_fail

Where the daemon is

OLLAMA_HOST takes a bare port, a host:port, or a full URL. One variable both configures the daemon and finds it, so ollama_url accepts all three spellings.

A loopback daemon must not go through an HTTP proxy. httpx reads $HTTPS_PROXY from the environment by default. In a sandbox that sends 127.0.0.1 somewhere with no route back. is_local decides whether a client trusts the environment.

---

source

is_local

def is_local(
    url
):

Return True when url points at this machine, whose daemon must be reached without any HTTP proxy.

---

source

ollama_url

def ollama_url(
    host:NoneType=None
):

Normalize an Ollama host (:11434, host:port, or a full URL) to a base URL. None reads $OLLAMA_HOST.

test_eq(ollama_url('http://127.0.0.1:11434'), 'http://127.0.0.1:11434')
test_eq(ollama_url('localhost:11434'), 'http://localhost:11434')
test_eq(ollama_url(':11434'), 'http://127.0.0.1:11434')          # the bare-port spelling
test_eq(ollama_url('box.local'), 'http://box.local:11434')       # port defaulted
test_eq(ollama_url('https://gpu.internal'), 'https://gpu.internal:443')

assert is_local(ollama_url()) and is_local(ollama_url('localhost'))
assert not is_local(ollama_url('https://gpu.internal'))          # that one does go through the proxy

Model ids

Ollama names models family:tag. It also serves any GGUF repo on the HuggingFace Hub under an hf.co/ prefix. ollama_model maps rishi’s ids onto that, with quant as the tag. The Qwen/Qwen3-4B-GGUF constant rishi.llama uses runs here unchanged.

A local .gguf file has no address in Ollama’s store. Importing one takes a blob upload and an /api/create. rishi says that rather than failing on the wire.

---

source

ollama_model

def ollama_model(
    model:NoneType=None, quant:str='Q4_K_M'
):

A rishi model id as Ollama addresses it: a hub GGUF repo becomes hf.co/<repo>:<quant>, an Ollama id passes through.

test_eq(ollama_model('qwen3:4b'), 'qwen3:4b')                    # an Ollama id is left alone
test_eq(ollama_model('gemma3'), 'gemma3')                          # ...tagless too
test_eq(ollama_model(), dflt_model)
test_eq(ollama_model('Qwen/Qwen3-4B-GGUF'), 'hf.co/Qwen/Qwen3-4B-GGUF:Q4_K_M')     # the rishi.llama id
test_eq(ollama_model('Qwen/Qwen3-4B-GGUF', 'Q8_0'), 'hf.co/Qwen/Qwen3-4B-GGUF:Q8_0')
test_eq(ollama_model('Qwen/Qwen3-4B-GGUF', None), 'hf.co/Qwen/Qwen3-4B-GGUF')      # let Ollama choose
test_eq(ollama_model('hf.co/bartowski/x-GGUF:IQ3_M'), 'hf.co/bartowski/x-GGUF:IQ3_M')
test_fail(lambda: ollama_model('/models/mine.gguf'), contains="runtime='llama'")

The wire

OllamaClient holds the HTTP surface rishi uses. /api/chat talks to a model. The management endpoints list the local store, pull what is missing, and name the models in memory. A refusal carries a JSON error field, and OllamaError repeats it.

/api/pull and a streamed /api/chat both answer in newline-delimited JSON, so one stream method serves both. An error can also arrive mid-stream, after a 200. The iterator checks for that too.

---

source

OllamaClient

def OllamaClient(
    host:NoneType=None, timeout:int=600, client:NoneType=None
):

One Ollama daemon over HTTP: /api/chat, plus the model-management endpoints rishi drives.

---

source

OllamaError

def OllamaError(
    *args, **kwargs
):

What the daemon said went wrong, or that nothing answered.

pull hands on Ollama’s own progress records. pull_progress renders them as one rewritten line, so a 4GB download looks like a download rather than a hang.

---

source

pull_progress

def pull_progress(
    out:NoneType=None
):

A pull callback printing one rewritten line: the status, and how far the bytes have got.

---

source

fmt_bytes

def fmt_bytes(
    n
):

Bytes as a short human string.

# the client against a scripted daemon: one `httpx.MockTransport` stands in for the whole HTTP
# surface, so requests and responses are exercised rather than mocked away
def mk_client(handler, **kw):
    "An `OllamaClient` whose requests are answered by `handler` instead of a daemon."
    return OllamaClient(client=httpx.Client(transport=httpx.MockTransport(handler),
                                            base_url='http://test'), **kw)

seen = []
def handler(req):
    seen.append((req.method, req.url.path, json.loads(req.content) if req.content else {}))
    if req.url.path == '/api/version': return httpx.Response(200, json={'version': '0.12.0'})
    if req.url.path == '/api/tags':    return httpx.Response(200, json={'models': [{'name': 'qwen3:0.6b'}]})
    if req.url.path == '/api/show':    return httpx.Response(200, json={'capabilities': ['completion','tools','vision']})
    if req.url.path == '/api/ps':      return httpx.Response(200, json={'models': [{'name': 'qwen3:0.6b'}]})
    if req.url.path == '/api/delete':  return httpx.Response(200, text='')
    return httpx.Response(404, json={'error': f'unknown path {req.url.path}'})

cl = mk_client(handler)
test_eq(cl.version(), '0.12.0')
test_eq(cl.up(), True)
test_eq(cl.models(), ['qwen3:0.6b'])
test_eq(cl.have('qwen3:0.6b'), True)
test_eq(cl.have('gemma3:1b'), False)
test_eq(cl.show('qwen3:0.6b')['capabilities'], ['completion', 'tools', 'vision'])
test_eq(len(cl.ps()), 1)
test_eq(cl.delete('qwen3:0.6b'), True)
test_eq(seen[-1][:2], ('DELETE', '/api/delete'))
test_fail(lambda: cl.get('/api/nope'), contains='unknown path')      # the daemon's own words

import io

# Connection failure returns `None`; streamed errors still raise.
def dead(req): raise httpx.ConnectError('nothing listening')
test_eq(mk_client(dead).version(), None)
test_eq(mk_client(dead).up(), False)

lines = [{'status': 'pulling manifest'}, {'status': 'downloading', 'completed': 512, 'total': 1024},
         {'error': 'model not found'}]
def puller(req): return httpx.Response(200, text='\n'.join(json.dumps(d) for d in lines))
got = []
test_fail(lambda: mk_client(puller).pull('nope', got.append), contains='model not found')
test_eq([d['status'] for d in got], ['pulling manifest', 'downloading'])   # progress up to the failure

buf = io.StringIO()
pull_progress(buf)({'status': 'downloading', 'completed': 512*1024, 'total': 1024*1024})
assert 'downloading 512.0KB/1.0MB' in buf.getvalue()
test_eq(fmt_bytes(900), '900B')
test_eq(fmt_bytes(1536), '1.5KB')

Installing Ollama

Ollama ships one release archive per platform, and nothing in it needs root or a package manager. The published one-liner pipes into /usr. The archive extracts anywhere. rishi keeps its own copy under ~/.cache/rishi/ollama and leaves the system alone. A machine with ollama on $PATH keeps using that one.

Linux archives are zstd. That is stdlib from Python 3.14, and before it rishi[ollama] brings zstandard. macOS ships gzip and Windows a zip.

---

source

ollama_bin

def ollama_bin(
    dir:NoneType=None, system:bool=True
):

An ollama binary: $OLLAMA_BIN, then rishi’s own install, then $PATH unless system=False. None if there is none.

---

source

install_url

def install_url(
    system:NoneType=None, machine:NoneType=None
):

URL of the Ollama release archive for this platform.

---

source

install_ollama

def install_ollama(
    dir:pathlib.PosixPath=Path('/home/runner/.cache/rishi/ollama'), url:NoneType=None, force:bool=False,
    on_progress:NoneType=None
):

Install Ollama under dir for this user: no root, no package manager, no system service. Returns the binary.

---

source

download

def download(
    url, dest, on_progress:NoneType=None, timeout:int=600
):

Stream url to dest, reporting (done, total) bytes to on_progress.

---

source

extract_archive

def extract_archive(
    src, dest
):

Extract a .tgz, .tar.zst or .zip release archive into dest.

test_eq(install_url('Linux', 'x86_64'),  'https://ollama.com/download/ollama-linux-amd64.tar.zst')
test_eq(install_url('Linux', 'aarch64'), 'https://ollama.com/download/ollama-linux-arm64.tar.zst')
test_eq(install_url('Darwin', 'arm64'),  'https://ollama.com/download/ollama-darwin.tgz')
test_eq(install_url('Windows', 'amd64'), 'https://ollama.com/download/ollama-windows-amd64.zip')
test_fail(lambda: install_url('Plan9', 'x86_64'), contains='no Ollama release')

# extraction and binary discovery, on a tarball built here. `bin/ollama` is what the real Linux
# and macOS archives lay down, and what `ollama_bin` has to find without consulting $PATH
import tempfile

with tempfile.TemporaryDirectory() as d:
    d = Path(d)
    (d/'bin').mkdir()
    (d/'bin'/'ollama').write_text('#!/bin/sh\necho fake')
    with tarfile.open(d/'a.tgz', 'w:gz') as t: t.add(d/'bin'/'ollama', arcname='bin/ollama')
    out = extract_archive(d/'a.tgz', d/'out')
    assert (out/'bin'/'ollama').exists()
    test_eq(ollama_bin(out, system=False), out/'bin'/'ollama')
    test_eq(install_ollama(out), out/'bin'/'ollama')       # already there, so nothing is downloaded
    test_eq(ollama_bin(d/'empty', system=False), None)

    # a flat archive (just `ollama`) is found too, and $OLLAMA_BIN wins over both
    (d/'flat').mkdir(); (d/'flat'/'ollama').write_text('x')
    test_eq(ollama_bin(d/'flat', system=False), d/'flat'/'ollama')
    os.environ['OLLAMA_BIN'] = str(d/'bin'/'ollama')
    try: test_eq(ollama_bin(d/'flat', system=False), d/'bin'/'ollama')
    finally: del os.environ['OLLAMA_BIN']

Running the daemon

OllamaServer starts ollama serve and waits for it to answer, installing Ollama first if the machine has none.

Ollama configures some things per process rather than per request. Those are constructor arguments here. They cover the model store, the default context length, how long a model stays resident, flash attention, KV cache quantization, and the parallel and loaded-model limits.

A daemon already listening is left alone. Someone else’s server is not rishi’s to restart or reconfigure. One rishi starts is stopped at interpreter exit, so a notebook leaks nothing. It gets /dev/null as stdin, so the daemon never shares the caller’s terminal.

---

source

OllamaServer

def OllamaServer(
    host:NoneType=None, bin:NoneType=None, dir:pathlib.PosixPath=Path('/home/runner/.cache/rishi/ollama'),
    models:NoneType=None, n_ctx:NoneType=None, keep_alive:NoneType=None, flash_attn:NoneType=None,
    kv_cache_type:NoneType=None, num_parallel:NoneType=None, max_loaded:NoneType=None, log:NoneType=None
):

An ollama serve process rishi starts and owns, installing Ollama first if the machine has none.

---

source

stop_ollama

def stop_ollama(
    host:NoneType=None
):

Stop the daemon rishi started at host, or every one of them when host is None.

---

source

ensure_ollama

def ensure_ollama(
    host:NoneType=None, install:bool=True, serve:bool=True, timeout:int=60, on_progress:NoneType=None,
    srv_kw:NoneType=None
):

A client for a live daemon at host: reuse one already listening, else start one, installing Ollama if needed.

# the process-level configuration, which is the half of Ollama that is not a request field
s = OllamaServer('localhost:11500', models='/data/models', n_ctx=8192, keep_alive='30m',
                 flash_attn=True, kv_cache_type='q8_0', num_parallel=2)
e = s.env()
test_eq(e['OLLAMA_HOST'], 'http://localhost:11500')
test_eq(e['OLLAMA_MODELS'], '/data/models')
test_eq(e['OLLAMA_CONTEXT_LENGTH'], '8192')
test_eq(e['OLLAMA_KEEP_ALIVE'], '30m')
test_eq(e['OLLAMA_FLASH_ATTENTION'], '1')            # a bool, in the spelling Ollama reads
test_eq(e['OLLAMA_KV_CACHE_TYPE'], 'q8_0')
test_eq(e['OLLAMA_NUM_PARALLEL'], '2')
assert 'OLLAMA_MAX_LOADED_MODELS' not in e           # unset stays unset, rather than guessing a default
test_eq(s.running, False)
test_eq(s.stop().running, False)                     # stopping one that never started is fine

# with nothing listening and no leave to start anything, the error names both ways out
test_fail(lambda: ensure_ollama('127.0.0.1:11599', serve=False), contains='ollama serve')
test_fail(lambda: OllamaServer('127.0.0.1:11599', bin='/nonexistent/ollama').start(install=False, timeout=1),
          contains='no ollama binary')
# the daemon gets end of file as stdin, so the caller's terminal keeps its waiting line and stays blocking
import os, pty, subprocess, sys, tempfile
from pathlib import Path
fake = Path(tempfile.mkdtemp())/'ollama'
fake.write_text(f'#!{sys.executable}\nimport os, sys; t = os.isatty(0); os.set_blocking(0, False); open(sys.argv[0] + ".tty", "w").write(str(t)); sys.exit(3)\n'); fake.chmod(0o755)
m, s = pty.openpty(); os.write(m, b'x\n')
probe = f"""import os, select
from rishi.ollama import OllamaServer
try: OllamaServer('127.0.0.1:11599', bin={str(fake)!r}).start(install=False, timeout=5)
except Exception as e: print(type(e).__name__)
print(open({str(fake)!r} + '.tty').read(), os.get_blocking(0), os.read(0, 9) if select.select([0], [], [], 1)[0] else b'')"""
got = subprocess.run([sys.executable, '-c', probe], stdin=s, capture_output=True, text=True, timeout=60).stdout.split()
os.close(s); os.close(m)
test_eq(got, ['OllamaError', 'False', 'True', "b'x\\n'"])

Messages

Ollama’s message shape is OpenAI’s, with two differences. Images sit in a sibling images list of bare base64 rather than in content parts. Thinking has its own thinking field rather than <think> tags in the text. to_ollama_msg converts rishi’s canonical history to that, and carries assistant thinking back out so a resumed conversation keeps it.

Images go out on every turn rather than collapsing to a placeholder, as on remote and unlike llama. The daemon holds the KV cache, so a repeat costs a prefix match rather than a re-encode.

Audio has nowhere to go here. rishi.llama (mtmd) and rishi.mlx both take it, and the error says so rather than dropping the clip.

---

source

to_ollama_msgs

def to_ollama_msgs(
    msgs, sp:str=''
):

Convert rishi history to Ollama messages, with sp as the leading system message.

---

source

to_ollama_msg

def to_ollama_msg(
    m
):

Convert one rishi history message to an Ollama chat message.

A reply carries thinking in a thinking field and calls in tool_calls. A model whose chat template was built for neither writes <think> and <tool_call> into the text instead. norm_ochat reads both forms, as Urai’s norm_resp does for llama.cpp, so an arbitrary GGUF from the Hub behaves like a model from Ollama’s own library.

Usage comes from Ollama’s eval counts. Ollama reports no figure for reuse, so use.cached_tokens reads zero even when the daemon is reusing the conversation’s prefix.

---

source

norm_ochat

def norm_ochat(
    r, model:NoneType=None
):

Convert one /api/chat response to a rishi Resp, reading <think> and <tool_call> text as well as the native fields.

---

source

ollama_usage

def ollama_usage(
    r, model:NoneType=None
):

Convert Ollama’s eval counts to a rishi usage dict.

# text, images and thinking, each into the field Ollama reads it from
test_eq(to_ollama_msg({'role': 'user', 'content': 'hi'}), {'role': 'user', 'content': 'hi'})
m = to_ollama_msg(mk_msg(['what is this?', Path('images.jpeg').read_bytes()]))
test_eq(m['content'], 'what is this?')
test_eq(len(m['images']), 1)
assert not m['images'][0].startswith('data:')          # bare base64, not a data URL
test_eq(to_ollama_msg({'role': 'assistant', 'content': 'a', 'channels': {'thought': 't'}})['thinking'], 't')

# a tool call goes out as an object, and its result comes back named
call_msg = {'role': 'assistant', 'content': '', 'tool_calls': [ToolCall(name='add', arguments={'a': 1, 'b': 2})]}
test_eq(to_ollama_msg(call_msg)['tool_calls'], [{'function': {'name': 'add', 'arguments': {'a': 1, 'b': 2}}}])
test_eq(to_ollama_msg({'role': 'tool', 'name': 'add', 'content': '3'}),
        {'role': 'tool', 'content': '3', 'tool_name': 'add'})

# the system prompt leads, and a system message already in `hist` is dropped rather than duplicated
ms = to_ollama_msgs([{'role': 'system', 'content': 'old'}, {'role': 'user', 'content': 'hi'}], sp='be brief')
test_eq(ms, [{'role': 'system', 'content': 'be brief'}, {'role': 'user', 'content': 'hi'}])

test_fail(lambda: to_ollama_msg(mk_msg([Path('speech.wav').read_bytes(), 'transcribe'])),
          contains="runtime='llama'")

# the native fields
r = norm_ochat({'model': 'qwen3:0.6b', 'done_reason': 'stop',
                'message': {'role': 'assistant', 'content': 'hello', 'thinking': 'thinking hard'},
                'prompt_eval_count': 10, 'eval_count': 4})
test_eq(resp_text(r), 'hello')
test_eq(thought(r), 'thinking hard')
test_eq(r['usage'], {'prompt_tokens': 10, 'completion_tokens': 4, 'total_tokens': 14, 'model': 'qwen3:0.6b'})
assert 'truncated' not in r
test_eq(truncated(norm_ochat({'message': {'content': 'x'}, 'done_reason': 'length'})), True)

# a native tool call: arguments as an object, and an id invented because Ollama sends none
r = norm_ochat({'message': {'tool_calls': [{'function': {'name': 'add', 'arguments': {'a': 1}}}]}})
test_eq(len(r['tool_calls']), 1)
test_eq(r['tool_calls'][0].name, 'add')
test_eq(r['tool_calls'][0].arguments, {'a': 1})
assert r['tool_calls'][0]['id']

# and a model that writes tags into the text instead, as an arbitrary GGUF off the Hub will
r = norm_ochat({'message': {'content': '<think>hmm</think>ok\n<tool_call>{"name": "add", "arguments": {"a": 1}}</tool_call>'}})
test_eq(resp_text(r), 'ok')
test_eq(thought(r), 'hmm')
test_eq(r['tool_calls'][0].name, 'add')

OllamaChat

The tool loop, approval, budget, callbacks and context handling come from Urai, as on every backend. OllamaChat adds the request and the daemon it goes to.

think is a field of the request here, not a switch in the system prompt. It takes True, False, or a level string such as 'low' or 'high', which is sent unchanged. A model that cannot think refuses the field; the chat then retries once without it and stops sending it.

The context window comes from the portable ctx option, which also accepts the rishi.llama name n_ctx, and goes out as Ollama’s num_ctx. n_gpu_layers goes out as num_gpu. Anything else Ollama’s sampler takes goes in options=. structured uses Ollama’s format field, so the reply is constrained to the schema instead of merely asked for it. With pull=True, the default, a missing model is downloaded when the chat is built, as the llama backend downloads a missing GGUF. keep_alive says how long the daemon keeps the model loaded after a turn, and unload() drops it at once.

ctx_limit is the window the chat believes it has. It follows ctx when you set one, else the daemon’s own default, which is $OLLAMA_CONTEXT_LENGTH or 4096 tokens whatever the model was trained for. Urai’s pct_full and SlidingWindowCallback depend on it being right.

---

source

OllamaChat

def OllamaChat(
    model:NoneType=None, *, runtime:NoneType=None, model_path:NoneType=None, opts:NoneType=None, host:NoneType=None,
    client:NoneType=None, # a daemon to use instead of finding or starting one
    quant:str='Q4_K_M', # which quantization to name when pulling
    pull:bool=True, install:bool=True, serve:bool=True, srv_kw:NoneType=None, on_progress:NoneType=None,
    n_gpu_layers:NoneType=None, keep_alive:NoneType=None, # how long the daemon holds the weights after a turn
    options:NoneType=None, # Ollama's own `options` block, merged last
    comp_kw:NoneType=None, # merged into the request body verbatim
    **kw
):

A Chat that sends each turn to an Ollama daemon, starting the daemon and pulling the model when needed.

---

source

dflt_ctx

def dflt_ctx():

The context length the daemon uses when a request names none: $OLLAMA_CONTEXT_LENGTH, else Ollama’s own 4096.

from dataclasses import dataclass
from typing import Literal
@dataclass
class Pick:
    "A choice."
    a: Literal['x', 'y']
c, seen = OllamaChat.__new__(OllamaChat), {}
c.model_id = 'm'
c._payload = lambda **kw: seen.update(kw) or kw
c._ask = lambda p: {'message': {'role': 'assistant', 'content': '{"a": "x"}'}, 'done': True}
test_eq(c._structured_call('q', Pick, ''), {'a': 'x'})
test_eq(seen['fmt']['properties']['a']['enum'], ['x', 'y'])   # the schema goes through urai's mk_toolspec, which spells a Literal as an enum

/api/show reports what a model can do, and it is more reliable than any guess Urai’s model_caps makes elsewhere, because the daemon has the weights. model_caps(model, runtime='ollama') asks it through ollama_caps, which maps Ollama’s vision capability to image input. When no daemon answers, or the model is not in its store, the answer falls back to text only.

---

source

ollama_caps

def ollama_caps(
    model:NoneType=None, client:NoneType=None, quant:str='Q4_K_M'
):

What an Ollama model accepts, from the daemon’s own /api/show. None when no daemon answers.

Against a real daemon

Everything above runs offline, against a scripted daemon. The cells below need Ollama and download a model on first use, so they are marked #| eval: false and the test run skips them. On a bare machine, OllamaChat installs Ollama under ~/.cache/rishi/ollama, starts it, pulls the model and answers.

In the saved output of the last cell, gemma3:4b reports text input only, with source default. That model was not in the daemon’s store, so /api/show failed and the capabilities fell back to Urai’s default.

chat = Chat('ollama/qwen3:0.6b', think=False, sp='You are concise.')
print(resp_text(chat('Give me one fact about lobsters.')))
print(chat.use)

Lobsters are marine animals with a soft, flexible body and a powerful, claw-like appendage.
total=55|in=33|out=22|turns=1|model=qwen3:0.6b

# thinking levels, tools and structured output, on the real thing
def add(a: int, b: int) -> int:
    "Add two integers."
    return a + b

from dataclasses import dataclass
@dataclass
class Person: name:str; age:int

t = Chat('ollama/qwen3:0.6b', tools=[add], think='low')
print(resp_text(t('What is 2 + 3? Use the add tool.')))
print(t.structured('Extract: John Smith is 30.', Person))
print(t.classify('I loved this film!', ['positive', 'negative']))
t.close()

Person(name='John Smith', age=30)
positive

# the daemon, driven from Python: what is downloaded, what is loaded, what a model can do
cl = ensure_ollama()
print(cl.version(), cl.models())
print([m['name'] for m in cl.ps()])
print(model_caps('qwen3:0.6b', runtime='ollama'))
print(model_caps('gemma3:4b', runtime='ollama'))     # vision, per the daemon itself

0.32.15 ['ornith-1.5:9b', 'qwen3:0.6b', 'qwen3.6:27b-coding-nvfp4']
['qwen3:0.6b']
Caps(in=text, out=text, runtime)
Caps(in=text, out=text, default)

What this backend does not do

Ollama’s chat API carries images but not audio; rishi.llama and rishi.mlx take audio. There is no saved KV cache either. rishi.llama and rishi.mlx write theirs to disk and read it back, but Ollama keeps its cache inside the daemon and offers no handle on it, so use.cached_tokens reads zero even while the daemon reuses the prefix.

Token counts before a turn are estimates, because Ollama has no tokenizer endpoint; the real count arrives with the reply, in use. Local .gguf paths are refused, because Ollama serves only what is in its own store, and runtime='llama' is the backend for a file. There is no broker. LlamaBroker exists because one Llama belongs to one process, while the Ollama daemon already serves many clients; size it with OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS.

Tests

The tests on this page cover host parsing, model ids, the HTTP client against a scripted daemon (httpx.MockTransport), the installer, the server’s environment, message conversion and response normalisation. None of them needs Ollama, so the test run executes all of them. OllamaChat itself runs only in the manual cells above.