nb

Learn with code in a notebook (leela or Jupyter): question cards, and code exercises that the kernel checks.

This module brings the tutor into a notebook. learn shows a question card in the output of a cell. exercise gives a code exercise, and its tests run on your function in the kernel. Both use the same Tutor and the same learning shelf as the web app.

These are the imports of the module.

Use

Leela shows ipywidgets, so the card is a widget in the output of the cell. Leela keeps cell metadata that it does not know. Thus the cells from write_exercise keep their metadata.abhyasa.

from abhyasa.nb import learn, exercise
learn('reciprocal-rank-fusion')      # a question card
ex = exercise('reciprocal-rank-fusion')
def rrf(lists, k=60): ...            # your code, in the next cell
ex.check(rrf)                        # runs the tests, records the attempt

The tutor

tutor makes the tutor of the notebook one time. A call with a store or a model makes a new one.

---

source

tutor

def tutor(
    learning:NoneType=None, ask:NoneType=None
)->abhyasa.tutor.Tutor:

The tutor of the notebook, made one time: the default vault and rishi_ask, unless you give others.

The question card

Card has a box for the answer, a box for the think-aloud, a confidence slider and four buttons.

---

source

Card

def Card(
    sess:abhyasa.tutor.Session
):

A question card widget over a Session: answer, think aloud, confidence, hint, and next.

fb_html shows a result in the card. learn starts a session in a card.

---

source

learn

def learn(
    focus:str=None, subject:str=None, limit:int=12
)->__main__.Card:

A question card for a session, on one concept when you name it.

---

source

fb_html

def fb_html(
    r
)->str:

A result as HTML.

After a miss on a concept that has a demo, the card shows the demo under the feedback. demo_view gives the demo as a notebook output. It uses show from rupaka, and leela draws the output in a sandboxed frame.

---

source

demo_view

def demo_view(
    r, learning
):

The demo of result r as a notebook output, or None.

A result with a demo gives an output with the rupaka mime and a full HTML page. A result without one gives nothing.

from abhyasa.store import temp_learning
from rupaka.spec import gallery
from rupaka.render import MIME
dl = temp_learning(); dl.put('Vowel sandhi', 'Vowels join by rule.', 'sanskrit')
dl.put_demo(gallery()['vowel-sandhi'], 'vowel-sandhi')
mb = demo_view(AttrDict(demo='vowel-sandhi'), dl)._repr_mimebundle_()
assert MIME in mb and '<script' in mb['text/html']
test_eq(demo_view(AttrDict(correct=True), dl), None)

Code exercises

The model writes a task (a small function to make), a stub and assert tests. The tests run on your function in the kernel. The exercise shows a failing test, but not the fix. A pass at depth 4 (apply) counts as a pass in the learner model.

CODE_SP and code_prompt ask the model for one exercise.

---

source

code_prompt

def code_prompt(
    c
):

The model writes the tests, so the notebook does not trust them. safe_test accepts only one assert. The assert can use only the function of the learner, literals, comparisons, arithmetic and a short list of safe builtins. It cannot use an import, a name or an attribute with a leading underscore, open, exec or eval. validate_exercise checks the shape of the reply of the model. A test runs with only the safe builtins. When a test runs on the main thread (as in a kernel), it also has a time limit. Thus an endless loop fails the test.

---

source

time_limit

def time_limit(
    secs:float=2.0
):

Raise TimeoutError after secs, on the main thread; elsewhere no limit is possible, so none is set.

---

source

validate_exercise

def validate_exercise(
    r
)->fastcore.basics.AttrDict:

The model’s exercise, checked: task, fname, stub and 1-12 safe tests. Raise ValueError with the reason.

---

source

safe_test

def safe_test(
    t:str, fname:str
)->bool:

Whether t is one plain assert that uses only fname, literals and the safe builtins.

Exercise keeps one exercise open on the tutor. check grades it with the same tries and lessons as a question.

---

source

Exercise

def Exercise(
    tut:abhyasa.tutor.Tutor, slug:str
):

One code exercise on a concept. check(fn) runs its tests on your function.

fb_text shows a result as text. exercise makes an exercise on the tutor of the notebook.

---

source

exercise

def exercise(
    slug:str
)->__main__.Exercise:

A code exercise on concept slug.

---

source

fb_text

def fb_text(
    r
):

Exercise notebooks

write_exercise writes the exercise into tagged cells of a new notebook file, for you to open later. It never writes to a notebook that is already there.

---

source

write_exercise

def write_exercise(
    ex:__main__.Exercise, path
)->pathlib.Path:

A new notebook at path with the task, a stub cell and the check. Do not write over a file.

Tests

The first tests check the safety rules. Then a fake model writes a full exercise.

safe_test refuses an import, a name with two leading underscores, a call to anything but the function and the safe builtins, and code that is not one assert.

assert safe_test("assert rrf([['a'], ['b']], k=0) == ['a', 'b']", 'rrf')
assert safe_test("assert all(x in 'ab' for x in rrf([['a', 'b']]))", 'rrf')
for bad in ("import os", "assert __import__('os')", "assert rrf.__globals__", "assert open('/etc/passwd')", "assert exec('1')",
            "rrf([])", "while True: pass", "assert rrf(1); import os", "assert (lambda: 1)()"):
    assert not safe_test(bad, 'rrf'), bad

validate_exercise refuses an empty or a text tests, a missing name, and an unsafe test, with a reason.

good = dict(task='t', fname='f', stub='def f(): ...', tests=['assert f() == 1'])
test_eq(validate_exercise(good).fname, 'f')
for bad, why in ((dict(good, tests=[]), 'list of 1 to 12'), (dict(good, tests='assert f()'), 'list of 1 to 12'),
                 ({k: v for k, v in good.items() if k != 'fname'}, 'fname'), (dict(good, tests=['import os']), 'not one plain assert'), ([], 'JSON object')):
    test_fail(lambda: validate_exercise(bad), contains=why)

A test that runs too long fails after the time limit, and the kernel continues. Off the main thread, a limit is not possible, so the code sets none.

import threading, time
if threading.current_thread() is threading.main_thread():   # a kernel runs cells here, so the limit works
    t0 = time.time()
    try:
        with time_limit(.2):
            for _ in range(10**10): pass   # long, but never endless, so no test runner can hang on it
        raise AssertionError('no timeout')
    except TimeoutError: assert time.time() - t0 < 2
else:   # nbdev-test runs cells on another thread: no signal is possible there, so the limit steps aside
    with time_limit(.2): pass

A fake model writes an exercise with two tests. The tutor of the notebook uses a temporary shelf.

from abhyasa.store import temp_learning
from abhyasa.llm import Fake
lr = temp_learning()
lr.put('Reciprocal rank fusion', 'Merge ranked lists by summing 1/(k + rank).', 'ml')
fake = Fake({'Return JSON: {"task"': dict(task='Write rrf(lists, k) that returns items by fused score.', fname='rrf',
             stub='def rrf(lists, k=60):\n    ...', tests=["assert rrf([['a','b'],['b','a']])[0] in 'ab'", "assert rrf([['a'],['b']], k=0) == ['a','b']"],
             solution='...'), 'Teach the smallest step': 'Sum 1/(k+r). Check: what if k is large?'},
            dflt=dict(question='Why ranks?', answer='Comparable.', rubric=['comparable']))
t = tutor(lr, fake)
ex = exercise('reciprocal-rank-fusion')

A wrong function fails both tests, and the exercise stays open.

def rrf(lists, k=60): return []
assert ex.check(rrf).startswith('0 of 2 tests pass')

A correct function passes after the failed try, so the outcome is fragile and the concept is shaky at depth 4.

def rrf(lists, k=60):
    s = {}
    for l in lists:
        for r, x in enumerate(l): s[x] = s.get(x, 0) + 1 / (k + r + 1)
    return sorted(s, key=lambda x: (-s[x], x))
out = ex.check(rrf)
assert out.startswith('fragile'), out     # passed after a failed try
test_eq(lr.concept('reciprocal-rank-fusion').shaky, 4)

write_exercise tags the cells and does not write over a file.

import tempfile
from fastcore.nbio import read_nb
ex2 = exercise('reciprocal-rank-fusion')
p = write_exercise(ex2, Path(tempfile.mkdtemp())/'rrf.ipynb')
nb = read_nb(p)
test_eq(nb.cells[1].metadata['abhyasa']['concept'], 'reciprocal-rank-fusion')
test_fail(lambda: write_exercise(ex2, p), exc=FileExistsError)

learn shows a card with the first question.

c = learn('reciprocal-rank-fusion')
assert 'Why ranks?' in c.q.value