walk

The runtime of one walk step: a short tutor note, then a check: a short answer or a code task.

A walk step shows a passage, then a short note from the tutor, then one check. The journey (journey.py) plans the steps and picks the check of each step. This module runs a step.

The note tells the learner what to look for in the passage. It does not answer the check. The short-answer check is a normal tutor question, with two changes: the step decides its way and depth, and each attempt keeps the walk, the step, the section and the page. A sitting log counts every check, also the checks that do not go through Session.answer.

In a code task, the learner writes one function. The tests run in a closed process with no network.

The note

The prompt for the note. It gets the idea, the passage and the kind of the next check. It never gets the check question. A step with no passage, as in a walk from a topic, uses another prompt: the note explains the idea itself, from its statement and the passages of your sources that name it, and it never says that a passage is missing.

The reply is cut to 3 sentences and to walks.note words (60 by default), even when the model writes more.

tutor_note writes the note once and keeps it on the step. A later call gives the kept note. A note that says that no passage was given is a failed note: the statement of the idea takes its place.

---

source

tutor_note

def tutor_note(
    lr, ask, j, i:int, sources:NoneType=None
)->str:

The note for step i of walk j: kept on the step, else written from the idea and the passage, or from the idea and sources when the step has no passage.

A short answer

Each attempt of a step keeps its place in the source. step_meta makes that record. In a side walk, from names the step that opened it, as <walk id>/<step>.

---

source

step_meta

def step_meta(
    j, i:int
)->dict:

The walk, the step, the section and the page of step i, for the record of each attempt. A side walk adds from, the step that opened it.

open_short opens the question of a step. The check of the step gives the way and the depth. The note goes into the question prompt. The question id stays on the step.

---

source

open_short

def open_short(
    tutor:abhyasa.tutor.Tutor, j, i:int, session:str
)->fastcore.basics.AttrDict:

Open the short-answer check of step i of walk j, and keep its id on the step.

The sitting log

A demo or an order prediction does not go through Session.answer. noted adds one row for it, in the same shape as the rows of a question.

---

source

Session.noted

def noted(
    a:dict, floor0:int
):

Add one log row for the attempt a, made outside Session.answer (a demo or an order prediction), and save the sitting.

A code task

The model writes the exercise: a task, a stub, a function name and the tests. code_task keeps the full exercise on the tutor. The page gets the question, the stub, the function name and the text of the visible tests. The solution and the hidden tests stay on the server.

---

source

code_task

def code_task(
    tutor:abhyasa.tutor.Tutor, c, session:str='', meta:dict=None
)->fastcore.basics.AttrDict:

Open a code task on concept c. Return the public question, the stub, the function name and the text of the visible tests.

_first_error gives the first failing test as text. A hidden test shows only its label and its status. A visible test shows its text, the error and the value that the function gave.

run_code grades one try. A syntax error uses no try: compile checks the code and runs none of it. A failing try keeps the task open. A pass or the last try closes it, as Exercise.check does.

---

source

run_code

def run_code(
    tutor:abhyasa.tutor.Tutor, qid:str, source:str, confidence:int=70
)->fastcore.basics.AttrDict:

Run source on the tests of the code task qid in a closed process, and grade it. A pass or the last try closes the task.

Session.code runs a try on the open question of the sitting. A close goes in the sitting log.

---

source

Session.code

def code(
    source:str, confidence:int=70
)->fastcore.basics.AttrDict:

Run source on the open code task of the sitting, log a close, and save the sitting.

Tests for the step runtime

These tests use the small paper in tests/walk_data.py and a fake model.

A walk with one fake model. It gives a note, a grade and a question.

lr = temp_learning(); did = add_paper(lr)
ask = Fake({'Return JSON: {"concepts"': EXTRACT_REPLY, 'You write a short note': 'Softmax makes weights from scores. Look at what a large score does.',
            'You grade one answer': dict(correct=True, grade=9, first_error='', misconception='', feedback='Yes.')},
           dflt=dict(question='Why do the weights add to 1?', answer='Each is divided by the sum.', rubric=['divide by the sum']))
j = journey_from_source(lr, ask, did)
t = Tutor(lr, ask, context=False)

The note is kept on its step. A second call makes no model call.

n = tutor_note(lr, ask, j, 1)
assert n.startswith('Softmax makes') and lr.journey(j.id).steps[1]['note'] == n
assert tutor_note(lr, Fake(), lr.journey(j.id), 1) == n

The note keeps at most 3 sentences and walks.note words (60 by default).

long = tutor_note(lr, Fake({'You write a short note': 'One. Two. Three. Four.'}), lr.journey(j.id), 0)
assert long == 'One. Two. Three.'
wide = tutor_note(lr, Fake({'You write a short note': ' '.join(['word'] * 40) + '. ' + ' '.join(['more'] * 40) + '.'}), lr.journey(j.id), 2)
assert len(wide.split()) <= 60

A step with no passage, as in a walk from a topic, gets a note that explains the idea itself, with the prompt walk.note_topic. The request has the quoted name and statement of the idea. It does not ask for a summary of a passage.

lr.walk_step(j.id, 0, read=None, note='')
ask2 = Fake({'You write a short note': 'Sequential computation runs one step after another. Look at the order of the steps.'})
n0 = tutor_note(lr, ask2, lr.journey(j.id), 0); p, sp = ask2.calls[-1]
assert n0.startswith('Sequential computation runs') and sp == prompt('walk.note_topic') and 'just read a passage' not in sp
assert 'The passage:' not in p and 'Idea: "Sequential computation"' in p and 'Statement: "A recurrent model computes' in p

With sources, the passages that name the idea go in the request as data, between <source> markers.

class Notes:
    name = 'notes'
    def search(self, q, k=4): return [passage('vault:notes', 'n1', 'My notes', 'Sequential computation stops parallel work.')]
lr.walk_step(j.id, 0, note=''); tutor_note(lr, ask2, lr.journey(j.id), 0, sources=[Notes()])
assert '<source title="My notes">Sequential computation stops parallel work.</source>' in ask2.calls[-1][0]

A note that still says that no passage was given is a failed note. The step gets the statement of the idea in its place.

for bad in ('No passage was provided, so there is nothing to summarize.', 'The text was not provided. Look for the order.'):
    lr.walk_step(j.id, 0, note='')
    got = tutor_note(lr, Fake({'You write a short note': bad}), lr.journey(j.id), 0)
    assert got == _trim(lr.concept('sequential-computation').statement) and lr.journey(j.id).steps[0]['note'] == got, got

The step decides the way and the depth. The question comes without its answer, and the prompt has the note.

s = Session(t, journey=j.id)
lr.walk_step(j.id, 1, check=dict(way='words', kind='short', depth=2, why='plain words fit'))
q = open_short(t, lr.journey(j.id), 1, s.id); s.q = q
assert q.method == 'words' and q.depth == 2
assert 'answer' not in q and 'rubric' not in q and lr.journey(j.id).steps[1]['qid'] == q.id
assert 'The learner just read this' in ask.calls[-1][0] and n in ask.calls[-1][0]

The attempt has its walk, step, section and page. The sitting log has the close.

r = s.answer('Each weight is divided by the sum of all.')
a = lr.attempts('softmax', limit=1)[0]
assert r.done and (a.walk, a.step, a.section, a.page) == (j.id, 1, '3.2.1 Scaled Dot-Product Attention', 4)
assert s.log[-1]['concept'] == 'softmax'

The review rule: the last way of a walk check is left out of the next question on the idea.

lr.update('softmax', due=0)
assert all(t.ask_on(lr.concept('softmax'), 3).method != a.method for _ in range(10))

A demo or an order prediction is not made in Session.answer. noted adds its row to the sitting log and saves it.

s.noted(dict(concept='softmax', question='Predict', correct=True, outcome='solid', method='demo', kind='demo'), floor0=2)
assert s.log[-1]['method'] == 'demo' and lr.session(s.id).log[-1]['method'] == 'demo'

Tests for a code task

A new walk and a sitting. The fake model writes the softmax exercise of tests/walk_data.py.

lr = temp_learning(); did = add_paper(lr)
ask = Fake({'Return JSON: {"concepts"': EXTRACT_REPLY, 'You write one small Python exercise': EXERCISE, 'Teach the smallest step': 'Take away the max. Check: what is softmax([5, 5])?'})
j = journey_from_source(lr, ask, did); t = Tutor(lr, ask, context=False); s = Session(t, journey=j.id)
k = code_task(t, lr.concept('softmax'), s.id, step_meta(j, 1)); s.q = k.q

The stub has no part of the solution. The page gets the two visible tests. The text of the hidden test stays on the server, and so do the answer and the rubric.

assert k.fname == 'softmax' and 'm = max(x)' not in k.stub and len(k.tests) == 2
assert 'answer' not in k.q and 'rubric' not in k.q

A syntax error uses no try. The message gives the line. A return outside a function and a null byte are syntax errors too, though ast.parse accepts the first.

r = s.code('def softmax(x, t=1.0)\n    return x')
assert r.syntax.startswith('line 1') and r.tries_left == 3 and not r.done
for bad in ('return 1', 'x = 1\0'): assert s.code(bad).syntax.startswith('line 1')
assert t.open[s.q.id].tries == []

The naive softmax fails on large scores. Two of three tests pass, and two tries are left. A hidden test shows no value.

r = s.code(NAIVE)
assert (r.passed, r.total, r.done, r.tries_left) == (2, 3, False, 2) and 'large scores' in r.first_error
assert not any(c['got'] for c in r.checks if c['hidden'])

A correct softmax closes the task. The attempt keeps the kind, the depth, the way and the walk. The sitting log has the close, and the way code gets the credit.

r = s.code(GOOD)
a = lr.attempts('softmax', limit=1)[0]
assert r.done and r.correct and (a.kind, a.depth, a.method, a.walk) == ('code', 4, 'code', j.id)
assert s.log[-1]['concept'] == 'softmax' and lr.stats().methods['*']['code'][0] > 0