demo

Interactive demos on the learning shelf: store them, make them in the background after repeated misses, and log each prediction.

A demo is one interactive figure from rupaka. It asks the learner for a prediction first. It shows the result after the learner locks in the prediction. This module keeps demos on the learning shelf. Concepts and attempts are on the same shelf. The module decides when the tutor makes a demo. It runs that job in the background, so the learner never waits.

rule what happens
a miss on a concept that has a demo the result of the question gets the demo, and the session shows it under the lesson
two misses with a saved misconception, or a concept with visual: true in its meta the tutor queues a job: a model writes a demo, and rupaka checks it
the learner asks (“Show me a demo” on the concept page) the same job starts immediately
a prediction in a demo an attempt with kind='demo', and an answer to the learner model: depth 2, or depth 3 with a “why” line

Import the modules that this notebook uses.

Storage

A demo is one vault document. Its source is learning://demo/<slug>, its kind is demo, and its meta holds the spec.

meta value
concept the slug of the concept, the link
spec the rupaka spec
schema, rupaka the schema of the spec, and the rupaka version that checked it
by model:<name> or hand
checked the time of the last passing check, or None
stats shown: predictions made; right: predictions that were right

The text of the document is the title, the question, the gate prompts and the captions, so hybrid search finds a demo.

source

demo_text

def demo_text(
    spec
)->str:

The searchable text of a demo: title, ask, gate prompts and captions.

put_demo adds a demo, or replaces the demo with the same slug. It keeps the stats of the old demo. The vault keeps only one document for each text. So the text ends with the slug and the concept, and two demos with the same spec stay two documents. A model picks the id of a spec. When another concept already uses that id, put_demo puts the slug of the new concept in front of the id.

source

Learning.put_demo

def put_demo(
    spec:dict, concept:str, by:str='hand', checked:float=None
)->fastcore.basics.AttrDict:

Store spec as a demo of concept.

demo gets one demo by slug. demos lists the demos of a concept, newest first. demo_for gives the newest one.

source

Learning.demo_for

def demo_for(
    concept:str
)->fastcore.basics.AttrDict | None:

The newest demo of concept, or None.

source

Learning.demos

def demos(
    concept:str=None
)->fastcore.foundation.L:

The demos of concept (or all demos), newest first.

source

Learning.demo

def demo(
    slug:str
)->fastcore.basics.AttrDict | None:

One demo by slug.

Tests for storage

Make a shelf in a temporary vault, with one concept and the sandhi demo from the rupaka gallery.

from abhyasa.store import temp_learning
lr = temp_learning()
lr.put('Vowel sandhi', 'Two vowels at a word boundary join by rule.', 'sanskrit', slug='sa-sandhi-vowels')
spec = gallery()['vowel-sandhi']
d = lr.put_demo(spec, 'sa-sandhi-vowels')
test_eq((d.slug, d.concept, d.by, d.stats), ('vowel-sandhi', 'sa-sandhi-vowels', 'hand', dict(shown=0, right=0)))

demo_for finds the demo by its concept. demos gives an empty list for an unknown concept. A search for the words of the caption finds the demo.

test_eq(lr.demo_for('sa-sandhi-vowels').spec, spec)
test_eq(lr.demos('nope'), [])
test_eq(lr.search('rule stops another starts', kind='demo')[0]['doc_id'], d.doc_id)

The same id for another concept gets a new slug, so it does not replace the first demo.

lr.put('Guna', 'a + i gives e.', 'sanskrit', slug='guna')
test_eq(lr.put_demo(spec, 'guna').slug, 'guna-vowel-sandhi')
test_eq(lr.demo('vowel-sandhi').concept, 'sa-sandhi-vowels')
lr.drop('guna')

A demo is not a concept, so the list of concepts stays the same.

test_eq(lr.concepts().attrgot('slug'), ['sa-sandhi-vowels'])

Predictions

The demo page sends each prediction. record_prediction stores it as an attempt with kind='demo'. The attempt has the gate, the pick, the right answer and the “why” line. It counts the prediction in the stats of the demo. A prediction also shows what the learner understands. So it goes to the learner model as an answer. The depth is 2 (recall), or 3 (explain) when the learner wrote why. The rules are the same as for a question. A right pick with high confidence is solid. A wrong pick with high confidence is a misconception.

source

Learning.record_prediction

def record_prediction(
    slug:str, gate:str, picked, ans, ok:bool, why:str='', context:str='', session:str='', confidence:int=None
)->fastcore.basics.AttrDict:

File one prediction in demo slug as an attempt, count it in the stats of the demo, and update the learner model of its concept.

Record a wrong prediction with a “why” line. Check the attempt fields, the question text and the stats of the demo.

c0 = lr.concept('sa-sandhi-vowels')
a = lr.record_prediction('vowel-sandhi', 'join', 'ai', 'e', False, why='a and i make a diphthong', context='For a + i:', session='s1')
test_eq((a.kind, a.correct, a.thinking, a.demo, a.expected), ('demo', False, 'a and i make a diphthong', 'vowel-sandhi', 'e'))
test_eq(a.question, 'For a + i: What do the two vowels become?')
test_eq(lr.demo('vowel-sandhi').stats, dict(shown=1, right=0))

The learner model of the concept gets the prediction. A wrong pick with a “why” line is an answer at depth 3. So the ceiling falls to 3, and the concept is seen one more time.

c1 = lr.concept('sa-sandhi-vowels')
test_eq((c1.ceiling, c1.seen), (3, c0.seen + 1))
test_eq(a.depth, 3)

When to make a demo

want_demo says if a concept should get a demo. It is true after DEMO_MISSES misses with a saved misconception, or when the concept meta has visual: true.

source

want_demo

def want_demo(
    c
)->bool:

True when concept c should get a demo.

Check the rule. One miss, or misses with no misconception, give no demo. Two misses with a misconception, or visual, give a demo.

assert not want_demo(dict(misses=1, misconceptions=['x'])) and not want_demo(dict(misses=3, misconceptions=[]))
assert want_demo(dict(misses=2, misconceptions=['x'])) and want_demo(dict(visual=True))

Make a demo

make_demo asks the model for a demo with rupaka.author, and stores the demo. The loop validates the spec. It checks the spec in headless Chrome when a browser is available. It repairs the spec up to two times. When the model cannot make a good demo, make_demo returns None and stores nothing. The text lesson stays.

source

make_demo

def make_demo(
    learning:abhyasa.store.Learning, ask, slug:str, first_error:str='', check:NoneType=None, log:list=None
)->fastcore.basics.AttrDict | None:

A new checked demo for concept slug, stored on the shelf; None when the model fails.

A fake model acts as the agent. It replies with a demo for the concept, or with text that is not JSON.

from abhyasa.llm import Fake
good = {**spec, 'id': 'sandhi-by-model'}
fk = Fake({'You design one interactive demo': good})
d = make_demo(lr, fk, 'sa-sandhi-vowels', check=False)
test_eq((d.slug, d.by, d.checked), ('sandhi-by-model', 'model:unknown', None))
test_eq(lr.demo_for('sa-sandhi-vowels').slug, 'sandhi-by-model')

When the model never gives a good spec, make_demo stores nothing. The log names the errors.

log = []
test_eq(make_demo(lr, Fake({'You design one interactive demo': 'no demo today'}), 'sa-sandhi-vowels', check=False, log=log), None)
test_eq(len(log), 3)
test_eq(len(lr.demos('sa-sandhi-vowels')), 2)

The background job

queue_demo runs make_demo in the thread pool of the tutor. It runs only one job at a time for each concept. A second call while the job runs gives the same job.

source

job_state

def job_state(
    tutor, slug:str
)->str:

‘running’, ‘failed’, ‘done’ or ‘none’ for the demo job of slug.

source

queue_demo

def queue_demo(
    tutor, slug:str, first_error:str=''
):

Start a background job that makes a demo for slug; the running job if there is one.

The tutor hook

on_close runs after each question closes (it is in CLOSE_HOOKS). After a miss, it attaches the newest demo of the concept to the result. When there is no demo yet and the concept wants one, it queues the job. Then it marks the result with demo_pending. A pass adds nothing.

source

on_close

def on_close(
    tutor, q, r
):

After a miss: attach the newest demo of the concept to the result, or queue a job when the concept wants a demo.

Tests for the hook

Make a session where each answer is a miss with high confidence and the same misconception. The fake model writes the questions, grades, lessons and demos.

from abhyasa.tutor import Tutor
from concurrent.futures import ThreadPoolExecutor
def grader(p, sp): return dict(correct=False, grade=1, first_error='You add the vowels.', misconception='vowels just add', feedback='Look at the rule.')
fk = Fake({'You grade one answer': grader, 'Teach the smallest step': 'One step. Check: what is a + u?',
           'You design one interactive demo': good}, dflt=dict(question='What is a + i?', answer='e', rubric=['e']))
lr2 = temp_learning()
lr2.put('Vowel sandhi', 'Two vowels join by rule.', 'sanskrit', slug='sa-sandhi-vowels')
t = Tutor(lr2, fk, context=False, pool=ThreadPoolExecutor(1))
def miss():
    q = t.next(focus='sa-sandhi-vowels')
    return t.give_up(q.id, confidence=90)

The first miss queues no job, because there is only one miss.

r1 = miss()
test_eq((r1.get('demo'), r1.get('demo_pending'), lr2.concept('sa-sandhi-vowels').misses), (None, None, 1))

A miss with a misconception saves the misconception. The second such miss queues one job. A third call while the job runs does not queue another job.

lr2.update('sa-sandhi-vowels', misconceptions=['vowels just add'])
r2 = miss()
j = t.jobs['sa-sandhi-vowels']
assert r2.demo_pending and (queue_demo(t, 'sa-sandhi-vowels') is j or j.done())
t.jobs['sa-sandhi-vowels'].result()
test_eq(job_state(t, 'sa-sandhi-vowels'), 'done')

Now the concept has a demo, so the next miss gets its slug in the result.

r3 = miss()
test_eq(r3.demo, 'sandhi-by-model')

A pass gets no demo.

q = t.next(focus='sa-sandhi-vowels')
r = t._close(t.open[q.id], True, 80, AttrDict(correct=True, grade=9))
assert 'demo' not in r