profile

What abhyasa knows about you: your background, goals, interests, the examples that help you, your pace, and what confuses you.

The tutor asks better questions when it knows the learner. A coder gets code examples. A Sanskrit reader gets a Sanskrit example. This module keeps one profile document on the learning shelf. The profile gets data in three ways: a short first conversation, edits by the learner, and changes that the tutor makes from evidence. The module keeps each change with who made it and why. So the learner can see each change and undo it. The profile stays in the vault on this machine.

Import the modules that this notebook uses.

The fields

A field is a list of short items, or one value. PACES names the three paces. EXAMPLES names the kinds of example a learner can like.

profile_text gives the profile as lines for a prompt. It is empty when the profile has no data. So the prompt for a new learner does not change.

---

source

profile_text

def profile_text(
    p
)->str:

The profile as prompt lines; empty when nothing is known.

clean makes a value fit its field. A list field takes a list or lines of text. The pace must be a known pace. The session length stays between 3 and 40.

---

source

clean

def clean(
    k, v
):

The value v made to fit field k.

The document

profile reads the document. It uses the defaults for the fields that are not set. known is False until the profile has a saved value.

---

source

Learning.profile

def profile()->fastcore.basics.AttrDict:

The learner profile, with its list of changes, newest first.

set_profile changes fields. It keeps each change with the old value, the new value, who made it (you, onboarding or tutor) and why.

---

source

Learning.set_profile

def set_profile(
    by:str='you', why:str='', **kv
)->list:

Change fields of the profile and keep each change with who made it and why. Return the changes.

undo_profile puts back the old value of one change. It finds the change by its time. The undo is a change too, so the list shows all the changes.

---

source

Learning.clear_profile

def clear_profile():

Remove the profile and its changes.

---

source

Learning.undo_profile

def undo_profile(
    at:float
)->list:

Put back the old value of the change made at at.

The first conversation

The learner answers four questions, by voice or by text. One model call turns the answers into fields.

---

source

onboard

def onboard(
    lr:abhyasa.store.Learning, ask, answers:dict
)->list:

Fill the profile from the onboarding answers. Return the changes.

---

source

onboard_prompt

def onboard_prompt(
    answers:dict
)->str:

The prompt that turns the onboarding answers into profile fields.

The tutor learns from evidence

observe reads the recent answers: right or not, confidence, hints, misconceptions and think-aloud. A small model call proposes at most three changes, each with its reason. observe applies them immediately and keeps each one with its reason. So the learner sees them and can undo them.

---

source

observe_prompt

def observe_prompt(
    p, ev:str, st:NoneType=None
)->str:

The prompt that proposes changes to profile p from the evidence ev and the method counts in the stats st.

---

source

evidence

def evidence(
    lr:abhyasa.store.Learning, n:int=12
)->str:

The last n final answers as short lines for a prompt: the outcome, the method, the first error, gaps you named, lessons you asked for, and what you said.

observe applies the changes. A list field takes add and remove. The pace takes value. The goals and the notes are for the learner only, so observe drops a change to them.

---

source

observe

def observe(
    lr:abhyasa.store.Learning, ask, n:int=12
)->list:

Change the profile from the recent answers. Return the changes, each with its reason.

How well you know what you know

A good learner knows what they know. When you feel sure and you are often wrong, you stop checking your own thinking. When you feel unsure and you are often right, you hold back ideas that work. The “Sure?” choice on each answer gives the data. calibration counts the first try of each question. It counts how many times you felt sure (60% or more), and how many of those answers were right. It counts the same for unsure. An answer with no confidence, a “don’t know” and a demo do not count.

---

source

calibration

def calibration(
    lr:abhyasa.store.Learning, n:int=300
)->fastcore.basics.AttrDict:

How often the first try of a question was right, when you felt sure and when you felt unsure.

calibration_text gives the calibration in plain words, with one line about what it means. With fewer than five answers, it gives an empty text.

---

source

calibration_text

def calibration_text(
    r
)->str:

The calibration r in plain words; empty with fewer than five answers.

Tests

A new shelf has an empty profile, and its prompt text is empty.

from abhyasa.store import temp_learning
from abhyasa.llm import Fake
lr = temp_learning()
p = lr.profile()
test_eq((p.known, p.pace, p.background, profile_text(p)), (False, 'steady', [], ''))

set_profile keeps each change, with who made it. Lines of text become a list.

ch = lr.set_profile(background='codes in Python\nknows Sanskrit grammar', pace='gentle')
test_eq([c['field'] for c in ch], ['background', 'pace'])
p = lr.profile()
test_eq((p.known, p.background, p.changes[0]['by']), (True, ['codes in Python', 'knows Sanskrit grammar'], 'you'))
assert 'Background: codes in Python; knows Sanskrit grammar' in profile_text(p) and 'Pace: gentle' in profile_text(p)

When a value stays the same, set_profile makes no change. A bad pace becomes steady, and the session length stays in its range.

test_eq(lr.set_profile(pace='gentle'), [])
test_eq((clean('pace', 'warp'), clean('session_len', '200'), clean('session_len', 'x')), ('steady', 40, 12))

undo_profile puts back the old value, and keeps the undo as a change.

at = [c for c in lr.profile().changes if c['field'] == 'pace'][0]['at']
lr.undo_profile(at)
test_eq((lr.profile().pace, lr.profile().changes[0]['why']), ('steady', 'you undid a change'))

onboard fills fields from the answers with one model call.

fake = Fake({'make a short learner profile': dict(goals=['understand LLM papers'], examples=['code', 'maths'], session_len=8, pace='steady', notes='')})
ch = onboard(lr, fake, dict(want='LLMs, to read papers', how='code please'))
p = lr.profile()
test_eq((p.goals, p.examples, p.session_len), (['understand LLM papers'], ['code', 'maths'], 8))
test_eq({c['by'] for c in ch}, {'onboarding'})

observe changes the profile from attempts and keeps the reason. It drops a change to a field that it must not change.

lr.put('Attention', 'Weights from query-key dot products.', 'ml')
lr.record('attention', 'What shape is QK^T?', answer='d by d', correct=False, confidence=80, depth=3, misconception='mixes up the axes', final=True)
fake.rules['You keep a short profile'] = dict(changes=[dict(field='confuses', add='tensor shapes in attention', why='You missed a shape question with high confidence.'),
                                                     dict(field='goals', add='hack the planet', why='no')])
ch = observe(lr, fake)
test_eq([c['field'] for c in ch], ['confuses'])
test_eq((lr.profile().confuses, lr.profile().changes[0]['by']), (['tensor shapes in attention'], 'tutor'))
assert 'misconception: mixes up the axes' in fake.calls[-1][0]

With no answers there is no evidence, and no model call.

n = len(fake.calls)
test_eq(observe(temp_learning(), fake), [])
test_eq(len(fake.calls), n)

calibration counts the first try of each question. A pass after a wrong try counts as wrong. A “don’t know” and an answer with no confidence do not count.

kl = temp_learning(); kl.put('Attention', 'Weights from query-key dot products.', 'ml')
def rec(q, ok, conf, **kw): kl.record('attention', q, correct=ok, confidence=conf, session='s1', **kw)
rec('q1', False, 90, final=False); rec('q1', True, 90, final=True)   # sure, but wrong on the first try
rec('q2', True, 90); rec('q3', True, 30); rec('q4', False, 10)
rec('q5', False, None, dont_know=True); rec('q6', True, None)
test_eq(calibration(kl), dict(sure=2, sure_right=1, unsure=2, unsure_right=1))
test_eq(calibration_text(calibration(kl)), '')                       # four answers: too few to say

With enough answers, the text gives the counts and one line about what they mean.

test_eq(calibration_text(AttrDict(sure=4, sure_right=1, unsure=1, unsure_right=0)),
        'When you felt sure, you were right in 1 of 4. When you felt unsure, you were right in 0 of 1. '
        'You are often sure and wrong. Before you answer, ask: how could this be wrong?')
assert calibration_text(AttrDict(sure=1, sure_right=1, unsure=4, unsure_right=4)).endswith('Trust your first idea a little more.')

evidence reads the full conversation of a question. It gets the first error from a nudge, and the ideas that you said you did not know. It also gets a lesson that you asked for, the method, and each thing you said.

el = temp_learning(); el.put('Neuron', 'z = w . x + b.', 'ml')
th = [dict(who='you', text="I don't know what the loop means"), dict(who='tutor', kind='nudge', text='A loop adds products.', first_error='You use squares.'),
      dict(who='you', text='2(w1x1 + w2x2 + b)')]
el.record('neuron', 'Why does z double?', answer='2z', final=True, outcome='fragile', method='maths', thread=th, gaps=['weighted sum as a loop'], taught=1, teach_method='code')
ev = evidence(el)
for s in ('taught with maths', 'first error: You use squares.', 'said they did not know: weighted sum as a loop', 'asked for a lesson after it (code)', "I don't know what the loop means / 2(w1x1"):
    assert s in ev, s

observe keeps the calibration in the stats document. So each new question can read it, and does not have to read all attempts.

observe(el, Fake({'keep a short profile': dict(changes=[])}))
test_eq(set(el.stats().calib), {'sure', 'sure_right', 'unsure', 'unsure_right'})
test_eq((clean('explore', 'often'), clean('explore', 'always')), ('often', 'sometimes'))

clear_profile removes the full profile and its list of changes.

lr.clear_profile()
test_eq((lr.profile().known, lr.profile().changes), (False, []))