# refactor ## Edits A plan is a list of `FileEdit` objects, one per file. Each object holds the path, the whole text before the edit, the whole text after the edit, and the number of changes. To apply a plan, write each `after`. To undo a plan, write each `before` back. A `before` of `None` means that the plan creates the file. To undo that plan, delete the file. ------------------------------------------------------------------------ ### FileEdit ``` python def FileEdit( path:str, before:str, after:str, edits:int )->None: ``` *The whole text of one file before and after an edit, and the number of changes. `before` is `None` for a new file.* ``` python FileEdit('a.py', 'x = 1\n', 'y = 1\n', 1).dict() ``` {'path': 'a.py', 'before': 'x = 1\n', 'after': 'y = 1\n', 'edits': 1} ## Text replacement `replace_plan(paths, query, replacement)` is find and replace across files. The search is literal and case-insensitive by default. `regex=True` treats `query` as a regular expression, and `replacement` can then refer to groups. `case=True` makes the match case-sensitive. `word=True` matches whole words only, so `koshas` stays unchanged in the example below. The function returns a list of `FileEdit` objects for the changed files, and a list of skipped paths. It raises `ValueError` when `query` is empty, when the regular expression is invalid, or when the replacement is invalid. `replace_plan` takes a list of paths and a `read` function, not a directory. `read` can return the text of files on disk, of a git revision, or of an editor’s unsaved buffers. `replace_plan` skips notebooks. It also skips a file that `read` cannot read, and a file whose text holds the Unicode replacement character. A binary file read as text holds that character. ------------------------------------------------------------------------ ### replace_plan ``` python def replace_plan( paths, query, replacement, regex:bool=False, case:bool=False, word:bool=False, read:function=_read ): ``` *Plan one find-and-replace across `paths`: the changed files as `FileEdit`s, and the paths skipped.* ``` python with TemporaryDirectory() as d: root = _tree(d, {'a.py': 'kosha = 1\nprint(koshas)\n', 'b.py': 'x = 2\n', 'nb.ipynb': '{}'}) rows, skipped = replace_plan(sorted(root.iterdir()), 'kosha', 'kosha2', word=True) print([Path(r.path).name for r in rows], [Path(p).name for p in skipped]) print(rows[0].after) ``` ['a.py'] ['nb.ipynb'] kosha2 = 1 print(koshas) ## Symbols and their spans The planners cut and paste text by character position. Python’s `ast` module reports a position as a line number and a column. The column counts UTF-8 bytes, not characters. On a line with an accented letter, every column after that letter is too large as a character offset. The private helpers here convert positions to character offsets. `top_symbols` lists the top-level functions and classes of a file. Each entry holds the name, the kind, the line and the span of text that the definition owns. A definition owns more than its own lines. It owns its decorators, the comment block directly above it, and the blank lines below it. Removing the span leaves no gap. The example shows `sep` with its comment and its `@cache` decorator. ------------------------------------------------------------------------ ### top_symbols ``` python def top_symbols( source ): ``` *Top-level functions and classes, each with the character span it owns.* ``` python SRC = '''import os # the separator, repeated @cache def sep(n): return os.sep * n class Thing: def go(self): return sep(1) ''' top_symbols(SRC) ``` [{'name': 'sep', 'kind': 'function', 'line': 5, 'start': 11, 'end': 76}, {'name': 'Thing', 'kind': 'class', 'line': 8, 'start': 76, 'end': 121}] ``` python sym = top_symbols(SRC)[0] print(SRC[sym['start']:sym['end']]) ``` # the separator, repeated @cache def sep(n): return os.sep * n ## Naming the module a file is `module_of` turns a file path into the dotted module name that an import uses. It finds the import root of the file with [`kosha.core.imp_root`](https://vedicreader.github.io/kosha/core.html#imp_root). The index uses the same rule for `mod_name`, and the two names agree. The `__init__.py` of a package gives the package name, `pkg`, and not `pkg.__init__`. ------------------------------------------------------------------------ ### module_of ``` python def module_of( path ): ``` *Dotted module name for a file, from the top of its `__init__.py` chain.* ``` python with TemporaryDirectory() as d: root = _tree(d, {'pkg/__init__.py': '', 'pkg/sub/__init__.py': '', 'pkg/sub/deep.py': '', 'loose.py': ''}) print([module_of(root/p) for p in ('pkg/sub/deep.py', 'pkg/__init__.py', 'loose.py')]) ``` ['pkg.sub.deep', 'pkg', 'loose'] ## Free names Moved code works only if every name that it reads but does not define is available in the new file. These names are the free names of the code. In the example, the function reads `BASE` and `os` from its module. The file that receives the function must provide both. `len` is not free because it is a builtin. `n` and `total` are not free because the function defines them. `_free` excludes the builtins by default. A module that defines its own `filter` or `id` must not exclude them. Otherwise the check treats a call to the `filter` of that module as satisfied by the builtin. `move_plan` handles that case, shown further down. ``` python _body = ast.parse('def f(n):\n total = BASE\n return len(total) + n + os.sep\n').body print(sorted(_free(_body)), '|', sorted(_free(_body, exclude=()) & {'len'})) ``` ['BASE', 'os'] | ['len'] ## Extract `extract(path, source, start, end, kind, name)` takes the text between the character offsets `start` and `end` of `source` and moves it into a new definition called `name`. `kind` has four values: - `'function'`: the selected statements become a function defined at the position of the selection, followed by a call to it. The selection must be whole statement lines. - `'method'`: the same, but the new method goes into the class, after the method that held the selection. The call is `self.name(...)`. The selection must be inside a method. - `'variable'`: the selected expression is assigned to `name` on the line above, and `name` replaces the expression. - `'constant'`: a literal value at module level becomes a constant. `extract` converts `name` to upper case. It puts the constant after the docstring and the `import` lines that follow it. `extract` raises `ValueError` with the reason when it cannot lift the selection. It refuses an empty selection, a selection that is not whole statements (function, method) or not one expression (variable, constant), a selection that returns or yields, an invalid `name`, an unknown `kind`, a constant that is not a literal, and a constant outside module scope. A `return` or `yield` would change meaning inside a new function. ------------------------------------------------------------------------ ### extract ``` python def extract( path, source, start, end, kind, name ): ``` *Plan lifting the selection `start:end` of `source` into a new function, method, variable or constant called `name`.* The new function’s parameters are the names the selection reads before it assigns them. Its return values are the names it assigns that the code after the selection still reads. Everything else stays local to the new function. Here `rows` becomes a parameter and `total`, which `print` reads afterwards, comes back as the return value. ``` python F = '''def report(rows): total = 0 for r in rows: total += r print(total) ''' sel = F.index(' total = 0'), F.index('total += r') + len('total += r') print(extract('r.py', F, *sel, 'function', 'tally').after) ``` def report(rows): def tally(rows): total = 0 for r in rows: total += r return total total = tally(rows) print(total) An extracted method sits beside the method that held the selection, not inside it. The call `self.step()` needs `step` to be a method of the class. ``` python M = '''class Report: def show(self): total = self.rows + 1 print(total) ''' print(extract('m.py', M, M.index(' total'), M.index('+ 1') + 3, 'method', 'step').after) ``` class Report: def show(self): total = self.step() print(total) def step(self): total = self.rows + 1 return total ## Inline `inline(path, source, pos)` is the reverse of extracting a variable. Put `pos` on a use of a variable. The variable must have exactly one assignment, written as a plain `name = value` on lines of its own. `inline` replaces every use of the variable with the value and deletes the assignment. It wraps a value in parentheses unless the value is a name, constant, attribute, subscript, call, list, dict, set, comprehension or f-string. A tuple value gets parentheses. `width = 1 + 2` inlined into `width * 3` gives `(1 + 2) * 3`, not `1 + 2 * 3`. `inline` raises `ValueError` when `pos` is not on a variable, or when the variable is assigned more than once, declared `global` or `nonlocal`, never read, or read before the assignment. It also raises `ValueError` when the assignment shares its lines with other code. `inline` refuses a substitution it cannot show to be safe, because it does not resolve scopes. The variable’s name must be bound nowhere else in the file, not even as a parameter. A value that reads other names needs every use in the assignment’s own scope, and each of those names bound once and before the assignment. A value that calls something, reads an attribute or item, builds a list, dict or set, or applies an operator, comparison or f-string to a name can run only once. `inline` accepts it only for a single use in the very next statement, outside any lambda, comprehension or conditional expression, outside the later operands of a chained comparison, and with no other call evaluated first. ------------------------------------------------------------------------ ### inline ``` python def inline( path, source, pos ): ``` *Replace a variable’s uses with its value and delete the assignment.* ``` python I = 'width = 1 + 2\nprint(width * 3, width)\n' inline('i.py', I, I.index('width')).after ``` 'print((1 + 2) * 3, (1 + 2))\n' ## Imports When a move adds an import, the import is relative if the importing module and the target share a package. Inside `pkg.c`, the module `pkg.b` is imported as `from .b import ...`. Modules with no shared package get an absolute import. Kosha resolves an existing relative import against the module that holds it. `from .a import x` inside `pkg.c` and `from pkg.a import x` inside `pkg.d` both import the module `pkg.a`. ``` python print(_stmt('pkg.b', 'pkg.c', ['shared'])) # same package: relative print(_stmt('other.b', 'pkg.c', ['shared'])) # nothing shared: absolute print(_absolute(ast.parse('from .a import x').body[0], 'pkg.c')) ``` from .b import shared from other.b import shared pkg.a A new import goes with the existing imports of the file. If the file already has a `from` import of the same module, the name is added to that import and the module is not repeated. A new import statement goes below the module docstring, the imports and the `__all__` that an nbdev module starts with. ``` python _t = '"doc"\nimport os\nfrom pkg.b import one\n\nprint(one)\n' _p = _parse('c.py', _t) print(_apply(_t, _add_imports(_t, _p, _starts(_t), 'pkg.c', {'pkg.b': {'two'}, 'pkg.d': {'three'}}, {'import sys'}))) ``` "doc" import os from .b import one, two from .d import three import sys print(one) ## Moving a definition A move must also keep the `__all__` of each module correct. `__all__` is the list of public names of a module. Kosha rewrites `__all__` when it is a plain list or tuple of strings. The rewritten value is a list. Kosha adds the moved names to the `__all__` of the destination. It removes them from the `__all__` of the source when `shim=False`. Kosha leaves `__all__` unchanged when the module builds it any other way, such as `['x'] + more`. A rewrite would need a guess. Every file that imported a moved name must import it from the destination module. `_repoint` changes two kinds of import. - `from src import name`: Kosha splits the statement. The names that stayed keep the old import, and the moved names get a new import from the destination. - `import src`, used as `src.name`: Kosha replaces each `src.name` with `name` and adds a direct import of `name`. It removes `import src`, or `src` from a longer `import` statement, only when nothing else in the file uses `src`. In the second kind, Kosha leaves the `src.name` uses unchanged when the file already binds `name` at its top level. Kosha adds a note in that case. A rewrite would point the uses at the wrong definition. `move_plan(src, names, dest, others)` combines these steps. It cuts the named top-level functions and classes out of `src`, with the comments and decorators that they own. It appends them to `dest`, and creates `dest` if it does not exist. It copies the imports that the moved code needs into `dest`. It imports the names that the moved code reads from `src` itself back from `src`. By default `src` keeps a re-export: an import of the moved names from `dest`. Code that still imports the names from `src` keeps working. `shim=False` leaves the re-export out. `move_plan` also changes the imports of the names in every file of `others`, as described above. Before it returns, `move_plan` checks every file in the plan for names that the edit would leave undefined. If it finds any, it raises `ValueError` with the file and the names. It returns a tuple of the plan and a list of notes. The notes describe things that `move_plan` did not do, such as a re-export that it left out. ------------------------------------------------------------------------ ### move_plan ``` python def move_plan( src, names, dest, others:tuple=(), shim:bool=True, read:function=_read ): ``` *Plan moving top-level definitions `names` from `src` into `dest` and repointing the imports in `others`.* ### A move, end to end The example project has three files in the package `pkg`. `a.py` defines `shared`. `b.py` is the destination. `c.py` imports `shared` by name. `show` prints each file of the plan as it would be after the move. ``` python PROJECT = {'pkg/__init__.py': '', 'pkg/a.py': ''' "paths" import os def helper(x): return x + 1 # the separator, repeated def shared(n): return os.sep * n ''', 'pkg/b.py': '"where shared is going"\n', 'pkg/c.py': 'from pkg.a import helper, shared\n\nprint(shared(2), helper(1))\n'} def show(rows, notes=()): for r in rows: print(f'==== {Path(r.path).name} ' + ('(new) ' if r.before is None else '') + '='*20); print(r.after) for n in notes: print('note:', n) ``` ``` python with TemporaryDirectory() as d: root = _tree(d, PROJECT) rows, notes = move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py', others=[root/'pkg/c.py']) show(rows, notes) ``` ==== a.py ==================== "paths" import os from .b import shared def helper(x): return x + 1 ==== b.py ==================== "where shared is going" import os # the separator, repeated def shared(n): return os.sep * n ==== c.py ==================== from pkg.a import helper from .b import shared print(shared(2), helper(1)) `a.py` keeps a re-export. Code that imported `shared` from `a.py` still works. With `shim=False`, `a.py` has no re-export. Only the files in `others` keep working. ``` python with TemporaryDirectory() as d: root = _tree(d, PROJECT) rows, notes = move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py', others=[root/'pkg/c.py'], shim=False) print(rows[0].after) ``` "paths" import os def helper(x): return x + 1 ### The hard cases `c.py` uses `import pkg.a` and reaches the moved function as `pkg.a.shared`. That use becomes a direct import of `shared` from `pkg.b`. The `import pkg.a` line stays because `c.py` still reads `pkg.a.helper`. `e.py` imports the module under an alias. `f.py` imports `shared` with a relative import. `move_plan` changes both files as well. ``` python with TemporaryDirectory() as d: root = _tree(d, dict(PROJECT, **{'pkg/c.py': 'import pkg.a\n\nprint(pkg.a.shared(2), pkg.a.helper(1))\n', 'pkg/e.py': 'import pkg.a as pa\n\nprint(pa.shared(2))\n', 'pkg/f.py': 'from .a import shared\n\nprint(shared(2))\n'})) others = [root/'pkg/c.py', root/'pkg/e.py', root/'pkg/f.py'] rows, notes = move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py', others=others) show(rows[2:], notes) ``` ==== c.py ==================== import pkg.a from .b import shared print(shared(2), pkg.a.helper(1)) ==== e.py ==================== from .b import shared print(shared(2)) ==== f.py ==================== from .b import shared print(shared(2)) A definition can have the same name as a builtin, and a check that ignores builtins goes wrong on it. `a.py` defines `filter`, and `use` calls it. When `filter` moves out of `a.py`, `a.py` must import it from the destination. A check that ignored builtins would decide that `filter` is always available. It would skip the import, and `use` would call the builtin `filter`. `move_plan` adds the import. ``` python SHADOW = {'pkg/__init__.py': '', 'pkg/a.py': 'def filter(xs): return [x for x in xs if x]\n\ndef use(xs): return len(filter(xs))\n', 'pkg/b.py': '"where filter is going"\n'} with TemporaryDirectory() as d: root = _tree(d, SHADOW) rows, notes = move_plan(root/'pkg/a.py', ['filter'], root/'pkg/b.py', shim=False) print(rows[0].after) ``` from .b import filter def use(xs): return len(filter(xs)) `move_plan` refuses a move that would make two modules import each other. Here the moved `shared` reads `BASE` from `a.py`. `helper` stays in `a.py` and calls `shared`. After the move, `b.py` would import `BASE` from `a.py`, and `a.py` would import `shared` from `b.py`. The `ValueError` tells you to move `BASE` as well. ``` python CIRCULAR = {'pkg/__init__.py': '', 'pkg/a.py': 'BASE = 3\n\ndef shared(n): return BASE * n\n\ndef helper(x): return shared(x) + 1\n', 'pkg/b.py': '"b"\n'} with TemporaryDirectory() as d: root = _tree(d, CIRCULAR) test_fail(lambda: move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py'), contains='pkg.a and pkg.b would import each other over BASE') print('refused') ``` refused If nothing left in `a.py` uses the moved definition, there is no real cycle. `b.py` still imports `BASE` from `a.py`. `move_plan` leaves the re-export out of `a.py` and adds a note. ``` python with TemporaryDirectory() as d: root = _tree(d, dict(CIRCULAR, **{'pkg/a.py': 'BASE = 3\n\ndef shared(n): return BASE * n\n\ndef helper(x): return x + 1\n'})) rows, notes = move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py') show(rows, notes) ``` ==== a.py ==================== BASE = 3 def helper(x): return x + 1 ==== b.py ==================== "b" from .a import BASE def shared(n): return BASE * n note: re-export left out: pkg.b imports BASE back from pkg.a Several definitions can move at once. They keep their order from the source file. A class moves with all its methods. With `shim=False`, `move_plan` updates `__all__` in both files. ``` python MANY = {'pkg/__init__.py': '', 'pkg/a.py': "__all__ = ['helper', 'shared', 'Thing']\nimport os\n\n" "def helper(x): return x + 1\n\ndef shared(n): return os.sep * n\n\n" "class Thing:\n def go(self): return shared(1)\n", 'pkg/b.py': "__all__ = ['already']\n\ndef already(): pass\n"} with TemporaryDirectory() as d: root = _tree(d, MANY) rows, notes = move_plan(root/'pkg/a.py', ['shared', 'Thing'], root/'pkg/b.py', shim=False) show(rows, notes) ``` ==== a.py ==================== __all__ = ['helper'] import os def helper(x): return x + 1 ==== b.py ==================== __all__ = ['already', 'shared', 'Thing'] import os def already(): pass def shared(n): return os.sep * n class Thing: def go(self): return shared(1) Each refusal raises `ValueError` with a message that states the problem. Most messages also name the file. The tests below check the messages.