refactor

Plan safe edits to Python files: find and replace, extract, inline, and moving a definition to another file.

Edits

A plan is a list of FileEdit objects, one per file. Each object holds the path, the whole text before the edit, the whole text after the edit, and the number of changes. To apply a plan, write each after. To undo a plan, write each before back. A before of None means that the plan creates the file. To undo that plan, delete the file.

---

FileEdit

def FileEdit(
    path:str, before:str, after:str, edits:int
)->None:

The whole text of one file before and after an edit, and the number of changes. before is None for a new file.

FileEdit('a.py', 'x = 1\n', 'y = 1\n', 1).dict()

{'path': 'a.py', 'before': 'x = 1\n', 'after': 'y = 1\n', 'edits': 1}

Text replacement

replace_plan(paths, query, replacement) is find and replace across files. The search is literal and case-insensitive by default. regex=True treats query as a regular expression, and replacement can then refer to groups. case=True makes the match case-sensitive. word=True matches whole words only, so koshas stays unchanged in the example below. The function returns a list of FileEdit objects for the changed files, and a list of skipped paths. It raises ValueError when query is empty, when the regular expression is invalid, or when the replacement is invalid.

replace_plan takes a list of paths and a read function, not a directory. read can return the text of files on disk, of a git revision, or of an editor’s unsaved buffers. replace_plan skips notebooks. It also skips a file that read cannot read, and a file whose text holds the Unicode replacement character. A binary file read as text holds that character.

---

replace_plan

def replace_plan(
    paths, query, replacement, regex:bool=False, case:bool=False, word:bool=False, read:function=_read
):

Plan one find-and-replace across paths: the changed files as FileEdits, and the paths skipped.

with TemporaryDirectory() as d:
    root = _tree(d, {'a.py': 'kosha = 1\nprint(koshas)\n', 'b.py': 'x = 2\n', 'nb.ipynb': '{}'})
    rows, skipped = replace_plan(sorted(root.iterdir()), 'kosha', 'kosha2', word=True)
    print([Path(r.path).name for r in rows], [Path(p).name for p in skipped])
    print(rows[0].after)

['a.py'] ['nb.ipynb']
kosha2 = 1
print(koshas)

Symbols and their spans

The planners cut and paste text by character position. Python’s ast module reports a position as a line number and a column. The column counts UTF-8 bytes, not characters. On a line with an accented letter, every column after that letter is too large as a character offset. The private helpers here convert positions to character offsets.

top_symbols lists the top-level functions and classes of a file. Each entry holds the name, the kind, the line and the span of text that the definition owns. A definition owns more than its own lines. It owns its decorators, the comment block directly above it, and the blank lines below it. Removing the span leaves no gap. The example shows sep with its comment and its @cache decorator.

---

top_symbols

def top_symbols(
    source
):

Top-level functions and classes, each with the character span it owns.

SRC = '''import os

# the separator, repeated
@cache
def sep(n): return os.sep * n


class Thing:
    def go(self): return sep(1)
'''
top_symbols(SRC)

[{'name': 'sep', 'kind': 'function', 'line': 5, 'start': 11, 'end': 76},
 {'name': 'Thing', 'kind': 'class', 'line': 8, 'start': 76, 'end': 121}]

sym = top_symbols(SRC)[0]
print(SRC[sym['start']:sym['end']])

# the separator, repeated
@cache
def sep(n): return os.sep * n

Naming the module a file is

module_of turns a file path into the dotted module name that an import uses. It finds the import root of the file with kosha.core.imp_root. The index uses the same rule for mod_name, and the two names agree. The __init__.py of a package gives the package name, pkg, and not pkg.__init__.

---

module_of

def module_of(
    path
):

Dotted module name for a file, from the top of its __init__.py chain.

with TemporaryDirectory() as d:
    root = _tree(d, {'pkg/__init__.py': '', 'pkg/sub/__init__.py': '', 'pkg/sub/deep.py': '', 'loose.py': ''})
    print([module_of(root/p) for p in ('pkg/sub/deep.py', 'pkg/__init__.py', 'loose.py')])

['pkg.sub.deep', 'pkg', 'loose']

Free names

Moved code works only if every name that it reads but does not define is available in the new file. These names are the free names of the code. In the example, the function reads BASE and os from its module. The file that receives the function must provide both. len is not free because it is a builtin. n and total are not free because the function defines them.

_free excludes the builtins by default. A module that defines its own filter or id must not exclude them. Otherwise the check treats a call to the filter of that module as satisfied by the builtin. move_plan handles that case, shown further down.

_body = ast.parse('def f(n):\n    total = BASE\n    return len(total) + n + os.sep\n').body
print(sorted(_free(_body)), '|', sorted(_free(_body, exclude=()) & {'len'}))

['BASE', 'os'] | ['len']

Extract

extract(path, source, start, end, kind, name) takes the text between the character offsets start and end of source and moves it into a new definition called name. kind has four values:

• 'function': the selected statements become a function defined at the position of the selection, followed by a call to it. The selection must be whole statement lines.
• 'method': the same, but the new method goes into the class, after the method that held the selection. The call is self.name(...). The selection must be inside a method.
• 'variable': the selected expression is assigned to name on the line above, and name replaces the expression.
• 'constant': a literal value at module level becomes a constant. extract converts name to upper case. It puts the constant after the docstring and the import lines that follow it.

extract raises ValueError with the reason when it cannot lift the selection. It refuses an empty selection, a selection that is not whole statements (function, method) or not one expression (variable, constant), a selection that returns or yields, an invalid name, an unknown kind, a constant that is not a literal, and a constant outside module scope. A return or yield would change meaning inside a new function.

---

extract

def extract(
    path, source, start, end, kind, name
):

Plan lifting the selection start:end of source into a new function, method, variable or constant called name.

The new function’s parameters are the names the selection reads before it assigns them. Its return values are the names it assigns that the code after the selection still reads. Everything else stays local to the new function. Here rows becomes a parameter and total, which print reads afterwards, comes back as the return value.

F = '''def report(rows):
    total = 0
    for r in rows: total += r
    print(total)
'''
sel = F.index('    total = 0'), F.index('total += r') + len('total += r')
print(extract('r.py', F, *sel, 'function', 'tally').after)

def report(rows):
    def tally(rows):
        total = 0
        for r in rows: total += r
        return total

    total = tally(rows)

    print(total)

An extracted method sits beside the method that held the selection, not inside it. The call self.step() needs step to be a method of the class.

M = '''class Report:
    def show(self):
        total = self.rows + 1
        print(total)
'''
print(extract('m.py', M, M.index('        total'), M.index('+ 1') + 3, 'method', 'step').after)

class Report:
    def show(self):
        total = self.step()

        print(total)

    def step(self):
        total = self.rows + 1
        return total

Inline

inline(path, source, pos) is the reverse of extracting a variable. Put pos on a use of a variable. The variable must have exactly one assignment, written as a plain name = value on lines of its own. inline replaces every use of the variable with the value and deletes the assignment. It wraps a value in parentheses unless the value is a name, constant, attribute, subscript, call, list, dict, set, comprehension or f-string. A tuple value gets parentheses. width = 1 + 2 inlined into width * 3 gives (1 + 2) * 3, not 1 + 2 * 3.

inline raises ValueError when pos is not on a variable, or when the variable is assigned more than once, declared global or nonlocal, never read, or read before the assignment. It also raises ValueError when the assignment shares its lines with other code.

inline refuses a substitution it cannot show to be safe, because it does not resolve scopes. The variable’s name must be bound nowhere else in the file, not even as a parameter. A value that reads other names needs every use in the assignment’s own scope, and each of those names bound once and before the assignment. A value that calls something, reads an attribute or item, builds a list, dict or set, or applies an operator, comparison or f-string to a name can run only once. inline accepts it only for a single use in the very next statement, outside any lambda, comprehension or conditional expression, outside the later operands of a chained comparison, and with no other call evaluated first.

---

inline

def inline(
    path, source, pos
):

Replace a variable’s uses with its value and delete the assignment.

I = 'width = 1 + 2\nprint(width * 3, width)\n'
inline('i.py', I, I.index('width')).after

'print((1 + 2) * 3, (1 + 2))\n'

Imports

When a move adds an import, the import is relative if the importing module and the target share a package. Inside pkg.c, the module pkg.b is imported as from .b import .... Modules with no shared package get an absolute import. Kosha resolves an existing relative import against the module that holds it. from .a import x inside pkg.c and from pkg.a import x inside pkg.d both import the module pkg.a.

print(_stmt('pkg.b', 'pkg.c', ['shared']))          # same package: relative
print(_stmt('other.b', 'pkg.c', ['shared']))        # nothing shared: absolute
print(_absolute(ast.parse('from .a import x').body[0], 'pkg.c'))

from .b import shared
from other.b import shared
pkg.a

A new import goes with the existing imports of the file. If the file already has a from import of the same module, the name is added to that import and the module is not repeated. A new import statement goes below the module docstring, the imports and the __all__ that an nbdev module starts with.

_t = '"doc"\nimport os\nfrom pkg.b import one\n\nprint(one)\n'
_p = _parse('c.py', _t)
print(_apply(_t, _add_imports(_t, _p, _starts(_t), 'pkg.c', {'pkg.b': {'two'}, 'pkg.d': {'three'}}, {'import sys'})))

"doc"
import os
from .b import one, two
from .d import three
import sys

print(one)

Moving a definition

A move must also keep the __all__ of each module correct. __all__ is the list of public names of a module. Kosha rewrites __all__ when it is a plain list or tuple of strings. The rewritten value is a list. Kosha adds the moved names to the __all__ of the destination. It removes them from the __all__ of the source when shim=False. Kosha leaves __all__ unchanged when the module builds it any other way, such as ['x'] + more. A rewrite would need a guess.

Every file that imported a moved name must import it from the destination module. _repoint changes two kinds of import.

• from src import name: Kosha splits the statement. The names that stayed keep the old import, and the moved names get a new import from the destination.
• import src, used as src.name: Kosha replaces each src.name with name and adds a direct import of name. It removes import src, or src from a longer import statement, only when nothing else in the file uses src.

In the second kind, Kosha leaves the src.name uses unchanged when the file already binds name at its top level. Kosha adds a note in that case. A rewrite would point the uses at the wrong definition.

move_plan(src, names, dest, others) combines these steps. It cuts the named top-level functions and classes out of src, with the comments and decorators that they own. It appends them to dest, and creates dest if it does not exist. It copies the imports that the moved code needs into dest. It imports the names that the moved code reads from src itself back from src. By default src keeps a re-export: an import of the moved names from dest. Code that still imports the names from src keeps working. shim=False leaves the re-export out. move_plan also changes the imports of the names in every file of others, as described above.

Before it returns, move_plan checks every file in the plan for names that the edit would leave undefined. If it finds any, it raises ValueError with the file and the names. It returns a tuple of the plan and a list of notes. The notes describe things that move_plan did not do, such as a re-export that it left out.

---

move_plan

def move_plan(
    src, names, dest, others:tuple=(), shim:bool=True, read:function=_read
):

Plan moving top-level definitions names from src into dest and repointing the imports in others.

A move, end to end

The example project has three files in the package pkg. a.py defines shared. b.py is the destination. c.py imports shared by name. show prints each file of the plan as it would be after the move.

PROJECT = {'pkg/__init__.py': '',
           'pkg/a.py': '''
                "paths"
                import os

                def helper(x): return x + 1

                # the separator, repeated
                def shared(n): return os.sep * n
                ''',
           'pkg/b.py': '"where shared is going"\n',
           'pkg/c.py': 'from pkg.a import helper, shared\n\nprint(shared(2), helper(1))\n'}

def show(rows, notes=()):
    for r in rows: print(f'==== {Path(r.path).name} ' + ('(new) ' if r.before is None else '') + '='*20); print(r.after)
    for n in notes: print('note:', n)

with TemporaryDirectory() as d:
    root = _tree(d, PROJECT)
    rows, notes = move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py', others=[root/'pkg/c.py'])
    show(rows, notes)

==== a.py ====================
"paths"
import os
from .b import shared

def helper(x): return x + 1

==== b.py ====================
"where shared is going"
import os


# the separator, repeated
def shared(n): return os.sep * n

==== c.py ====================
from pkg.a import helper
from .b import shared

print(shared(2), helper(1))

a.py keeps a re-export. Code that imported shared from a.py still works. With shim=False, a.py has no re-export. Only the files in others keep working.

with TemporaryDirectory() as d:
    root = _tree(d, PROJECT)
    rows, notes = move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py', others=[root/'pkg/c.py'], shim=False)
    print(rows[0].after)

"paths"
import os

def helper(x): return x + 1

The hard cases

c.py uses import pkg.a and reaches the moved function as pkg.a.shared. That use becomes a direct import of shared from pkg.b. The import pkg.a line stays because c.py still reads pkg.a.helper. e.py imports the module under an alias. f.py imports shared with a relative import. move_plan changes both files as well.

with TemporaryDirectory() as d:
    root = _tree(d, dict(PROJECT, **{'pkg/c.py': 'import pkg.a\n\nprint(pkg.a.shared(2), pkg.a.helper(1))\n',
                                     'pkg/e.py': 'import pkg.a as pa\n\nprint(pa.shared(2))\n',
                                     'pkg/f.py': 'from .a import shared\n\nprint(shared(2))\n'}))
    others = [root/'pkg/c.py', root/'pkg/e.py', root/'pkg/f.py']
    rows, notes = move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py', others=others)
    show(rows[2:], notes)

==== c.py ====================
import pkg.a
from .b import shared

print(shared(2), pkg.a.helper(1))

==== e.py ====================
from .b import shared

print(shared(2))

==== f.py ====================
from .b import shared

print(shared(2))

A definition can have the same name as a builtin, and a check that ignores builtins goes wrong on it. a.py defines filter, and use calls it. When filter moves out of a.py, a.py must import it from the destination. A check that ignored builtins would decide that filter is always available. It would skip the import, and use would call the builtin filter. move_plan adds the import.

SHADOW = {'pkg/__init__.py': '',
          'pkg/a.py': 'def filter(xs): return [x for x in xs if x]\n\ndef use(xs): return len(filter(xs))\n',
          'pkg/b.py': '"where filter is going"\n'}

with TemporaryDirectory() as d:
    root = _tree(d, SHADOW)
    rows, notes = move_plan(root/'pkg/a.py', ['filter'], root/'pkg/b.py', shim=False)
    print(rows[0].after)

from .b import filter

def use(xs): return len(filter(xs))

move_plan refuses a move that would make two modules import each other. Here the moved shared reads BASE from a.py. helper stays in a.py and calls shared. After the move, b.py would import BASE from a.py, and a.py would import shared from b.py. The ValueError tells you to move BASE as well.

CIRCULAR = {'pkg/__init__.py': '',
            'pkg/a.py': 'BASE = 3\n\ndef shared(n): return BASE * n\n\ndef helper(x): return shared(x) + 1\n',
            'pkg/b.py': '"b"\n'}

with TemporaryDirectory() as d:
    root = _tree(d, CIRCULAR)
    test_fail(lambda: move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py'),
              contains='pkg.a and pkg.b would import each other over BASE')
    print('refused')

refused

If nothing left in a.py uses the moved definition, there is no real cycle. b.py still imports BASE from a.py. move_plan leaves the re-export out of a.py and adds a note.

with TemporaryDirectory() as d:
    root = _tree(d, dict(CIRCULAR, **{'pkg/a.py': 'BASE = 3\n\ndef shared(n): return BASE * n\n\ndef helper(x): return x + 1\n'}))
    rows, notes = move_plan(root/'pkg/a.py', ['shared'], root/'pkg/b.py')
    show(rows, notes)

==== a.py ====================
BASE = 3

def helper(x): return x + 1

==== b.py ====================
"b"
from .a import BASE


def shared(n): return BASE * n

note: re-export left out: pkg.b imports BASE back from pkg.a

Several definitions can move at once. They keep their order from the source file. A class moves with all its methods. With shim=False, move_plan updates __all__ in both files.

MANY = {'pkg/__init__.py': '',
        'pkg/a.py': "__all__ = ['helper', 'shared', 'Thing']\nimport os\n\n"
                    "def helper(x): return x + 1\n\ndef shared(n): return os.sep * n\n\n"
                    "class Thing:\n    def go(self): return shared(1)\n",
        'pkg/b.py': "__all__ = ['already']\n\ndef already(): pass\n"}

with TemporaryDirectory() as d:
    root = _tree(d, MANY)
    rows, notes = move_plan(root/'pkg/a.py', ['shared', 'Thing'], root/'pkg/b.py', shim=False)
    show(rows, notes)

==== a.py ====================
__all__ = ['helper']
import os

def helper(x): return x + 1

==== b.py ====================
__all__ = ['already', 'shared', 'Thing']
import os

def already(): pass


def shared(n): return os.sep * n

class Thing:
    def go(self): return shared(1)

Each refusal raises ValueError with a message that states the problem. Most messages also name the file. The tests below check the messages.