nbio

Reading, writing, and running Jupyter notebooks

Reading a notebook

A notebook is just a json file.

Exported source

def _read_json(self, encoding=None, errors=None):
    return loads(Path(self).read_text(encoding=encoding, errors=errors))

minimal_fn = Path('../tests/minimal.ipynb')
minimal_txt = AttrDict(_read_json(minimal_fn))

It contains two sections, the metadata…:

minimal_txt.metadata

{'solveit_dialog_mode': 'learning', 'solveit_ver': 2}

…and, more importantly, the cells:

minimal_txt.cells

[{'cell_type': 'markdown',
  'id': '801558df',
  'metadata': {},
  'source': ['## A minimal notebook']},
 {'cell_type': 'code',
  'execution_count': None,
  'id': 'e2147a69',
  'metadata': {'time_run': '2026-01-04T20:52:49.901559+00:00'},
  'outputs': [{'data': {'text/plain': ['2']},
    'execution_count': 0,
    'metadata': {},
    'output_type': 'execute_result'}],
  'source': ['# Do some arithmetic\n', '1+1']}]

The second cell here is a code cell, however it contains no outputs, because it hasn’t been executed yet. To execute a notebook, we first need to convert it into a format suitable for nbclient (which expects some dict keys to be available as attrs, and some available as regular dict keys). Normally, nbformat is used for this step, but it’s rather slow and inflexible, so we’ll write our own function based on fastcore’s handy dict2obj, which makes all keys available as both attrs and keys.

nb_lang

def nb_lang(
    nb
):

Call self as a function.

Each notebook language has its own comment character(s), taken from Quarto’s table; nb_lang reads the notebook’s language from its kernelspec, defaulting to Python. Cells record their language in lang_ (a per-cell metadata.language overrides the notebook default), which the directive functions below use to recognize comments.

NbCell

def NbCell(
    idx, cell, lang:str='python'
):

dict subclass that also provides access to keys as attrs, and has a pretty markdown repr

We use an AttrDict subclass which has some basic functionality for accessing notebook cells.

Two cells are equal if they have the same id, even when their content differs – id is what identifies which cell this is, so this is what lets list.index/list.remove (used by Notebook’s cell-moving methods) find the right cell even when another cell elsewhere has identical source. Cells lacking an id fall back to comparing source/cell_type.

c1 = NbCell(0, dict(cell_type='code', source='a', id='x'))
c2 = NbCell(0, dict(cell_type='code', source='b', id='x'))
c3 = NbCell(0, dict(cell_type='code', source='a', id='y'))
test_eq(c1, c2)      # same id, different content -- still equal
assert c1 != c3      # same content, different id -- not equal

cells = [c1, c3]  # duplicate content ('a'), different ids
test_eq(cells.index(c3), 1)  # finds the actual cell, not the first with matching content

dict2nb

def dict2nb(
    js:NoneType=None, **kwargs
):

Convert dict js to an AttrDict,

We can now convert our JSON into this nbclient-compatible format, which pretty prints the source code of cells in notebooks.

minimal = dict2nb(minimal_txt)
cell = minimal.cells[1]
cell

{ 'cell_type': 'code',
  'execution_count': None,
  'id': 'e2147a69',
  'idx_': 1,
  'lang_': 'python',
  'metadata': {'time_run': '2026-01-04T20:52:49.901559+00:00'},
  'outputs': [ { 'data': {'text/plain': '2'},
                 'execution_count': 0,
                 'metadata': {},
                 'output_type': 'execute_result'}],
  'source': '# Do some arithmetic\n1+1'}

The abstract syntax tree of source code cells is available in the parsed_ property:

cell.parsed_(), cell.parsed_()[0].value.op

([<ast.Expr>], <ast.Add>)

read_nb

def read_nb(
    path
):

Return notebook at path (expands ~)

This reads the JSON for the file at path and converts it with dict2nb. For instance:

minimal = read_nb(minimal_fn)
str(minimal.cells[0])

"{'cell_type': 'markdown', 'id': '801558df', 'metadata': {}, 'source': '## A minimal notebook', 'idx_': 0, 'lang_': 'python'}"

The file name read is stored in path_:

minimal.path_

'../tests/minimal.ipynb'

Creating a notebook

mk_cell

def mk_cell(
    text, # `source` attr in cell
    cell_type:str='code', # `cell_type` attr in cell
    **kwargs
):

Create an NbCell containing text

mk_cell('print(1)', execution_count=0)

{ 'cell_type': 'code',
  'directives_': {},
  'execution_count': 0,
  'id': '3b0a226b',
  'idx_': 0,
  'lang_': 'python',
  'metadata': {},
  'outputs': [],
  'source': 'print(1)'}

new_nb

def new_nb(
    cells:NoneType=None, meta:NoneType=None, nbformat:int=4, nbformat_minor:int=5
):

Returns an empty new notebook

nb_frontmatter merges a notebook’s frontmatter from three sources, lowest precedence first: the notebook’s own metadata.nbdev mapping, its first markdown cell, and its first raw cell anywhere. A cell that is entirely a literal --- block parses as YAML (so ----prefixed prose never half-parses), and a markdown cell that isn’t one contributes # title, > description, and - key: value lines instead. Content beating metadata mirrors how directives reads a cell (comments win over the meta form), so both scopes tell one precedence story. Metadata values pass through verbatim: the 'true'→'' bare-directive normalization belongs to cell directives, where bareness must be canonical, and has no meaning in a mapping where every key carries a value. It takes anything whose cells items carry cell_type and source, which includes aidialog/solveit dialogs (their messages duck-type as cells; ones without a metadata attr simply contribute nothing from that source); strvals=True keeps every YAML scalar a string apart from frontmatter’s bool exception, and malformed YAML in a literal block raises. cell_frontmatter and md_frontmatter are the per-cell pieces, shared with nbdev’s FrontmatterProc.

nb_frontmatter

def nb_frontmatter(
    nb, strvals:bool=False
):

Frontmatter from nb, merged lowest-to-highest from its metadata.nbdev mapping (values verbatim), first markdown cell (literal --- block, or # title synthesis), and first raw cell anywhere

md_frontmatter

def md_frontmatter(
    s:str, strvals:bool=False
):

Frontmatter synthesized from an H1-formatted markdown cell: # title, > description, and - key: value lines

cell_frontmatter

def cell_frontmatter(
    s:str, strvals:bool=False
):

Frontmatter mapping from a cell source that is entirely a literal --- block, else {}

fm_nb = new_nb([mk_cell('---\ntitle: T\nformdata:\n  who: Sam\n  n: 1000\n---', 'raw'), mk_cell('Body', 'markdown')])
test_eq(nb_frontmatter(fm_nb), {'title':'T', 'formdata':{'who':'Sam', 'n':1000}})
test_eq(nb_frontmatter(fm_nb, strvals=True)['formdata'], {'who':'Sam', 'n':'1000'})
test_eq(nb_frontmatter(new_nb([])), {})
h1 = mk_cell('# My title\n\n> A subtitle\n\n- author: Zoë\n\nProse.', 'markdown')
test_eq(nb_frontmatter(new_nb([h1])), {'title':'My title', 'description':'A subtitle', 'author':'Zoë'})
test_eq(md_frontmatter('# T\n- n: 1000\n- f: true', strvals=True), dict(title='T', n='1000', f=True))  # strvals honored via the `- key: value` path too
both = new_nb([h1, mk_cell('code'), mk_cell('---\ntitle: Raw wins\n---', 'raw')])
test_eq(nb_frontmatter(both)['title'], 'Raw wins')                                   # first raw anywhere; raw beats markdown
test_eq(nb_frontmatter(both)['description'], 'A subtitle')                           # markdown keys merge beneath
meta_nb = new_nb([mk_cell('---\neval: false\n---', 'raw')], meta={'nbdev': {'eval':'true', 'export':'true'}})
test_eq(nb_frontmatter(meta_nb), {'eval':False, 'export':'true'})                    # metadata.nbdev merges lowest, values verbatim (no bare-'' form)
test_eq(nb_frontmatter(new_nb([mk_cell('---\nt: v\n---\ntrailing', 'raw')])), {})    # whole-cell anchoring: a body disqualifies
test_eq(nb_frontmatter(new_nb([mk_cell('---\nt: v\n---', 'markdown')])), {'t':'v'})   # a markdown cell that IS a literal block parses
test_fail(lambda: nb_frontmatter(new_nb([mk_cell('---\nbad: "unclosed\n---', 'raw')])))

Use this function when creating a new notebook. Useful for when you don’t want to create a notebook on disk first and then read it.

test_eq(new_nb().cells, [])

Directives

nbdev and Quarto put directives in comments at the top of a cell: lines like #| export or #| eval: false, ending at the first non-comment line. All spellings are equivalent: #| foo: bar, #| foo:bar, and #| foo bar parse to the same thing, and a directive’s value is the raw text after its name, so nothing is lost to tokenization. A value of true means the same as no value at all: #| hide, #| hide: true, and #| hide: are one directive (which also means no directive can take the literal word true as its value). Directives can also be stored in cell metadata, as a dict under an nbdev key, with the same value domain: every value is a str, \"true\" meaning bare (non-str values raise, so a JSON false fails loudly rather than silently differing from \"false\"). When the same name appears in both places, the comment wins.

first_code_ln

def first_code_ln(
    code_list, re_pattern:NoneType=None, lang:str='python'
):

get first line number where code occurs, where code_list is a list of code

_tst = """ 
#| default_exp
 #| export
#| hide_input
foo
"""
test_eq(first_code_ln(_tst.splitlines(True)), 4)

_directive parses one directive line into its name and value. The value is the raw text after the name (and optional colon), byte-exact apart from one leading space and trailing whitespace, so both spellings parse identically; a value of true collapses to '', the same as a bare directive. Non-directive lines (including cell magics) parse to None.

test_eq(_directive('#| export: utils'), ('export','utils'))
test_eq(_directive('#| export utils'),  ('export','utils'))
test_eq(_directive('#| export:utils'),  ('export','utils'))
test_eq(_directive('#| hide'), ('hide',''))
test_eq(_directive('#| hide:'), ('hide',''))
test_eq(_directive('#| hide: true'), ('hide',''))
test_eq(_directive(' # | woo:baz'), ('woo','baz'))
test_eq(_directive('#| fig-cap: "two  spaces: kept"'), ('fig-cap','"two  spaces: kept"'))
test_eq(_directive('#| filter_stream secret apikey'), ('filter_stream','secret apikey'))
test_eq(_directive('%%timeit'), None)
test_eq(_directive('# plain comment'), None)

dir_tag renders meta-form directives as the compact bracket tag that summary rows splice after the type character (c[export]:...). CellRow uses it below, and aidialog’s message previews share the same form.

dir_tag

def dir_tag(
    meta
):

Meta-form nbdev directives in meta as a compact [k k=v] bracket tag, or '' if none

test_eq(dir_tag({}), '')
test_eq(dir_tag({'nbdev': {'export': 'utils', 'hide': 'true'}}), '[export=utils hide]')
dir_tag({'nbdev': {'export': 'true'}})

'[export]'

directives reads as a plain dict of name to value string: {'export': 'utils', 'hide': ''} (bare directives have value ''). The getter returns a copy, so to edit, modify the copy and assign it back; the setter regenerates the comment block in canonical colon form (cell magics stay first, untouched). Directives that came from cell metadata are tracked by name and written back to metadata rather than to comments; names added through the setter become comments.

c = mk_cell('#| export: utils\n#| hide\ndef f(): pass')
test_eq(c.directives, {'export':'utils', 'hide':''})

d = c.directives
d.pop('hide')
d['eval'] = 'false'
c.directives = d
test_eq(c.source, '#| export: utils\n#| eval: false\ndef f(): pass')

Metadata directives merge in (comments win on conflict), and edits route back to where each directive came from. The setter writes bare directives to metadata as "true":

c = mk_cell('#| export: utils\n1', metadata=dict(nbdev=dict(export='other', eval='false')))
test_eq(c.directives, {'export':'utils', 'eval':'false'})

d = c.directives
d['eval'] = ''
d['hide'] = ''
c.directives = d
test_eq(c.metadata['nbdev'], {'eval': 'true'})
test_eq(c.source, '#| export: utils\n#| hide\n1')
test_eq(c.directives, {'export':'utils', 'hide':'', 'eval':''})

with expect_fail(TypeError, 'must be str'): mk_cell('1', metadata=dict(nbdev=dict(eval=False))).directives

NbCell.remove_directives

def remove_directives(
    quarto:bool=False
):

Strip directives from source, keeping cell magics; with quarto, instead materialize every directive (metadata included) as a Quarto option line

NbCell.has_directive

def has_directive(
    name
):

Call self as a function.

NbCell.directive

def directive(
    name, default:NoneType=None
):

Value of directive name ('' if bare), or default if absent

directive answers “what value?” and has_directive answers “is it there?”: a bare directive’s value is '', which is falsy, so presence tests need has_directive. remove_directives serves the two pipeline consumers: the default strips every directive line (module export needs clean source), while quarto=True instead rewrites the cell with every directive (comments and metadata alike) as a Quarto option line, bare ones as name: true; Quarto consumes the options it knows and quietly turns unknown ones into data-* attributes.

c = mk_cell('%%time\n#| exports: utils\n#| hide\n#| code-fold: show\nslow()', metadata=dict(nbdev=dict(echo='false')))
test_eq(c.directive('exports'), 'utils')
assert c.has_directive('hide') and not c.has_directive('eval')

c.remove_directives(quarto=True)
test_eq(c.source, '%%time\n#| exports: utils\n#| hide: true\n#| code-fold: show\n#| echo: false\nslow()')

c = mk_cell('%%time\n#| exports: utils\n#| hide\n#| code-fold: show\nslow()', metadata=dict(nbdev=dict(echo='false')))
c.remove_directives()
test_eq(c.source, '%%time\nslow()')

Writing a notebook

nb2dict

def nb2dict(
    d, k:NoneType=None
):

Convert parsed notebook to dict

This returns the exact same dict as is read from the notebook JSON.

minimal_fn = Path('../tests/minimal.ipynb')
minimal = read_nb(minimal_fn)
minimal_dict = _read_json(minimal_fn)
assert minimal_dict==nb2dict(minimal)

nb2str

def nb2str(
    nb
):

Convert nb to a str

To save a notebook we first need to convert it to a str:

print(nb2str(minimal)[:45])

{
 "cells": [
  {
   "cell_type": "markdown",

write_nb

def write_nb(
    nb, path
):

Write nb to path (expands ~)

This returns the exact same string as saved by Jupyter.

tmp = Path('tmp.ipynb')
try:
    minimal_txt = minimal_fn.read_text()
    write_nb(minimal, tmp)
    test_eq(minimal_txt, tmp.read_text())
finally: tmp.unlink()

Cell tools

Cell tools apply fastcore.tools’ string editing primitives to one notebook cell’s source, addressed by path and cell id, mirroring that module’s file tools: the same operations and parameters, with path, cell_id in place of path. Each editor (including the structural cell_ast_replace) returns a diff of the change, and view_cell shows a cell’s source with optional line numbers or exhash addresses.

Naming and parameter conventions shared across the editing toolkit are documented in fastcore.editskill, which also re-exports this module’s editing tools.

Everything in this family addresses its unit by id, with one contract: exact match, or any unique prefix, and a loud KeyError naming the problem otherwise. find_id is that contract as a function, shared by the notebook tools here and the dialog tools built on them:

find_id

def find_id(
    items, # Items carrying `.id` attrs
    k:str, # Id to find: exact, or unique prefix
    unit:str='item', # Unit name for error messages
):

The item with id k; a missing or ambiguous id raises KeyError naming the problem

_its = [AttrDict(id='aa12'), AttrDict(id='ab34')]
test_eq(find_id(_its, 'ab'), _its[1])
with expect_fail(KeyError, 'ambiguous item id'): find_id(_its, 'a')
with expect_fail(KeyError, 'no thing id'): find_id(_its, 'zz', 'thing')

cell_edit

def cell_edit(
    f, name:NoneType=None
):

Wrap text editor f as a cell editing function: path, cell_id addressing, diff-or-error return

The spliced editors inherit fastcore.tools’ signatures, so the toolkit-wide replace_params group collapses the same six options in doc() overviews here:

view_cell

def view_cell(
    path:str, # Notebook file to read (expands `~`)
    cell_id:str, # Id of the cell to view (exact, or unique prefix)
    start_line:int=1, # Starting line to view
    end_line:int=None, # End line (defaults to last line if None; may be past EOF, which clamps to the last line)
    nums:bool=True, # Show line numbers?
    lnhashs:bool=False, # Show exhash `lineno|hash|` addresses instead of line numbers?
    incl_out:bool=False, # Append the cell's outputs in an `<out>` block?
    trunc_out:bool=True, # Truncate included outputs to ~512 chars?
):

View a cell’s source, optionally limited to 1-based line range

A scratch notebook gives the editors something to work on. view_cell reads one cell’s source by id: plain line numbers by default, a line range, or lnhash addresses ready for cell_exhash:

tmp_nb = Path('tmp_cells.ipynb')
write_nb(new_nb([mk_cell('a=1\nprint(a)'), mk_cell('# title', 'markdown')]), tmp_nb)
cid = read_nb(tmp_nb).cells[0].id
test_eq(str(view_cell(tmp_nb, cid)), '1: a=1\n2: print(a)')
test_eq(str(view_cell(tmp_nb, cid, start_line=2, nums=False)), 'print(a)')
test_eq(str(view_cell(tmp_nb, cid, lnhashs=True)).splitlines()[0], lnhash(1,'a=1')+'a=1')

Each editor is a transaction: it applies the edit, writes the file, and returns a diff – the diff is the verification, no second read needed:

res = cell_str_replace(tmp_nb, cid, 'a=1', 'a=2')
assert '-a=1' in str(res) and '+a=2' in str(res)
test_eq(read_nb(tmp_nb).cells[0].source, 'a=2\nprint(a)')

The line-addressed ops keep the text primitives’ conventions: insert_line at 0 prepends, deleting that line undoes it, and cell_replace_lines with no range replaces the whole source – yielding a line block, so the result carries a trailing newline:

cell_insert_line(tmp_nb, cid, 0, 'import sys')
cell_del_lines(tmp_nb, cid, 1, 1)
test_eq(read_nb(tmp_nb).cells[0].source, 'a=2\nprint(a)')
cell_replace_lines(tmp_nb, cid, new_content='b=3')
test_eq(read_nb(tmp_nb).cells[0].source, 'b=3\n')

Failures are loud: an unmatched str_replace reports an error instead of writing, and a missing cell id raises:

assert str(cell_str_replace(tmp_nb, cid, 'q', 'r')).startswith('error:')
with expect_fail(KeyError, 'no cell id'): view_cell(tmp_nb, 'zzzz')
tmp_nb.unlink()

Here’s how to put all the pieces of fastcore.nbio together:

nb = new_nb([mk_cell('print(1)')])
path = Path('test.ipynb')
write_nb(nb, path)
nb2 = read_nb(path)
print(nb2.cells)
path.unlink()

[{'cell_type': 'code', 'execution_count': None, 'id': 'd0c314de', 'metadata': {}, 'outputs': [], 'source': 'print(1)', 'idx_': 0, 'lang_': 'python'}]

Notebook files on disk store multiline text as lists of lines (nbformat’s split_lines), for source, stream text, textual mime data, and attachments. Like nbformat’s own reader, we join these to plain strings on read, and split them back on write, so in-memory code always sees strings while files round-trip byte-identically with Jupyter’s. JSON mimes and error tracebacks are genuinely structured, so they pass through untouched.

disk = dict(nbformat=4, nbformat_minor=5, metadata={}, cells=[
    dict(cell_type='code', id='c0', metadata={}, execution_count=1, source=['a=1\n','a'],
         outputs=[dict(output_type='execute_result', metadata={}, execution_count=1,
                       data={'text/plain':['hi\n','there'], 'text/markdown':['single'],
                             'application/json':{'a':[1,2]}, 'image/png':'iVBOR\nw0KG=='}),
                  dict(output_type='stream', name='stdout', text=['s1\n','s2\n']),
                  dict(output_type='error', ename='E', evalue='e', traceback=['t1','t2'])]),
    dict(cell_type='markdown', id='m0', metadata={}, source=['# t\n','x'],
         attachments={'im.txt':{'text/plain':['a\n','b']}, 'im.png':{'image/png':'aGk='}})])
tmp = Path('tmp.ipynb')
try:
    tmp.write_text(dumps(disk))
    nb = read_nb(tmp)
    res,strm,err = nb.cells[0].outputs
    test_eq(res['data']['text/plain'], 'hi\nthere')        # textual data joined on read
    test_eq(res['data']['text/markdown'], 'single')        # one-element lists too
    test_eq(res['data']['application/json'], {'a':[1,2]})  # JSON mimes untouched
    test_eq(res['data']['image/png'], 'iVBOR\nw0KG==')
    test_eq(strm['text'], 's1\ns2\n')                      # stream text joined
    test_eq(err['traceback'], ['t1','t2'])                 # tracebacks stay lists
    test_eq(nb.cells[1].attachments['im.txt']['text/plain'], 'a\nb')
    test_eq(nb2dict(nb), disk)                             # splitting on save restores the disk form exactly
finally: tmp.unlink()

Diffing cell sequences

diff_cells compares two versions of a cell sequence – an open notebook against its file on disk, a dialog against an edited copy – and yields the edits that turn one into the other. Items are aligned by id (any objects with .id work, not just cells), and each edit is a block: a contiguous run of deletions, insertions, or in-place changes, so an applier can handle a whole run at once and inserted items keep their order. Identity questions ride the ids; whether an aligned pair changed is a content question, answered by comparing key(item) on each side.

diff_cells

def diff_cells(
    a, # Old sequence of items carrying `.id` (cells, messages, ...)
    b, # New sequence
    key:function=noop, # Projection to comparable data, for change detection
):

Yield block CellEdits that turn a into b, aligned by item id

Five cells, then a copy with one deleted, one edited, and one inserted. Edits arrive in a-order: old and new are lists (parallel for change), and an insert’s idx anchors the block in a – the new items belong after a[idx-1], or at the front when idx is 0:

a = [mk_cell(f'x={i}', id=c) for i,c in enumerate('abcde')]
b = copy.deepcopy(a)
del b[1]
b[1].source = 'x=99'
b.insert(3, mk_cell('y=0', id='f'))
edits = list(diff_cells(a, b))
d,c,i = edits
test_eq(d.op, 'delete'); test_eq([o.id for o in d.old], ['b'])
test_eq(c.op, 'change'); test_eq(c.new[0].source, 'x=99')
test_eq(i.op, 'insert'); test_eq(i.idx, 4)
edits

The default key compares whole items, which for cells means their full dict – outputs included. Pass a projection to name the content that matters: with key=attrgetter('source'), an outputs-only difference is no edit at all. The same parameter serves hosts whose items don’t compare by content at all (aidialog messages, say, whose == checks ids), by projecting each item to its comparable form:

b2 = copy.deepcopy(a)
b2[0].outputs = [dict(output_type='stream', name='stdout', text='hi\n')]
test_eq([e.op for e in diff_cells(a, b2)], ['change'])
list(diff_cells(a, b2, key=attrgetter('source')))

Validation

Direct edits to cell dicts can produce notebooks that Jupyter and other tools reject: an outputs key on a markdown cell, a code cell missing execution_count, duplicate ids. validate_cell and validate_nb are cheap structural checks for exactly those mistakes. They raise ValueError naming the offending cell, and never repair: fixing is the caller’s decision. This is not the full nbformat schema, just the rules whose violation breaks notebooks in practice.

validate_nb

def validate_nb(
    nb
):

Raise ValueError for structural problems in notebook nb; returns it unchanged if fine

validate_cell

def validate_cell(
    cell, idx:NoneType=None
):

Raise ValueError for structural problems in notebook cell dict cell; returns it unchanged if fine

A valid notebook passes through unchanged, and each rule fails loudly, naming the cell (the markdown-with-outputs case is a real one: stray keys from hand-edited files):

vnb = new_nb([mk_cell('1+1'), mk_cell('a note', 'markdown')])
test_eq(validate_nb(vnb), vnb)
vnb.cells[1]['outputs'] = []
with expect_fail(ValueError, 'not allowed in a markdown cell'): validate_nb(vnb)
del vnb.cells[1]['outputs'], vnb.cells[0]['execution_count']
with expect_fail(ValueError, 'requires execution_count'): validate_nb(vnb)

dup = new_nb([mk_cell('a', id='x1'), mk_cell('b', id='x1')])
with expect_fail(ValueError, 'duplicate cell id'): validate_nb(dup)
with expect_fail(ValueError, 'unknown cell_type'): validate_cell(dict(cell_type='wat', source=''))
with expect_fail(ValueError, 'str or list of str'): validate_cell(dict(cell_type='raw', source=[1,2]))

repair_nb

def repair_nb(
    nb
):

Fix deterministic structural problems in nb, returning a list of repairs made

repair_cell

def repair_cell(
    cell, idx:NoneType=None
):

Fix deterministic structural problems in cell, returning a list of repairs made

Each validation rule has a deterministic repair, applied by repair_nb: non-code cells lose stray outputs/execution_count, code cells gain missing ones, broken source/metadata are coerced, missing notebook fields are added, and duplicate cell ids are regenerated (the first occurrence keeps the id). Unknown cell_types are left alone, since no repair can know the intent. The returned list says what was done, so callers can report or count repairs; a valid notebook returns [] and is untouched.

bad = dict2nb(dict(cells=[
    dict(cell_type='markdown', source='hi', outputs=[], execution_count=1, id='x1'),
    dict(cell_type='code', source='1+1', id='x1'),
], metadata={}, nbformat=4, nbformat_minor=5))
with expect_fail(ValueError): validate_nb(bad)

repairs = repair_nb(bad)
validate_nb(bad)
test_eq(len(repairs), 5)
assert bad.cells[0].id != bad.cells[1].id
test_eq(repair_nb(bad), [])

Code outputs have required structure too. Each entry must be a dict with one of the four standard output_types, carrying its required fields: stream a name and text, the two display types a data dict and metadata, execute_result also an execution_count, and error an ename, evalue, and traceback. Repair fills the deterministic omissions (metadata, execution_count) and removes entries too malformed to keep, reporting each removal with its reason.

onb = new_nb([mk_cell('1+1')])
onb.cells[0].outputs = [dict(output_type='execute_result', data={'text/plain':['2']}, metadata={}, execution_count=1)]
test_eq(validate_nb(onb), onb)
onb.cells[0].outputs = [dict(output_type='wat')]
with expect_fail(ValueError, 'unknown output_type'): validate_nb(onb)
onb.cells[0].outputs = [dict(output_type='stream', name='stdout')]
with expect_fail(ValueError, 'text'): validate_nb(onb)
onb.cells[0].outputs = [dict(output_type='execute_result', data={})]
with expect_fail(ValueError, 'metadata'): validate_nb(onb)
onb.cells[0].outputs = [dict(output_type='error', ename='E', evalue='boom', traceback='not a list')]
with expect_fail(ValueError, 'traceback'): validate_nb(onb)

rnb = new_nb([mk_cell('1+1')])
rnb.cells[0].outputs = [
    dict(output_type='execute_result', data={'text/plain':'2'}),
    dict(output_type='stream', name='stdout', text='hi'),
    dict(output_type='error', ename='E', evalue='boom'),
    'junk']
repairs = repair_nb(rnb)
validate_nb(rnb)
test_eq(len(rnb.cells[0].outputs), 2)
test_eq(rnb.cells[0].outputs[0]['metadata'], {})
test_eq(rnb.cells[0].outputs[0]['execution_count'], None)
test_eq(repair_nb(rnb), [])
repairs

['cell 58d10327 output 0: set metadata',
 'cell 58d10327 output 0: set execution_count',
 'cell 58d10327 output 2: removed output (error requires a traceback list of str)',
 'cell 58d10327 output 3: removed output (output must be a dict)']

Output rendering

preferred_out

def preferred_out(
    data, html1st:bool=True, include_imgs:bool=False
):

Call self as a function.

preferred_out selects the best MIME type from an output’s data dict, preferring HTML by default:

data = dict(text_plain=['42'], **{'text/html': ['<b>42</b>'], 'text/plain': ['42']})
wdata = {'image/webp': 'AAAA', 'text/plain': ['im']}
test_eq(preferred_out(wdata, include_imgs=True)[0], 'image/webp')
test_eq(preferred_out(wdata)[0], 'text/plain')
preferred_out(data), preferred_out(data, html1st=False)

(('text/html', ['<b>42</b>']), ('text/html', ['<b>42</b>']))

join_out

def join_out(
    d
):

Join Jupyter’s list-of-lines output data into one string

mk_error

def mk_error(
    traceback, ename:str='', evalue:str=''
):

Helper to create an error output dict

mk_display

def mk_display(
    metadata:NoneType=None, **data
):

Helper to create a display_data output dict

mk_result

def mk_result(
    metadata:NoneType=None, **data
):

Helper to create an execute_result output dict

mk_stream

def mk_stream(
    name, text
):

Helper to create an output stream dict

concat_streams

def concat_streams(
    outputs
):

Concatenate stream outputs by name (stdout/stderr), preserving execute_result at end

concat_streams merges consecutive stream outputs by name and moves execute_results to the end, like standard jupyter output rendering:

outs = [mk_result(text_plain=['42']),
        mk_stream('stdout', 'hello '), mk_stream('stdout', 'world\n'), mk_stream('stderr', 'warn\n')]
outs

[{'output_type': 'execute_result',
  'data': {'text/plain': ['42']},
  'metadata': {}},
 {'output_type': 'stream', 'name': 'stdout', 'text': 'hello '},
 {'output_type': 'stream', 'name': 'stdout', 'text': 'world\n'},
 {'output_type': 'stream', 'name': 'stderr', 'text': 'warn\n'}]

concat_streams(outs)

[{'output_type': 'stream', 'name': 'stdout', 'text': 'hello world\n'},
 {'output_type': 'stream', 'name': 'stderr', 'text': 'warn\n'},
 {'output_type': 'execute_result',
  'data': {'text/plain': ['42']},
  'metadata': {}}]

Carriage-return overwrites apply across chunk boundaries, as they would on a live terminal — a progress line ending in \r is overwritten by the next chunk, but a final \r (nothing follows it) leaves the text visible:

prog = [mk_stream('stdout', 'step 1\r'), mk_stream('stdout', 'step 2\r'), mk_stream('stdout', 'done\n')]
test_eq(concat_streams(prog), [mk_stream('stdout', 'done\n')])
test_eq(concat_streams([mk_stream('stdout', 'working\r')]), [mk_stream('stdout', 'working')])

preferred_msg_out

def preferred_msg_out(
    out, html1st:bool=True, include_imgs:bool=False
):

Preferred mime type and content for any Jupyter output dict (stream, error, or data-bearing)

preferred_msg_out extends preferred_out to any output dict: streams and errors are always plain text, while data-bearing outputs go through mime preference:

test_eq(preferred_msg_out(mk_stream('stdout', ['a\n','b\n'])), ('text/plain', 'a\nb\n'))
test_eq(preferred_msg_out(mk_result(text_html=['<b>4</b>'], text_plain=['4'])), ('text/html', ['<b>4</b>']))
test_eq(preferred_msg_out(mk_result(text_markdown=['*4*'], text_html=['<b>4</b>']), html1st=False)[0], 'text/markdown')

render_output

def render_output(
    out
):

Convert a single output dict to an HTML string

print(render_output(mk_result(text_plain=['42'])))

<pre class="!border-0 !rounded-none !my-0 !p-0"><code class="nohighlight">42</code></pre>

render_outputs

def render_outputs(
    outputs
):

Render a full list of outputs, concatenating streams first.

print(render_outputs(outs))

<pre class="!border-0 !rounded-none !my-0 !p-0"><code class="nohighlight">hello world
</code></pre>
<pre class="!border-0 !rounded-none !my-0 !p-0"><code class="nohighlight">warn
</code></pre>
<pre class="!border-0 !rounded-none !my-0 !p-0"><code class="nohighlight">42</code></pre>

render_text

def render_text(
    outputs, html1st:bool=False, tb_maxlen:NoneType=None
):

Render notebook outputs to concise ANSI-stripped text, using XML-ish tags when multiple outputs are present; tb_maxlen caps over-long error-traceback lines

A single output renders as plain text directly:…

print(render_text([outs[0]]))

…but multiple outputs get wrapped in XML-ish tags so they stay distinguishable. For rich outputs we prefer markdown by default; pass html1st=True to prefer HTML instead. Outputs without a text representation (e.g. images alone) render as empty.

print(render_text(outs))

<stdout>
hello world
</stdout>
<stderr>
warn
</stderr>
<execute_result>
42
</execute_result>

Errors arrive ANSI-colored from IPython, so render_text always strips escape sequences. A traceback line can also be enormous - a cell magic’s transformed source echoes its whole payload on one line - so tb_maxlen caps over-long traceback lines, keeping File/Cell locations and the final chunk (the exception message) whole, and dropping caret-decoration rows that mean nothing once cut.

tb = ['\x1b[31mFile /a/b.py:1\x1b[0m\n' + 'x'*200 + '\n~~~~^^^' + '~'*200, 'ValueError: ' + 'y'*200]
t = render_text([mk_error(tb, 'ValueError', 'boom')])
test_eq('\x1b' in t, False)
assert 'x'*200 in t
t2 = render_text([mk_error(tb, 'ValueError', 'boom')], tb_maxlen=120)
assert 'x'*200 not in t2 and 'File /a/b.py:1' in t2 and '~~~~^^^' not in t2 and 'y'*200 in t2
print(t2)

view_cell composes with output rendering: incl_out=True appends the rendered outputs in an <out> block (trunc_out=False lifts the ~512-char cap).

vnb = Path('tmp_out.ipynb')
vouts = [dict(output_type='execute_result', metadata={}, data={'text/plain': ['1']}, execution_count=1)]
write_nb(new_nb([mk_cell('a=1\nprint(a)', outputs=vouts)]), vnb)
vcid = read_nb(vnb).cells[0].id
test_eq(str(view_cell(vnb, vcid, incl_out=True)), '1: a=1\n2: print(a)\n<out>\n1\n</out>')
assert '<out>' not in str(view_cell(vnb, vcid))
vnb.unlink()

disp = [mk_display(text_html=['<b>42</b>'], text_markdown=['**42**'], text_plain=['42'])]
test_eq(render_text(disp), '**42**')
test_eq(render_text(disp, html1st=True), '<b>42</b>')

test_eq(render_text([mk_display(image_png='abc')]), '')
test_eq(render_text([mk_error('oops')]), 'oops')

nb.cells[0].outputs = [mk_stream('stdout', '1\n')]

Notebook class

item2xml

def item2xml(
    typ, # Tag name: the cell or message type, e.g. 'code', 'markdown', 'raw', 'prompt'
    content:str='', # The item's source text
    out:str='', # Rendered output text
    id:NoneType=None, # Optional id attribute
    meta:NoneType=None, # Cell/message metadata: directives in its `nbdev` dict render as attrs, bare ones as bare attrs
    **attrs
):

A notebook cell or dialog message as concise XML: content, then an <out> section when out is non-empty

item2xml renders notebook cells and dialog messages to LLM-friendly XML (cell2xml below and aidialog’s message renderers build on it). The content sits directly inside the type tag, with no wrapper of its own, and an <out> section marks where output begins - so an item without output carries no extra tags at all. Passing meta renders the metadata’s nbdev directives as attributes, so every projection built on item2xml shows them the same way.

test_eq(to_xml(item2xml('code', 'x*2', '42', id='ab')), '<code id="ab">x*2<out>42</out></code>')
to_xml(item2xml('markdown', '# hi', id='cd'))

'<markdown id="cd"># hi</markdown>'

Falsy attrs are dropped, literal True attrs have no value:

test_eq(to_xml(item2xml('code', 'x', time='', kind='system', foo=True)), '<code kind="system" foo>x</code>')
test_eq(to_xml(item2xml('code', 'x', meta=dict(nbdev=dict(hide='true', eval='false')))), '<code hide eval="false">x</code>')

cells2xml

def cells2xml(
    cells, wrap:partial=functools.partial(<function ft at 0x7fca2dfb2480>, 'nb'), ids:bool=True, incl_out:bool=True,
    **kw
):

Convert notebook cells to XML format

cell2xml

def cell2xml(
    cell, ids:bool=True, incl_out:bool=True
):

Convert NbCell to concise XML format

We can view any notebook as concise XML. For instance, our minimal notebook:

print(cells2xml(nb.cells, incl_out=False))

<nb><code id="c0">a=1
a</code><markdown id="m0"># t
x</markdown></nb>

repr(cell2xml(nb.cells[0], incl_out=False))

'<code id="c0">a=1\na</code>'

repr(cell2xml(nb.cells[0], incl_out=True))

'<code id="c0">a=1\na<out>1\n</out></code>'

# Metadata directives render as attrs: bare as bare, others verbatim (names unhyphenated)
c = mk_cell('1+1', metadata=dict(nbdev=dict(hide='true', default_exp='core')))
test_eq(repr(cell2xml(c, incl_out=False)), f'<code id="{c.id}" hide default_exp="core">1+1</code>')

Notebook

def Notebook(
    nb, path:NoneType=None
):

Read, query, and edit Jupyter notebooks

We can now open a notebook and access its metadata and cells:

nbo = Notebook.open(minimal_fn)
list(nbo.meta), len(nbo.cells), len(nbo)

(['solveit_dialog_mode', 'solveit_ver'], 2, 2)

nbo.path.name

'minimal.ipynb'

[o.id for o in nbo]

['801558df', 'e2147a69']

'e2147a69' in nbo, 'nonexistent' in nbo

(True, False)

Notebooks’ repr is their xml:

nbo

<nb path="/Users/jhoward/aai-ws/fastcore/tests/minimal.ipynb"><markdown id="801558df">## A minimal notebook</markdown><code id="e2147a69"># Do some arithmetic
1+1<out>2</out></code></nb>

You can also get a more concise version that doesn’t include outputs or the full path:

print(nbo.concise)

<nb path="minimal.ipynb"><markdown id="801558df">## A minimal notebook</markdown><code id="e2147a69"># Do some arithmetic
1+1</code></nb>

Cells can be accessed by integer index or by their string id – exact, or any unique prefix. A missing or ambiguous id raises a KeyError naming the problem:

nbo[0].source

'## A minimal notebook'

nbo['e2147a69'].source

'# Do some arithmetic\n1+1'

test_eq(nbo['e214'].source, nbo['e2147a69'].source)
with expect_fail(KeyError, 'no cell id'): nbo['nope']

You can directly set a cell’s source by id or index:

nbo['e2147a69'] = '2+2'
nbo['e2147a69'].source

'2+2'

Cells also carry the shared edit family as methods - the same operations as the cell_* functions, minus the address arguments, since the cell in hand is the carrier. These are in-memory edits: the diff comes back for verification, and nothing touches disk until the notebook is saved.

d = nbo['e2147a69'].str_replace('2+2', '3+3')
test_eq(nbo['e2147a69'].source, '3+3')
print(d)

@@ -1 +1 @@
-2+2
+3+3

You can also update outputs and metadata directly on a cell:

nbo['e2147a69'].outputs = [{'output_type': 'execute_result', 'data': {'text/plain': ['4']}}]
nbo['e2147a69'].outputs

[{'output_type': 'execute_result', 'data': {'text/plain': ['4']}}]

nbo['e2147a69'].metadata['custom'] = True
nbo['e2147a69'].metadata

{'custom': True, 'time_run': '2026-01-04T20:52:49.901559+00:00'}

The add method inserts a new cell at a given position (defaulting to the end):

Notebook.add

def add(
    source, cell_type:str='code', idx:NoneType=None, after:NoneType=None, before:NoneType=None, **kwargs
):

Add a new cell with source at idx (default: end), or after/before a cell id

nbo.add('print("hello")')
nbo.add('# A heading', cell_type='markdown', idx=0)
len(nbo), nbo[0].source

(4, '## A minimal notebook')

Cells can also be inserted relative to an existing cell by id:

cid = nbo[0].id
nbo.add('# After first', cell_type='markdown', after=cid)
nbo.add('# Before first', cell_type='markdown', before=cid)
[c.source for c in nbo[:3]]

['# Before first', '## A minimal notebook', '# After first']

Notebook.md

def md(
    source, idx:NoneType=None, after:NoneType=None, before:NoneType=None, **kwargs
):

Add a new cell with source at idx (default: end), or after/before a cell id

md is a shortcut to add(..., cell_type='markdown')

nbo.md('A note')
len(nbo), nbo[-1].cell_type

(7, 'markdown')

You can delete by id or index:

prev_len = len(nbo)
del nbo[0]
len(nbo) == prev_len - 1

True

Notebook.move

def move(
    src_ids, after:NoneType=None, before:NoneType=None
):

Move cells with src_ids after/before a cell id, or to end

Cells can be moved by id, either relative to another cell or to the end:

nbo = Notebook.open(minimal_fn)
c0,c1 = nbo[0].id,nbo[1].id
nbo.move(c1, before=c0)
[c.id for c in nbo] == [c1, c0]

True

Use save to write to disk:

nbo.save('path.ipynb')

If no path is passed, the path used in open() will be re-used.

Notebook.view_cell

def view_cell(
    id, nums:bool=True, incl_out:bool=False, trunc_out:bool=True
):

Show cell source with optional line numbers

The view_cell method displays a cell’s source with optional line numbers:

print(nbo.view_cell('e2147a69', incl_out=True))
assert nbo.view_cell('e2147a69', incl_out=True).endswith('<out>\n2\n</out>')
assert '<out>' not in nbo.view_cell('e2147a69')

     1 │ # Do some arithmetic
     2 │ 1+1
<out>
2
</out>

The path-taking twins of the query methods return snapshots, not live cells: CellRow records id, type, source, and the cell meta, and its summary line id:t[directives]:source (t: c=code m=markdown r=raw) shows any nbdev directives, so export state is visible at a glance. Rows are data to read and addresses to act on - edits then go through the cell_* functions or cell_exhash. The correspondence is mechanical: the function is the method with a path argument standing where the held notebook was.

CellRows

def CellRows(
    *args, **kwargs
):

Built-in mutable sequence.

If no argument is given, the constructor creates a new empty list. The argument must be an iterable if specified.

CellRow

def CellRow(
    c, maxlen:int=120, kind:str='match'
):

Snapshot of one cell, shown as id:t[directives]:source (t: c=code m=markdown r=raw); a context row from a find shows - in place of its final :

Notebook.to_dict

def to_dict():

The plain dict form of the held notebook (nb2dict): the representation layer

summary_nb

def summary_nb(
    path, # Notebook file to read
    maxlen:int=120, # Maximum source characters per line
):

One snapshot line per cell of the notebook at path

Notebook.summary

def summary(
    maxlen:int=120
):

One CellRow line per cell

test_eq(type(Notebook.open(minimal_fn).to_dict()), dict)
summary_nb(minimal_fn)

801558df:m:## A minimal notebook
e2147a69:c:# Do some arithmetic\n1+1

find_cells searches cell sources by regex, and by default includes one neighbouring cell of context around each match, because in a documented notebook the neighbouring markdown usually explains the match. Results are FoundCells – the Found mixin over a plain list, so sibling toolkits (aidialog’s message finds) share the identical contract: index them by cell id (exact or unique prefix), never by position – a context row can sit at any index, so fc[0] may be a neighbour of the match, and integer indexing raises an error saying so. In the display, a context row shows - in place of its final :.

Found

def Found(
    items:NoneType=None, matched:NoneType=None
):

Mixin for find-result containers: matched ids, id-only indexing, and match/context kinds

FoundCells

def FoundCells(
    items:NoneType=None, matched:NoneType=None
):

Find results: cells indexed by id (exact or unique prefix), shown as CellRow lines with context rows marked

Notebook.find_cells

def find_cells(
    pat:str='', # Regex over cell source
    cell_type:str=None, # Optional limit by type ('code', 'markdown', or 'raw')
    ids:str='', # Optional limit by cell ids (comma-separated str, or list); exact or unique prefixes
    context:int=None, # Cells of context around matches (default 1)
):

Find live cells matching all the given criteria, plus context neighbouring cells

nbo = Notebook.open(minimal_fn)
fc = nbo.find_cells(r'\d\+\d')
fc

The match is the code cell; its markdown neighbour rides along as context, marked with -. Positional indexing fails with an error teaching the id idiom, and context=0 returns only matches:

with expect_fail(TypeError, 'context rows'): fc[0]
test_eq(fc['e214'].source, '# Do some arithmetic\n1+1')
test_eq([c.id for c in nbo.find_cells(r'\d\+\d', context=0)], ['e2147a69'])
test_eq(fc.matched, {'e2147a69'})

The path-taking twin returns CellRow snapshots in a FoundCells, with the same indexing contract:

find_cells

def find_cells(
    path, # Notebook file to search
    pat:str='', # Regex over cell source
    cell_type:str=None, # Optional limit by type ('code', 'markdown', or 'raw')
    ids:str='', # Optional limit by cell ids (comma-separated str, or list); exact or unique prefixes
    context:int=None, # Cells of context around matches (default 1)
):

Snapshot FoundCells for matching cells in the notebook at path

rows = find_cells(minimal_fn, r'\d\+\d')
test_eq(type(rows['e2147a69']), CellRow)
test_eq(rows.matched, {'e2147a69'})
rows

update_cell is the transaction for a cell’s attributes rather than its source’s lines: plain keywords assign (cell_type=, metadata= replacing wholesale), mergemeta deep-merges into the metadata, and export sets the nbdev export directive. There is no negative export directive – a cell is exported by having it or not by lacking it – so export=False means removal, in whichever form (comment or metadata) the cell uses, and export=True adds it. deep_merge treats a None value as deletion:

deep_merge

def deep_merge(
    d:dict, # Base dict
    u:dict, # Updates: nested dicts merge recursively, and a `None` value deletes its key
):

Copy of d updated by u

test_eq(deep_merge(dict(a=1,b=dict(c=2,d=3)), dict(b=dict(c=None,e=4), a=None)), dict(b=dict(d=3,e=4)))

update_cell

def update_cell(
    path:str, # Notebook file to modify
    cell_id:str, # Id of the cell to update (exact, or unique prefix)
    mergemeta:dict=None, # `deep_merge` into cell metadata; a `None` value deletes its key
    export:bool=None, # Add (True) or remove (False) the nbdev export directive, in whichever form the cell uses
    **kwargs
):

Update a cell’s attributes, metadata, or export directive

The diff covers the cell’s concise XML and its metadata, so every change this function can make is visible in its return value. Removing the directive from a comment-form cell edits its source; adding one to a bare cell writes the comment form too, keeping the cell’s existing style:

up = Path('tmp_upd.ipynb')
write_nb(new_nb([mk_cell('#| export\ndef f(): pass', id='aa11'), mk_cell('x=1', id='bb22')]), up)
print(update_cell(up, 'aa11', export=False))
print(update_cell(up, 'bb22', mergemeta=dict(tags=['demo'])))

test_eq(read_nb(up).cells[0].directives, {})
test_eq(read_nb(up).cells[1].metadata['tags'], ['demo'])
print(update_cell(up, 'bb22', metadata={}))
up.unlink()

Selecting cells to run

Kernels and notebook tools (execnb’s CaptureShell.run_all, aidialog’s %nbrun magic) need to pick code cells from a notebook by cell id. select_cells matches an id or unique prefix, optionally including cells above/below the match or all code cells, and optionally filtering to nbdev-exported cells. Whether a selected cell then participates follows the eval cascade: the cell’s own eval: directive wins, else the notebook-level eval directive, else the default_eval argument. Cells named by id always run — #| eval: false means “not by default”, not “never” — and ignore_eval=True skips the cascade entirely, like run-all in Jupyter:

select_cells

def select_cells(
    nb, # A notebook read with `read_nb`
    *msgids:str, # Cell ids, or unique prefixes, to match
    above:bool=False, # Include each matched cell and all cells above it?
    below:bool=False, # Include each matched cell and all cells below it?
    all:bool=False, # Include all code cells (ignores `msgids`)?
    exported:bool=False, # Only cells with `#| export` or `#| exports`?
    default_eval:bool=True, # Participation default when neither the cell nor the notebook has an `eval` directive
    ignore_eval:bool=False, # Skip `eval` filtering entirely: every selected cell runs
):

Select code cells from nb by cell id or unique prefix, in the order given; cells named in msgids always run, others follow the eval cascade

does_cell_eval

def does_cell_eval(
    cell, default:bool
):

Does cell participate in a non-interactive run? Decided by its eval directive, or default when the directive is absent or unrecognized

fm_default_eval

def fm_default_eval(
    fm, default_eval:bool=True
):

Participation default for cells without their own eval directive: the notebook-level eval directive in frontmatter mapping fm if given, else default_eval

snb = new_nb([mk_cell('# intro', 'markdown')] + [mk_cell(f'x{i} = {i}', id=f'aa{i}{i}0000') for i in range(4)])
codes = [c for c in snb.cells if c.cell_type=='code']
c1 = codes[1]
test_eq(select_cells(snb, 'aa11'), [c1])
test_eq(select_cells(snb, 'aa110000'), [c1])
test_eq(select_cells(snb, 'aa11', above=True), codes[:2])
test_eq(select_cells(snb, 'aa11', below=True), codes[1:])
test_eq(select_cells(snb, all=True), codes)
test_eq(select_cells(snb, 'aa33', 'aa00'), [codes[3], codes[0]])  # several ids, in the order given
c1.source = '#| export\n'+c1.source
test_eq(select_cells(snb, all=True, exported=True), [c1])
c1.source = '#| eval: false\n'+c1.source
test_eq(select_cells(snb, all=True), [c for c in codes if c is not c1])   # `eval: false` drops out of bulk selection
test_eq(select_cells(snb, 'aa11'), [c1])                                  # but a cell named by id always runs
codes[2].source = 'nbdev_export'+'()'
test_eq(select_cells(snb, all=True), [codes[0], codes[3]])
test_eq(select_cells(snb, all=True, ignore_eval=True), codes)             # Jupyter-style literal run-all
with expect_fail(Exception, 'aa'): select_cells(snb, 'aa')
with expect_fail(Exception, 'msgids'): select_cells(snb)
select_cells(snb, 'aa11')

[{'cell_type': 'code',
  'source': '#| eval: false\n#| export\nx1 = 1',
  'directives_': {},
  'id': 'aa110000',
  'metadata': {},
  'outputs': [],
  'execution_count': None,
  'idx_': 2,
  'lang_': 'python',
  '_meta_names_': set(),
  '_directives_': {'eval': 'false', 'export': ''}}]

Directive lookalikes inside strings are not directives, while metadata-form directives count like comment-form ones:

sc = mk_cell('x = """\n#| eval: false\n"""')
assert sc in select_cells(new_nb([sc]), all=True)
mc = mk_cell('y = 1', metadata=dict(nbdev=dict(eval='false')))
assert mc not in select_cells(new_nb([mc]), all=True)

The full participation rule is the eval cascade: the cell’s own directive, else the notebook-level eval directive (any nb_frontmatter source — frontmatter block, # title list line, or notebook metadata.nbdev), else the default_eval argument. fm_default_eval resolves the notebook level, recognizing true/True/false/False as bools or strings and ignoring anything else; does_cell_eval applies the cell level. Passing default_eval=False turns a run opt-in — only cells marked #| eval: true participate — which is how template fill runs a dialog’s cells while batch testing keeps the run-by-default policy:

onb = new_nb([mk_cell('a = 1', id='p1'), mk_cell('#| eval: true\nb = 2', id='m1')])
test_eq(len(select_cells(onb, all=True)), 2)                                    # nothing says otherwise: everything runs
test_eq([c.id for c in select_cells(onb, all=True, default_eval=False)], ['m1'])  # opt-in: only marked cells
fnb = new_nb([mk_cell('---\neval: false\n---', 'raw'), *onb.cells])
test_eq([c.id for c in select_cells(fnb, all=True)], ['m1'])                    # the nb-level directive fills silence
test_eq(fm_default_eval(nb_frontmatter(fnb)), False)
test_eq(fm_default_eval({'eval':'True'}, default_eval=False), True)             # four spellings, str or bool
test_eq(fm_default_eval({'eval':'maybe'}), True)                                # unrecognized: ignored

Running a cell

run_cell

async def run_cell(
    shell, # An `InteractiveShell`-compatible object: `transform_cell`, `run_cell_async`, `events`
    raw_cell:str, # Python/IPython source for one cell: magics, `!` commands, and top-level `await` all work
    store_history:bool=False, # Store the cell in the shell's history? (enables native `;` suppression and execution counts)
    silent:bool=False, # Suppress displayhook and `post_run_cell` event? (the result value stays unset)
    shell_futures:bool=True, # Share `__future__` imports with the shell?
    cell_id:NoneType=None, # Optional cell id, passed through to `run_cell_async`
):

Run one cell on shell: transform, await run_cell_async on the calling loop, and fire the post events; returns the ExecutionResult

run_cell is the shared execution core. It transforms the source itself, since IPython no longer auto-transforms, then awaits the shell’s own run_cell_async, and fires the post events that run_cell_async omits. It is duck-typed over the shell, needing only transform_cell, run_cell_async, and events, so any InteractiveShell-compatible object works and fastcore needs no IPython import. What happens to displays, streams, and tracebacks is decided by the shell you pass and the silent flag, never here. A capturing shell records, a kernel shell publishes, and silent=True suppresses the displayhook and post_run_cell while streams still flow, which also leaves the result value unset. With store_history=True the shell’s native trailing-; suppression and execution counts work exactly as in a live kernel.

One run returns the value and fires post_run_cell. A failing run records its exception in the result rather than raising. A silent run fires neither the displayhook nor post_run_cell, so the value goes unrecorded. A trailing ; suppresses as in a notebook:

sh = InteractiveShell()
events = []
sh.events.register('post_run_cell', lambda result: events.append(result))
r = await run_cell(sh, 'y = 6\ny*7', store_history=True)
test_eq((r.result, r.error_in_exec, len(events)), (42, None, 1))
r2 = await run_cell(sh, '1/0', store_history=True)
test_eq((type(r2.error_in_exec), len(events)), (ZeroDivisionError, 2))
rs = await run_cell(sh, '9*9', silent=True, store_history=True)
test_eq((rs.result, len(events)), (None, 2))
r3 = await run_cell(sh, 'y*7;', store_history=True)
test_eq(r3.result, None)
r

A fresh InteractiveShell shows the contract: source in, ExecutionResult out, state kept in the shell’s namespace. With store_history=True the shell’s native trailing-; suppression applies, and magics, ! commands, and top-level await all work.

sh = InteractiveShell()
r = await run_cell(sh, 'x = 6\nx*7', store_history=True)
test_eq((sh.user_ns['x'], r.result), (6, 42))
test_eq((await run_cell(sh, 'x;', store_history=True)).result, None)
test_eq((await run_cell(sh, '%who_ls int', store_history=True)).result, ['x'])
(await run_cell(sh, 'import asyncio\nawait asyncio.sleep(0.001)\n"slept"', store_history=True)).result

Prints and displays go through the ordinary user channels, so an enclosing capture sees them. On a plain shell the displayhook writes the result to stdout as well, since displaying is exactly what a non-silent run does. silent=True runs the same code with no display and no populated result.

with capture_output() as cap: r = await run_cell(sh, 'print("inner"); 5', store_history=True)
assert cap.stdout.startswith('inner\n') and 'Out' in cap.stdout
test_eq(r.result, 5)
with capture_output() as cap2: rs = await run_cell(sh, 'print("quiet"); 5', silent=True)
test_eq((cap2.stdout, rs.result), ('quiet\n', None))

The execution events fire around the cell, so extensions that display from them work. matplotlib’s inline backend flushes figures on post_execute, and since a bare InteractiveShell has no GUI loop, events are all it needs.

sh.enable_gui = lambda gui=None: None
await run_cell(sh, '%matplotlib inline')
with capture_output() as cap: await run_cell(sh, 'import matplotlib.pyplot as plt\nplt.plot([1,4,2]);', store_history=True)
test_eq(len(cap.outputs), 1)
cap.outputs[0]

Message outputs

Consumers of kernel messages usually want the iopub outputs in the shape a notebook file stores (the wire framing itself lives in jupywire), so downstream code (renderers, dialog models) needn’t know about protocol wrapping. msg2out converts one message, with nbformat.v4.output_from_msg’s semantics but no nbformat dependency: fields are selected explicitly, so the protocol-only transient never survives into the output dict, metadata tolerates being absent or None, and a non-output message raises ValueError. The msg_type may sit at the message top level (as jupyter_client-shaped dicts carry it) or in the header (as wire messages do). msgs2outs converts a whole drained list, skipping non-output chatter such as status and execute_input.

msgs2outs

def msgs2outs(
    msgs
):

nbformat-style output dicts for the output messages in msgs, skipping other message types

msg2out

def msg2out(
    msg
):

nbformat-style output dict for Jupyter iopub message dict msg

def _hdr_msg(typ, **c): return dict(header=dict(msg_type=typ), content=c)  # wire style
def _top_msg(typ, **c): return dict(msg_type=typ, content=c)  # jupyter_client style

test_eq(msg2out(_hdr_msg('stream', name='stdout', text='hi\n')), dict(output_type='stream', name='stdout', text='hi\n'))
test_eq(msg2out(_top_msg('error', ename='E', evalue='boom', traceback=['t'])),
        dict(output_type='error', ename='E', evalue='boom', traceback=['t']))
test_eq(msg2out(_hdr_msg('execute_result', data={'text/plain':'42'}, metadata={}, execution_count=1, transient={'display_id':'x'})),
        dict(output_type='execute_result', metadata={}, data={'text/plain':'42'}, execution_count=1))
test_eq(msg2out(_top_msg('display_data', data={'text/plain':'hi'}, metadata=None, transient={})),
        dict(output_type='display_data', metadata={}, data={'text/plain':'hi'}))
with expect_fail(ValueError): msg2out(_hdr_msg('status', execution_state='idle'))

msgs = [_hdr_msg('status', execution_state='busy'), _hdr_msg('execute_input', code='1+1'),
        _hdr_msg('stream', name='stdout', text='hi\n'), _top_msg('execute_result', data={'text/plain':'2'}, metadata={})]
msgs2outs(msgs)

[{'output_type': 'stream', 'name': 'stdout', 'text': 'hi\n'},
 {'output_type': 'execute_result',
  'metadata': {},
  'data': {'text/plain': '2'},
  'execution_count': None}]