Basic functionality

Basic functionality used in the fastai library

This module has an unusually large API surface, so the symbol listing in doc(fastcore.basics) is suppressed via __pyskill_sigs__: the exported cells below assemble into a curated module docstring instead.

Basics

The staples used throughout fastai code, each replacing a common multi-line pattern with a one-liner: ifnone(a,b) is b if a is None else a (though both args are always evaluated); listify and tuplify convert anything to a list or tuple the way you’d mean it (None becomes [], a str or dict stays a single item, a generator is consumed); and basic_repr('a,b') gives a class a deterministic key=value repr with no memory address, so notebook and git diffs stay clean.

ifnone

def ifnone(
    a, b
):

b if a is None else a

Since b if a is None else a is such a common pattern, we wrap it in a function. However, be careful, because python will evaluate both a and b when calling ifnone (which it doesn’t do if using the if version directly).

test_eq(ifnone(None,1), 1)
test_eq(ifnone(2   ,1), 2)

req

def req(
    x, msg:NoneType=None, exc:type=ValueError
):

x unless it’s None, in which case raise exc(msg)

The complement of ifnone for the cases where None is not an acceptable answer: many lookups signal “not found” with None (first on an empty sequence, dict.get, a C API returning a null handle), and the caller’s next line is often if x is None: raise .... req folds that guard into the lookup expression itself.

test_eq(req(2, 'missing'), 2)
test_eq(req(False, 'missing'), False)
test_fail(lambda: req(None, 'missing'), contains='missing')
test_fail(lambda: req(None, 'gone', exc=KeyError), exc=KeyError)

maybe_attr

def maybe_attr(
    o, attr
):

getattr(o,attr,o)

Return the attribute attr for object o. If the attribute doesn’t exist, then return the object o instead.

class myobj: myattr='foo'

test_eq(maybe_attr(myobj, 'myattr'), 'foo')
test_eq(maybe_attr(myobj, 'another_attr'), myobj)

basic_repr

def basic_repr(
    flds:NoneType=None
):

Minimal __repr__

In types which provide rich display functionality in Jupyter, their __repr__ is also called in order to provide a fallback text representation. Unfortunately, this includes a memory address which changes on every invocation, making it non-deterministic. This causes diffs to get messy and creates conflicts in git. To fix this, put __repr__=basic_repr() inside your class.

class SomeClass: __repr__=basic_repr()
repr(SomeClass())

'SomeClass()'

If you pass a list of attributes (flds) of an object, then this will generate a string with the name of each attribute and its corresponding value. The format of this string is key=value, where key is the name of the attribute, and value is the value of the attribute. For each value, attempt to use the __name__ attribute, otherwise fall back to using the value’s __repr__ when constructing the string.

class SomeClass:
    a=1
    b='foo'
    __repr__=basic_repr('a,b')

repr(SomeClass())

"SomeClass(a=1, b='foo')"

Nested objects work too:

class AnotherClass:
    c=SomeClass()
    d='bar'
    __repr__=basic_repr(['c', 'd'])

repr(AnotherClass())

"AnotherClass(c=SomeClass(a=1, b='foo'), d='bar')"

Instance variables (but not class variables) are shown if basic_repr is called with no arguments:

class SomeClass:
    def __init__(self, a=1, b='foo'): self.a,self.b = a,b
    __repr__=basic_repr()

repr(SomeClass())

"SomeClass(a=1, b='foo')"

BasicRepr

def BasicRepr(
    *args, **kwargs
):

Base class for objects needing a basic __repr__

As a shortcut for creating a __repr__ for instance variables, you can inherit from BasicRepr:

class SomeClass(BasicRepr):
    def __init__(self, a=1, b='foo'): self.a,self.b = a,b

repr(SomeClass())

"SomeClass(a=1, b='foo')"

is_array

def is_array(
    x
):

True if x supports __array__ or iloc

is_array covers numpy arrays and anything array-like, and pandas objects through iloc:

is_array(np.array(1)),is_array([1])

(True, False)

listify

def listify(
    o:NoneType=None, *rest, use_list:bool=False, match:NoneType=None
):

Convert o to a list

Conversion is designed to “do what you mean”, e.g:

test_eq(listify('hi'), ['hi'])
test_eq(listify(b'hi'), [b'hi'])
test_eq(listify(array(1)), [array(1)])
test_eq(listify(1), [1])
test_eq(listify([1,2]), [1,2])
test_eq(listify(range(3)), [0,1,2])
test_eq(listify(None), [])
test_eq(listify(1,2), [1,2])

arr = np.arange(9).reshape(3,3)
listify(arr)

[array([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])]

listify(array([1,2]))

[array([1, 2])]

Generators are turned into lists too:

gen = (o for o in range(3))
test_eq(listify(gen), [0,1,2])

Use match to provide a length to match:

test_eq(listify(1,match=3), [1,1,1])

If match is a sequence, it’s length is used:

test_eq(listify(1,match=range(3)), [1,1,1])

If the listified item is not of length 1, it must be the same length as match:

test_eq(listify([1,1,1],match=3), [1,1,1])
with expect_fail(): listify([1,1],match=3)

tuplify

def tuplify(
    o, use_list:bool=False, match:NoneType=None
):

Make o a tuple

tuplify is listify returning a tuple:

test_eq(tuplify(None),())
test_eq(tuplify([1,2,3]),(1,2,3))
test_eq(tuplify(1,match=[1,2,3]),(1,1,1))

true

def true(
    x
):

Test whether x is truthy; collections with >0 elements are considered True

true avoids the ambiguous truth value of arrays by using their length:

[(o, true(o)) for o in (array(0), array(1), array([0]), array([0, 1]), 1, 0, '', None)]

[(array(0), False),
 (array(1), True),
 (array([0]), True),
 (array([0, 1]), True),
 (1, True),
 (0, False),
 ('', False),
 (None, False)]

NullType

def NullType(
    *args, **kwargs
):

An object that is False and can be called, chained, and indexed

null is the single NullType instance. Any attribute, call or index on it returns null again, so a chain never raises:

bool(null.hi().there[3])

False

tonull

def tonull(
    x
):

Convert None to null

tonull lets a possibly-None value take part in such a chain:

bool(tonull(None).hi().there[3])

False

get_class builds a class whose fields default to None, with __init__, __eq__ and a basic_repr derived from them:

get_class

def get_class(
    nm, *fld_names, sup:NoneType=None, doc:NoneType=None, funcs:NoneType=None, anno:NoneType=None, **flds
):

Dynamically create a class, optionally inheriting from sup, containing fld_names

_t = get_class('_t', 'a', b=2, anno={'b':int})
t = _t()
test_eq(t.a, None)
test_eq(t.b, 2)

'_t(a=1, b=3)'

Fields can be filled by keyword or by position:

t = _t(1, b=3)
test_eq(t.a, 1)
test_eq(t.b, 3)
t = _t(1, 3)
test_eq(t.a, 1)
test_eq(t.b, 3)

Instances pickle, unannotated fields default to Any, and there is a repr:

test_eq(t, pickle.loads(pickle.dumps(t)))
test_eq(_t.__annotations__, {'b':int, 'a':typing.Any})
repr(t)

Most often you’ll want to call mk_class, since it adds the class to your module. See mk_class for more details and examples of use (which also apply to get_class).

mk_class

def mk_class(
    nm, *fld_names, sup:NoneType=None, doc:NoneType=None, funcs:NoneType=None, mod:NoneType=None, anno:NoneType=None,
    **flds
):

Create a class using get_class and add to the caller’s module

Any kwargs will be added as class attributes, and sup is an optional (tuple of) base classes.

mk_class('_t', a=1, sup=dict)
t = _t()
test_eq(t.a, 1)
assert(isinstance(t,dict))

A __init__ is provided that sets attrs for any kwargs, and for any args (matching by position to fields), along with a __repr__ which prints all attrs. The docstring is set to doc. You can pass funcs which will be added as attrs with the function names.

def foo(self): return 1
mk_class('_t', 'a', sup=dict, doc='test doc', funcs=foo)

t = _t(3, b=2)
test_eq(t.a, 3)
test_eq(t.b, 2)
test_eq(t.foo(), 1)
test_eq(t.__doc__, 'test doc')
t

{}

wrap_class

def wrap_class(
    nm, *fld_names, sup:NoneType=None, doc:NoneType=None, funcs:NoneType=None, **flds
):

Decorator: makes function a method of a new class nm passing parameters to mk_class

wrap_class turns the decorated function into a method of a new class, made by mk_class in the function’s module:

@wrap_class('_t', a=2)
def bar(self,x): return x+1

t = _t()
test_eq(t.a, 2)
test_eq(t.bar(3), 4)

ignore_exceptions swallows anything raised inside the block:

ignore_exceptions

def ignore_exceptions(
    *args, **kwargs
):

Context manager to ignore exceptions

with ignore_exceptions(): raise Exception

exec_local

def exec_local(
    code, var_name
):

Call exec on code and return the var var_name

exec_local runs code in a fresh local namespace and returns one variable from it:

test_eq(exec_local("a=1", "a"), 1)

risinstance

def risinstance(
    types, obj:NoneType=None
):

Curried isinstance but with args reversed

risinstance takes the types first, so it curries into a predicate, and accepts type names as strings, matched against the object’s MRO:

assert risinstance(int, 1)
assert not risinstance(str, 0)
assert risinstance(int)(1)
assert not risinstance(int)(None)

types can also be strings:

assert risinstance(('str','int'), 'a')
assert risinstance('str', 'a')
assert not risinstance('int', 'a')

ver2tuple

def ver2tuple(
    v:str
)->tuple:

ver2tuple reads up to three numeric parts of a version string, treating missing parts as 0:

test_eq(ver2tuple('3.8.1'), (3,8,1))
test_eq(ver2tuple('3.1'), (3,1,0))
test_eq(ver2tuple('3.'), (3,0,0))
test_eq(ver2tuple('3'), (3,0,0))

NoOp

These are used when you need a pass-through function.

noop

def noop(
    x:NoneType=None, *args, **kwargs
):

Do nothing

noop()
test_eq(noop(1),1)

noops is the method form, taking self and returning its argument:

noops

def noops(
    x:NoneType=None, *args, **kwargs
):

Do nothing (method)

class _t: foo=noops
test_eq(_t().foo(1),1)

Infinite Lists

These lists are useful for things like padding an array or adding index column(s) to arrays.

Inf defines the following properties:

count: itertools.count()
zeros: itertools.cycle([0])
ones : itertools.cycle([1])
nones: itertools.cycle([None])

test_eq([o for i, o in zip(range(5), Inf.count)], [0, 1, 2, 3, 4])

test_eq([o for i, o in zip(range(5), Inf.zeros)], [0] * 5)

test_eq([o for i, o in zip(range(5), Inf.ones)], [1] * 5)

test_eq([o for i, o in zip(range(5), Inf.nones)], [None] * 5)

Operator Functions

Curried versions of the comparison and arithmetic functions in Python’s operator module: lt gt le ge eq ne add sub mul truediv is_ is_not in_ mod. With two args they work like operator’s versions; with one arg they return a partial that binds it as the second argument, so lt(3) means “is less than 3” and in_(vals) means “is contained in vals”. They read especially well as cmp arguments to fastcore.test.test, e.g. test(x, valid, in_).

in_

def in_(
    x, a
):

True if x in a

The remaining operators are generated from the operator module by _mk_op, so they share _oper’s behaviour. in_ is a membership test, and each operator can also be used as a partial with the collection given first:

assert in_('c', ('b', 'c', 'a'))
assert in_(4, [2,3,4,5])
assert in_('t', 'fastai')
assert not in_('h', 'fastai')

assert in_('fastai')('t')
assert in_([2,3,4,5])(4)
assert not in_('fastai')('h')

In addition to in_, the following functions are provided matching the behavior of the equivalent versions in operator: lt gt le ge eq ne add sub mul truediv is_ is_not mod.

lt(3,5),gt(3,5),is_(None,None),in_(0,[1,2]),mod(3,2)

(True, False, True, False, 1)

Similarly to in_, they also have additional functionality: if you only pass one param, they return a partial function that passes that param as the second positional parameter.

lt(5)(3),gt(5)(3),is_(None)(None),in_([1,2])(0),mod(2)(3)

(True, False, True, False, 1)

ret_true

def ret_true(
    *args, **kwargs
):

Predicate: always True

ret_true and ret_false accept any arguments and ignore them, which makes them safe defaults for predicate parameters:

assert ret_true(1,2,3)
assert ret_true(False)

ret_false

def ret_false(
    *args, **kwargs
):

Predicate: always False

ret_false is the complement:

stop

def stop(
    e:type=StopIteration
):

Raises exception e (by default StopIteration)

stop is a function that raises, so it can end an iteration from inside an expression such as a lambda or a map:

gen

def gen(
    func, seq, cond:function=ret_true
):

Map func over seq, stopping at the first result that fails cond; handles StopIteration

gen maps func over seq and stops at the first result that fails cond, so it can take a slice of an infinite sequence:

test_eq(gen(noop, Inf.count, lt(5)), range(5))
test_eq(gen(operator.neg, Inf.count, gt(-5)), [0, -1, -2, -3, -4])
test_eq(gen(lambda o: o if o < 5 else stop(), Inf.count), range(5))

chunked

def chunked(
    it, chunk_sz:NoneType=None, drop_last:bool=False, n_chunks:NoneType=None, pad:bool=False, pad_val:NoneType=None
):

Return batches from iterator it of size chunk_sz (or return n_chunks total)

Note that you must pass either chunk_sz, or n_chunks, but not both.

t = list(range(10))
test_eq(chunked(t,3),      [[0,1,2], [3,4,5], [6,7,8], [9]])
test_eq(chunked(t,3,True), [[0,1,2], [3,4,5], [6,7,8],    ])

A generator is consumed as it goes, so stop inside one ends the chunks:

t = map(lambda o:stop() if o==6 else o, Inf.count)
test_eq(chunked(t,3), [[0, 1, 2], [3, 4, 5]])
t = map(lambda o:stop() if o==7 else o, Inf.count)
test_eq(chunked(t,3), [[0, 1, 2], [3, 4, 5], [6]])

Arrays chunk the same way:

t = np.arange(10)
test_eq(chunked(t,3),      [[0,1,2], [3,4,5], [6,7,8], [9]])
test_eq(chunked(t,3,True), [[0,1,2], [3,4,5], [6,7,8],    ])

An empty input gives no chunks, however it is sized:

test_eq(chunked([], 3),          [])
test_eq(chunked([], n_chunks=3), [])

Pass pad=True and an optional pad_val to pad the last chunk:

t = list(range(10))
test_eq(chunked(t,3,pad=True), [[0,1,2], [3,4,5], [6,7,8], [9,None,None]])
test_eq(chunked(t,3,pad=True,pad_val=0), [[0,1,2], [3,4,5], [6,7,8], [9,0,0]])
test_eq(chunked(t,4,pad=True,pad_val=-1), [[0,1,2,3], [4,5,6,7], [8,9,-1,-1]])
test_eq(chunked(range(5),2,pad=True), [[0,1], [2,3], [4,None]])

otherwise

def otherwise(
    x, tst, y
):

y if tst(x) else x

otherwise replaces x with y only when the test passes:

test_eq(otherwise(2+1, gt(3), 4), 3)
test_eq(otherwise(2+1, gt(2), 4), 4)

Attribute Helpers

AttrDict is a dict whose keys are also attributes, so d.foo reads and writes d['foo'] (to convert a whole nested structure at once, see dict2obj in fastcore.xtras); NS is the same idea built on SimpleNamespace, adding indexing and iteration. store_attr(), called inside __init__, stores the function’s arguments as attributes of self in one line:

These functions reduce boilerplate when setting or manipulating attributes or properties of objects.

custom_dir

def custom_dir(
    c, add
):

Implement custom __dir__, adding add to cls

custom_dir allows you extract the __dict__ property of a class and appends the list add to it.

class _T: 
    def f(): pass

s = custom_dir(_T(), add=['foo', 'bar'])
assert {'foo', 'bar', 'f'}.issubset(s)

adict

def adict(
    *args, **kwargs
):

dict subclass that also provides access to keys as attrs

adict reads and writes keys as attributes, and missing keys still go through get:

d = adict(a=1,b="two")
test_eq(d.a, 1)
test_eq(d['b'], 'two')
test_eq(d.get('c','nope'), 'nope')

Assignment works both ways too, and the keys show up in dir:

d.b = 2
test_eq(d.b, 2)
test_eq(d['b'], 2)
d['b'] = 3
test_eq(d['b'], 3)
test_eq(d.b, 3)
assert 'a' in dir(d)

AttrDict

def AttrDict(
    *args, **kwargs
):

dict subclass that also provides access to keys as attrs, and has a pretty markdown repr

AttrDict will pretty print in Jupyter Notebooks:

d = AttrDict(a=1,b="two")

_test_dict = dict(a=1, b={'c':1, 'd':2}, c={'c':1, 'd':2}, d={'c':1, 'd':2}, e={'c':1, 'd':2}, f=dict(c=1, d=2, e=4, f=[1,2,3,4,5]))
AttrDict(_test_dict)

{ 'a': 1,
  'b': {'c': 1, 'd': 2},
  'c': {'c': 1, 'd': 2},
  'd': {'c': 1, 'd': 2},
  'e': {'c': 1, 'd': 2},
  'f': {'c': 1, 'd': 2, 'e': 4, 'f': [1, 2, 3, 4, 5]}}

AttrDictDefault

def AttrDictDefault(
    *args, default_:NoneType=None, **kwargs
):

AttrDict subclass that returns default_ for missing attrs

AttrDictDefault answers missing attributes with default_ instead of raising:

d = AttrDictDefault(a=1,b="two", default_='nope')
test_eq(d.a, 1)
test_eq(d['b'], 'two')
test_eq(d.c, 'nope')

NS

def NS(
    *args, **kwargs
):

SimpleNamespace subclass that also adds iter and dict support

This is very similar to AttrDict, but since it starts with SimpleNamespace, it has some differences in behavior. You can use it just like SimpleNamespace:

d = NS(**_test_dict)
d

namespace(a=1,
          b={'c': 1, 'd': 2},
          c={'c': 1, 'd': 2},
          d={'c': 1, 'd': 2},
          e={'c': 1, 'd': 2},
          f={'c': 1, 'd': 2, 'e': 4, 'f': [1, 2, 3, 4, 5]})

…but you can also index it to get/set:

d['a']

…and iterate t:

list(d)

['a', 'b', 'c', 'd', 'e', 'f']

get_annotations_ex

def get_annotations_ex(
    obj, *, globals:NoneType=None, locals:NoneType=None
):

Backport of py3.10 get_annotations that returns globals/locals

In Python 3.10 inspect.get_annotations was added. However previous versions of Python are unable to evaluate type annotations correctly if from future import __annotations__ is used. Furthermore, all annotations are evaluated, even if only some subset are needed. get_annotations_ex provides the same functionality as inspect.get_annotations, but works on earlier versions of Python, and returns the globals and locals needed to evaluate types.

eval_type

def eval_type(
    t, glb, loc
):

eval a type or collection of types, if needed, for annotations in py3.10+

In py3.10, or if from future import __annotations__ is used, a is a str:

class _T2a: pass
def func(a: _T2a): pass
ann,glb,loc = get_annotations_ex(func)

eval_type(ann['a'], glb, loc)

__main__._T2a

| is supported for defining Union types when using eval_type even for python versions prior to 3.9:

class _T2b: pass
def func(a: _T2a|_T2b): pass
ann,glb,loc = get_annotations_ex(func)

eval_type(ann['a'], glb, loc)

__main__._T2a | __main__._T2b

type_hints

def type_hints(
    f
):

Like typing.get_type_hints but returns {} if not allowed type

For example, type func is allowed so type_hints returns the same value as typing.get_hints:

def f(a:int)->bool: ...
exp = {'a':int,'return':bool}
test_eq(type_hints(f), typing.get_type_hints(f))
test_eq(type_hints(f), exp)

However, class is not an allowed type, so type_hints returns {}:

class _T:
    def __init__(self, a:int=0)->bool: ...
assert not type_hints(_T)

annotations

def annotations(
    o
):

Annotations for o, or type(o)

This supports a wider range of situations than type_hints, by checking type() and __init__ for annotations too:

for o in _T,_T(),_T.__init__,f: test_eq(annotations(o), exp)
assert not annotations(int)
assert not annotations(print)

anno_ret

def anno_ret(
    func
):

Get the return annotation of func

anno_ret is the return entry of annotations, or None when there is no annotation or no function:

def f(x) -> float: return x
test_eq(anno_ret(f), float)

def f(x) -> typing.Tuple[float,float]: return x
assert anno_ret(f)==typing.Tuple[float,float]

If your return annotation is None, anno_ret will return NoneType (and not None):

def f(x) -> None: return x

test_eq(anno_ret(f), NoneType)
assert anno_ret(f) is not None

If your function does not have a return type, or if you pass in None instead of a function, then anno_ret returns None:

def f(x): return x

test_eq(anno_ret(f), None)
test_eq(anno_ret(None), None)

signature_ex

def signature_ex(
    obj, eval_str:bool=False
):

Like inspect.signature, but a declared Parameter.empty default is kept as a default

inspect uses Parameter.empty both as its “no default” marker and as a value a function may declare as an actual default (a common sentinel idiom, e.g. fasthtml.core.add_sig_param). Plain signature() reads such a declaration back as “no default”, producing a signature that fails validation as soon as it is rebuilt, e.g. by Signature.replace. signature_ex keeps the declared default distinct, so the signature stays valid and renders as the source declared it:

def _f(a, b=1, c=Parameter.empty, *, d=Parameter.empty): ...
with expect_fail(ValueError, contains='non-default'): signature(_f).replace()
sig = signature_ex(_f)
assert sig.parameters['c'].default is not Parameter.empty
assert sig.parameters['d'].default is not Parameter.empty
sig.replace()
sig

union2tuple

def union2tuple(
    t
):

The member types of a Union or X|Y annotation, otherwise t unchanged

union2tuple unpacks a Union or X|Y annotation into its member types and returns anything else unchanged:

test_eq(union2tuple(Union[int,str]), (int,str))
test_eq(union2tuple(int), int)
assert union2tuple(Tuple[int,str])==Tuple[int,str]
test_eq(union2tuple((int,str)), (int,str))
if UnionType: test_eq(union2tuple(int|str), (int,str))

argnames

def argnames(
    f, frame:bool=False
):

Names of arguments to function or frame f

argnames reads positional and keyword-only names from a function, or from a frame with frame=True:

test_eq(argnames(f), ['x'])

with_cast

def with_cast(
    f
):

Decorator which uses any parameter annotations as preprocessing functions

with_cast calls each annotation on its argument before the function runs, defaults included, and applies the return annotation to the result:

@with_cast
def _f(a, b:Path, c:str='', d=0): return (a,b,c,d)

test_eq(_f(1, '.', 3), (1,Path('.'),'3',0))
test_eq(_f(1, '.'), (1,Path('.'),'',0))

@with_cast
def _g(a:int=0)->str: return a

test_eq(_g(4.0), '4')
test_eq(_g(4.4), '4')
test_eq(_g(2), '2')

store_attr

def store_attr(
    names:NoneType=None, but:str='', cast:bool=False, **attrs
):

Store params named in comma-separated names from calling context into attrs in self

store_attr reads the caller’s arguments by name from its frame, so the class below needs no assignments:

class Point:
    def __init__(self, x, y, scale=1): store_attr()
p = Point(1, 2)
test_eq((p.x, p.y, p.scale), (1, 2, 1))

In it’s most basic form, you can use store_attr to shorten code like this:

class T:
    def __init__(self, a,b,c): self.a,self.b,self.c = a,b,c

…to this:

class T:
    def __init__(self, a,b,c): store_attr('a,b,c', self)

This class behaves as if we’d used the first form:

t = T(1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2

class T1:
    def __init__(self, a,b,c): store_attr()

init_args

def init_args(
    o
):

The __init__ parameters of o that it holds as attributes, with their current values

init_args reads an object’s constructor arguments back from the attributes it stored, which is what a class using store_attr holds. Class attributes and methods that share a parameter’s name are left out. Use it to display or log how an object was made:

t = T1(1,c=2,b=3)
test_eq(init_args(t), dict(a=1, b=3, c=2))
init_args(t)

Since you normally want to use the first argument (often called self) for storing attributes, it’s optional:

class T:
    def __init__(self, a,b,c:str): store_attr('a,b,c')

t = T(1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2

With cast=True any parameter annotations will be used as preprocessing functions for the corresponding arguments:

class T:
    def __init__(self, a:listify, b, c:str): store_attr('a,b,c', cast=True)

t = T(1,c=2,b=3)
assert t.a==[1] and t.b==3 and t.c=='2'

You can inherit from a class using store_attr, and just call it again to add in any new attributes added in the derived class:

class T2(T):
    def __init__(self, d, **kwargs):
        super().__init__(**kwargs)
        store_attr('d')

t = T2(d=1,a=2,b=3,c=4)
assert t.a==2 and t.b==3 and t.c==4 and t.d==1

You can skip passing a list of attrs to store. In this case, all arguments passed to the method are stored:

class T:
    def __init__(self, a,b,c): store_attr()

t = T(1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2

class T4(T):
    def __init__(self, d, **kwargs):
        super().__init__(**kwargs)
        store_attr()

t = T4(4, a=1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2 and t.d==4

class T4:
    def __init__(self, *, a: int, b: float = 1): store_attr()
        
t = T4(a=3)
assert t.a==3 and t.b==1
t = T4(a=3, b=2)
assert t.a==3 and t.b==2

You can skip some attrs by passing but:

class T:
    def __init__(self, a,b,c): store_attr(but='a')

t = T(1,c=2,b=3)
assert t.b==3 and t.c==2
assert not hasattr(t,'a')

You can also pass keywords to store_attr, which is identical to setting the attrs directly:

class T:
    def __init__(self): store_attr(a=1)

t = T()
assert t.a==1

You can also use store_attr inside functions.

def create_T(a, b):
    t = SimpleNamespace()
    store_attr(self=t)
    return t

t = create_T(a=1, b=2)
assert t.a==1 and t.b==2

attrdict

def attrdict(
    o, *ks, default:NoneType=None
):

Dict from each k in ks to getattr(o,k)

attrdict picks named attributes into a dict, with default for any that are missing:

class T:
    def __init__(self, a,b,c): store_attr()

t = T(1,c=2,b=3)
test_eq(attrdict(t,'b','c'), {'b':3, 'c':2})

properties

def properties(
    cls, *ps
):

Change attrs in cls with names in ps to properties

properties turns the named methods of cls into properties in place:

class T:
    def a(self): return 1
    def b(self): return 2
properties(T,'a')

test_eq(T().a,1)
test_eq(T().b(),2)

The case-convention functions below share one splitter. id_words segments an identifier into its words and never changes case: punctuation characters in splits are treated as separators, and the letter c enables splitting at case boundaries, including after acronym runs. Constrain splits when a convention gives characters different meanings, such as CSS pseudo-selectors where : must not split. Separator runs come back as empty words, so names like __init__ survive a round trip.

id_words

def id_words(
    s:str, splits:str='-_ :c'
)->list:

Split identifier s into words: punctuation chars in splits are separators (runs kept as empty words, so they survive a round trip), ‘c’ splits case boundaries

test_eq(id_words('beforeRequest'), ['before','Request'])
test_eq(id_words('validation:validate'), ['validation','validate'])
test_eq(id_words('HTMLElement'), ['HTML','Element'])
test_eq(id_words('HX-Request-Type'), ['HX','Request','Type'])
test_eq(id_words('a__b'), ['a','','b'])
test_eq(id_words('a-b_c', 'c'), ['a-b_c'])
id_words('parseHTMLTree')

Each to_* emitter owns its casing policy: to_camel keeps acronym words intact (so an already-camel name round-trips exactly, which HtmxOn-style consumers rely on), to_pascal Title-cases every word (the historic snake2camel output), and to_kebab/to_snake lowercase throughout.

to_snake

def to_snake(
    s:str, splits:str='-_ :c'
)->str: # chkstyle: ignore

snake_case form of s

to_kebab

def to_kebab(
    s:str, splits:str='-_ :c'
)->str:

kebab-case form of s

to_pascal

def to_pascal(
    s:str, splits:str='-_ :c'
)->str:

PascalCase form of s; every word Title-cased (acronyms flatten)

to_camel

def to_camel(
    s:str, splits:str='-_ :c'
)->str:

camelCase form of s; acronym words after the first keep their casing

test_eq(to_camel('before-request'), 'beforeRequest')
test_eq(to_camel('parseHTMLTree'), 'parseHTMLTree')
test_eq(to_pascal('a_b_cc'), 'ABCc')
test_eq(to_pascal('parseHTMLTree'), 'ParseHtmlTree')
test_eq(to_kebab('beforeRequest'), 'before-request')
test_eq(to_kebab('noSSESourceError'), 'no-sse-source-error')
test_eq(to_snake('HX-Request-Type'), 'hx_request_type')
test_eq(to_snake('__init__'), '__init__')
test_eq(to_kebab('a__b'), 'a--b')
to_camel('no-sse-source-error'), to_kebab('validation:validate')

camel2words

def camel2words(
    s, space:str=' '
):

Convert CamelCase to ‘spaced words’

camel2words splits on case changes only, so hyphens and underscores stay as they are:

test_eq(camel2words('ClassAreCamel'), 'Class Are Camel')

camel2snake

def camel2snake(
    name
):

Convert CamelCase to snake_case

camel2snake is to_snake splitting on case changes only:

test_eq(camel2snake('ClassAreCamel'), 'class_are_camel')
test_eq(camel2snake('Already_Snake'), 'already_snake')

snake2camel

def snake2camel(
    s
):

Convert snake_case to CamelCase

snake2camel is to_pascal splitting on underscores only, so the first letter is capitalised too:

test_eq(snake2camel('a_b_cc'), 'ABCc')

humanize

def humanize(
    x
):

Concise human-readable x, e.g. 9200 -> ‘9.2k’

humanize scales by thousands up to T, keeps one decimal, and drops a trailing .0:

test_eq(humanize(950), '950')
test_eq(humanize(0), '0')
test_eq(humanize(9200), '9.2k')
test_eq(humanize(61234), '61.2k')
test_eq(humanize(1_300_000), '1.3M')
test_eq(humanize(-2500), '-2.5k')
test_eq(humanize(2_000_000_000_000_000), '2000T')

class2attr

def class2attr(
    cls_name
):

Return the snake-cased name of the class; strip ending cls_name if it exists.

class2attr snake-cases the class name, dropping cls_name when it is the suffix, which suits naming an attribute after a subclass:

class Parent:
    @property
    def name(self): return class2attr(self, 'Parent')

class ChildOfParent(Parent): pass
class ParentChildOf(Parent): pass

A trailing Parent is dropped and the rest snake-cased, while a leading one stays:

p = Parent()
cp = ChildOfParent()
cp2 = ParentChildOf()

test_eq(p.name, 'parent')
test_eq(cp.name, 'child_of')
test_eq(cp2.name, 'parent_child_of')

getcallable

def getcallable(
    o, attr
):

Calls getattr with a default of noop

getcallable returns noop when the attribute is missing, so the result can always be called:

class Math:
    def addition(self,a,b): return a+b

m = Math()

test_eq(getcallable(m, "addition")(a=1,b=2), 3)
test_eq(getcallable(m, "subtraction")(a=1,b=2), None)

getattrs

def getattrs(
    o, *attrs, default:NoneType=None
):

List of all attrs in o

getattrs reads several attributes at once, in order:

from fractions import Fraction

getattrs(Fraction(1,2), 'numerator', 'denominator')

[1, 2]

hasattrs

def hasattrs(
    o, attrs
):

Test whether o contains all attrs

hasattrs is hasattr over a sequence of names:

assert hasattrs(1,('imag','real'))
assert not hasattrs(1,('imag','foo'))

setattrs

def setattrs(
    dest, flds, src
):

Set fields flds on dest from src, a dict or an object

setattrs copies the comma-separated fields flds from src, a dict or an object, onto dest:

d = dict(a=1,bb="2",ignore=3)
o = SimpleNamespace()
setattrs(o, "a,bb", d)
test_eq(o.a, 1)
test_eq(o.bb, "2")

d = SimpleNamespace(a=1,bb="2",ignore=3)
o = SimpleNamespace()
setattrs(o, "a,bb", d)
test_eq(o.a, 1)
test_eq(o.bb, "2")

try_attrs

def try_attrs(
    obj, *attrs
):

Return first attr that exists in obj

try_attrs returns the first attribute found, and raises AttributeError naming all of them when none is:

test_eq(try_attrs(1, 'real'), 1)
test_eq(try_attrs(1, 'foobar', 'real'), 1)

DepProp

def DepProp(
    fchange, fnorm:NoneType=None
):

Property decorator with dependency update triggering

DepProp is a descriptor that stores a value and calls a “change” function whenever that value changes or is deleted. This is useful for invalidating caches or triggering side effects when a dependency is updated. An optional normalizer can preprocess values before storage.

class Square:
    @DepProp
    def width(self): self._area_cache = None

    @property
    def area(self):
        if self._area_cache is None: self._area_cache = self.width**2
        return self._area_cache

r = Square()
r.width = 3
test_eq(r.area, 9)
r.width = 5
test_eq(r.area, 25)

Use the .norm decorator to add a normalizer that preprocesses values before storage:

class T:
    @DepProp
    def name(self): self.log = f'changed to {self.name}'

    @name.norm
    def name(self, v): return v.strip().lower()

t = T()
t.name = '  Hello '
test_eq(t.name, 'hello')
test_eq(t.log, 'changed to hello')

Setting the same value again does not trigger the change function:

t.log = 'not called'
t.name = 'hello'
test_eq(t.log, 'not called')

Deleting the property removes the backing attribute and calls fchange:

del t.name
test_eq(t.log, 'changed to None')
test_eq(t.name, None)

Attribute Delegation

GetAttrBase

def GetAttrBase(
    *args, **kwargs
):

Basic delegation of __getattr__ and __dir__

Inherit from GetAttr to have attr access passed down to an instance attribute. This makes it easy to create composites that don’t require callers to know about their components. For a more detailed discussion of how this works as well as relevant context, we suggest reading the delegated composition section of this blog article.

You can customise the behaviour of GetAttr in subclasses via; - _default - By default, this is set to 'default', so attr access is passed down to self.default - _default can be set to the name of any instance attribute that does not start with dunder __ - _xtra - By default, this is None, so all attr access is passed down - You can limit which attrs get passed down by setting _xtra to a list of attribute names

GetAttr

def GetAttr(
    *args, **kwargs
):

Inherit from this to have all attr accesses in self._xtra passed down to self.default

To illuminate the utility of GetAttr, suppose we have the following two classes, _WebPage which is a superclass of _ProductPage, which we wish to compose like so:

class _WebPage:
    def __init__(self, title, author="Jeremy"): self.title,self.author = title,author

class _ProductPage:
    def __init__(self, page, price): self.page,self.price = page,price
        
page = _WebPage('Soap', author="Sylvain")
p = _ProductPage(page, 15.0)

How do we make it so we can just write p.author, instead of p.page.author to access the author attribute? We can use GetAttr, of course! First, we subclass GetAttr when defining _ProductPage. Next, we set self.default to the object whose attributes we want to be able to access directly, which in this case is the page argument passed on initialization:

class _ProductPage(GetAttr):
    def __init__(self, page, price): self.default,self.price = page,price

p = _ProductPage(page, 15.0)

Now, we can access the author attribute directly from the instance:

test_eq(p.author, 'Sylvain')

If you wish to store the object you are composing in an attribute other than self.default, you can set the class attribute _data as shown below. This is useful in the case where you might have a name collision with self.default:

class _C(GetAttr):
    _default = '_data'
    def __init__(self,a): self._data = a
    def foo(self): noop

t = _C('Hi')
test_eq(t._data, 'Hi') 
with expect_fail(): t.default
test_eq(t.lower(), 'hi')
test_eq(t.upper(), 'HI')
assert 'lower' in dir(t)
assert 'upper' in dir(t)

By default, all attributes and methods of the object you are composing are retained. In the below example, we compose a str object with the class _C. This allows us to directly call string methods on instances of class _C, such as str.lower() or str.upper():

class _C(GetAttr):
    def __init__(self,a): self.default = a
    def foo(self): noop

t = _C('Hi')
test_eq(t.lower(), 'hi')
test_eq(t.upper(), 'HI')
assert 'lower' in dir(t)
assert 'upper' in dir(t)

However, you can choose which attributes or methods to retain by defining a class attribute _xtra, which is a list of allowed attribute and method names to delegate. In the below example, we only delegate the lower method from the composed str object when defining class _C:

class _C(GetAttr):
    _xtra = ['lower']
    def __init__(self,a): self.default = a
    def foo(self): noop

t = _C('Hi')
test_eq(t.default, 'Hi')
test_eq(t.lower(), 'hi')
with expect_fail(): t.upper()
assert 'lower' in dir(t)
assert 'upper' not in dir(t)

You must be careful to properly set an instance attribute in __init__ that corresponds to the class attribute _default. The below example sets the class attribute _default to data, but erroneously fails to define self.data (and instead defines self.default).

Failing to properly set instance attributes leads to errors when you try to access methods directly:

class _C(GetAttr):
    _default = 'data'
    def __init__(self,a): self.default = a
    def foo(self): noop
        
t = _C('Hi')
test_eq(t.default, 'Hi')
with expect_fail(): t.data
with expect_fail(): t.lower()
with expect_fail(): t.upper()
with expect_fail(): dir(t)

delegate_attr

def delegate_attr(
    k, to
):

Use in __getattr__ to delegate to attr to without inheriting from GetAttr

delegate_attr is a functional way to delegate attributes, and is an alternative to GetAttr. We recommend reading the documentation of GetAttr for more details around delegation.

You can use achieve delegation when you define __getattr__ by using delegate_attr:

class _C:
    def __init__(self, o): self.o = o
    def __getattr__(self, k): return delegate_attr(self, k, to='o')
    

t = _C('HELLO')
test_eq(t.lower(), 'hello')

t = _C(np.array([5,4,3]))
test_eq(t.sum(), 12)

t = _C(pd.DataFrame({'a': [1,2], 'b': [3,4]}))
test_eq(t.b.max(), 4)

Extensible Types

ShowPrint is a base class that defines a show method, which is used primarily for callbacks in fastai that expect this method to be defined.

Int, Float, and Str extend int, float and str respectively by adding an additional show method by inheriting from ShowPrint.

The code for Int is shown below:

Examples:

Int(0).show()
Float(2.0).show()
Str('Hello').show()

0
2.0
Hello

Collection functions

Functions that manipulate popular python collections.

partition

def partition(
    coll, f
):

Partition a collection by a predicate

ts,fs = partition(range(10), mod(2))
test_eq(fs, [0,2,4,6,8])
test_eq(ts, [1,3,5,7,9])

partition_dict

def partition_dict(
    d, f
):

Partition a dict by a predicate that takes key/value params

partition_dict returns the matching pairs first, then the rest, keeping each dict’s order:

d = dict(a=1, b=2, c=3, d=4)
ts,fs = partition_dict(d, lambda k,v: v%2)
test_eq(fs, {'b':2, 'd':4})
test_eq(ts, {'a':1, 'c':3})

ts,fs = partition_dict(d, lambda k,v: k in 'bc')
test_eq(ts, {'b':2, 'c':3})
test_eq(fs, {'a':1, 'd':4})

flatten

def flatten(
    o
):

Concatenate all collections and items as a generator

flatten recurses into nested collections, leaving strings whole, and yields a flat stream. concat collects it into a list:

concat

def concat(
    colls
)->list:

Concatenate all collections and items as a list

concat([(o for o in range(2)),[2,3,4], 5])

[0, 1, 2, 3, 4, 5]

concat([["abc", "xyz"], ["foo", "bar"]])

['abc', 'xyz', 'foo', 'bar']

strcat

def strcat(
    its, sep:str=''
)->str:

Concatenate stringified items its

strcat stringifies each item before joining:

test_eq(strcat(['a',2]), 'a2')
test_eq(strcat(['a',2], ';'), 'a;2')

detuplify

def detuplify(
    x
):

If x is a tuple with one thing, extract it

detuplify unwraps a single item, gives None for an empty tuple, and leaves longer tuples and multi-dimensional arrays alone:

test_eq(detuplify(()),None)
test_eq(detuplify([1]),1)
test_eq(detuplify([1,2]), [1,2])
test_eq(detuplify(np.array([[1,2]])), np.array([[1,2]]))

replicate

def replicate(
    item, match
):

Create tuple of item copied len(match) times

replicate repeats item to the length of match:

t = [1,1]
test_eq(replicate([1,2], t),([1,2],[1,2]))
test_eq(replicate(1, t),(1,1))

setify

def setify(
    o
):

Turn any list like-object into a set.

setify accepts None, a string, which is kept whole, or any iterable:

test_eq(setify(None),set())
test_eq(setify('abc'),{'abc'})
test_eq(setify([1,2,2]),{1,2})
test_eq(setify(range(0,3)),{0,1,2})
test_eq(setify({1,2}),{1,2})

merge

def merge(
    *ds
):

Merge all dictionaries in ds

merge combines dicts left to right, with later keys winning, and skips None:

test_eq(merge(), {})
test_eq(merge(dict(a=1,b=2)), dict(a=1,b=2))
test_eq(merge(dict(a=1,b=2), dict(b=3,c=4), None), dict(a=1, b=3, c=4))

groupby

def groupby(
    x, key, val:function=noop
):

Like itertools.groupby but doesn’t need to be sorted, and isn’t lazy, plus some extensions

The result is a dict from key to list of values, in first-seen order:

test_eq(groupby('aa ab bb'.split(), itemgetter(0)), {'a':['aa','ab'], 'b':['bb']})

You can use an int as key or val (which uses itemgetter; passing a str will use attrgetter), eg:

test_eq(groupby('aa ab bb'.split(), 0), {'a':['aa','ab'], 'b':['bb']})

…and you can use a tuple as key or val (which creates a tuple from the provided keys or vals), eg:

test_eq(groupby('aaa abc bba'.split(), 0, (1,2)), {'a':[('a','a'),('b','c')], 'b':[('b','a')]})

Here’s an example of how to invert a grouping, and using a val function:

d = {0: [1, 3, 7], 2: [3], 3: [5], 4: [8], 5: [4], 7: [5]}
groupby(((o,k) for k,v in d.items() for o in v), 0, 1)

{1: [0], 3: [0, 2], 7: [0], 5: [3, 7], 8: [4], 4: [5]}

last_index

def last_index(
    x, o
):

Finds the last index of occurence of x in o (returns -1 if no occurence)

last_index searches from the end and gives -1 when x is absent:

test_eq(last_index(9, [1, 2, 9, 3, 4, 9, 10]), 5)
test_eq(last_index(6, [1, 2, 9, 3, 4, 9, 10]), -1)

filter_dict

def filter_dict(
    d, func
):

Filter a dict using func, applied to keys and values

filter_dict, filter_keys and filter_values differ only in what the predicate sees:

letters = {o:chr(o) for o in range(65,73)}
letters

{65: 'A', 66: 'B', 67: 'C', 68: 'D', 69: 'E', 70: 'F', 71: 'G', 72: 'H'}

filter_dict(letters, lambda k,v: k<67 or v in 'FG')

{65: 'A', 66: 'B', 70: 'F', 71: 'G'}

filter_keys

def filter_keys(
    d, func
):

Filter a dict using func, applied to keys

filter_keys passes only the key:

filter_keys(letters, lt(67))

{65: 'A', 66: 'B'}

filter_values

def filter_values(
    d, func
):

Filter a dict using func, applied to values

filter_values passes only the value:

filter_values(letters, in_('FG'))

{70: 'F', 71: 'G'}

cycle

def cycle(
    o
):

Like itertools.cycle except creates list of Nones if o is empty

cycle never stops, and an empty input cycles None rather than raising:

test_eq(itertools.islice(cycle([1,2,3]),5), [1,2,3,1,2])
test_eq(itertools.islice(cycle([]),3), [None]*3)
test_eq(itertools.islice(cycle(None),3), [None]*3)
test_eq(itertools.islice(cycle(1),3), [1,1,1])

zip_cycle

def zip_cycle(
    x, *args
):

Like itertools.zip_longest but cycles through elements of all but first argument

zip_cycle runs to the length of the first argument, repeating the others as needed:

test_eq(zip_cycle([1,2,3,4],list('abc')), [(1, 'a'), (2, 'b'), (3, 'c'), (4, 'a')])

sorted_ex

def sorted_ex(
    iterable, key:NoneType=None, reverse:bool=False, cmp:NoneType=None, **kwargs
):

Like sorted, but if key is str use attrgetter; if int use itemgetter; use cmp comparator function or key with kwargs

Attributes can be used for sorting by passing their name as a string:

class TestObj:
    def __init__(self, x): self.x = x
objs = [TestObj(i) for i in [3,1,2]]
test_eq([o.x for o in sorted_ex(objs, 'x')], [1,2,3])

Tuple/list items can be sorted by index position:

items = [(1,'c'), (2,'b'), (3,'a')]
test_eq(sorted_ex(items, 1), [(3,'a'), (2,'b'), (1,'c')])

A custom key function transforms values:

test_eq(sorted_ex([3,1,2], lambda x: -x), [3,2,1])

You can use a comparison function (returning -1/1/0):

test_eq(sorted_ex([3,1,2], cmp=lambda a,b: 1 if a>b else -1 if a<b else 0), [1,2,3])

Additional parameters can be passed to key/cmp functions:

def key_with_kwargs(x, offset=0): return x + offset
test_eq(sorted_ex([3,1,2], key=key_with_kwargs, offset=10), [1,2,3])

Reverse sort capability:

test_eq(sorted_ex([1,2,3], reverse=True), [3,2,1])

not_

def not_(
    f
):

Create new function that negates result of f

not_ wraps a predicate so it answers the opposite:

def f(a): return a>0
test_eq(f(1),True)
test_eq(not_(f)(1),False)
test_eq(not_(f)(a=-1),True)

argwhere

def argwhere(
    iterable, f, negate:bool=False, **kwargs
):

Like filter_ex, but return indices for matching items

argwhere gives the positions of matching items, with the same negate and kwargs handling as filter_ex:

test_eq(argwhere([1,2,3,4], gt(2)), [2,3])

filter_ex

def filter_ex(
    iterable, f:function=noop, negate:bool=False, gen:bool=False, **kwargs
):

Like filter, but passing kwargs to f, defaulting f to noop, and adding negate and gen

filter_ex passes kwargs through to f, and negate inverts the test:

test_eq(filter_ex([1,2,3,4], gt(2), negate=True), [1,2])

range_of

def range_of(
    a, b:NoneType=None, step:NoneType=None
):

All indices of collection a, if a is a collection, otherwise range

range_of gives the indices of a collection, or behaves as range when given numbers:

test_eq(range_of([1,1,1,1]), [0,1,2,3])
test_eq(range_of(4), [0,1,2,3])

renumerate

def renumerate(
    iterable, start:int=0
):

Same as enumerate, but returns index as 2nd element instead of 1st

renumerate is enumerate with the item first and its index second:

test_eq(renumerate('abc'), (('a',0),('b',1),('c',2)))

first

def first(
    x, f:NoneType=None, negate:bool=False, **kwargs
):

First element of x, optionally filtered by f, or None if missing

first takes any iterable and returns None when nothing qualifies, so it never raises:

test_eq(first(['a', 'b', 'c', 'd', 'e']), 'a')
test_eq(first([False]), False)
test_eq(first([False], noop), None)

last

def last(
    x, f:NoneType=None, negate:bool=False, **kwargs
):

Last element of x, optionally filtered by f, or None if missing

last walks the whole iterable, so it works on generators too:

test_eq(last(['a', 'b', 'c', 'd', 'e']), 'e')
test_eq(last([False]), False)
test_eq(last([False], noop), None)

only

def only(
    o
):

Return the only item of o, raise if o doesn’t have exactly one item

only insists on exactly one item, and its error says whether there were none or several:

nested_attr

def nested_attr(
    o, attr, default:NoneType=None
):

Same as getattr, but if attr includes a ., then looks inside nested objects

nested_attr follows a dotted path through attributes or keys, returning default on the first miss:

class CustomIndexable:
    def __init__(self): self.data = dict(a=1, b='v', c={'d':5})
    def __getitem__(self, key): return self.data[key]

custom_indexable = CustomIndexable()
test_eq(nested_attr(custom_indexable,'a'),1)
test_eq(nested_attr(custom_indexable,'c.d'),5)
test_eq(nested_attr(custom_indexable,'e'),None)

class TestObj: def init(self): self.nested = {‘key’: [1, 2, {‘inner’: ‘value’}]} test_obj = TestObj()

test_eq(nested_attr(test_obj, ‘nested.key.2.inner’),‘value’) test_eq(nested_attr([1, 2, 3], ‘1’),2)

b = dict(a=1, b='v', c={'d':5})
test_eq(nested_attr(b,'b'),'v')
test_eq(nested_attr(b,'c.d'),5)

a = SimpleNamespace(b=(SimpleNamespace(c=1)))
test_eq(nested_attr(a, 'b.c'), getattr(getattr(a, 'b'), 'c'))
test_eq(nested_attr(a, 'b.d'), None)
test_eq(nested_attr(b, 'a'), 1)

nested_setdefault

def nested_setdefault(
    o, attr, default
):

Same as setdefault, but if attr includes a ., then looks inside nested objects

nested_setdefault creates each missing level as an empty container of the same type as o:

nested_callable

def nested_callable(
    o, attr
):

Same as nested_attr but if not found will return noop

nested_callable is nested_attr defaulting to noop, so the result can always be called:

a = SimpleNamespace(b=(SimpleNamespace(c=1)))
test_eq(nested_callable(a, 'b.c'), getattr(getattr(a, 'b'), 'c'))
test_eq(nested_callable(a, 'b.d'), noop)

nested_idx

def nested_idx(
    coll, *idxs
):

Index into nested collections, dicts, etc, with idxs

nested_idx walks idxs through nested dicts, lists and objects, returning None for a missing step:

a = {'b':[1,{'c':2}]}
test_eq(nested_idx(a, 'nope'), None)
test_eq(nested_idx(a, 'nope', 'nup'), None)
test_eq(nested_idx(a, 'b', 3), None)
test_eq(nested_idx(a), a)
test_eq(nested_idx(a, 'b'), [1,{'c':2}])
test_eq(nested_idx(a, 'b', 1), {'c':2})
test_eq(nested_idx(a, 'b', 1, 'c'), 2)

a = SimpleNamespace(b=[1,{'c':2}])
test_eq(nested_idx(a, 'nope'), None)
test_eq(nested_idx(a, 'nope', 'nup'), None)
test_eq(nested_idx(a, 'b', 3), None)
test_eq(nested_idx(a), a)
test_eq(nested_idx(a, 'b'), [1,{'c':2}])
test_eq(nested_idx(a, 'b', 1), {'c':2})
test_eq(nested_idx(a, 'b', 1, 'c'), 2)

set_nested_idx

def set_nested_idx(
    coll, value, *idxs
):

Set value indexed like `nested_idx

set_nested_idx assigns at the same kind of path:

set_nested_idx(a, 3, 'b', 0)
test_eq(nested_idx(a, 'b', 0), 3)

val2idx

def val2idx(
    x
):

Dict from value to index

val2idx inverts a sequence into a value-to-position dict:

test_eq(val2idx([1,2,3]), {3:2,1:0,2:1})

uniqueify

def uniqueify(
    x, sort:bool=False, bidir:bool=False, start:NoneType=None
):

Unique elements in x, optional sort, optional return reverse correspondence, optional prepend with elements.

uniqueify keeps first occurrences in order. sort sorts them, start prepends items, and bidir also returns the value-to-index dict:

t = [1,1,0,5,0,3]
test_eq(uniqueify(t),[1,0,5,3])
test_eq(uniqueify(t, sort=True),[0,1,3,5])
test_eq(uniqueify(t, start=[7,8,6]), [7,8,6,1,0,5,3])
v,o = uniqueify(t, bidir=True)
test_eq(v,[1,0,5,3])
test_eq(o,{1:0, 0: 1, 5: 2, 3: 3})
v,o = uniqueify(t, sort=True, bidir=True)
test_eq(v,[0,1,3,5])
test_eq(o,{0:0, 1: 1, 3: 2, 5: 3})

loop_first_last

def loop_first_last(
    values
):

Iterate and generate a tuple with a flag for first and last value.

loop_first_last yields (is_first, is_last, value), which saves a length check when rendering sequences:

test_eq(loop_first_last(range(3)), [(True,False,0), (False,False,1), (False,True,2)])

loop_first

def loop_first(
    values
):

Iterate and generate a tuple with a flag for first value.

loop_first keeps just the first flag:

test_eq(loop_first(range(3)), [(True,0), (False,1), (False,2)])

loop_last

def loop_last(
    values
):

Iterate and generate a tuple with a flag for last value.

loop_last keeps just the last flag:

test_eq(loop_last(range(3)), [(False,0), (False,1), (True,2)])

first_match

def first_match(
    lst, f, default:NoneType=None
):

Index of the first element of lst matching predicate f, or default if none

first_match returns the index of the first match, not the element:

a = [0,2,4,5,6,7,10]
test_eq(first_match(a, lambda o:o%2), 3)

last_match

def last_match(
    lst, f, default:NoneType=None
):

Index of the last element of lst matching predicate f, or default if none

last_match searches from the end:

test_eq(last_match(a, lambda o:o%2), 5)

joins

def joins(
    sep, its
):

Sugar for sep.join(map(str, its))

Plain str.join needs strings, so ','.join([1,2,3]) raises a TypeError. joins maps str over the items first:

joins(",", [1,2,3])

'1,2,3'

fastuple

A tuple with extended functionality.

fastuple takes its items as separate arguments, and every arithmetic and comparison operator in num_methods is applied elementwise, cycling shorter arguments:

fastuple

def fastuple(
    *args, **kwargs
):

A tuple with elementwise ops and more friendly init behavior

Friendly init behavior

Common failure modes when trying to initialize a tuple in python:

tuple(3)
> TypeError: 'int' object is not iterable

or

tuple(3, 4)
> TypeError: tuple expected at most 1 arguments, got 2

However, fastuple allows you to define tuples like this and in the usual way:

test_eq(fastuple(3), (3,))
test_eq(fastuple(3,4), (3, 4))
test_eq(fastuple((3,4)), (3, 4))

Elementwise operations

fastuple.add

def add(
    *args
):

+ is already defined in tuple for concat, so use add instead

test_eq(fastuple.add((1,1),(2,2)), (3,3))
test_eq_type(fastuple(1,1).add(2), fastuple(3,3))
test_eq(fastuple('1','2').add('2'), fastuple('12','22'))

mul scales elementwise, by another tuple or by a scalar:

fastuple.mul

def mul(
    *args
):

* is already defined in tuple for replicating, so use mul instead

test_eq_type(fastuple(1,1).mul(2), fastuple(2,2))

Other Elementwise Operations

Additionally, the following elementwise operations are available: - le: less than or equal - eq: equal - gt: greater than - min: minimum of

test_eq(fastuple(3,1).le(1), (False, True))
test_eq(fastuple(3,1).eq(1), (False, True))
test_eq(fastuple(3,1).gt(1), (True, False))
test_eq(fastuple(3,1).min(2), (2,1))

You can also do other elementwise operations like negate a fastuple, or subtract two fastuples:

test_eq(-fastuple(1,2), (-1,-2))
test_eq(~fastuple(1,0,1), (False,True,False))

test_eq(fastuple(1,1)-fastuple(2,2), (-1,-1))

test_eq(type(fastuple(1)), fastuple)
test_eq_type(fastuple(1,2), fastuple(1,2))
test_ne(fastuple(1,2), fastuple(1,3))
test_eq(fastuple(), ())

Functions on Functions

Tools for making and transforming functions: compose(f,g,...) chains functions left to right; bind is partial extended with arg0,arg1,… placeholders for reordering positional arguments; and fail_clean re-raises exceptions with the library’s own traceback frames stripped, for errors that are part of a function’s contract rather than bugs.

Utilities for functional programming or for defining, modifying, or debugging functions.

bind is the same as partial, but also allows you to reorder positional arguments using variable name(s) arg{i} where i refers to the zero-indexed positional argument. bind as implemented currently only supports reordering of up to the first 5 positional arguments.

Consider the function myfunc below, which has 3 positional arguments. These arguments can be referenced as arg0, arg1, and arg1, respectively.

bind

def bind(
    func, *pargs, **pkwargs
):

Same as partial, except you can use arg0 arg1 etc param placeholders

def myfn(a,b,c,d=1,e=2): return(a,b,c,d,e)

In the below example we bind the positional arguments of myfn as follows:

The second input 14, referenced by arg1, is substituted for the first positional argument.
We supply a default value of 17 for the second positional argument.
The first input 19, referenced by arg0, is subsituted for the third positional argument.

test_eq(bind(myfn, arg1, 17, arg0, e=3)(19,14), (14,17,19,1,3))

In this next example:

We set the default value to 17 for the first positional argument.
The first input 19 refrenced by arg0, becomes the second positional argument.
The second input 14 becomes the third positional argument.
We override the default the value for named argument e to 3.

test_eq(bind(myfn, 17, arg0, e=3)(19,14), (17,19,14,1,3))

This is an example of using bind like partial and do not reorder any arguments:

test_eq(bind(myfn)(17,19,14), (17,19,14,1,2))

bind can also be used to change default values. In the below example, we use the first input 3 to override the default value of the named argument e, and supply default values for the first three positional arguments:

test_eq(bind(myfn, 17,19,14,e=arg0)(3), (17,19,14,1,3))

Every arg{i} placeholder refers to the i-th call-time argument, in positional and keyword bindings alike; arguments past the highest placeholder index are appended:

test_eq(bind(myfn, arg1, d=arg0)(19,14,8,5), (14,8,5,19,2))
test_eq(bind(myfn, c=arg0, d=arg1)(19,14,3,7), (3,7,19,14,2))
test_eq(bind(myfn, arg0, arg0)(3,5), (3,3,5,1,2))

mapt

def mapt(
    func, *iterables
):

Tuplified map

mapt is map returning a tuple:

t = [0,1,2,3]
test_eq(mapt(operator.neg, t), (0,-1,-2,-3))

map_ex

def map_ex(
    iterable, f, *args, gen:bool=False, **kwargs
):

Like map, but use bind, and supports str and indexing

map_ex returns a list by default, and f may be a callable, a format string, or anything indexable:

test_eq(map_ex(t,operator.neg), [0,-1,-2,-3])

If f is a string then it is treated as a format string to create the mapping:

test_eq(map_ex(t, '#{}#'), ['#0#','#1#','#2#','#3#'])

If f is a dictionary (or anything supporting __getitem__) then it is indexed to create the mapping:

test_eq(map_ex(t, list('abcd')), list('abcd'))

You can also pass the same arg params that bind accepts:

def f(a=None,b=None): return b
test_eq(map_ex(t, f, b=arg0), range(4))

compose

def compose(
    *funcs, order:NoneType=None
):

Create a function that composes all functions in funcs, passing along remaining *args and **kwargs to all

compose applies funcs left to right, threading the extra arguments into each call, and order sorts them by that attribute first:

f1 = lambda o,p=0: (o*2)+p
f2 = lambda o,p=1: (o+1)/p
test_eq(f2(f1(3)), compose(f1,f2)(3))
test_eq(f2(f1(3,p=3),p=3), compose(f1,f2)(3,p=3))
test_eq(f2(f1(3,  3),  3), compose(f1,f2)(3,  3))

f1.order = 1
test_eq(f1(f2(3)), compose(f1,f2, order="order")(3))

maps

def maps(
    *args, retain:function=noop
):

Like map, except funcs are composed first

maps composes all but the last argument and maps the result over the last, with retain deciding how each output relates to its input:

test_eq(maps([1]), [1])
test_eq(maps(operator.neg, [1,2]), [-1,-2])
test_eq(maps(operator.neg, operator.neg, [1,2]), [1,2])

partialler

def partialler(
    f, *args, order:NoneType=None, **kwargs
):

Like functools.partial but also copies over docstring

partialler is partial plus an order attribute, copied from f unless given, and it keeps f’s docstring:

def _f(x,a=1):
    "test func"
    return x-a
_f.order=1

f = partialler(_f, 2)
test_eq(f.order, 1)
test_eq(f(3), -1)

Keyword arguments and order can be given too, and the docstring comes across:

f = partialler(_f, a=2, order=3)
test_eq(f.__doc__, "test func")
test_eq(f.order, 3)
test_eq(f(3), _f(3,2))

class partial0:
    "Like `partialler`, but args passed to callable are inserted at started, instead of at end"
    def __init__(self, f, *args, order=None, **kwargs):
        self.f,self.args,self.kwargs = f,args,kwargs
        self.order = ifnone(order, getattr(f,'order',None))
        self.__doc__ = f.__doc__

    def __call__(self, *args, **kwargs): return self.f(*args, *self.args, **kwargs, **self.kwargs)

partial0 puts the call-time argument first, so f(3) is _f(3, 2), unlike the partialler example above:

f = partial0(_f, 2)
test_eq(f.order, 1)
test_eq(f(3), 1)

instantiate

def instantiate(
    t
):

Instantiate t if it’s a type, otherwise do nothing

instantiate calls a type and leaves an instance alone:

test_eq_type(instantiate(int), 0)
test_eq_type(instantiate(1), 1)

using_attr

def using_attr(
    f, attr
):

Construct a function which applies f to the argument’s attribute attr

using_attr lifts f to act on an attribute of its argument:

t = Path('/a/b.txt')
f = using_attr(str.upper, 'name')
test_eq(f(t), 'B.TXT')

negate

def negate(
    f
):

Returns the negation of f

negate also rewrites the docstring to say what it negates:

def true():
    'Returns True'
    return True

false = negate(true)
print(false.__doc__)
test_eq(false(), not true())

Returns `not true(...)`

Original: Returns True

fail_clean

def fail_clean(
    *excs
):

Re-raise excs (default: Exception) without internal traceback frames

fail_clean marks a boundary where errors are part of a function’s contract rather than something to debug: exceptions of the listed types (any Exception by default) are re-raised with the internal traceback frames removed, so the caller sees the message and their own call site instead of the library’s plumbing. The original exception object is preserved, including its attributes. Exceptions not listed keep their full stack for debugging. Use @fail_clean bare, or pass exception types like @fail_clean(ValueError):

The traceback is cut down to the caller and the wrapper, with the internal frames gone.

import traceback

def _boom(): raise ValueError('boom')
def _mid(): return _boom()

@fail_clean
def g(): return _mid()

try: g()
except ValueError as e:
    test_eq(str(e), 'boom')
    test_eq(e.__cause__, None)
    test_eq(len(traceback.extract_tb(e.__traceback__)), 2)

@fail_clean(KeyError)
def h(): return _mid()
try: h()
except ValueError as e: assert len(traceback.extract_tb(e.__traceback__)) > 2

dstar

def dstar(
    f
):

Wrap f to accept a single dict, unpacking it as keyword args

dstar wraps a function to accept a single dictionary, unpacking it as keyword arguments (the ** counterpart of star in fastcore.foundation). For instance:

def greet(name, greeting='Hello'): return f'{greeting}, {name}!'
configs = [{'name': 'Alice'}, {'name': 'Bob', 'greeting': 'Hi'}]
list(map(dstar(greet), configs))

['Hello, Alice!', 'Hi, Bob!']

A more realistic example showing API request configuration. Each request dictionary may have different keys present, and dspread handles this naturally (missing keys use the function’s defaults):

# chkstyle: skip
def api_request(endpoint, method='GET', timeout=30, headers=None):
    return f"{method} {endpoint} (timeout={timeout})"

requests = [
    {'endpoint': '/users', 'method': 'POST', 'timeout': 60},
    {'endpoint': '/data'},
    {'endpoint': '/health', 'method': 'HEAD', 'timeout': 5}
]
list(map(dstar(api_request), requests))

['POST /users (timeout=60)',
 'GET /data (timeout=30)',
 'HEAD /health (timeout=5)']

Self (with an uppercase S)

A Concise Way To Create Lambdas

_all_ adds Self to __all__, since it is made by instantiating _SelfCls rather than defined with def or class:

~Self is a concise alternative to lambda for a function that operates on a single object (note the capitalization!). Write the chain of attribute accesses, method calls, and indexing just as you would after a variable name, with ~Self in its place, and the result is a plain function that runs the chain on its argument:

~Self.sum() is lambda o: o.sum()
~Self.imag is lambda o: o.imag
~Self[1] is lambda o: o[1]
~Self.sum().real is lambda o: o.sum().real
~Self alone is the identity, lambda o: o

Since ., (), and [] bind tighter than ~, the whole chain builds first and ~ then converts it to a function, so the chain never needs its own parentheses: map(~Self.imag, nums) works as written.

The functions ~Self returns are ordinary functions: safe to introspect, and independent of any later chains built from the same prefix. Parens mean what they mean in plain Python: a step written with () is called, so a chain that calls a non-callable attribute raises TypeError rather than quietly fetching it:

Self.imag reads an attribute and Self.imag() calls it, as in plain Python, so calling a non-callable attribute fails with the usual TypeError.

test_eq((~Self.imag)(3), 0)
with expect_fail(TypeError, 'not callable'): (~Self.imag())(3)
test_eq(list(map(~Self.real, [3,4])), [3,4])
x = np.array([3.,1])
test_eq((~Self.sum().real)(x), 4.)
test_eq((~Self)(x) is x, True)
test_eq((~Self[1])(x), 1)
test_eq((~Self.strip()[0])(' abc '), 'a')

g1 = Self.strip()
g2 = g1.upper()
test_eq((~g1)('  hi  '), 'hi')
test_eq((~g2)('  hi  '), 'HI')

assert not hasattr(Self.sum(), '__wrapped__')

with expect_fail(TypeError, 'applied'): Self.strip()()
with expect_fail(TypeError, '~Self'): bool(Self.imag)

Calling Self directly adds a call step to the chain, so ~Self(1,b=2) is a function which calls whatever it’s applied to with those arguments:

def f(a, b=3): return a+b+2
def g(a, b=3): return a*b
fg = ~Self(1,b=2)
list(map(fg, [f,g]))

[5, 2]

Patching

@patch adds a function to an existing class as a method, using the function’s self: type annotation to pick the class (a union annotation patches several classes at once); @patch_to(Cls) is the same with the class passed explicitly. Both take as_prop, set_prop, and cls_method. fastai code uses this to build classes incrementally across a notebook, so expect to find a class’s methods defined far from the class itself:

copy_func

def copy_func(
    f
):

Copy a non-builtin function (NB copy.copy does not work for this)

Sometimes it may be desirable to make a copy of a function that doesn’t point to the original object. When you use Python’s built in copy.copy or copy.deepcopy to copy a function, you get a reference to the original object:

copy.copy and copy.deepcopy both return the same function object, so an attribute set on one shows up on the other.

import copy as cp

def foo(): pass
a = cp.copy(foo)
b = cp.deepcopy(foo)

a.someattr = 'hello'
test_eq(b.someattr, 'hello')

assert a is foo and b is foo

However, with copy_func, you can retrieve a copy of a function without a reference to the original object:

c = copy_func(foo)
assert c is not foo

def g(x, *, y=3): return x+y
test_eq(copy_func(g)(4), 7)

patch_to

def patch_to(
    cls, as_prop:bool=False, cls_method:bool=False, set_prop:bool=False, static_method:bool=False, nm:NoneType=None,
    glb:NoneType=None
):

Decorator: add f to cls

The @patch_to decorator allows you to monkey patch a function into a class as a method:

class _T3(int): pass  

@patch_to(_T3)
def func1(self, a): return self+a

t = _T3(1)
test_eq(t.func1(2), 3)

if sys.version_info >= (3,11): test_eq(_T3.func1.__code__.co_qualname, '_T3.func1')

You can access instance properties in the usual way via self:

class _T4():
    def __init__(self, g): self.g = g
        
@patch_to(_T4)
def greet(self, x): return self.g + x
        
t = _T4('hello ')
test_eq(t.greet('world'), 'hello world')

You can instead specify that the method should be a class method by setting cls_method=True:

class _T5(int): attr = 3
    
@patch_to(_T5, cls_method=True)
def func(cls, x): return cls.attr + x

test_eq(_T5.func(4), 7)

Additionally you can specify that the function you want to patch should be a class attribute with as_prop=True:

@patch_to(_T5, as_prop=True)
def add_ten(self): return self + 10

t = _T5(4)
test_eq(t.add_ten, 14)

Once you have a property, you can assign a setter with set_prop=True:

class _T2():
    def __init__(self, val): self._val = val

@patch_to(_T2, as_prop=True)
def val(self): return self._val

t = _T2(2)
test_eq(t.val, 2)

@patch_to(_T2, set_prop=True)
def val(self, val): self._val = val

t.val = 3
test_eq(t.val, 3)

Instead of passing one class to the @patch_to decorator, you can pass multiple classes in a tuple to simulteanously patch more than one class with the same method:

class _T6(int): pass
class _T7(int): pass

@patch_to((_T6,_T7))
def func_mult(self, a): return self*a

t = _T6(2)
test_eq(t.func_mult(4), 8)
t = _T7(2)
test_eq(t.func_mult(4), 8)

You can also rename the function in the patched class:

class _T8(int): pass  

@patch_to(_T8, nm='add_value')
def func2(self, a): return self+a

t = _T8(1)
test_eq(t.add_value(2), 3)
test_eq(_T8.add_value.__name__, 'add_value')
assert not hasattr(t, 'func2')

patch

def patch(
    f:NoneType=None, *, as_prop:bool=False, cls_method:bool=False, static_method:bool=False, set_prop:bool=False,
    nm:NoneType=None
):

Decorator: add f to the first parameter’s class (based on f’s type annotations)

patch finds the class from the first parameter’s annotation, and cls_method=True reads it from cls instead:

@patch
def total(self:Point): return self.x + self.y
test_eq(Point(3, 4).total(), 7)

@patch is an alternative to @patch_to that allows you similarly monkey patch class(es) by using type annotations:

class _T8(int): pass  

@patch
def func(self:_T8, a): return self+a

t = _T8(1)
test_eq(t.func(3), 4)
test_eq(t.func.__qualname__, '_T8.func')
if sys.version_info >= (3,11): test_eq(_T8.func.__code__.co_qualname, '_T8.func')

class MyMath: pass

@patch_to(MyMath, static_method=True)
def add(a, b): return a + b

@patch(static_method=True)
def mul(a:MyMath, b): return a * b

test_eq(MyMath.add(2, 3), 5)
test_eq(MyMath.mul(2, 3), 6)

Similarly to patch_to, you can supply a union of classes instead of a single class in your type annotations to patch multiple classes:

class _T9(int): pass 

@patch
def func2(x:_T8|_T9, a): return x*a

t = _T8(2)
test_eq(t.func2(4), 8)
test_eq(t.func2.__qualname__, '_T8.func2')

t = _T9(2)
test_eq(t.func2(4), 8)
test_eq(t.func2.__qualname__, '_T9.func2')

Just like patch_to decorator you can use as_prop, set_prop, and cls_method parameters with patch decorator:

@patch(as_prop=True)
def add_ten(self:_T5): return self + 10

t = _T5(4)
test_eq(t.add_ten, 14)

class _T2():
    def __init__(self, val): self._val = val

@patch(as_prop=True)
def val(self:_T2): return self._val

t = _T2(2)
test_eq(t.val, 2)

@patch(set_prop=True)
def val(self:_T2, val): self._val = val

t.val = 3
test_eq(t.val, 3)

class _T5(int): attr = 3
    
@patch(cls_method=True)
def func(cls:_T5, x): return cls.attr + x

test_eq(_T5.func(4), 7)

nm renames the patched method, as with patch_to:

class _T8(int): pass  

@patch(nm='add_value')
def func2(self:_T8, a): return self+a

t = _T8(1)
test_eq(t.add_value(2), 3)
test_eq(_T8.add_value.__name__, 'add_value')
assert not hasattr(t, 'func2')

Patching classmethod shouldn’t affect how python’s inheritance works

class FastParent: pass

@patch(cls_method=True)
def type_cls(cls: FastParent): return cls

class FastChild(FastParent): pass

parent = FastParent()
test_eq(parent.type_cls(), FastParent)

child = FastChild()
test_eq(child.type_cls(), FastChild)

It also works on iterable classes like Enums:

class Color(Enum):
    RED = 1
    GREEN = 2

@patch(cls_method=True)
def from_name(cls: Color, s): return cls[s.upper()]

test_eq(Color.from_name.__name__, 'from_name')
test_eq(Color.from_name('red'), Color.RED)
test_eq(Color.from_name('GREEN'), Color.GREEN)

Other Helpers

extend_enum

def extend_enum(
    cls, # Enum class to modify
    n, # Name of the new enum member
    v, # Value of the new enum member
):

Add new member n with value v to enum class cls at runtime

extend_enum mutates an existing enum class by constructing a new member, registering it in the enum’s internal lookup tables, and attaching it as a class attribute, so it behaves like a normal enum member created in the original class definition.

from enum import Enum

class Color(Enum): red = 1; blue = 2

extend_enum(Color, 'green', 3)
Color.green, Color['green'], Color(3)

(<Color.green: 3>, <Color.green: 3>, <Color.green: 3>)

compile_re

def compile_re(
    pat
):

Compile pat if it’s not None

compile_re lets None mean no pattern:

assert compile_re(None) is None
assert compile_re('a').match('ab')

imports() injects the members into the caller’s namespace, so they can be used as bare names:

ImportEnum

def ImportEnum(
    *args, **kwds
):

An Enum that can have its values imported

_T = ImportEnum('_T', {'foobar':1, 'goobar':2})
_T.imports()
test_eq(foobar, _T.foobar)
test_eq(goobar, _T.goobar)

StrEnum members are strings, and print as their names:

StrEnum

def StrEnum(
    *args, **kwds
):

An ImportEnum that behaves like a str

str_enum

def str_enum(
    name, *vals
):

Simplified creation of StrEnum types

str_enum makes a StrEnum whose values equal their names, registered in the caller’s module so that it pickles:

ValEnum prints the value rather than the name:

ValEnum

def ValEnum(
    *args, **kwds
):

An ImportEnum that stringifies using values

import pickle

_T = str_enum('_T', 'a', 'b')
test_eq(f'{_T.a}', 'a')
test_eq(_T.a, 'a')
test_eq(list(_T.__members__), ['a','b'])
test_eq(_T.__module__, __name__)
test_eq(pickle.loads(pickle.dumps(_T.a)), _T.a)
print(_T.a, _T.a.upper())

a A

Stateful keeps anything in self._state out of pickles, so unpicklable resources can be recreated on load:

Stateful

def Stateful(
    *args, **kwargs
):

A base class/mixin for objects that should not serialize all their state

class _T(Stateful):
    def __init__(self):
        super().__init__()
        self.a=1
        self._state['test']=2

After a pickle round trip, _state is empty while ordinary attributes survive:

t = _T()
t2 = pickle.loads(pickle.dumps(t))
test_eq(t.a,1)
test_eq(t._state['test'],2)
test_eq(t2.a,1)
test_eq(t2._state,{})

Override _init_state to do any necessary setup steps that are required during __init__ or during deserialization (e.g. pickle.load). Here’s an example of how Stateful simplifies the official Python example for Handling Stateful Objects.

class TextReader(Stateful):  # chkstyle: ignore
    'Print and number lines in a text file.'
    _stateattrs=('file',)
    def __init__(self, filename):
        self.filename,self.lineno = filename,0
        super().__init__()

    def readline(self):
        self.lineno += 1
        line = self.file.readline()
        if line: return f"{self.lineno}: {line.strip()}"

    def _init_state(self):
        self.file = open(self.filename)
        for _ in range(self.lineno): self.file.readline()

reader = TextReader("00_test.ipynb")
print(reader.readline())
print(reader.readline())

new_reader = pickle.loads(pickle.dumps(reader))
print(reader.readline())

1: {
2: "cells": [
3: {

NotStr

def NotStr(
    s
):

Behaves like a str, but isn’t an instance of one

NotStr forwards string methods through GetAttr while failing isinstance(s, str), for places that treat real strings specially:

s = NotStr("hello")
assert not isinstance(s, str)
test_eq(s, 'hello')
test_eq(s*2, 'hellohello')
test_eq(len(s), 5)
test_eq(s[:2], "he")
test_eq(s[2], "l")

Allow strings with special characters to render properly in Jupyter. Without calling print() strings with special characters are displayed like so:

PrettyString

def PrettyString(
    *args, **kwargs
):

Little hack to get strings to show properly in Jupyter.

with_special_chars='a string\nwith\nnew\nlines and\ttabs'
with_special_chars

'a string\nwith\nnew\nlines and\ttabs'

We can correct this with PrettyString:

PrettyString(with_special_chars)

a string
with
new
lines and   tabs

even_mults

def even_mults(
    start, stop, n
):

Build log-stepped array from start to stop in n steps.

even_mults spaces n values so each is the same multiple of the one before:

test_eq(even_mults(2,8,3), [2,4,8])
test_eq(even_mults(2,32,5), [2,4,8,16,32])
test_eq(even_mults(2,8,1), 8)

num_cpus

def num_cpus():

Get number of cpus

num_cpus respects the process’s CPU affinity where the platform reports it, and sets defaults.cpus:

num_cpus()

add_props

def add_props(
    f, g:NoneType=None, n:int=2
):

Create properties passing each of range(n) to f

add_props builds n properties in one go, each getter called with its index, with an optional setter:

class _T(): a,b = add_props(lambda i,x:i*2)

t = _T()
test_eq(t.a,0)
test_eq(t.b,2)

class _T(): 
    def __init__(self, v): self.v=v
    def _set(i, self, v): self.v[i] = v
    a,b = add_props(lambda i,x: x.v[i], _set)

Reading goes through the getter and assignment through the setter:

t = _T([0,2])
test_eq(t.a,0)
test_eq(t.b,2)
t.a = t.a+1
t.b = 3
test_eq(t.a,1)
test_eq(t.b,3)

str2bool

def str2bool(
    s
):

Case-insensitive convert string s too a bool (y,yes,t,true,on,1->True)

True values are ‘y’, ‘yes’, ‘t’, ‘true’, ‘on’, and ‘1’; false values are ‘n’, ‘no’, ‘f’, ‘false’, ‘off’, and ‘0’. Raises ValueError if ‘val’ is anything else.

for o in "y YES t True on 1".split(): assert str2bool(o)
for o in "n no FALSE off 0".split(): assert not str2bool(o)
for o in 0,None,'',False: assert not str2bool(o)
for o in 1,True: assert str2bool(o)

str2int

def str2int(
    s
)->int:

Convert s to an int

str2int treats an empty string and none as 0, and on and off as 1 and 0:

str2float

def str2float(
    s:str
):

Convert s to a float

str2float treats an empty string as 0.0:

str2list

def str2list(
    s:str
):

Convert s to a list

str2list parses a Python list literal, adding the brackets if they are missing, and gives [] for an empty string:

str2date

def str2date(
    s:str
)->datetime.date:

date.fromisoformat with empty string handling

str2date parses ISO dates and returns None for an empty string:

str2dt

def str2dt(
    s:str
)->datetime.datetime:

datetime.fromisoformat with Z suffix and empty string handling

str2dt also accepts a trailing Z for UTC:

to_date

def to_date(
    arg
):

to_list

def to_list(
    arg, # chkstyle: ignore
):

to_float

def to_float(
    arg
):

to_int

def to_int(
    arg
):

to_bool

def to_bool(
    arg
):

The to_* functions apply the str2* parsers to strings and the plain constructors to anything else. type_map is what typed uses to cast:

typed

def typed(
    _func:NoneType=None, *, cast:bool=False
):

Decorator to check param and return types at runtime, with optional casting

typed validates argument types at runtime. This is in contrast to MyPy which only offers static type checking.

For example, a TypeError will be raised if we try to pass an integer into the first argument of the below function:

@typed
def discount(price:int, pct:float) -> float: return (1-pct) * price

with ExceptionExpected(TypeError): discount(100.0, .1)

You can have automatic casting based on heuristics by specifying typed(cast=True). If casting is not possible, a TypeError is raised.

A float is truncated to an int, and a string is stripped and parsed.

@typed(cast=True)
def discount(price:int, pct:float) -> float: return (1-pct) * price

assert 90.0 == discount(100.5, .1)
assert 90.0 == discount(' 100 ', .1)
with ExceptionExpected(TypeError): discount("a", .1)

We can also optionally allow multiple types by enumarating the types in a tuple as illustrated below:

@typed
def discount(price:int|float, pct:float): return (1-pct) * price

assert 90.0 == discount(100.0, .1)

@typed(cast=True)
def discount(price:int|None, pct:float): return (1-pct) * price

assert 90.0 == discount(100.0, .1)

We currently do not support union types when casting.

@typed(cast=True)
def discount(price:int|float, pct:float): return (1-pct) * price

with ExceptionExpected(AssertionError): assert 90.0 == discount("100.0", .1)

typed works with classes, too:

class Foo:
    @typed
    def __init__(self, a:int, b: int, c:str): pass
    @typed(cast=True)
    def test(cls, d:str): return d

with ExceptionExpected(TypeError): Foo(1, 2, 3) 
assert isinstance(Foo(1,2, 'a string').test(10), str)

It also works with custom types.

@typed
def test_foo(foo: Foo): pass

with ExceptionExpected(TypeError): test_foo(1)
test_foo(Foo(1, 2, 'a string'))

class Bar:
    @typed
    def __init__(self, a:int): self.a = a
@typed(cast=True)
def test_bar(bar: Bar): return bar

assert isinstance(test_bar(1), Bar)
test_eq(test_bar(1).a, 1)
with ExceptionExpected(TypeError): test_bar("foobar")

exec_new

def exec_new(
    code
):

Execute code in a new environment and return it

exec_new runs code in a fresh globals dict that carries the current package, so relative imports work, and returns that dict:

g = exec_new('a=1')
test_eq(g['a'], 1)

exec_import

def exec_import(
    mod, sym
):

Import sym from mod in a new environment

exec_import is exec_new for one import, giving a namespace with sym in it:

g = exec_import('math', 'pi')
test_eq(g['pi'], math.pi)

sig_with_params

def sig_with_params(
    sig, remove:NoneType=None, **updates
):

Copy of sig with removed params dropped and updates added, kind-sorted

kindsort

def kindsort(
    ps
):

Params sorted by Parameter.kind

sig_with_params lets you modify a function signature by adding, replacing, or removing parameters. This is useful when creating wrapper functions or decorators that need to adjust the signature of the wrapped function.

You can remove parameters by name:

def foo(a, b, c=3): pass
sig = signature(foo)

new_sig = sig_with_params(sig, remove=['b'])
test_eq(list(new_sig.parameters.keys()), ['a', 'c'])

You can also add new parameters:

new_param = Parameter('d', Parameter.KEYWORD_ONLY, default=4)
new_sig = sig_with_params(sig, d=new_param)
test_eq(list(new_sig.parameters.keys()), ['a', 'b', 'c', 'd'])

The result is kind-sorted (kindsort: a stable sort on Parameter.kind, Python’s own ordering rule), so composing groups of params can never produce an illegal signature: a collector in sig stays last however many params are added after it:

A signature that is already in kind order comes back unchanged.

def bar(a, *, b=2, **kwargs): pass
new_sig = sig_with_params(signature(bar), c=Parameter('c', Parameter.KEYWORD_ONLY, default=3))
test_eq(str(new_sig), '(a, *, b=2, c=3, **kwargs)')
test_eq([p.name for p in kindsort(new_sig.parameters.values())], ['a', 'b', 'c', 'kwargs'])

fdelegates

def fdelegates(
    to
):

This is a simplified version of fastcore.meta.delegates that supports only regular functions.

xdumps

def xdumps(
    o, **kwargs
):

json.dumps, using __json__ methods when available

Python’s json.dumps can handle custom objects through a default callback, but it does not look for serialization methods on the objects themselves. Therefore even a class is not normally serializable:

class Person:
    def __init__(self, name): self.name = name
    def __json__(self): return dict(name=self.name)

with expect_fail(TypeError, 'Person is not JSON serializable'): json.dumps(Person('Alyssa'))

xdumps recognizes __json__ and serializes the value it returns. The protocol is applied recursively, so custom objects can appear inside ordinary lists and dictionaries:

people = dict(team=[Person('Alyssa'), Person('Ben')])
test_eq(json.loads(xdumps(people)), dict(team=[dict(name='Alyssa'), dict(name='Ben')]))

xdumps also supports default like dumps does:

test_eq(json.loads(xdumps(1j, default=str)), '1j')

Some standard library types cannot implement __json__, since builtin types cannot gain methods, so xdumps handles the common ones directly: datetimes, dates and times become ISO8601 strings, UUID and Path become strings, and enums become their values. An object’s own __json__ still comes first, and default remains the escape hatch for anything else:

class Priority(enum.Enum): high = 1
d = dict(t=datetime(2026,8,2,10,30), day=date(2026,8,2), u=UUID(int=18), p=Path('a/b'), pri=Priority.high)
test_eq(json.loads(xdumps(d)), dict(t='2026-08-02T10:30:00', day='2026-08-02', u='00000000-0000-0000-0000-000000000012', p='a/b', pri=1))

revive_dates is the inverse for the datetime case: it walks a decoded JSON tree and converts any string in ISO8601 form (Z suffix included) back into a datetime. Conversion is by shape, so a string that merely looks like a timestamp converts too - apply it to trees where date strings can only be dates, such as message headers, rather than to arbitrary user content:

revive_dates

def revive_dates(
    o
):

Recursively convert ISO8601-formatted strings in a decoded JSON tree into datetime objects

msg = json.loads(xdumps(dict(date=datetime(2026,8,2,10,30), note='sent on 2026-08-02, not a timestamp')))
test_eq(revive_dates(dict(d='2026-08-02T10:30:00Z'))['d'], datetime.fromisoformat('2026-08-02T10:30:00+00:00'))
revive_dates(msg)

Notebook functions

ipython_shell

def ipython_shell():

Same as get_ipython but returns False if not in IPython

in_ipython

def in_ipython():

Check if code is running in some kind of IPython environment

in_colab

def in_colab():

Check if the code is running in Google Colaboratory

in_jupyter and in_notebook tell a notebook front end apart from a plain IPython shell:

in_jupyter

def in_jupyter():

Check if the code is running in a jupyter notebook

in_notebook

def in_notebook():

Check if the code is running in a jupyter notebook

These variables are available as booleans in fastcore.basics as IN_IPYTHON, IN_JUPYTER, IN_COLAB and IN_NOTEBOOK.

IN_IPYTHON, IN_JUPYTER, IN_COLAB, IN_NOTEBOOK

(False, False, False, False)