Basic functionality used in the fastai library
from fastcore.test import *
from nbdev.showdoc import *
from fastcore.nb_imports import *

Basics

ifnone[source]

ifnone(a, b)

b if a is None else a

Since b if a is None else a is such a common pattern, we wrap it in a function. However, be careful, because python will evaluate both a and b when calling ifnone (which it doesn't do if using the if version directly).

test_eq(ifnone(None,1), 1)
test_eq(ifnone(2   ,1), 2)

maybe_attr[source]

maybe_attr(o, attr)

getattr(o,attr,o)

Return the attribute attr for object o. If the attribute doesn't exist, then return the object o instead.

class myobj: myattr='foo'

test_eq(maybe_attr(myobj, 'myattr'), 'foo')
test_eq(maybe_attr(myobj, 'another_attr'), myobj)

basic_repr[source]

basic_repr(flds=None)

Lookup a user-supplied list of attributes (flds) of an object and generate a string with the name of each attribute and its corresponding value. The format of this string is key=value, where key is the name of the attribute, and value is the value of the attribute. For each value, attempt to use the __name__ attribute, otherwise fall back to using the value's __repr__ when constructing the string.

class SomeClass:
    a=1
    b='foo'
    __repr__=basic_repr('a,b')
    __name__='some-class'
    
class AnotherClass:
    c=SomeClass()
    d='bar'
    __repr__=basic_repr(['c', 'd'])
    
sc = SomeClass()    
ac = AnotherClass()

test_eq(repr(sc), "SomeClass(a=1, b='foo')")
test_eq(repr(ac), "AnotherClass(c=SomeClass(a=1, b='foo'), d='bar')")

is_array[source]

is_array(x)

True if x supports __array__ or iloc

is_array(np.array(1)),is_array([1])
(True, False)

listify[source]

listify(o=None, *rest, use_list=False, match=None)

Convert o to a list

Conversion is designed to "do what you mean", e.g:

test_eq(listify('hi'), ['hi'])
test_eq(listify(array(1)), [array(1)])
test_eq(listify(1), [1])
test_eq(listify([1,2]), [1,2])
test_eq(listify(range(3)), [0,1,2])
test_eq(listify(None), [])
test_eq(listify(1,2), [1,2])
arr = np.arange(9).reshape(3,3)
listify(arr)
[array([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])]
listify(array([1,2]))
[array([1, 2])]

Generators are turned into lists too:

gen = (o for o in range(3))
test_eq(listify(gen), [0,1,2])

Use match to provide a length to match:

test_eq(listify(1,match=3), [1,1,1])

If match is a sequence, it's length is used:

test_eq(listify(1,match=range(3)), [1,1,1])

If the listified item is not of length 1, it must be the same length as match:

test_eq(listify([1,1,1],match=3), [1,1,1])
test_fail(lambda: listify([1,1],match=3))

tuplify[source]

tuplify(o, use_list=False, match=None)

Make o a tuple

test_eq(tuplify(None),())
test_eq(tuplify([1,2,3]),(1,2,3))
test_eq(tuplify(1,match=[1,2,3]),(1,1,1))

true[source]

true(*args, **kwargs)

Predicate: always True

[(o,true(o)) for o in
 (array(0),array(1),array([0]),array([0,1]),1,0,'',None)]
[(array(0), False),
 (array(1), True),
 (array([0]), True),
 (array([0, 1]), True),
 (1, True),
 (0, False),
 ('', False),
 (None, False)]

class NullType[source]

NullType()

An object that is False and can be called, chained, and indexed

bool(null.hi().there[3])
False

tonull[source]

tonull(x)

Convert None to null

bool(tonull(None).hi().there[3])
False

get_class[source]

get_class(nm, *fld_names, sup=None, doc=None, funcs=None, **flds)

Dynamically create a class, optionally inheriting from sup, containing fld_names

_t = get_class('_t', 'a', b=2)
t = _t()
test_eq(t.a, None)
test_eq(t.b, 2)
t = _t(1, b=3)
test_eq(t.a, 1)
test_eq(t.b, 3)
t = _t(1, 3)
test_eq(t.a, 1)
test_eq(t.b, 3)
test_eq(repr(t), '_t(a=1, b=3)')
test_eq(t, pickle.loads(pickle.dumps(t)))

Most often you'll want to call mk_class, since it adds the class to your module. See mk_class for more details and examples of use (which also apply to get_class).

mk_class[source]

mk_class(nm, *fld_names, sup=None, doc=None, funcs=None, mod=None, **flds)

Create a class using get_class and add to the caller's module

Any kwargs will be added as class attributes, and sup is an optional (tuple of) base classes.

mk_class('_t', a=1, sup=dict)
t = _t()
test_eq(t.a, 1)
assert(isinstance(t,dict))

A __init__ is provided that sets attrs for any kwargs, and for any args (matching by position to fields), along with a __repr__ which prints all attrs. The docstring is set to doc. You can pass funcs which will be added as attrs with the function names.

def foo(self): return 1
mk_class('_t', 'a', sup=dict, doc='test doc', funcs=foo)

t = _t(3, b=2)
test_eq(t.a, 3)
test_eq(t.b, 2)
test_eq(t.foo(), 1)
test_eq(t.__doc__, 'test doc')
t
{}

wrap_class[source]

wrap_class(nm, *fld_names, sup=None, doc=None, funcs=None, **flds)

Decorator: makes function a method of a new class nm passing parameters to mk_class

@wrap_class('_t', a=2)
def bar(self,x): return x+1

t = _t()
test_eq(t.a, 2)
test_eq(t.bar(3), 4)

class ignore_exceptions[source]

ignore_exceptions()

Context manager to ignore exceptions

with ignore_exceptions(): 
    # Exception will be ignored
    raise Exception

exec_local[source]

exec_local(code, var_name)

Call exec on code and return the var `var_name

test_eq(exec_local("a=1", "a"), 1)

risinstance[source]

risinstance(types, obj=None)

Curried isinstance but with args reversed

assert risinstance(int, 1)
assert not risinstance(str, 0)
assert risinstance(int)(1)

NoOp

These are used when you need a pass-through function.

noop[source]

noop(x=None, *args, **kwargs)

Do nothing

noop()
test_eq(noop(1),1)

noops[source]

noops(x=None, *args, **kwargs)

Do nothing (method)

class _t: foo=noops
test_eq(_t().foo(1),1)

Infinite Lists

These lists are useful for things like padding an array or adding index column(s) to arrays.

class Inf[source]

Inf()

Infinite lists

Inf defines the following properties:

  • count: itertools.count()
  • zeros: itertools.cycle([0])
  • ones : itertools.cycle([1])
  • nones: itertools.cycle([None])
test_eq([o for i,o in zip(range(5), Inf.count)],
        [0, 1, 2, 3, 4])

test_eq([o for i,o in zip(range(5), Inf.zeros)],
        [0]*5)

test_eq([o for i,o in zip(range(5), Inf.ones)],
        [1]*5)

test_eq([o for i,o in zip(range(5), Inf.nones)],
        [None]*5)

Operator Functions

in_[source]

in_(a, b=<object object at 0x7fbf07fe4320>)

Same as operator.in_, or returns partial if 1 arg

assert in_('c', ('b', 'c', 'a'))
assert in_(4, [2,3,4,5])
assert in_('t', 'fastai')
test_fail(in_('h', 'fastai'))

# use in_ as a partial
assert in_('fastai')('t')
assert in_([2,3,4,5])(4)
test_fail(in_('fastai')('h'))

In addition to in_, the following functions are provided matching the behavior of the equivalent versions in operator: lt gt le ge eq ne add sub mul truediv is_ is_not.

lt(3,5),gt(3,5),is_(None,None),in_(0,[1,2])
(True, False, True, False)

Similarly to _in, they also have additional functionality: if you only pass one param, they return a partial function that passes that param as the second positional parameter.

lt(5)(3),gt(5)(3),is_(None)(None),in_([1,2])(0)
(True, False, True, False)

true[source]

true(*args, **kwargs)

Predicate: always True

assert true(1,2,3)
assert true(False)
assert true(None)
assert true([])

stop[source]

stop(e=StopIteration)

Raises exception e (by default StopException)

gen[source]

gen(func, seq, cond=true)

Like (func(o) for o in seq if cond(func(o))) but handles StopIteration

test_eq(gen(noop, Inf.count, lt(5)),
        range(5))
test_eq(gen(operator.neg, Inf.count, gt(-5)),
        [0,-1,-2,-3,-4])
test_eq(gen(lambda o:o if o<5 else stop(), Inf.count),
        range(5))

chunked[source]

chunked(it, chunk_sz=None, drop_last=False, n_chunks=None)

Return batches from iterator it of size chunk_sz (or return n_chunks total)

Note that you must pass either chunk_sz, or n_chunks, but not both.

t = list(range(10))
test_eq(chunked(t,3),      [[0,1,2], [3,4,5], [6,7,8], [9]])
test_eq(chunked(t,3,True), [[0,1,2], [3,4,5], [6,7,8],    ])

t = map(lambda o:stop() if o==6 else o, Inf.count)
test_eq(chunked(t,3), [[0, 1, 2], [3, 4, 5]])
t = map(lambda o:stop() if o==7 else o, Inf.count)
test_eq(chunked(t,3), [[0, 1, 2], [3, 4, 5], [6]])

t = np.arange(10)
test_eq(chunked(t,3),      [[0,1,2], [3,4,5], [6,7,8], [9]])
test_eq(chunked(t,3,True), [[0,1,2], [3,4,5], [6,7,8],    ])

otherwise[source]

otherwise(x, tst, y)

y if tst(x) else x

test_eq(otherwise(2+1, gt(3), 4), 3)
test_eq(otherwise(2+1, gt(2), 4), 4)

Attribute Helpers

These functions reduce boilerplate when setting or manipulating attributes or properties of objects.

custom_dir[source]

custom_dir(c, add:list)

Implement custom __dir__, adding add to cls

custom_dir allows you extract the __dict__ property of a class and appends the list add to it.

class _T: 
    def f(): pass

s = custom_dir(_T, add=['foo', 'bar']) # a list of everything in `__dict__` of `_T` with ['foo', 'bar'] appended.
assert {'foo', 'bar', 'f'}.issubset(s)

class AttrDict[source]

AttrDict() :: dict

dict subclass that also provides access to keys as attrs

d = AttrDict(a=1,b="two")
test_eq(d.a, 1)
test_eq(d['b'], 'two')
test_eq(d.get('c','nope'), 'nope')
d.b = 2
test_eq(d.b, 2)
test_eq(d['b'], 2)
d['b'] = 3
test_eq(d['b'], 3)
test_eq(d.b, 3)
assert 'a' in dir(d)

type_hints[source]

type_hints(f)

Same as typing.get_type_hints but returns {} if not allowed type

def type_hints(f):
    "Same as `typing.get_type_hints` but returns `{}` if not allowed type"
    return typing.get_type_hints(f) if isinstance(f, typing._allowed_types) else {}

Below is a list of allowed types for type hints in python:

list(typing._allowed_types)
[function,
 builtin_function_or_method,
 method,
 module,
 wrapper_descriptor,
 method-wrapper,
 method_descriptor]

For example, type func is allowed so type_hints returns the same value as typing.get_hints:

def f(a:int)->bool: ... # a function with type hints (allowed)
exp = {'a':int,'return':bool}
test_eq(type_hints(f), typing.get_type_hints(f))
test_eq(type_hints(f), exp)

However, class is not an allowed type, so type_hints returns {}:

class _T:
    def __init__(self, a:int=0)->bool: ...
assert not type_hints(_T)

annotations[source]

annotations(o)

Annotations for o, or type(o)

This supports a wider range of situations than type_hints, by checking type() and __init__ for annotations too:

for o in _T,_T(),_T.__init__,f: test_eq(annotations(o), exp)
assert not annotations(int)
assert not annotations(print)

anno_ret[source]

anno_ret(func)

Get the return annotation of func

def f(x) -> float: return x
test_eq(anno_ret(f), float)

def f(x) -> typing.Tuple[float,float]: return x
test_eq(anno_ret(f), typing.Tuple[float,float])

If your return annotation is None, anno_ret will return NoneType (and not None):

def f(x) -> None: return x

test_eq(anno_ret(f), NoneType)
assert anno_ret(f) is not None # returns NoneType instead of None

If your function does not have a return type, or if you pass in None instead of a function, then anno_ret returns None:

def f(x): return x

test_eq(anno_ret(f), None)
test_eq(anno_ret(None), None) # instead of passing in a func, pass in None

argnames[source]

argnames(f, frame=False)

Names of arguments to function or frame f

test_eq(argnames(f), ['x'])

with_cast[source]

with_cast(f)

Decorator which uses any parameter annotations as preprocessing functions

@with_cast
def _f(a, b:Path, c:str='', d=0)->bool: return (a,b,c,d)

test_eq(_f(1, '.', 3), (1,Path('.'),'3',0))
test_eq(_f(1, '.'), (1,Path('.'),'',0))

store_attr[source]

store_attr(names=None, but='', cast=False, store_args=None, **attrs)

Store params named in comma-separated names from calling context into attrs in self

In it's most basic form, you can use store_attr to shorten code like this:

class T:
    def __init__(self, a,b,c): self.a,self.b,self.c = a,b,c

...to this:

class T:
    def __init__(self, a,b,c): store_attr('a,b,c', self)

This class behaves as if we'd used the first form:

t = T(1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2

In addition, it stores the attrs as a dict in __stored_args__, which you can use for display, logging, and so forth.

test_eq(t.__stored_args__, {'a':1, 'b':3, 'c':2})

Since you normally want to use the first argument (often called self) for storing attributes, it's optional:

class T:
    def __init__(self, a,b,c:str): store_attr('a,b,c')

t = T(1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2

With cast=True any parameter annotations will be used as preprocessing functions for the corresponding arguments:

class T:
    def __init__(self, a:listify, b, c:str): store_attr('a,b,c', cast=True)

t = T(1,c=2,b=3)
assert t.a==[1] and t.b==3 and t.c=='2'

You can inherit from a class using store_attr, and just call it again to add in any new attributes added in the derived class:

class T2(T):
    def __init__(self, d, **kwargs):
        super().__init__(**kwargs)
        store_attr('d')

t = T2(d=1,a=2,b=3,c=4)
assert t.a==2 and t.b==3 and t.c==4 and t.d==1

You can skip passing a list of attrs to store. In this case, all arguments passed to the method are stored:

class T:
    def __init__(self, a,b,c): store_attr()

t = T(1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2
class T4(T):
    def __init__(self, d, **kwargs):
        super().__init__(**kwargs)
        store_attr()

t = T4(4, a=1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2 and t.d==4
class T4:
    def __init__(self, *, a: int, b: float = 1):
        store_attr()
        
t = T4(a=3)
assert t.a==3 and t.b==1
t = T4(a=3, b=2)
assert t.a==3 and t.b==2

You can skip some attrs by passing but:

class T:
    def __init__(self, a,b,c): store_attr(but='a')

t = T(1,c=2,b=3)
assert t.b==3 and t.c==2
assert not hasattr(t,'a')

You can also pass keywords to store_attr, which is identical to setting the attrs directly, but also stores them in __stored_args__.

class T:
    def __init__(self): store_attr(a=1)

t = T()
assert t.a==1

You can also use store_attr inside functions.

def create_T(a, b):
    t = SimpleNamespace()
    store_attr(self=t)
    return t

t = create_T(a=1, b=2)
assert t.a==1 and t.b==2

attrdict[source]

attrdict(o, *ks, default=None)

Dict from each k in ks to getattr(o,k)

class T:
    def __init__(self, a,b,c): store_attr()

t = T(1,c=2,b=3)
test_eq(attrdict(t,'b','c'), {'b':3, 'c':2})

properties[source]

properties(*ps)

Change attrs in cls with names in ps to properties

class T:
    def a(self): return 1
    def b(self): return 2
properties(T,'a')

test_eq(T().a,1)
test_eq(T().b(),2)

camel2snake[source]

camel2snake(name)

Convert CamelCase to snake_case

test_eq(camel2snake('ClassAreCamel'), 'class_are_camel')
test_eq(camel2snake('Already_Snake'), 'already__snake')

snake2camel[source]

snake2camel(s)

Convert snake_case to CamelCase

test_eq(snake2camel('a_b_cc'), 'ABCc')

class2attr[source]

class2attr(cls_name)

Return the snake-cased name of the class; strip ending cls_name if it exists.

class Parent:
    @property
    def name(self): return class2attr(self, 'Parent')

class ChildOfParent(Parent): pass
class ParentChildOf(Parent): pass

p = Parent()
cp = ChildOfParent()
cp2 = ParentChildOf()

test_eq(p.name, 'parent')
test_eq(cp.name, 'child_of')
test_eq(cp2.name, 'parent_child_of')

getattrs[source]

getattrs(o, *attrs, default=None)

List of all attrs in o

from fractions import Fraction
getattrs(Fraction(1,2), 'numerator', 'denominator')
[1, 2]

hasattrs[source]

hasattrs(o, attrs)

Test whether o contains all attrs

assert hasattrs(1,('imag','real'))
assert not hasattrs(1,('imag','foo'))

setattrs[source]

setattrs(dest, flds, src)

d = dict(a=1,bb="2",ignore=3)
o = SimpleNamespace()
setattrs(o, "a,bb", d)
test_eq(o.a, 1)
test_eq(o.bb, "2")
d = SimpleNamespace(a=1,bb="2",ignore=3)
o = SimpleNamespace()
setattrs(o, "a,bb", d)
test_eq(o.a, 1)
test_eq(o.bb, "2")

try_attrs[source]

try_attrs(obj, *attrs)

Return first attr that exists in obj

test_eq(try_attrs(1, 'real'), 1)
test_eq(try_attrs(1, 'foobar', 'real'), 1)

Extensible Types

ShowPrint is a base class that defines a show method, which is used primarily for callbacks in fastai that expect this method to be defined.

Int, Float, and Str extend int, float and str respectively by adding an additional show method by inheriting from ShowPrint.

The code for Int is shown below:

Examples:

Int(0).show()
Float(2.0).show()
Str('Hello').show()
0
2.0
Hello

Collection functions

Functions that manipulate popular python collections.

detuplify[source]

detuplify(x)

If x is a tuple with one thing, extract it

test_eq(detuplify(()),None)
test_eq(detuplify([1]),1)
test_eq(detuplify([1,2]), [1,2])
test_eq(detuplify(np.array([[1,2]])), np.array([[1,2]]))

replicate[source]

replicate(item, match)

Create tuple of item copied len(match) times

t = [1,1]
test_eq(replicate([1,2], t),([1,2],[1,2]))
test_eq(replicate(1, t),(1,1))

setify[source]

setify(o)

Turn any list like-object into a set.

test_eq(setify(None),set())
test_eq(setify('abc'),{'abc'})
test_eq(setify([1,2,2]),{1,2})
test_eq(setify(range(0,3)),{0,1,2})
test_eq(setify({1,2}),{1,2})

merge[source]

merge(*ds)

Merge all dictionaries in ds

test_eq(merge(), {})
test_eq(merge(dict(a=1,b=2)), dict(a=1,b=2))
test_eq(merge(dict(a=1,b=2), dict(b=3,c=4), None), dict(a=1, b=3, c=4))

range_of[source]

range_of(a, b=None, step=None)

All indices of collection a, if a is a collection, otherwise range

test_eq(range_of([1,1,1,1]), [0,1,2,3])

groupby[source]

groupby(x, key, val=noop)

Like itertools.groupby but doesn't need to be sorted, and isn't lazy, plus some extensions

test_eq(groupby('aa ab bb'.split(), itemgetter(0)), {'a':['aa','ab'], 'b':['bb']})

Here's an example of how to invert a grouping, using an int as key (which uses itemgetter; passing a str will use attrgetter), and using a val function:

d = {0: [1, 3, 7], 2: [3], 3: [5], 4: [8], 5: [4], 7: [5]}
groupby(((o,k) for k,v in d.items() for o in v), 0, 1)
{1: [0], 3: [0, 2], 7: [0], 5: [3, 7], 8: [4], 4: [5]}

last_index[source]

last_index(x, o)

Finds the last index of occurence of x in o (returns -1 if no occurence)

test_eq(last_index(9, [1, 2, 9, 3, 4, 9, 10]), 5)
test_eq(last_index(6, [1, 2, 9, 3, 4, 9, 10]), -1)

filter_dict[source]

filter_dict(d, func)

Filter a dict using func, applied to keys and values

letters = {o:chr(o) for o in range(65,73)}
letters
{65: 'A', 66: 'B', 67: 'C', 68: 'D', 69: 'E', 70: 'F', 71: 'G', 72: 'H'}
filter_dict(letters, lambda k,v: k<67 or v in 'FG')
{65: 'A', 66: 'B', 70: 'F', 71: 'G'}

filter_keys[source]

filter_keys(d, func)

Filter a dict using func, applied to keys

filter_keys(letters, lt(67))
{65: 'A', 66: 'B'}

filter_values[source]

filter_values(d, func)

Filter a dict using func, applied to values

filter_values(letters, in_('FG'))
{70: 'F', 71: 'G'}

cycle[source]

cycle(o)

Like itertools.cycle except creates list of Nones if o is empty

test_eq(itertools.islice(cycle([1,2,3]),5), [1,2,3,1,2])
test_eq(itertools.islice(cycle([]),3), [None]*3)
test_eq(itertools.islice(cycle(None),3), [None]*3)
test_eq(itertools.islice(cycle(1),3), [1,1,1])

zip_cycle[source]

zip_cycle(x, *args)

Like itertools.zip_longest but cycles through elements of all but first argument

test_eq(zip_cycle([1,2,3,4],list('abc')), [(1, 'a'), (2, 'b'), (3, 'c'), (4, 'a')])

sorted_ex[source]

sorted_ex(iterable, key=None, reverse=False)

Like sorted, but if key is str use attrgetter; if int use itemgetter

not_[source]

not_(f)

Create new function that negates result of f

def f(a): return a>0
test_eq(f(1),True)
test_eq(not_(f)(1),False)
test_eq(not_(f)(a=-1),True)

argwhere[source]

argwhere(iterable, f, negate=False, **kwargs)

Like filter_ex, but return indices for matching items

filter_ex[source]

filter_ex(iterable, f=noop, negate=False, gen=False, **kwargs)

Like filter, but passing kwargs to f, defaulting f to noop, and adding negate and gen

range_of[source]

range_of(a, b=None, step=None)

All indices of collection a, if a is a collection, otherwise range

test_eq(range_of([1,1,1,1]), [0,1,2,3])
test_eq(range_of(4), [0,1,2,3])

renumerate[source]

renumerate(iterable, start=0)

Same as enumerate, but returns index as 2nd element instead of 1st

test_eq(renumerate('abc'), (('a',0),('b',1),('c',2)))

first[source]

first(x, f=None, negate=False, **kwargs)

First element of x, optionally filtered by f, or None if missing

test_eq(first(['a', 'b', 'c', 'd', 'e']), 'a')
test_eq(first([False]), False)
test_eq(first([False], noop), None)

nested_attr[source]

nested_attr(o, attr, default=None)

Same as getattr, but if attr includes a ., then looks inside nested objects

a = SimpleNamespace(b=(SimpleNamespace(c=1)))
test_eq(nested_attr(a, 'b.c'), getattr(getattr(a, 'b'), 'c'))
test_eq(nested_attr(a, 'b.d'), None)

nested_idx[source]

nested_idx(coll, *idxs)

Index into nested collections, dicts, etc, with idxs

a = {'b':[1,{'c':2}]}
test_eq(nested_idx(a), a)
test_eq(nested_idx(a, 'b'), [1,{'c':2}])
test_eq(nested_idx(a, 'b', 1), {'c':2})
test_eq(nested_idx(a, 'b', 1, 'c'), 2)

val2idx[source]

val2idx(x)

Dict from value to index

test_eq(val2idx([1,2,3]), {3:2,1:0,2:1})

uniqueify[source]

uniqueify(x, sort=False, bidir=False, start=None)

Unique elements in x, optionally sort-ed, optionally return reverse correspondence, optionally prepend with elements.

t = [1,1,0,5,0,3]
test_eq(uniqueify(t),[1,0,5,3])
test_eq(uniqueify(t, sort=True),[0,1,3,5])
test_eq(uniqueify(t, start=[7,8,6]), [7,8,6,1,0,5,3])
v,o = uniqueify(t, bidir=True)
test_eq(v,[1,0,5,3])
test_eq(o,{1:0, 0: 1, 5: 2, 3: 3})
v,o = uniqueify(t, sort=True, bidir=True)
test_eq(v,[0,1,3,5])
test_eq(o,{0:0, 1: 1, 3: 2, 5: 3})

fastuple

A tuple with extended functionality.

class fastuple[source]

fastuple(x=None, *rest) :: tuple

A tuple with elementwise ops and more friendly init behavior

Friendly init behavior

Common failure modes when trying to initialize a tuple in python:

tuple(3)
> TypeError:'int' object is not iterable

or

tuple(3, 4)
> TypeError:tuple expected at most 1 arguments, got 2

However, fastuple allows you to define tuples like this and in the usual way:

test_eq(fastuple(3), (3,))
test_eq(fastuple(3,4), (3, 4))
test_eq(fastuple((3,4)), (3, 4))

Elementwise operations

fastuple.add[source]

fastuple.add(*args)

+ is already defined in tuple for concat, so use add instead

test_eq(fastuple.add((1,1),(2,2)), (3,3))
test_eq_type(fastuple(1,1).add(2), fastuple(3,3))
test_eq(fastuple('1','2').add('2'), fastuple('12','22'))
fastuple.mul[source]

fastuple.mul(*args)

* is already defined in tuple for replicating, so use mul instead

test_eq_type(fastuple(1,1).mul(2), fastuple(2,2))

Other Elementwise Operations

Additionally, the following elementwise operations are available:

  • le: less than
  • eq: equal
  • gt: greater than
  • min: minimum of
test_eq(fastuple(3,1).le(1), (False, True))
test_eq(fastuple(3,1).eq(1), (False, True))
test_eq(fastuple(3,1).gt(1), (True, False))
test_eq(fastuple(3,1).min(2), (2,1))

You can also do other elemntwise operations like negate a fastuple, or subtract two fastuples:

test_eq(-fastuple(1,2), (-1,-2))
test_eq(~fastuple(1,0,1), (False,True,False))

test_eq(fastuple(1,1)-fastuple(2,2), (-1,-1))
test_eq(type(fastuple(1)), fastuple)
test_eq_type(fastuple(1,2), fastuple(1,2))
test_ne(fastuple(1,2), fastuple(1,3))
test_eq(fastuple(), ())

Functions on Functions

Utilities for functional programming or for defining, modifying, or debugging functions.

class bind[source]

bind(func, *pargs, **pkwargs)

Same as partial, except you can use arg0 arg1 etc param placeholders

bind is the same as partial, but also allows you to reorder positional arguments using variable name(s) arg{i} where i refers to the zero-indexed positional argument. bind as implemented currently only supports reordering of up to the first 5 positional arguments.

Consider the function myfunc below, which has 3 positional arguments. These arguments can be referenced as arg0, arg1, and arg1, respectively.

def myfn(a,b,c,d=1,e=2): return(a,b,c,d,e)

In the below example we bind the positional arguments of myfn as follows:

  • The second input 14, referenced by arg1, is substituted for the first positional argument.
  • We supply a default value of 17 for the second positional argument.
  • The first input 19, referenced by arg0, is subsituted for the third positional argument.
test_eq(bind(myfn, arg1, 17, arg0, e=3)(19,14), (14,17,19,1,3))

In this next example:

  • We set the default value to 17 for the first positional argument.
  • The first input 19 refrenced by arg0, becomes the second positional argument.
  • The second input 14 becomes the third positional argument.
  • We override the default the value for named argument e to 3.
test_eq(bind(myfn, 17, arg0, e=3)(19,14), (17,19,14,1,3))

This is an example of using bind like partial and do not reorder any arguments:

test_eq(bind(myfn)(17,19,14), (17,19,14,1,2))

bind can also be used to change default values. In the below example, we use the first input 3 to override the default value of the named argument e, and supply default values for the first three positional arguments:

test_eq(bind(myfn, 17,19,14,e=arg0)(3), (17,19,14,1,3))

map_ex[source]

map_ex(iterable, f, *args, gen=False, **kwargs)

Like map, but use bind, and supports str and indexing

t = [0,1,2,3]
test_eq(map_ex(t,operator.neg), [0,-1,-2,-3])

If f is a string then it is treated as a format string to create the mapping:

test_eq(map_ex(t, '#{}#'), ['#0#','#1#','#2#','#3#'])

If f is a dictionary (or anything supporting __getitem__) then it is indexed to create the mapping:

test_eq(map_ex(t, list('abcd')), list('abcd'))

You can also pass the same arg params that bind accepts:

def f(a=None,b=None): return b
test_eq(map_ex(t, f, b=arg0), range(4))

compose[source]

compose(*funcs, order=None)

Create a function that composes all functions in funcs, passing along remaining *args and **kwargs to all

f1 = lambda o,p=0: (o*2)+p
f2 = lambda o,p=1: (o+1)/p
test_eq(f2(f1(3)), compose(f1,f2)(3))
test_eq(f2(f1(3,p=3),p=3), compose(f1,f2)(3,p=3))
test_eq(f2(f1(3,  3),  3), compose(f1,f2)(3,  3))

f1.order = 1
test_eq(f1(f2(3)), compose(f1,f2, order="order")(3))

maps[source]

maps(*args, retain=noop)

Like map, except funcs are composed first

test_eq(maps([1]), [1])
test_eq(maps(operator.neg, [1,2]), [-1,-2])
test_eq(maps(operator.neg, operator.neg, [1,2]), [1,2])

partialler[source]

partialler(f, *args, order=None, **kwargs)

Like functools.partial but also copies over docstring

def _f(x,a=1):
    "test func"
    return x-a
_f.order=1

f = partialler(_f, 2)
test_eq(f.order, 1)
test_eq(f(3), -1)
f = partialler(_f, a=2, order=3)
test_eq(f.__doc__, "test func")
test_eq(f.order, 3)
test_eq(f(3), _f(3,2))
class partial0:
    "Like `partialler`, but args passed to callable are inserted at started, instead of at end"
    def __init__(self, f, *args, order=None, **kwargs):
        self.f,self.args,self.kwargs = f,args,kwargs
        self.order = ifnone(order, getattr(f,'order',None))
        self.__doc__ = f.__doc__

    def __call__(self, *args, **kwargs): return self.f(*args, *self.args, **kwargs, **self.kwargs)
f = partial0(_f, 2)
test_eq(f.order, 1)
test_eq(f(3), 1) # NB: different to `partialler` example

instantiate[source]

instantiate(t)

Instantiate t if it's a type, otherwise do nothing

test_eq_type(instantiate(int), 0)
test_eq_type(instantiate(1), 1)

using_attr[source]

using_attr(f, attr)

Change function f to operate on attr

t = Path('/a/b.txt')
f = using_attr(str.upper, 'name')
test_eq(f(t), 'B.TXT')

Self (with an uppercase S)

A Concise Way To Create Lambdas

This is a concise way to create lambdas that are calling methods on an object (note the capitalization!)

Self.sum()), for instance, is a shortcut for lambda o: o.sum().

f = Self.sum()
x = np.array([3.,1])
test_eq(f(x), 4.)

# This is equivalent to above
f = lambda o: o.sum()
x = np.array([3.,1])
test_eq(f(x), 4.)

f = Self.argmin()
arr = np.array([1,2,3,4,5])
test_eq(f(arr), arr.argmin())

f = Self.sum().is_integer()
x = np.array([3.,1])
test_eq(f(x), True)

f = Self.sum().real.is_integer()
x = np.array([3.,1])
test_eq(f(x), True)

f = Self.imag()
test_eq(f(3), 0)

f = Self[1]
test_eq(f(x), 1)

Self is also callable, which creates a function which calls any function passed to it, using the arguments passed to Self:

def f(a, b=3): return a+b+2
def g(a, b=3): return a*b
fg = Self(1,b=2)
list(map(fg, [f,g]))
[5, 2]

Patching

copy_func[source]

copy_func(f)

Copy a non-builtin function (NB copy.copy does not work for this)

Sometimes it may be desirable to make a copy of a function that doesn't point to the original object. When you use Python's built in copy.copy or copy.deepcopy to copy a function, you get a reference to the original object:

import copy as cp
def foo(): pass
a = cp.copy(foo)
b = cp.deepcopy(foo)

a.someattr = 'hello' # since a and b point at the same object, updating a will update b
test_eq(b.someattr, 'hello')

assert a is foo and b is foo

However, with copy_func, you can retrieve a copy of a function without a reference to the original object:

c = copy_func(foo) # c is an indpendent object
assert c is not foo
def g(x, *, y=3):
    return x+y
test_eq(copy_func(g)(4), 7)

patch_to[source]

patch_to(as_prop=False, cls_method=False)

Decorator: add f to cls

The @patch_to decorator allows you to monkey patch a function into a class as a method:

class _T3(int): pass  

@patch_to(_T3)
def func1(self, a): return self+a

t = _T3(1) # we initilized `t` to a type int = 1
test_eq(t.func1(2), 3) # we add 2 to `t`, so 2 + 1 = 3

You can access instance properties in the usual way via self:

class _T4():
    def __init__(self, g): self.g = g
        
@patch_to(_T4)
def greet(self, x): return self.g + x
        
t = _T4('hello ') # this sets self.g = 'helllo '
test_eq(t.greet('world'), 'hello world') #t.greet('world') will append 'world' to 'hello '

You can instead specify that the method should be a class method by setting cls_method=True:

class _T5(int): attr = 3 # attr is a class attribute we will access in a later method
    
@patch_to(_T5, cls_method=True)
def func(cls, x): return cls.attr + x # you can access class attributes in the normal way

test_eq(_T5.func(4), 7)

Additionally you can specify that the function you want to patch should be a class attribute with as_prop = False

@patch_to(_T5, as_prop=True)
def add_ten(self): return self + 10

t = _T5(4)
test_eq(t.add_ten, 14)

Instead of passing one class to the @patch_to decorator, you can pass multiple classes in a tuple to simulteanously patch more than one class with the same method:

class _T6(int): pass
class _T7(int): pass

@patch_to((_T6,_T7))
def func_mult(self, a): return self*a

t = _T6(2)
test_eq(t.func_mult(4), 8)
t = _T7(2)
test_eq(t.func_mult(4), 8)

patch[source]

patch(f=None, as_prop=False, cls_method=False)

Decorator: add f to the first parameter's class (based on f's type annotations)

@patch is an alternative to @patch_to that allows you similarly monkey patch class(es) by using type annotations:

class _T8(int): pass  

@patch
def func(self:_T8, a): return self+a

t = _T8(1)  # we initilized `t` to a type int = 1
test_eq(t.func(3), 4) # we add 3 to `t`, so 3 + 1 = 4
test_eq(t.func.__qualname__, '_T8.func')

Similarly to patch_to, you can supply a tuple of classes instead of a single class in your type annotations to patch multiple classes:

class _T9(int): pass 

@patch
def func2(x:(_T8,_T9), a): return x*a # will patch both _T8 and _T9

t = _T8(2)
test_eq(t.func2(4), 8)
test_eq(t.func2.__qualname__, '_T8.func2')

t = _T9(2)
test_eq(t.func2(4), 8)
test_eq(t.func2.__qualname__, '_T9.func2')

Just like patch_to decorator you can use as_propas_prop and cls_method parameters with patch decorator:

@patch(as_prop=True)
def add_ten(self:_T5): return self + 10

t = _T5(4)
test_eq(t.add_ten, 14)
class _T5(int): attr = 3 # attr is a class attribute we will access in a later method
    
@patch(cls_method=True)
def func(cls:_T5, x): return cls.attr + x # you can access class attributes in the normal way

test_eq(_T5.func(4), 7)

patch_property[source]

patch_property(f)

Deprecated; use patch(as_prop=True) instead

Other Helpers

ImportEnum[source]

Enum = []

An Enum that can have its values imported

_T = ImportEnum('_T', {'foobar':1, 'goobar':2})
_T.imports()
test_eq(foobar, _T.foobar)
test_eq(goobar, _T.goobar)

StrEnum[source]

Enum = []

An ImportEnum that behaves like a str

str_enum[source]

str_enum(name, *vals)

Simplified creation of StrEnum types

_T = str_enum('_T', 'a', 'b')
test_eq(f'{_T.a}', 'a')
test_eq(_T.a, 'a')
test_eq(list(_T.__members__), ['a','b'])
print(_T.a, _T.a.upper())
a A

class Stateful[source]

Stateful(*args, **kwargs)

A base class/mixin for objects that should not serialize all their state

class _T(Stateful):
    def __init__(self):
        super().__init__()
        self.a=1
        self._state['test']=2

t = _T()
t2 = pickle.loads(pickle.dumps(t))
test_eq(t.a,1)
test_eq(t._state['test'],2)
test_eq(t2.a,1)
test_eq(t2._state,{})

Override _init_state to do any necessary setup steps that are required during __init__ or during deserialization (e.g. pickle.load). Here's an example of how Stateful simplifies the official Python example for Handling Stateful Objects.

class TextReader(Stateful):
    """Print and number lines in a text file."""
    _stateattrs=('file',)
    def __init__(self, filename):
        self.filename,self.lineno = filename,0
        super().__init__()

    def readline(self):
        self.lineno += 1
        line = self.file.readline()
        if line: return f"{self.lineno}: {line.strip()}"

    def _init_state(self):
        self.file = open(self.filename)
        for _ in range(self.lineno): self.file.readline()
reader = TextReader("00_test.ipynb")
print(reader.readline())
print(reader.readline())

new_reader = pickle.loads(pickle.dumps(reader))
print(reader.readline())
1: {
2: "cells": [
3: {

class PrettyString[source]

PrettyString() :: str

Little hack to get strings to show properly in Jupyter.

Allow strings with special characters to render properly in Jupyter. Without calling print() strings with special characters are displayed like so:

with_special_chars='a string\nwith\nnew\nlines and\ttabs'
with_special_chars
'a string\nwith\nnew\nlines and\ttabs'

We can correct this with PrettyString:

PrettyString(with_special_chars)
a string
with
new
lines and	tabs

even_mults[source]

even_mults(start, stop, n)

Build log-stepped array from start to stop in n steps.

test_eq(even_mults(2,8,3), [2,4,8])
test_eq(even_mults(2,32,5), [2,4,8,16,32])
test_eq(even_mults(2,8,1), 8)

num_cpus[source]

num_cpus()

Get number of cpus

num_cpus()
64

add_props[source]

add_props(f, g=None, n=2)

Create properties passing each of range(n) to f

class _T(): a,b = add_props(lambda i,x:i*2)

t = _T()
test_eq(t.a,0)
test_eq(t.b,2)
class _T(): 
    def __init__(self, v): self.v=v
    def _set(i, self, v): self.v[i] = v
    a,b = add_props(lambda i,x: x.v[i], _set)

t = _T([0,2])
test_eq(t.a,0)
test_eq(t.b,2)
t.a = t.a+1
t.b = 3
test_eq(t.a,1)
test_eq(t.b,3)

typed[source]

typed(f)

Decorator to check param and return types at runtime

typed validates argument types at runtime. This is in contrast to MyPy which only offers static type checking.

For example, a TypeError will be raised if we try to pass an integer into the first argument of the below function:

@typed
def discount(price:int, pct:float): 
    return (1-pct) * price

with ExceptionExpected(TypeError): discount(100.0, .1)

We can also optionally allow multiple types by enumarating the types in a tuple as illustrated below:

def discount(price:(int,float), pct:float): 
    return (1-pct) * price

assert 90.0 == discount(100.0, .1)
@typed
def foo(a:int, b:str='a'): return a
test_eq(foo(1, '2'), 1)

with ExceptionExpected(TypeError): foo(1,2)

@typed
def foo()->str: return 1
with ExceptionExpected(TypeError): foo()

@typed
def foo()->str: return '1'
assert foo()

typed works with classes, too:

class Foo:
    @typed
    def __init__(self, a:int, b: int, c:str): pass
    @typed
    def test(cls, d:str): return d

with ExceptionExpected(TypeError): Foo(1, 2, 3) 
with ExceptionExpected(TypeError): Foo(1,2, 'a string').test(10)

Notebook functions

ipython_shell[source]

ipython_shell()

Same as get_ipython but returns False if not in IPython

in_ipython[source]

in_ipython()

Check if code is running in some kind of IPython environment

in_colab[source]

in_colab()

Check if the code is running in Google Colaboratory

in_jupyter[source]

in_jupyter()

Check if the code is running in a jupyter notebook

in_notebook[source]

in_notebook()

Check if the code is running in a jupyter notebook

These variables are available as booleans in fastcore.basics as IN_IPYTHON, IN_JUPYTER, IN_COLAB and IN_NOTEBOOK.

IN_IPYTHON, IN_JUPYTER, IN_COLAB, IN_NOTEBOOK
(True, True, False, True)