Basic functionality

Basic functionality used in the fastai library

Basics

---

source

ifnone

 ifnone (a, b)

b if a is None else a

Since b if a is None else a is such a common pattern, we wrap it in a function. However, be careful, because python will evaluate both a and b when calling ifnone (which it doesn’t do if using the if version directly).

test_eq(ifnone(None,1), 1)
test_eq(ifnone(2   ,1), 2)

---

source

maybe_attr

 maybe_attr (o, attr)

getattr(o,attr,o)

Return the attribute attr for object o. If the attribute doesn’t exist, then return the object o instead.

class myobj: myattr='foo'

test_eq(maybe_attr(myobj, 'myattr'), 'foo')
test_eq(maybe_attr(myobj, 'another_attr'), myobj)

---

source

basic_repr

 basic_repr (flds=None)

Minimal __repr__

In types which provide rich display functionality in Jupyter, their __repr__ is also called in order to provide a fallback text representation. Unfortunately, this includes a memory address which changes on every invocation, making it non-deterministic. This causes diffs to get messy and creates conflicts in git. To fix this, put __repr__=basic_repr() inside your class.

class SomeClass: __repr__=basic_repr()
repr(SomeClass())

'SomeClass()'

If you pass a list of attributes (flds) of an object, then this will generate a string with the name of each attribute and its corresponding value. The format of this string is key=value, where key is the name of the attribute, and value is the value of the attribute. For each value, attempt to use the __name__ attribute, otherwise fall back to using the value’s __repr__ when constructing the string.

class SomeClass:
    a=1
    b='foo'
    __repr__=basic_repr('a,b')
    __name__='some-class'

repr(SomeClass())

"SomeClass(a=1, b='foo')"

Nested objects work too:

class AnotherClass:
    c=SomeClass()
    d='bar'
    __repr__=basic_repr(['c', 'd'])

repr(AnotherClass())

"AnotherClass(c=SomeClass(a=1, b='foo'), d='bar')"

Instance variables (but not class variables) are shown if basic_repr is called with no arguments:

class SomeClass:
    def __init__(self, a=1, b='foo'): self.a,self.b = a,b
    __repr__=basic_repr()

repr(SomeClass())

"SomeClass(a=1, b='foo')"

---

source

BasicRepr

 BasicRepr ()

Base class for objects needing a basic __repr__

As a shortcut for creating a __repr__ for instance variables, you can inherit from BasicRepr:

class SomeClass(BasicRepr):
    def __init__(self, a=1, b='foo'): self.a,self.b = a,b

repr(SomeClass())

"SomeClass(a=1, b='foo')"

---

source

is_array

 is_array (x)

True if x supports __array__ or iloc

is_array(np.array(1)),is_array([1])

(True, False)

---

source

listify

 listify (o=None, *rest, use_list=False, match=None)

Convert o to a list

Conversion is designed to “do what you mean”, e.g:

test_eq(listify('hi'), ['hi'])
test_eq(listify(b'hi'), [b'hi'])
test_eq(listify(array(1)), [array(1)])
test_eq(listify(1), [1])
test_eq(listify([1,2]), [1,2])
test_eq(listify(range(3)), [0,1,2])
test_eq(listify(None), [])
test_eq(listify(1,2), [1,2])

arr = np.arange(9).reshape(3,3)
listify(arr)

[array([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])]

listify(array([1,2]))

[array([1, 2])]

Generators are turned into lists too:

gen = (o for o in range(3))
test_eq(listify(gen), [0,1,2])

Use match to provide a length to match:

test_eq(listify(1,match=3), [1,1,1])

If match is a sequence, it’s length is used:

test_eq(listify(1,match=range(3)), [1,1,1])

If the listified item is not of length 1, it must be the same length as match:

test_eq(listify([1,1,1],match=3), [1,1,1])
test_fail(lambda: listify([1,1],match=3))

---

source

tuplify

 tuplify (o, use_list=False, match=None)

Make o a tuple

test_eq(tuplify(None),())
test_eq(tuplify([1,2,3]),(1,2,3))
test_eq(tuplify(1,match=[1,2,3]),(1,1,1))

---

source

true

 true (x)

Test whether x is truthy; collections with >0 elements are considered True

[(o,true(o)) for o in
 (array(0),array(1),array([0]),array([0,1]),1,0,'',None)]

[(array(0), False),
 (array(1), True),
 (array([0]), True),
 (array([0, 1]), True),
 (1, True),
 (0, False),
 ('', False),
 (None, False)]

---

source

NullType

 NullType ()

An object that is False and can be called, chained, and indexed

bool(null.hi().there[3])

False

---

source

tonull

 tonull (x)

Convert None to null

bool(tonull(None).hi().there[3])

False

---

source

get_class

 get_class (nm, *fld_names, sup=None, doc=None, funcs=None, anno=None,
            **flds)

Dynamically create a class, optionally inheriting from sup, containing fld_names

_t = get_class('_t', 'a', b=2, anno={'b':int})
t = _t()
test_eq(t.a, None)
test_eq(t.b, 2)
t = _t(1, b=3)
test_eq(t.a, 1)
test_eq(t.b, 3)
t = _t(1, 3)
test_eq(t.a, 1)
test_eq(t.b, 3)
test_eq(t, pickle.loads(pickle.dumps(t)))
test_eq(_t.__annotations__, {'b':int, 'a':typing.Any})
repr(t)

'_t(a=1, b=3)'

Most often you’ll want to call mk_class, since it adds the class to your module. See mk_class for more details and examples of use (which also apply to get_class).

---

source

mk_class

 mk_class (nm, *fld_names, sup=None, doc=None, funcs=None, mod=None,
           anno=None, **flds)

Create a class using get_class and add to the caller’s module

Any kwargs will be added as class attributes, and sup is an optional (tuple of) base classes.

mk_class('_t', a=1, sup=dict)
t = _t()
test_eq(t.a, 1)
assert(isinstance(t,dict))

A __init__ is provided that sets attrs for any kwargs, and for any args (matching by position to fields), along with a __repr__ which prints all attrs. The docstring is set to doc. You can pass funcs which will be added as attrs with the function names.

def foo(self): return 1
mk_class('_t', 'a', sup=dict, doc='test doc', funcs=foo)

t = _t(3, b=2)
test_eq(t.a, 3)
test_eq(t.b, 2)
test_eq(t.foo(), 1)
test_eq(t.__doc__, 'test doc')
t

{}

---

source

wrap_class

 wrap_class (nm, *fld_names, sup=None, doc=None, funcs=None, **flds)

Decorator: makes function a method of a new class nm passing parameters to mk_class

@wrap_class('_t', a=2)
def bar(self,x): return x+1

t = _t()
test_eq(t.a, 2)
test_eq(t.bar(3), 4)

---

source

ignore_exceptions

 ignore_exceptions ()

Context manager to ignore exceptions

with ignore_exceptions(): 
    # Exception will be ignored
    raise Exception

---

source

exec_local

 exec_local (code, var_name)

Call exec on code and return the var var_name

test_eq(exec_local("a=1", "a"), 1)

---

source

risinstance

 risinstance (types, obj=None)

Curried isinstance but with args reversed

assert risinstance(int, 1)
assert not risinstance(str, 0)
assert risinstance(int)(1)
assert not risinstance(int)(None)

types can also be strings:

assert risinstance(('str','int'), 'a')
assert risinstance('str', 'a')
assert not risinstance('int', 'a')

---

source

ver2tuple

 ver2tuple (v:str)

test_eq(ver2tuple('3.8.1'), (3,8,1))
test_eq(ver2tuple('3.1'), (3,1,0))
test_eq(ver2tuple('3.'), (3,0,0))
test_eq(ver2tuple('3'), (3,0,0))

NoOp

These are used when you need a pass-through function.

---

noop

 noop (x=None, *args, **kwargs)

Do nothing

noop()
test_eq(noop(1),1)

---

noops

 noops (x=None, *args, **kwargs)

Do nothing (method)

class _t: foo=noops
test_eq(_t().foo(1),1)

Infinite Lists

These lists are useful for things like padding an array or adding index column(s) to arrays.

Inf defines the following properties:

• count: itertools.count()
• zeros: itertools.cycle([0])
• ones : itertools.cycle([1])
• nones: itertools.cycle([None])

test_eq([o for i,o in zip(range(5), Inf.count)],
        [0, 1, 2, 3, 4])

test_eq([o for i,o in zip(range(5), Inf.zeros)],
        [0]*5)

test_eq([o for i,o in zip(range(5), Inf.ones)],
        [1]*5)

test_eq([o for i,o in zip(range(5), Inf.nones)],
        [None]*5)

Operator Functions

---

source

in_

 in_ (x, a)

True if x in a

# test if element is in another
assert in_('c', ('b', 'c', 'a'))
assert in_(4, [2,3,4,5])
assert in_('t', 'fastai')
test_fail(in_('h', 'fastai'))

# use in_ as a partial
assert in_('fastai')('t')
assert in_([2,3,4,5])(4)
test_fail(in_('fastai')('h'))

In addition to in_, the following functions are provided matching the behavior of the equivalent versions in operator: lt gt le ge eq ne add sub mul truediv is_ is_not mod.

lt(3,5),gt(3,5),is_(None,None),in_(0,[1,2]),mod(3,2)

(True, False, True, False, 1)

Similarly to _in, they also have additional functionality: if you only pass one param, they return a partial function that passes that param as the second positional parameter.

lt(5)(3),gt(5)(3),is_(None)(None),in_([1,2])(0),mod(2)(3)

(True, False, True, False, 1)

---

source

ret_true

 ret_true (*args, **kwargs)

Predicate: always True

assert ret_true(1,2,3)
assert ret_true(False)

---

source

ret_false

 ret_false (*args, **kwargs)

Predicate: always False

---

source

stop

 stop (e=<class 'StopIteration'>)

Raises exception e (by default StopIteration)

---

source

gen

 gen (func, seq, cond=<function ret_true>)

Like (func(o) for o in seq if cond(func(o))) but handles StopIteration

test_eq(gen(noop, Inf.count, lt(5)),
        range(5))
test_eq(gen(operator.neg, Inf.count, gt(-5)),
        [0,-1,-2,-3,-4])
test_eq(gen(lambda o:o if o<5 else stop(), Inf.count),
        range(5))

---

source

chunked

 chunked (it, chunk_sz=None, drop_last=False, n_chunks=None)

Return batches from iterator it of size chunk_sz (or return n_chunks total)

---

source

chunked

 chunked (it, chunk_sz=None, drop_last=False, n_chunks=None, pad=False,
          pad_val=None)

Return batches from iterator it of size chunk_sz (or return n_chunks total)

Note that you must pass either chunk_sz, or n_chunks, but not both.

t = list(range(10))
test_eq(chunked(t,3),      [[0,1,2], [3,4,5], [6,7,8], [9]])
test_eq(chunked(t,3,True), [[0,1,2], [3,4,5], [6,7,8],    ])

t = map(lambda o:stop() if o==6 else o, Inf.count)
test_eq(chunked(t,3), [[0, 1, 2], [3, 4, 5]])
t = map(lambda o:stop() if o==7 else o, Inf.count)
test_eq(chunked(t,3), [[0, 1, 2], [3, 4, 5], [6]])

t = np.arange(10)
test_eq(chunked(t,3),      [[0,1,2], [3,4,5], [6,7,8], [9]])
test_eq(chunked(t,3,True), [[0,1,2], [3,4,5], [6,7,8],    ])

test_eq(chunked([], 3),          [])
test_eq(chunked([], n_chunks=3), [])

Pass pad=True and an optional pad_val to pad the last chunk:

t = list(range(10))
test_eq(chunked(t,3,pad=True), [[0,1,2], [3,4,5], [6,7,8], [9,None,None]])
test_eq(chunked(t,3,pad=True,pad_val=0), [[0,1,2], [3,4,5], [6,7,8], [9,0,0]])
test_eq(chunked(t,4,pad=True,pad_val=-1), [[0,1,2,3], [4,5,6,7], [8,9,-1,-1]])
test_eq(chunked(range(5),2,pad=True), [[0,1], [2,3], [4,None]])

---

source

otherwise

 otherwise (x, tst, y)

y if tst(x) else x

test_eq(otherwise(2+1, gt(3), 4), 3)
test_eq(otherwise(2+1, gt(2), 4), 4)

Attribute Helpers

These functions reduce boilerplate when setting or manipulating attributes or properties of objects.

---

source

custom_dir

 custom_dir (c, add)

Implement custom __dir__, adding add to cls

custom_dir allows you extract the __dict__ property of a class and appends the list add to it.

class _T: 
    def f(): pass

s = custom_dir(_T(), add=['foo', 'bar'])
assert {'foo', 'bar', 'f'}.issubset(s)

---

source

AttrDict

dict subclass that also provides access to keys as attrs

d = AttrDict(a=1,b="two")
test_eq(d.a, 1)
test_eq(d['b'], 'two')
test_eq(d.get('c','nope'), 'nope')
d.b = 2
test_eq(d.b, 2)
test_eq(d['b'], 2)
d['b'] = 3
test_eq(d['b'], 3)
test_eq(d.b, 3)
assert 'a' in dir(d)

AttrDict will pretty print in Jupyter Notebooks:

_test_dict = {'a':1, 'b': {'c':1, 'd':2}, 'c': {'c':1, 'd':2}, 'd': {'c':1, 'd':2},
              'e': {'c':1, 'd':2}, 'f': {'c':1, 'd':2, 'e': 4, 'f':[1,2,3,4,5]}}
AttrDict(_test_dict)

{ 'a': 1,
  'b': {'c': 1, 'd': 2},
  'c': {'c': 1, 'd': 2},
  'd': {'c': 1, 'd': 2},
  'e': {'c': 1, 'd': 2},
  'f': {'c': 1, 'd': 2, 'e': 4, 'f': [1, 2, 3, 4, 5]}}

---

source

AttrDictDefault

 AttrDictDefault (*args, default_=None, **kwargs)

AttrDict subclass that returns None for missing attrs

d = AttrDictDefault(a=1,b="two", default_='nope')
test_eq(d.a, 1)
test_eq(d['b'], 'two')
test_eq(d.c, 'nope')

---

source

NS

SimpleNamespace subclass that also adds iter and dict support

This is very similar to AttrDict, but since it starts with SimpleNamespace, it has some differences in behavior. You can use it just like SimpleNamespace:

d = NS(**_test_dict)
d

namespace(a=1,
          b={'c': 1, 'd': 2},
          c={'c': 1, 'd': 2},
          d={'c': 1, 'd': 2},
          e={'c': 1, 'd': 2},
          f={'c': 1, 'd': 2, 'e': 4, 'f': [1, 2, 3, 4, 5]})

…but you can also index it to get/set:

d['a']

1

…and iterate t:

list(d)

['a', 'b', 'c', 'd', 'e', 'f']

---

source

get_annotations_ex

 get_annotations_ex (obj, globals=None, locals=None)

Backport of py3.10 get_annotations that returns globals/locals

In Python 3.10 inspect.get_annotations was added. However previous versions of Python are unable to evaluate type annotations correctly if from future import __annotations__ is used. Furthermore, all annotations are evaluated, even if only some subset are needed. get_annotations_ex provides the same functionality as inspect.get_annotations, but works on earlier versions of Python, and returns the globals and locals needed to evaluate types.

---

source

eval_type

 eval_type (t, glb, loc)

eval a type or collection of types, if needed, for annotations in py3.10+

In py3.10, or if from future import __annotations__ is used, a is a str:

class _T2a: pass
def func(a: _T2a): pass
ann,glb,loc = get_annotations_ex(func)

eval_type(ann['a'], glb, loc)

__main__._T2a

| is supported for defining Union types when using eval_type even for python versions prior to 3.9:

class _T2b: pass
def func(a: _T2a|_T2b): pass
ann,glb,loc = get_annotations_ex(func)

eval_type(ann['a'], glb, loc)

typing.Union[__main__._T2a, __main__._T2b]

---

source

type_hints

 type_hints (f)

Like typing.get_type_hints but returns {} if not allowed type

def type_hints(f):
    "Like `typing.get_type_hints` but returns `{}` if not allowed type"
    if not isinstance(f, _allowed_types): return {}
    ann,glb,loc = get_annotations_ex(f)
    return {k:_eval_type(v,glb,loc) for k,v in ann.items()}

For example, type func is allowed so type_hints returns the same value as typing.get_hints:

def f(a:int)->bool: ... # a function with type hints (allowed)
exp = {'a':int,'return':bool}
test_eq(type_hints(f), typing.get_type_hints(f))
test_eq(type_hints(f), exp)

However, class is not an allowed type, so type_hints returns {}:

class _T:
    def __init__(self, a:int=0)->bool: ...
assert not type_hints(_T)

---

source

annotations

 annotations (o)

Annotations for o, or type(o)

This supports a wider range of situations than type_hints, by checking type() and __init__ for annotations too:

for o in _T,_T(),_T.__init__,f: test_eq(annotations(o), exp)
assert not annotations(int)
assert not annotations(print)

---

source

anno_ret

 anno_ret (func)

Get the return annotation of func

def f(x) -> float: return x
test_eq(anno_ret(f), float)

def f(x) -> typing.Tuple[float,float]: return x
assert anno_ret(f)==typing.Tuple[float,float]

If your return annotation is None, anno_ret will return NoneType (and not None):

def f(x) -> None: return x

test_eq(anno_ret(f), NoneType)
assert anno_ret(f) is not None # returns NoneType instead of None

If your function does not have a return type, or if you pass in None instead of a function, then anno_ret returns None:

def f(x): return x

test_eq(anno_ret(f), None)
test_eq(anno_ret(None), None) # instead of passing in a func, pass in None

---

source

signature_ex

 signature_ex (obj, eval_str:bool=False)

Backport of inspect.signature(..., eval_str=True to <py310

---

source

union2tuple

 union2tuple (t)

test_eq(union2tuple(Union[int,str]), (int,str))
test_eq(union2tuple(int), int)
assert union2tuple(Tuple[int,str])==Tuple[int,str]
test_eq(union2tuple((int,str)), (int,str))
if UnionType: test_eq(union2tuple(int|str), (int,str))

---

source

argnames

 argnames (f, frame=False)

Names of arguments to function or frame f

test_eq(argnames(f), ['x'])

---

source

with_cast

 with_cast (f)

Decorator which uses any parameter annotations as preprocessing functions

@with_cast
def _f(a, b:Path, c:str='', d=0): return (a,b,c,d)

test_eq(_f(1, '.', 3), (1,Path('.'),'3',0))
test_eq(_f(1, '.'), (1,Path('.'),'',0))

@with_cast
def _g(a:int=0)->str: return a

test_eq(_g(4.0), '4')
test_eq(_g(4.4), '4')
test_eq(_g(2), '2')

---

source

store_attr

 store_attr (names=None, but='', cast=False, store_args=None, **attrs)

Store params named in comma-separated names from calling context into attrs in self

In it’s most basic form, you can use store_attr to shorten code like this:

class T:
    def __init__(self, a,b,c): self.a,self.b,self.c = a,b,c

…to this:

class T:
    def __init__(self, a,b,c): store_attr('a,b,c', self)

This class behaves as if we’d used the first form:

t = T(1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2

class T1:
    def __init__(self, a,b,c): store_attr()

In addition, it stores the attrs as a dict in __stored_args__, which you can use for display, logging, and so forth.

test_eq(t.__stored_args__, {'a':1, 'b':3, 'c':2})

Since you normally want to use the first argument (often called self) for storing attributes, it’s optional:

class T:
    def __init__(self, a,b,c:str): store_attr('a,b,c')

t = T(1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2

With cast=True any parameter annotations will be used as preprocessing functions for the corresponding arguments:

class T:
    def __init__(self, a:listify, b, c:str): store_attr('a,b,c', cast=True)

t = T(1,c=2,b=3)
assert t.a==[1] and t.b==3 and t.c=='2'

You can inherit from a class using store_attr, and just call it again to add in any new attributes added in the derived class:

class T2(T):
    def __init__(self, d, **kwargs):
        super().__init__(**kwargs)
        store_attr('d')

t = T2(d=1,a=2,b=3,c=4)
assert t.a==2 and t.b==3 and t.c==4 and t.d==1

You can skip passing a list of attrs to store. In this case, all arguments passed to the method are stored:

class T:
    def __init__(self, a,b,c): store_attr()

t = T(1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2

class T4(T):
    def __init__(self, d, **kwargs):
        super().__init__(**kwargs)
        store_attr()

t = T4(4, a=1,c=2,b=3)
assert t.a==1 and t.b==3 and t.c==2 and t.d==4

class T4:
    def __init__(self, *, a: int, b: float = 1):
        store_attr()
        
t = T4(a=3)
assert t.a==3 and t.b==1
t = T4(a=3, b=2)
assert t.a==3 and t.b==2

You can skip some attrs by passing but:

class T:
    def __init__(self, a,b,c): store_attr(but='a')

t = T(1,c=2,b=3)
assert t.b==3 and t.c==2
assert not hasattr(t,'a')

You can also pass keywords to store_attr, which is identical to setting the attrs directly, but also stores them in __stored_args__.

class T:
    def __init__(self): store_attr(a=1)

t = T()
assert t.a==1

You can also use store_attr inside functions.

def create_T(a, b):
    t = SimpleNamespace()
    store_attr(self=t)
    return t

t = create_T(a=1, b=2)
assert t.a==1 and t.b==2

---

source

attrdict

 attrdict (o, *ks, default=None)

Dict from each k in ks to getattr(o,k)

class T:
    def __init__(self, a,b,c): store_attr()

t = T(1,c=2,b=3)
test_eq(attrdict(t,'b','c'), {'b':3, 'c':2})

---

source

properties

 properties (cls, *ps)

Change attrs in cls with names in ps to properties

class T:
    def a(self): return 1
    def b(self): return 2
properties(T,'a')

test_eq(T().a,1)
test_eq(T().b(),2)

---

source

camel2words

 camel2words (s, space=' ')

Convert CamelCase to ‘spaced words’

test_eq(camel2words('ClassAreCamel'), 'Class Are Camel')

---

source

camel2snake

 camel2snake (name)

Convert CamelCase to snake_case

test_eq(camel2snake('ClassAreCamel'), 'class_are_camel')
test_eq(camel2snake('Already_Snake'), 'already__snake')

---

source

snake2camel

 snake2camel (s)

Convert snake_case to CamelCase

test_eq(snake2camel('a_b_cc'), 'ABCc')

---

source

class2attr

 class2attr (cls_name)

Return the snake-cased name of the class; strip ending cls_name if it exists.

class Parent:
    @property
    def name(self): return class2attr(self, 'Parent')

class ChildOfParent(Parent): pass
class ParentChildOf(Parent): pass

p = Parent()
cp = ChildOfParent()
cp2 = ParentChildOf()

test_eq(p.name, 'parent')
test_eq(cp.name, 'child_of')
test_eq(cp2.name, 'parent_child_of')

---

source

getcallable

 getcallable (o, attr)

Calls getattr with a default of noop

class Math:
    def addition(self,a,b): return a+b

m = Math()

test_eq(getcallable(m, "addition")(a=1,b=2), 3)
test_eq(getcallable(m, "subtraction")(a=1,b=2), None)

---

source

getattrs

 getattrs (o, *attrs, default=None)

List of all attrs in o

from fractions import Fraction

getattrs(Fraction(1,2), 'numerator', 'denominator')

[1, 2]

---

source

hasattrs

 hasattrs (o, attrs)

Test whether o contains all attrs

assert hasattrs(1,('imag','real'))
assert not hasattrs(1,('imag','foo'))

---

source

setattrs

 setattrs (dest, flds, src)

d = dict(a=1,bb="2",ignore=3)
o = SimpleNamespace()
setattrs(o, "a,bb", d)
test_eq(o.a, 1)
test_eq(o.bb, "2")

d = SimpleNamespace(a=1,bb="2",ignore=3)
o = SimpleNamespace()
setattrs(o, "a,bb", d)
test_eq(o.a, 1)
test_eq(o.bb, "2")

---

source

try_attrs

 try_attrs (obj, *attrs)

Return first attr that exists in obj

test_eq(try_attrs(1, 'real'), 1)
test_eq(try_attrs(1, 'foobar', 'real'), 1)

Attribute Delegation

---

source

GetAttrBase

 GetAttrBase ()

Basic delegation of __getattr__ and __dir__

---

source

GetAttr

 GetAttr ()

Inherit from this to have all attr accesses in self._xtra passed down to self.default

Inherit from GetAttr to have attr access passed down to an instance attribute. This makes it easy to create composites that don’t require callers to know about their components. For a more detailed discussion of how this works as well as relevant context, we suggest reading the delegated composition section of this blog article.

You can customise the behaviour of GetAttr in subclasses via; - _default - By default, this is set to 'default', so attr access is passed down to self.default - _default can be set to the name of any instance attribute that does not start with dunder __ - _xtra - By default, this is None, so all attr access is passed down - You can limit which attrs get passed down by setting _xtra to a list of attribute names

To illuminate the utility of GetAttr, suppose we have the following two classes, _WebPage which is a superclass of _ProductPage, which we wish to compose like so:

class _WebPage:
    def __init__(self, title, author="Jeremy"):
        self.title,self.author = title,author

class _ProductPage:
    def __init__(self, page, price): self.page,self.price = page,price
        
page = _WebPage('Soap', author="Sylvain")
p = _ProductPage(page, 15.0)

How do we make it so we can just write p.author, instead of p.page.author to access the author attribute? We can use GetAttr, of course! First, we subclass GetAttr when defining _ProductPage. Next, we set self.default to the object whose attributes we want to be able to access directly, which in this case is the page argument passed on initialization:

class _ProductPage(GetAttr):
    def __init__(self, page, price): self.default,self.price = page,price #self.default allows you to access page directly.

p = _ProductPage(page, 15.0)

Now, we can access the author attribute directly from the instance:

test_eq(p.author, 'Sylvain')

If you wish to store the object you are composing in an attribute other than self.default, you can set the class attribute _data as shown below. This is useful in the case where you might have a name collision with self.default:

class _C(GetAttr):
    _default = '_data' # use different component name; `self._data` rather than `self.default`
    def __init__(self,a): self._data = a
    def foo(self): noop

t = _C('Hi')
test_eq(t._data, 'Hi') 
test_fail(lambda: t.default) # we no longer have self.default
test_eq(t.lower(), 'hi')
test_eq(t.upper(), 'HI')
assert 'lower' in dir(t)
assert 'upper' in dir(t)

By default, all attributes and methods of the object you are composing are retained. In the below example, we compose a str object with the class _C. This allows us to directly call string methods on instances of class _C, such as str.lower() or str.upper():

class _C(GetAttr):
    # allow all attributes and methods to get passed to `self.default` (by leaving _xtra=None)
    def __init__(self,a): self.default = a
    def foo(self): noop

t = _C('Hi')
test_eq(t.lower(), 'hi')
test_eq(t.upper(), 'HI')
assert 'lower' in dir(t)
assert 'upper' in dir(t)

However, you can choose which attributes or methods to retain by defining a class attribute _xtra, which is a list of allowed attribute and method names to delegate. In the below example, we only delegate the lower method from the composed str object when defining class _C:

class _C(GetAttr):
    _xtra = ['lower'] # specify which attributes get passed to `self.default`
    def __init__(self,a): self.default = a
    def foo(self): noop

t = _C('Hi')
test_eq(t.default, 'Hi')
test_eq(t.lower(), 'hi')
test_fail(lambda: t.upper()) # upper wasn't in _xtra, so it isn't available to be called
assert 'lower' in dir(t)
assert 'upper' not in dir(t)

You must be careful to properly set an instance attribute in __init__ that corresponds to the class attribute _default. The below example sets the class attribute _default to data, but erroneously fails to define self.data (and instead defines self.default).

Failing to properly set instance attributes leads to errors when you try to access methods directly:

class _C(GetAttr):
    _default = 'data' # use a bad component name; i.e. self.data does not exist
    def __init__(self,a): self.default = a
    def foo(self): noop
        
# TODO: should we raise an error when we create a new instance ...
t = _C('Hi')
test_eq(t.default, 'Hi')
# ... or is it enough for all GetAttr features to raise errors
test_fail(lambda: t.data)
test_fail(lambda: t.lower())
test_fail(lambda: t.upper())
test_fail(lambda: dir(t))

---

source

delegate_attr

 delegate_attr (k, to)

Use in __getattr__ to delegate to attr to without inheriting from GetAttr

delegate_attr is a functional way to delegate attributes, and is an alternative to GetAttr. We recommend reading the documentation of GetAttr for more details around delegation.

You can use achieve delegation when you define __getattr__ by using delegate_attr:

class _C:
    def __init__(self, o): self.o = o # self.o corresponds to the `to` argument in delegate_attr.
    def __getattr__(self, k): return delegate_attr(self, k, to='o')
    

t = _C('HELLO') # delegates to a string
test_eq(t.lower(), 'hello')

t = _C(np.array([5,4,3])) # delegates to a numpy array
test_eq(t.sum(), 12)

t = _C(pd.DataFrame({'a': [1,2], 'b': [3,4]})) # delegates to a pandas.DataFrame
test_eq(t.b.max(), 4)

Extensible Types

ShowPrint is a base class that defines a show method, which is used primarily for callbacks in fastai that expect this method to be defined.

Int, Float, and Str extend int, float and str respectively by adding an additional show method by inheriting from ShowPrint.

The code for Int is shown below:

Examples:

Int(0).show()
Float(2.0).show()
Str('Hello').show()

0
2.0
Hello

Collection functions

Functions that manipulate popular python collections.

---

source

partition

 partition (coll, f)

Partition a collection by a predicate

ts,fs = partition(range(10), mod(2))
test_eq(fs, [0,2,4,6,8])
test_eq(ts, [1,3,5,7,9])

---

source

partition_dict

 partition_dict (d, f)

Partition a dict by a predicate that takes key/value params

d = {'a':1, 'b':2, 'c':3, 'd':4}
ts,fs = partition_dict(d, lambda k,v: v%2)
test_eq(fs, {'b':2, 'd':4})
test_eq(ts, {'a':1, 'c':3})

ts,fs = partition_dict(d, lambda k,v: k in 'bc')
test_eq(ts, {'b':2, 'c':3})
test_eq(fs, {'a':1, 'd':4})

---

source

flatten

 flatten (o)

Concatenate all collections and items as a generator

---

source

concat

 concat (colls)

Concatenate all collections and items as a list

concat([(o for o in range(2)),[2,3,4], 5])

[0, 1, 2, 3, 4, 5]

concat([["abc", "xyz"], ["foo", "bar"]])

['abc', 'xyz', 'foo', 'bar']

---

source

strcat

 strcat (its, sep:str='')

Concatenate stringified items its

test_eq(strcat(['a',2]), 'a2')
test_eq(strcat(['a',2], ';'), 'a;2')

---

source

detuplify

 detuplify (x)

If x is a tuple with one thing, extract it

test_eq(detuplify(()),None)
test_eq(detuplify([1]),1)
test_eq(detuplify([1,2]), [1,2])
test_eq(detuplify(np.array([[1,2]])), np.array([[1,2]]))

---

source

replicate

 replicate (item, match)

Create tuple of item copied len(match) times

t = [1,1]
test_eq(replicate([1,2], t),([1,2],[1,2]))
test_eq(replicate(1, t),(1,1))

---

source

setify

 setify (o)

Turn any list like-object into a set.

# test
test_eq(setify(None),set())
test_eq(setify('abc'),{'abc'})
test_eq(setify([1,2,2]),{1,2})
test_eq(setify(range(0,3)),{0,1,2})
test_eq(setify({1,2}),{1,2})

---

source

merge

 merge (*ds)

Merge all dictionaries in ds

test_eq(merge(), {})
test_eq(merge(dict(a=1,b=2)), dict(a=1,b=2))
test_eq(merge(dict(a=1,b=2), dict(b=3,c=4), None), dict(a=1, b=3, c=4))

---

source

range_of

 range_of (x)

All indices of collection x (i.e. list(range(len(x))))

test_eq(range_of([1,1,1,1]), [0,1,2,3])

---

source

groupby

 groupby (x, key, val=<function noop>)

Like itertools.groupby but doesn’t need to be sorted, and isn’t lazy, plus some extensions

test_eq(groupby('aa ab bb'.split(), itemgetter(0)), {'a':['aa','ab'], 'b':['bb']})

You can use an int as key or val (which uses itemgetter; passing a str will use attrgetter), eg:

test_eq(groupby('aa ab bb'.split(), 0), {'a':['aa','ab'], 'b':['bb']})

…and you can use a tuple as key or val (which creates a tuple from the provided keys or vals), eg:

test_eq(groupby('aaa abc bba'.split(), 0, (1,2)), {'a':[('a','a'),('b','c')], 'b':[('b','a')]})

Here’s an example of how to invert a grouping, and using a val function:

d = {0: [1, 3, 7], 2: [3], 3: [5], 4: [8], 5: [4], 7: [5]}
groupby(((o,k) for k,v in d.items() for o in v), 0, 1)

{1: [0], 3: [0, 2], 7: [0], 5: [3, 7], 8: [4], 4: [5]}

---

source

last_index

 last_index (x, o)

Finds the last index of occurence of x in o (returns -1 if no occurence)

test_eq(last_index(9, [1, 2, 9, 3, 4, 9, 10]), 5)
test_eq(last_index(6, [1, 2, 9, 3, 4, 9, 10]), -1)

---

source

filter_dict

 filter_dict (d, func)

Filter a dict using func, applied to keys and values

letters = {o:chr(o) for o in range(65,73)}
letters

{65: 'A', 66: 'B', 67: 'C', 68: 'D', 69: 'E', 70: 'F', 71: 'G', 72: 'H'}

filter_dict(letters, lambda k,v: k<67 or v in 'FG')

{65: 'A', 66: 'B', 70: 'F', 71: 'G'}

---

source

filter_keys

 filter_keys (d, func)

Filter a dict using func, applied to keys

filter_keys(letters, lt(67))

{65: 'A', 66: 'B'}

---

source

filter_values

 filter_values (d, func)

Filter a dict using func, applied to values

filter_values(letters, in_('FG'))

{70: 'F', 71: 'G'}

---

source

cycle

 cycle (o)

Like itertools.cycle except creates list of Nones if o is empty

test_eq(itertools.islice(cycle([1,2,3]),5), [1,2,3,1,2])
test_eq(itertools.islice(cycle([]),3), [None]*3)
test_eq(itertools.islice(cycle(None),3), [None]*3)
test_eq(itertools.islice(cycle(1),3), [1,1,1])

---

source

zip_cycle

 zip_cycle (x, *args)

Like itertools.zip_longest but cycles through elements of all but first argument

test_eq(zip_cycle([1,2,3,4],list('abc')), [(1, 'a'), (2, 'b'), (3, 'c'), (4, 'a')])

---

source

sorted_ex

 sorted_ex (iterable, key=None, reverse=False, cmp=None, **kwargs)

Like sorted, but if key is str use attrgetter; if int use itemgetter; use cmp comparator function or key with kwargs

Attributes can be used for sorting by passing their name as a string:

class TestObj:
    def __init__(self, x): self.x = x
objs = [TestObj(i) for i in [3,1,2]]
test_eq([o.x for o in sorted_ex(objs, 'x')], [1,2,3])

Tuple/list items can be sorted by index position:

items = [(1,'c'), (2,'b'), (3,'a')]
test_eq(sorted_ex(items, 1), [(3,'a'), (2,'b'), (1,'c')])

A custom key function transforms values:

test_eq(sorted_ex([3,1,2], lambda x: -x), [3,2,1])

You can use a comparison function (returning -1/1/0):

test_eq(sorted_ex([3,1,2], cmp=lambda a,b: 1 if a>b else -1 if a<b else 0), [1,2,3])

Additional parameters can be passed to key/cmp functions:

def key_with_kwargs(x, offset=0): return x + offset
test_eq(sorted_ex([3,1,2], key=key_with_kwargs, offset=10), [1,2,3])

Reverse sort capability:

test_eq(sorted_ex([1,2,3], reverse=True), [3,2,1])

---

source

not_

 not_ (f)

Create new function that negates result of f

def f(a): return a>0
test_eq(f(1),True)
test_eq(not_(f)(1),False)
test_eq(not_(f)(a=-1),True)

---

source

argwhere

 argwhere (iterable, f, negate=False, **kwargs)

Like filter_ex, but return indices for matching items

---

source

filter_ex

 filter_ex (iterable, f=<function noop>, negate=False, gen=False,
            **kwargs)

Like filter, but passing kwargs to f, defaulting f to noop, and adding negate and gen

---

source

range_of

 range_of (a, b=None, step=None)

All indices of collection a, if a is a collection, otherwise range

test_eq(range_of([1,1,1,1]), [0,1,2,3])
test_eq(range_of(4), [0,1,2,3])

---

source

renumerate

 renumerate (iterable, start=0)

Same as enumerate, but returns index as 2nd element instead of 1st

test_eq(renumerate('abc'), (('a',0),('b',1),('c',2)))

---

source

first

 first (x, f=None, negate=False, **kwargs)

First element of x, optionally filtered by f, or None if missing

test_eq(first(['a', 'b', 'c', 'd', 'e']), 'a')
test_eq(first([False]), False)
test_eq(first([False], noop), None)

---

source

last

 last (x, f=None, negate=False, **kwargs)

Last element of x, optionally filtered by f, or None if missing

test_eq(last(['a', 'b', 'c', 'd', 'e']), 'e')
test_eq(last([False]), False)
test_eq(last([False], noop), None)

---

source

only

 only (o)

Return the only item of o, raise if o doesn’t have exactly one item

---

source

nested_attr

 nested_attr (o, attr, default=None)

Same as getattr, but if attr includes a ., then looks inside nested objects

class CustomIndexable:
    def __init__(self): self.data = {'a':1,'b':'v','c':{'d':5}}
    def __getitem__(self, key): return self.data[key]

custom_indexable = CustomIndexable()
test_eq(nested_attr(custom_indexable,'a'),1)
test_eq(nested_attr(custom_indexable,'c.d'),5)
test_eq(nested_attr(custom_indexable,'e'),None)

class TestObj: def init(self): self.nested = {‘key’: [1, 2, {‘inner’: ‘value’}]} test_obj = TestObj()

test_eq(nested_attr(test_obj, ‘nested.key.2.inner’),‘value’) test_eq(nested_attr([1, 2, 3], ‘1’),2)

b = {'a':1,'b':'v','c':{'d':5}}
test_eq(nested_attr(b,'b'),'v')
test_eq(nested_attr(b,'c.d'),5)

a = SimpleNamespace(b=(SimpleNamespace(c=1)))
test_eq(nested_attr(a, 'b.c'), getattr(getattr(a, 'b'), 'c'))
test_eq(nested_attr(a, 'b.d'), None)
test_eq(nested_attr(b, 'a'), 1)

---

source

nested_setdefault

 nested_setdefault (o, attr, default)

Same as setdefault, but if attr includes a ., then looks inside nested objects

---

source

nested_callable

 nested_callable (o, attr)

Same as nested_attr but if not found will return noop

a = SimpleNamespace(b=(SimpleNamespace(c=1)))
test_eq(nested_callable(a, 'b.c'), getattr(getattr(a, 'b'), 'c'))
test_eq(nested_callable(a, 'b.d'), noop)

---

source

nested_idx

 nested_idx (coll, *idxs)

Index into nested collections, dicts, etc, with idxs

a = {'b':[1,{'c':2}]}
test_eq(nested_idx(a, 'nope'), None)
test_eq(nested_idx(a, 'nope', 'nup'), None)
test_eq(nested_idx(a, 'b', 3), None)
test_eq(nested_idx(a), a)
test_eq(nested_idx(a, 'b'), [1,{'c':2}])
test_eq(nested_idx(a, 'b', 1), {'c':2})
test_eq(nested_idx(a, 'b', 1, 'c'), 2)

a = SimpleNamespace(b=[1,{'c':2}])
test_eq(nested_idx(a, 'nope'), None)
test_eq(nested_idx(a, 'nope', 'nup'), None)
test_eq(nested_idx(a, 'b', 3), None)
test_eq(nested_idx(a), a)
test_eq(nested_idx(a, 'b'), [1,{'c':2}])
test_eq(nested_idx(a, 'b', 1), {'c':2})
test_eq(nested_idx(a, 'b', 1, 'c'), 2)

---

source

set_nested_idx

 set_nested_idx (coll, value, *idxs)

Set value indexed like `nested_idx

set_nested_idx(a, 3, 'b', 0)
test_eq(nested_idx(a, 'b', 0), 3)

---

source

val2idx

 val2idx (x)

Dict from value to index

test_eq(val2idx([1,2,3]), {3:2,1:0,2:1})

---

source

uniqueify

 uniqueify (x, sort=False, bidir=False, start=None)

Unique elements in x, optional sort, optional return reverse correspondence, optional prepend with elements.

t = [1,1,0,5,0,3]
test_eq(uniqueify(t),[1,0,5,3])
test_eq(uniqueify(t, sort=True),[0,1,3,5])
test_eq(uniqueify(t, start=[7,8,6]), [7,8,6,1,0,5,3])
v,o = uniqueify(t, bidir=True)
test_eq(v,[1,0,5,3])
test_eq(o,{1:0, 0: 1, 5: 2, 3: 3})
v,o = uniqueify(t, sort=True, bidir=True)
test_eq(v,[0,1,3,5])
test_eq(o,{0:0, 1: 1, 3: 2, 5: 3})

---

source

loop_first_last

 loop_first_last (values)

Iterate and generate a tuple with a flag for first and last value.

test_eq(loop_first_last(range(3)), [(True,False,0), (False,False,1), (False,True,2)])

---

source

loop_first

 loop_first (values)

Iterate and generate a tuple with a flag for first value.

test_eq(loop_first(range(3)), [(True,0), (False,1), (False,2)])

---

source

loop_last

 loop_last (values)

Iterate and generate a tuple with a flag for last value.

test_eq(loop_last(range(3)), [(False,0), (False,1), (True,2)])

---

source

first_match

 first_match (lst, f, default=None)

First element of lst matching predicate f, or default if none

a = [0,2,4,5,6,7,10]
test_eq(first_match(a, lambda o:o%2), 3)

---

source

last_match

 last_match (lst, f, default=None)

Last element of lst matching predicate f, or default if none

test_eq(last_match(a, lambda o:o%2), 5)

fastuple

A tuple with extended functionality.

---

source

fastuple

 fastuple (x=None, *rest)

A tuple with elementwise ops and more friendly init behavior

Friendly init behavior

Common failure modes when trying to initialize a tuple in python:

tuple(3)
> TypeError: 'int' object is not iterable

or

tuple(3, 4)
> TypeError: tuple expected at most 1 arguments, got 2

However, fastuple allows you to define tuples like this and in the usual way:

test_eq(fastuple(3), (3,))
test_eq(fastuple(3,4), (3, 4))
test_eq(fastuple((3,4)), (3, 4))

Elementwise operations

---

source

fastuple.add

 fastuple.add (*args)

+ is already defined in tuple for concat, so use add instead

test_eq(fastuple.add((1,1),(2,2)), (3,3))
test_eq_type(fastuple(1,1).add(2), fastuple(3,3))
test_eq(fastuple('1','2').add('2'), fastuple('12','22'))

---

source

fastuple.mul

 fastuple.mul (*args)

* is already defined in tuple for replicating, so use mul instead

test_eq_type(fastuple(1,1).mul(2), fastuple(2,2))

Other Elementwise Operations

Additionally, the following elementwise operations are available: - le: less than or equal - eq: equal - gt: greater than - min: minimum of

test_eq(fastuple(3,1).le(1), (False, True))
test_eq(fastuple(3,1).eq(1), (False, True))
test_eq(fastuple(3,1).gt(1), (True, False))
test_eq(fastuple(3,1).min(2), (2,1))

You can also do other elementwise operations like negate a fastuple, or subtract two fastuples:

test_eq(-fastuple(1,2), (-1,-2))
test_eq(~fastuple(1,0,1), (False,True,False))

test_eq(fastuple(1,1)-fastuple(2,2), (-1,-1))

test_eq(type(fastuple(1)), fastuple)
test_eq_type(fastuple(1,2), fastuple(1,2))
test_ne(fastuple(1,2), fastuple(1,3))
test_eq(fastuple(), ())

Functions on Functions

Utilities for functional programming or for defining, modifying, or debugging functions.

---

source

bind

 bind (func, *pargs, **pkwargs)

Same as partial, except you can use arg0 arg1 etc param placeholders

bind is the same as partial, but also allows you to reorder positional arguments using variable name(s) arg{i} where i refers to the zero-indexed positional argument. bind as implemented currently only supports reordering of up to the first 5 positional arguments.

Consider the function myfunc below, which has 3 positional arguments. These arguments can be referenced as arg0, arg1, and arg1, respectively.

def myfn(a,b,c,d=1,e=2): return(a,b,c,d,e)

In the below example we bind the positional arguments of myfn as follows:

• The second input 14, referenced by arg1, is substituted for the first positional argument.
• We supply a default value of 17 for the second positional argument.
• The first input 19, referenced by arg0, is subsituted for the third positional argument.

test_eq(bind(myfn, arg1, 17, arg0, e=3)(19,14), (14,17,19,1,3))

In this next example:

• We set the default value to 17 for the first positional argument.
• The first input 19 refrenced by arg0, becomes the second positional argument.
• The second input 14 becomes the third positional argument.
• We override the default the value for named argument e to 3.

test_eq(bind(myfn, 17, arg0, e=3)(19,14), (17,19,14,1,3))

This is an example of using bind like partial and do not reorder any arguments:

test_eq(bind(myfn)(17,19,14), (17,19,14,1,2))

bind can also be used to change default values. In the below example, we use the first input 3 to override the default value of the named argument e, and supply default values for the first three positional arguments:

test_eq(bind(myfn, 17,19,14,e=arg0)(3), (17,19,14,1,3))

---

source

mapt

 mapt (func, *iterables)

Tuplified map

t = [0,1,2,3]
test_eq(mapt(operator.neg, t), (0,-1,-2,-3))

---

source

map_ex

 map_ex (iterable, f, *args, gen=False, **kwargs)

Like map, but use bind, and supports str and indexing

test_eq(map_ex(t,operator.neg), [0,-1,-2,-3])

If f is a string then it is treated as a format string to create the mapping:

test_eq(map_ex(t, '#{}#'), ['#0#','#1#','#2#','#3#'])

If f is a dictionary (or anything supporting __getitem__) then it is indexed to create the mapping:

test_eq(map_ex(t, list('abcd')), list('abcd'))

You can also pass the same arg params that bind accepts:

def f(a=None,b=None): return b
test_eq(map_ex(t, f, b=arg0), range(4))

---

source

compose

 compose (*funcs, order=None)

Create a function that composes all functions in funcs, passing along remaining *args and **kwargs to all

f1 = lambda o,p=0: (o*2)+p
f2 = lambda o,p=1: (o+1)/p
test_eq(f2(f1(3)), compose(f1,f2)(3))
test_eq(f2(f1(3,p=3),p=3), compose(f1,f2)(3,p=3))
test_eq(f2(f1(3,  3),  3), compose(f1,f2)(3,  3))

f1.order = 1
test_eq(f1(f2(3)), compose(f1,f2, order="order")(3))

---

source

maps

 maps (*args, retain=<function noop>)

Like map, except funcs are composed first

test_eq(maps([1]), [1])
test_eq(maps(operator.neg, [1,2]), [-1,-2])
test_eq(maps(operator.neg, operator.neg, [1,2]), [1,2])

---

source

partialler

 partialler (f, *args, order=None, **kwargs)

Like functools.partial but also copies over docstring

def _f(x,a=1):
    "test func"
    return x-a
_f.order=1

f = partialler(_f, 2)
test_eq(f.order, 1)
test_eq(f(3), -1)
f = partialler(_f, a=2, order=3)
test_eq(f.__doc__, "test func")
test_eq(f.order, 3)
test_eq(f(3), _f(3,2))

class partial0:
    "Like `partialler`, but args passed to callable are inserted at started, instead of at end"
    def __init__(self, f, *args, order=None, **kwargs):
        self.f,self.args,self.kwargs = f,args,kwargs
        self.order = ifnone(order, getattr(f,'order',None))
        self.__doc__ = f.__doc__

    def __call__(self, *args, **kwargs): return self.f(*args, *self.args, **kwargs, **self.kwargs)

f = partial0(_f, 2)
test_eq(f.order, 1)
test_eq(f(3), 1) # NB: different to `partialler` example

---

source

instantiate

 instantiate (t)

Instantiate t if it’s a type, otherwise do nothing

test_eq_type(instantiate(int), 0)
test_eq_type(instantiate(1), 1)

---

source

using_attr

 using_attr (f, attr)

Construct a function which applies f to the argument’s attribute attr

t = Path('/a/b.txt')
f = using_attr(str.upper, 'name')
test_eq(f(t), 'B.TXT')

Self (with an uppercase S)

A Concise Way To Create Lambdas

This is a concise way to create lambdas that are calling methods on an object (note the capitalization!)

Self.sum(), for instance, is a shortcut for lambda o: o.sum().

f = Self.sum()
x = np.array([3.,1])
test_eq(f(x), 4.)

# This is equivalent to above
f = lambda o: o.sum()
x = np.array([3.,1])
test_eq(f(x), 4.)

f = Self.argmin()
arr = np.array([1,2,3,4,5])
test_eq(f(arr), arr.argmin())

f = Self.sum().is_integer()
x = np.array([3.,1])
test_eq(f(x), True)

f = Self.sum().real.is_integer()
x = np.array([3.,1])
test_eq(f(x), True)

f = Self.imag()
test_eq(f(3), 0)

f = Self[1]
test_eq(f(x), 1)

Self is also callable, which creates a function which calls any function passed to it, using the arguments passed to Self:

def f(a, b=3): return a+b+2
def g(a, b=3): return a*b
fg = Self(1,b=2)
list(map(fg, [f,g]))

[5, 2]

Patching

---

source

copy_func

 copy_func (f)

Copy a non-builtin function (NB copy.copy does not work for this)

Sometimes it may be desirable to make a copy of a function that doesn’t point to the original object. When you use Python’s built in copy.copy or copy.deepcopy to copy a function, you get a reference to the original object:

import copy as cp

def foo(): pass
a = cp.copy(foo)
b = cp.deepcopy(foo)

a.someattr = 'hello' # since a and b point at the same object, updating a will update b
test_eq(b.someattr, 'hello')

assert a is foo and b is foo

However, with copy_func, you can retrieve a copy of a function without a reference to the original object:

c = copy_func(foo) # c is an indpendent object
assert c is not foo

def g(x, *, y=3): return x+y
test_eq(copy_func(g)(4), 7)

---

source

patch_to

 patch_to (cls, as_prop=False, cls_method=False, set_prop=False)

Decorator: add f to cls

The @patch_to decorator allows you to monkey patch a function into a class as a method:

class _T3(int): pass  

@patch_to(_T3)
def func1(self, a): return self+a

t = _T3(1) # we initialized `t` to a type int = 1
test_eq(t.func1(2), 3) # we add 2 to `t`, so 2 + 1 = 3

You can access instance properties in the usual way via self:

class _T4():
    def __init__(self, g): self.g = g
        
@patch_to(_T4)
def greet(self, x): return self.g + x
        
t = _T4('hello ') # this sets self.g = 'hello '
test_eq(t.greet('world'), 'hello world') #t.greet('world') will append 'world' to 'hello '

You can instead specify that the method should be a class method by setting cls_method=True:

class _T5(int): attr = 3 # attr is a class attribute we will access in a later method
    
@patch_to(_T5, cls_method=True)
def func(cls, x): return cls.attr + x # you can access class attributes in the normal way

test_eq(_T5.func(4), 7)

Additionally you can specify that the function you want to patch should be a class attribute with as_prop=True:

@patch_to(_T5, as_prop=True)
def add_ten(self): return self + 10

t = _T5(4)
test_eq(t.add_ten, 14)

Once you have a property, you can assign a setter with set_prop=True:

class _T2():
    def __init__(self, val): self._val = val

@patch_to(_T2, as_prop=True)
def val(self): return self._val

t = _T2(2)
test_eq(t.val, 2)

@patch_to(_T2, set_prop=True)
def val(self, val): self._val = val

t.val = 3
test_eq(t.val, 3)

Instead of passing one class to the @patch_to decorator, you can pass multiple classes in a tuple to simulteanously patch more than one class with the same method:

class _T6(int): pass
class _T7(int): pass

@patch_to((_T6,_T7))
def func_mult(self, a): return self*a

t = _T6(2)
test_eq(t.func_mult(4), 8)
t = _T7(2)
test_eq(t.func_mult(4), 8)

---

source

patch

 patch (f=None, as_prop=False, cls_method=False, set_prop=False)

Decorator: add f to the first parameter’s class (based on f’s type annotations)

@patch is an alternative to @patch_to that allows you similarly monkey patch class(es) by using type annotations:

class _T8(int): pass  

@patch
def func(self:_T8, a): return self+a

t = _T8(1)  # we initilized `t` to a type int = 1
test_eq(t.func(3), 4) # we add 3 to `t`, so 3 + 1 = 4
test_eq(t.func.__qualname__, '_T8.func')

Similarly to patch_to, you can supply a union of classes instead of a single class in your type annotations to patch multiple classes:

class _T9(int): pass 

@patch
def func2(x:_T8|_T9, a): return x*a # will patch both _T8 and _T9

t = _T8(2)
test_eq(t.func2(4), 8)
test_eq(t.func2.__qualname__, '_T8.func2')

t = _T9(2)
test_eq(t.func2(4), 8)
test_eq(t.func2.__qualname__, '_T9.func2')

Just like patch_to decorator you can use as_prop, set_prop, and cls_method parameters with patch decorator:

@patch(as_prop=True)
def add_ten(self:_T5): return self + 10

t = _T5(4)
test_eq(t.add_ten, 14)

class _T2():
    def __init__(self, val): self._val = val

@patch(as_prop=True)
def val(self:_T2): return self._val

t = _T2(2)
test_eq(t.val, 2)

@patch(set_prop=True)
def val(self:_T2, val): self._val = val

t.val = 3
test_eq(t.val, 3)

class _T5(int): attr = 3 # attr is a class attribute we will access in a later method
    
@patch(cls_method=True)
def func(cls:_T5, x): return cls.attr + x # you can access class attributes in the normal way

test_eq(_T5.func(4), 7)

---

source

patch_property

 patch_property (f)

Deprecated; use patch(as_prop=True) instead

Patching classmethod shouldn’t affect how python’s inheritance works

class FastParent: pass

@patch(cls_method=True)
def type_cls(cls: FastParent): return cls

class FastChild(FastParent): pass

parent = FastParent()
test_eq(parent.type_cls(), FastParent)

child = FastChild()
test_eq(child.type_cls(), FastChild)

Other Helpers

---

source

compile_re

 compile_re (pat)

Compile pat if it’s not None

assert compile_re(None) is None
assert compile_re('a').match('ab')

---

source

ImportEnum

 ImportEnum (value, names=None, module=None, qualname=None, type=None,
             start=1)

An Enum that can have its values imported

_T = ImportEnum('_T', {'foobar':1, 'goobar':2})
_T.imports()
test_eq(foobar, _T.foobar)
test_eq(goobar, _T.goobar)

---

source

StrEnum

 StrEnum (value, names=None, module=None, qualname=None, type=None,
          start=1)

An ImportEnum that behaves like a str

---

source

str_enum

 str_enum (name, *vals)

Simplified creation of StrEnum types

---

source

ValEnum

 ValEnum (value, names=None, module=None, qualname=None, type=None,
          start=1)

An ImportEnum that stringifies using values

_T = str_enum('_T', 'a', 'b')
test_eq(f'{_T.a}', 'a')
test_eq(_T.a, 'a')
test_eq(list(_T.__members__), ['a','b'])
print(_T.a, _T.a.upper())

a A

---

source

Stateful

 Stateful (*args, **kwargs)

A base class/mixin for objects that should not serialize all their state

class _T(Stateful):
    def __init__(self):
        super().__init__()
        self.a=1
        self._state['test']=2

t = _T()
t2 = pickle.loads(pickle.dumps(t))
test_eq(t.a,1)
test_eq(t._state['test'],2)
test_eq(t2.a,1)
test_eq(t2._state,{})

Override _init_state to do any necessary setup steps that are required during __init__ or during deserialization (e.g. pickle.load). Here’s an example of how Stateful simplifies the official Python example for Handling Stateful Objects.

class TextReader(Stateful):
    """Print and number lines in a text file."""
    _stateattrs=('file',)
    def __init__(self, filename):
        self.filename,self.lineno = filename,0
        super().__init__()

    def readline(self):
        self.lineno += 1
        line = self.file.readline()
        if line: return f"{self.lineno}: {line.strip()}"

    def _init_state(self):
        self.file = open(self.filename)
        for _ in range(self.lineno): self.file.readline()

reader = TextReader("00_test.ipynb")
print(reader.readline())
print(reader.readline())

new_reader = pickle.loads(pickle.dumps(reader))
print(reader.readline())

1: {
2: "cells": [
3: {

---

source

NotStr

 NotStr (s)

Behaves like a str, but isn’t an instance of one

s = NotStr("hello")
assert not isinstance(s, str)
test_eq(s, 'hello')
test_eq(s*2, 'hellohello')
test_eq(len(s), 5)

---

source

PrettyString

Little hack to get strings to show properly in Jupyter.

Allow strings with special characters to render properly in Jupyter. Without calling print() strings with special characters are displayed like so:

with_special_chars='a string\nwith\nnew\nlines and\ttabs'
with_special_chars

'a string\nwith\nnew\nlines and\ttabs'

We can correct this with PrettyString:

PrettyString(with_special_chars)

a string
with
new
lines and   tabs

---

source

even_mults

 even_mults (start, stop, n)

Build log-stepped array from start to stop in n steps.

test_eq(even_mults(2,8,3), [2,4,8])
test_eq(even_mults(2,32,5), [2,4,8,16,32])
test_eq(even_mults(2,8,1), 8)

---

source

num_cpus

 num_cpus ()

Get number of cpus

num_cpus()

22

---

source

add_props

 add_props (f, g=None, n=2)

Create properties passing each of range(n) to f

class _T(): a,b = add_props(lambda i,x:i*2)

t = _T()
test_eq(t.a,0)
test_eq(t.b,2)

class _T(): 
    def __init__(self, v): self.v=v
    def _set(i, self, v): self.v[i] = v
    a,b = add_props(lambda i,x: x.v[i], _set)

t = _T([0,2])
test_eq(t.a,0)
test_eq(t.b,2)
t.a = t.a+1
t.b = 3
test_eq(t.a,1)
test_eq(t.b,3)

---

source

str2bool

 str2bool (s)

Case-insensitive convert string s too a bool (y,yes,t,true,on,1->True)

True values are ‘y’, ‘yes’, ‘t’, ‘true’, ‘on’, and ‘1’; false values are ‘n’, ‘no’, ‘f’, ‘false’, ‘off’, and ‘0’. Raises ValueError if ‘val’ is anything else.

for o in "y YES t True on 1".split(): assert str2bool(o)
for o in "n no FALSE off 0".split(): assert not str2bool(o)
for o in 0,None,'',False: assert not str2bool(o)
for o in 1,True: assert str2bool(o)

---

source

str2int

 str2int (s)

Convert s to an int

---

source

str2float

 str2float (s:str)

Convert s to a float

---

source

str2list

 str2list (s:str)

Convert s to a list

---

source

str2date

 str2date (s:str)

date.fromisoformat with empty string handling

---

source

to_date

 to_date (arg)

---

source

to_list

 to_list (arg)

---

source

to_float

 to_float (arg)

---

source

to_int

 to_int (arg)

---

source

to_bool

 to_bool (arg)

---

source

typed

 typed (_func=None, cast=False)

Decorator to check param and return types at runtime, with optional casting

typed validates argument types at runtime. This is in contrast to MyPy which only offers static type checking.

For example, a TypeError will be raised if we try to pass an integer into the first argument of the below function:

@typed
def discount(price:int, pct:float) -> float:
    return (1-pct) * price

with ExceptionExpected(TypeError): discount(100.0, .1)

You can have automatic casting based on heuristics by specifying typed(cast=True). If casting is not possible, a TypeError is raised.

@typed(cast=True)
def discount(price:int, pct:float) -> float:
    return (1-pct) * price

assert 90.0 == discount(100.5, .1) # will auto cast 100.5 to the int 100
assert 90.0 == discount(' 100 ', .1) # will auto cast the str "100" to the int 100
with ExceptionExpected(TypeError): discount("a", .1)

We can also optionally allow multiple types by enumarating the types in a tuple as illustrated below:

@typed
def discount(price:int|float, pct:float): 
    return (1-pct) * price

assert 90.0 == discount(100.0, .1)

@typed(cast=True)
def discount(price:int|None, pct:float):
    return (1-pct) * price

assert 90.0 == discount(100.0, .1)

We currently do not support union types when casting.

@typed(cast=True)
def discount(price:int|float, pct:float):
    return (1-pct) * price

with ExceptionExpected(AssertionError): assert 90.0 == discount("100.0", .1)

typed works with classes, too:

class Foo:
    @typed
    def __init__(self, a:int, b: int, c:str): pass
    @typed(cast=True)
    def test(cls, d:str): return d

with ExceptionExpected(TypeError): Foo(1, 2, 3) 
assert isinstance(Foo(1,2, 'a string').test(10), str)

It also works with custom types.

@typed
def test_foo(foo: Foo): pass

with ExceptionExpected(TypeError): test_foo(1)
test_foo(Foo(1, 2, 'a string'))

class Bar:
    @typed
    def __init__(self, a:int): self.a = a
@typed(cast=True)
def test_bar(bar: Bar): return bar

assert isinstance(test_bar(1), Bar)
test_eq(test_bar(1).a, 1)
with ExceptionExpected(TypeError): test_bar("foobar")

---

source

exec_new

 exec_new (code)

Execute code in a new environment and return it

g = exec_new('a=1')
test_eq(g['a'], 1)

---

source

exec_import

 exec_import (mod, sym)

Import sym from mod in a new environment

Notebook functions

---

ipython_shell

 ipython_shell ()

Same as get_ipython but returns False if not in IPython

---

in_ipython

 in_ipython ()

Check if code is running in some kind of IPython environment

---

in_colab

 in_colab ()

Check if the code is running in Google Colaboratory

---

in_jupyter

 in_jupyter ()

Check if the code is running in a jupyter notebook

---

in_notebook

 in_notebook ()

Check if the code is running in a jupyter notebook

These variables are available as booleans in fastcore.basics as IN_IPYTHON, IN_JUPYTER, IN_COLAB and IN_NOTEBOOK.

IN_IPYTHON, IN_JUPYTER, IN_COLAB, IN_NOTEBOOK

(True, True, False, True)