Foundation

The L class and helpers for it

Foundational Functions

working_directory

 working_directory (path)

Change working directory to path and return to previous on exit.

source

add_docs

 add_docs (cls, cls_doc=None, **docs)

Copy values from docs to cls docstrings, and confirm all public methods are documented

add_docs allows you to add docstrings to a class and its associated methods. This function allows you to group docstrings together seperate from your code, which enables you to define one-line functions as well as organize your code more succintly. We believe this confers a number of benefits which we discuss in our style guide.

Suppose you have the following undocumented class:

class T:
    def foo(self): pass
    def bar(self): pass

You can add documentation to this class like so:

add_docs(T, cls_doc="A docstring for the class.",
            foo="The foo method.",
            bar="The bar method.")

Now, docstrings will appear as expected:

test_eq(T.__doc__, "A docstring for the class.")
test_eq(T.foo.__doc__, "The foo method.")
test_eq(T.bar.__doc__, "The bar method.")

add_docs also validates that all of your public methods contain a docstring. If one of your methods is not documented, it will raise an error:

class T:
    def foo(self): pass
    def bar(self): pass

f=lambda: add_docs(T, "A docstring for the class.", foo="The foo method.")
test_fail(f, contains="Missing docs")

source

docs

 docs (cls)

Decorator version of add_docs, using _docs dict

Instead of using add_docs, you can use the decorator docs as shown below. Note that the docstring for the class can be set with the argument cls_doc:

@docs
class _T:
    def f(self): pass
    def g(cls): pass
    
    _docs = dict(cls_doc="The class docstring", 
                 f="The docstring for method f.",
                 g="A different docstring for method g.")

    
test_eq(_T.__doc__, "The class docstring")
test_eq(_T.f.__doc__, "The docstring for method f.")
test_eq(_T.g.__doc__, "A different docstring for method g.")

For either the docs decorator or the add_docs function, you can still define your docstrings in the normal way. Below we set the docstring for the class as usual, but define the method docstrings through the _docs attribute:

@docs
class _T:
    "The class docstring"
    def f(self): pass
    _docs = dict(f="The docstring for method f.")

    
test_eq(_T.__doc__, "The class docstring")
test_eq(_T.f.__doc__, "The docstring for method f.")

is_iter

 is_iter (o)

Test whether o can be used in a for loop

assert is_iter([1])
assert not is_iter(array(1))
assert is_iter(array([1,2]))
assert (o for o in range(3))

source

coll_repr

 coll_repr (c, max_n=20)

String repr of up to max_n items of (possibly lazy) collection c

coll_repr is used to provide a more informative __repr__ about list-like objects. coll_repr and is used by L to build a __repr__ that displays the length of a list in addition to a preview of a list.

Below is an example of the __repr__ string created for a list of 1000 elements:

test_eq(coll_repr(range(1000),10), '(#1000) [0,1,2,3,4,5,6,7,8,9...]')
test_eq(coll_repr(range(1000), 5), '(#1000) [0,1,2,3,4...]')
test_eq(coll_repr(range(10),   5), '(#10) [0,1,2,3,4...]')
test_eq(coll_repr(range(5),    5), '(#5) [0,1,2,3,4]')

source

is_bool

 is_bool (x)

Check whether x is a bool or None

source

mask2idxs

 mask2idxs (mask)

Convert bool mask or index list to index L

test_eq(mask2idxs([False,True,False,True]), [1,3])
test_eq(mask2idxs(array([False,True,False,True])), [1,3])
test_eq(mask2idxs(array([1,2,3])), [1,2,3])

source

cycle

 cycle (o)

Like itertools.cycle except creates list of Nones if o is empty

test_eq(itertools.islice(cycle([1,2,3]),5), [1,2,3,1,2])
test_eq(itertools.islice(cycle([]),3), [None]*3)
test_eq(itertools.islice(cycle(None),3), [None]*3)
test_eq(itertools.islice(cycle(1),3), [1,1,1])

source

zip_cycle

 zip_cycle (x, *args)

Like itertools.zip_longest but cycles through elements of all but first argument

test_eq(zip_cycle([1,2,3,4],list('abc')), [(1, 'a'), (2, 'b'), (3, 'c'), (4, 'a')])

source

is_indexer

 is_indexer (idx)

Test whether idx will index a single item in a list

You can, for example index a single item in a list with an integer or a 0-dimensional numpy array:

assert is_indexer(1)
assert is_indexer(np.array(1))

However, you cannot index into single item in a list with another list or a numpy array with ndim > 0.

assert not is_indexer([1, 2])
assert not is_indexer(np.array([[1, 2], [3, 4]]))

source

product

 product (xs)

The product of elements of xs, with Nones removed

product([None, 3, 4, 5])

product([])

sum([])

`L` helpers

source

CollBase

 CollBase (items)

Base class for composing a list of items

ColBase is a base class that emulates the functionality of a python list:

class _T(CollBase): pass
l = _T([1,2,3,4,5])

test_eq(len(l), 5) # __len__
test_eq(l[-1], 5); test_eq(l[0], 1) #__getitem__
l[2] = 100; test_eq(l[2], 100)      # __set_item__
del l[0]; test_eq(len(l), 4)        # __delitem__
test_eq(str(l), '[2, 100, 4, 5]')   # __repr__

source

L

 L (x=None, *args, **kwargs)

Behaves like a list of items but can also index with list of indices or masks

L is a drop in replacement for a python list. Inspired by NumPy, L, supports advanced indexing and has additional methods (outlined below) that provide additional functionality and encourage simple expressive code. For example, the code below takes a list of pairs, selects the second item of each pair, takes its absolute value, filters items greater than 4, and adds them up:

from fastcore.utils import gt

d = dict(a=1,b=-5,d=6,e=9).items()
test_eq(L(d).itemgot(1).map(abs).filter(gt(4)).sum(), 20) # abs(-5) + abs(6) + abs(9) = 20; 1 was filtered out.

Read this overview section for a quick tutorial of L, as well as background on the name.

You can create an L from an existing iterable (e.g. a list, range, etc) and access or modify it with an int list/tuple index, mask, int, or slice. All list methods can also be used with L.

t = L(range(12))
test_eq(t, list(range(12)))
test_ne(t, list(range(11)))
t.reverse()
test_eq(t[0], 11)
t[3] = "h"
test_eq(t[3], "h")
t[3,5] = ("j","k")
test_eq(t[3,5], ["j","k"])
test_eq(t, L(t))
test_eq(L(L(1,2),[3,4]), ([1,2],[3,4]))
t

(#12) [11,10,9,'j',7,'k',5,4,3,2,1,0]

Any L is a Sequence so you can use it with methods like random.sample:

assert isinstance(t, Sequence)

import random

random.seed(0)
random.sample(t, 3)

[5, 0, 11]

There are optimized indexers for arrays, tensors, and DataFrames.

import pandas as pd

arr = np.arange(9).reshape(3,3)
t = L(arr, use_list=None)
test_eq(t[1,2], arr[[1,2]])

df = pd.DataFrame({'a':[1,2,3]})
t = L(df, use_list=None)
test_eq(t[1,2], L(pd.DataFrame({'a':[2,3]}, index=[1,2]), use_list=None))

You can also modify an L with append, +, and *.

t = L()
test_eq(t, [])
t.append(1)
test_eq(t, [1])
t += [3,2]
test_eq(t, [1,3,2])
t = t + [4]
test_eq(t, [1,3,2,4])
t = 5 + t
test_eq(t, [5,1,3,2,4])
test_eq(L(1,2,3), [1,2,3])
test_eq(L(1,2,3), L(1,2,3))
t = L(1)*5
t = t.map(operator.neg)
test_eq(t,[-1]*5)
test_eq(~L([True,False,False]), L([False,True,True]))
t = L(range(4))
test_eq(zip(t, L(1).cycle()), zip(range(4),(1,1,1,1)))
t = L.range(100)
test_shuffled(t,t.shuffle())

test_eq(L([]).sum(), 0)
test_eq(L([]).product(), 1)

def _f(x,a=0): return x+a
t = L(1)*5
test_eq(t.map(_f), t)
test_eq(t.map(_f,1), [2]*5)
test_eq(t.map(_f,a=2), [3]*5)

An L can be constructed from anything iterable, although tensors and arrays will not be iterated over on construction, unless you pass use_list to the constructor.

test_eq(L([1,2,3]),[1,2,3])
test_eq(L(L([1,2,3])),[1,2,3])
test_ne(L([1,2,3]),[1,2,])
test_eq(L('abc'),['abc'])
test_eq(L(range(0,3)),[0,1,2])
test_eq(L(o for o in range(0,3)),[0,1,2])
test_eq(L(array(0)),[array(0)])
test_eq(L([array(0),array(1)]),[array(0),array(1)])
test_eq(L(array([0.,1.1]))[0],array([0.,1.1]))
test_eq(L(array([0.,1.1]), use_list=True), [array(0.),array(1.1)])  # `use_list=True` to unwrap arrays/arrays

If match is not None then the created list is same len as match, either by:

If len(items)==1 then items is replicated,
Otherwise an error is raised if match and items are not already the same size.

test_eq(L(1,match=[1,2,3]),[1,1,1])
test_eq(L([1,2],match=[2,3]),[1,2])
test_fail(lambda: L([1,2],match=[1,2,3]))

If you create an L from an existing L then you’ll get back the original object (since L uses the NewChkMeta metaclass).

test_is(L(t), t)

An L is considred equal to a list if they have the same elements. It’s never considered equal to a str a set or a dict even if they have the same elements/keys.

test_eq(L(['a', 'b']), ['a', 'b'])
test_ne(L(['a', 'b']), 'ab')
test_ne(L(['a', 'b']), {'a':1, 'b':2})

`L` Methods

source

L.getitem

 L.__getitem__ (idx)

Retrieve idx (can be list of indices, or mask, or int) items

t = L(range(12))
test_eq(t[1,2], [1,2])                # implicit tuple
test_eq(t[[1,2]], [1,2])              # list
test_eq(t[:3], [0,1,2])               # slice
test_eq(t[[False]*11 + [True]], [11]) # mask
test_eq(t[array(3)], 3)

source

L.setitem

 L.__setitem__ (idx, o)

Set idx (can be list of indices, or mask, or int) items to o (which is broadcast if not iterable)

t[4,6] = 0
test_eq(t[4,6], [0,0])
t[4,6] = [1,2]
test_eq(t[4,6], [1,2])

source

L.unique

 L.unique (sort=False, bidir=False, start=None)

Unique items, in stable order

test_eq(L(4,1,2,3,4,4).unique(), [4,1,2,3])

source

L.val2idx

 L.val2idx ()

Dict from value to index

test_eq(L(1,2,3).val2idx(), {3:2,1:0,2:1})

source

L.groupby

 L.groupby (key, val=<function noop>)

Same as fastcore.basics.groupby

words = L.split('aaa abc bba')
test_eq(words.groupby(0, (1,2)), {'a':[('a','a'),('b','c')], 'b':[('b','a')]})

source

L.filter

 L.filter (f=<function noop>, negate=False, **kwargs)

Create new L filtered by predicate f, passing args and kwargs to f

list(t)

[0, 1, 2, 3, 1, 5, 2, 7, 8, 9, 10, 11]

test_eq(t.filter(lambda o:o<5), [0,1,2,3,1,2])
test_eq(t.filter(lambda o:o<5, negate=True), [5,7,8,9,10,11])

source

L.argwhere

 L.argwhere (f, negate=False, **kwargs)

Like filter, but return indices for matching items

test_eq(t.argwhere(lambda o:o<5), [0,1,2,3,4,6])

source

L.argfirst

 L.argfirst (f, negate=False)

Return index of first matching item

test_eq(t.argfirst(lambda o:o>4), 5)
test_eq(t.argfirst(lambda o:o>4,negate=True),0)

source

L.map

 L.map (f, *args, **kwargs)

Create new L with f applied to all items, passing args and kwargs to f

test_eq(L.range(4).map(operator.neg), [0,-1,-2,-3])

If f is a string then it is treated as a format string to create the mapping:

test_eq(L.range(4).map('#{}#'), ['#0#','#1#','#2#','#3#'])

If f is a dictionary (or anything supporting __getitem__) then it is indexed to create the mapping:

test_eq(L.range(4).map(list('abcd')), list('abcd'))

You can also pass the same arg params that bind accepts:

def f(a=None,b=None): return b
test_eq(L.range(4).map(f, b=arg0), range(4))

source

L.map_dict

 L.map_dict (f=<function noop>, *args, **kwargs)

Like map, but creates a dict from items to function results

test_eq(L(range(1,5)).map_dict(), {1:1, 2:2, 3:3, 4:4})
test_eq(L(range(1,5)).map_dict(operator.neg), {1:-1, 2:-2, 3:-3, 4:-4})

source

L.zip

 L.zip (cycled=False)

Create new L with zip(*items)

t = L([[1,2,3],'abc'])
test_eq(t.zip(), [(1, 'a'),(2, 'b'),(3, 'c')])

t = L([[1,2,3,4],['a','b','c']])
test_eq(t.zip(cycled=True ), [(1, 'a'),(2, 'b'),(3, 'c'),(4, 'a')])
test_eq(t.zip(cycled=False), [(1, 'a'),(2, 'b'),(3, 'c')])

source

L.map_zip

 L.map_zip (f, *args, cycled=False, **kwargs)

Combine zip and starmap

t = L([1,2,3],[2,3,4])
test_eq(t.map_zip(operator.mul), [2,6,12])

source

L.zipwith

 L.zipwith (*rest, cycled=False)

Create new L with self zip with each of *rest

b = [[0],[1],[2,2]]
t = L([1,2,3]).zipwith(b)
test_eq(t, [(1,[0]), (2,[1]), (3,[2,2])])

source

L.map_zipwith

 L.map_zipwith (f, *rest, cycled=False, **kwargs)

Combine zipwith and starmap

test_eq(L(1,2,3).map_zipwith(operator.mul, [2,3,4]), [2,6,12])

source

L.itemgot

 L.itemgot (*idxs)

Create new L with item idx of all items

test_eq(t.itemgot(1), b)

source

L.attrgot

 L.attrgot (k, default=None)

Create new L with attr k (or value k for dicts) of all items.

# Example when items are not a dict
a = [SimpleNamespace(a=3,b=4),SimpleNamespace(a=1,b=2)]
test_eq(L(a).attrgot('b'), [4,2])

#Example of when items are a dict
b =[{'id': 15, 'name': 'nbdev'}, {'id': 17, 'name': 'fastcore'}]
test_eq(L(b).attrgot('id'), [15, 17])

source

L.sorted

 L.sorted (key=None, reverse=False, cmp=None, **kwargs)

New L sorted by key, using sort_ex. If key is str use attrgetter; if int use itemgetter

test_eq(L(a).sorted('a').attrgot('b'), [2,4])

source

L.split

 L.split (s, sep=None, maxsplit=-1)

Class Method: Same as str.split, but returns an L

test_eq(L.split('a b c'), list('abc'))

source

L.range

 L.range (a, b=None, step=None)

Class Method: Same as range, but returns L. Can pass collection for a, to use len(a)

test_eq_type(L.range([1,1,1]), L(range(3)))
test_eq_type(L.range(5,2,2), L(range(5,2,2)))

source

L.concat

 L.concat ()

Concatenate all elements of list

test_eq(L([0,1,2,3],4,L(5,6)).concat(), range(7))

source

L.copy

 L.copy ()

Same as list.copy, but returns an L

t = L([0,1,2,3],4,L(5,6)).copy()
test_eq(t.concat(), range(7))

source

L.map_first

 L.map_first (f=<function noop>, g=<function noop>, *args, **kwargs)

First element of map_filter

t = L(0,1,2,3)
test_eq(t.map_first(lambda o:o*2 if o>2 else None), 6)

source

L.setattrs

 L.setattrs (attr, val)

Call setattr on all items

t = L(SimpleNamespace(),SimpleNamespace())
t.setattrs('foo', 'bar')
test_eq(t.attrgot('foo'), ['bar','bar'])

Config

source

save_config_file

 save_config_file (file, d, **kwargs)

Write settings dict to a new config file, or overwrite the existing one.

source

read_config_file

 read_config_file (file, **kwargs)

Config files are saved and read using Python’s configparser.ConfigParser, inside the DEFAULT section.

_d = dict(user='fastai', lib_name='fastcore', some_path='test', some_bool=True, some_num=3)
try:
    save_config_file('tmp.ini', _d)
    res = read_config_file('tmp.ini')
finally: os.unlink('tmp.ini')
dict(res)

{'user': 'fastai',
 'lib_name': 'fastcore',
 'some_path': 'test',
 'some_bool': 'True',
 'some_num': '3'}

source

Config

 Config (cfg_path, cfg_name, create=None, save=True, extra_files=None,
         types=None, **cfg_kwargs)

Reading and writing ConfigParser ini files

Config is a convenient wrapper around ConfigParser ini files with a single section (DEFAULT).

Instantiate a Config from an ini file at cfg_path/cfg_name:

save_config_file('../tmp.ini', _d)
try: cfg = Config('..', 'tmp.ini')
finally: os.unlink('../tmp.ini')
cfg

{'user': 'fastai', 'lib_name': 'fastcore', 'some_path': 'test', 'some_bool': 'True', 'some_num': '3'}

You can create a new file if one doesn’t exist by providing a create dict:

try: cfg = Config('..', 'tmp.ini', create=_d)
finally: os.unlink('../tmp.ini')
cfg

{'user': 'fastai', 'lib_name': 'fastcore', 'some_path': 'test', 'some_bool': 'True', 'some_num': '3'}

If you additionally pass save=False, the Config will contain the items from create without writing a new file:

cfg = Config('..', 'tmp.ini', create=_d, save=False)
test_eq(cfg.user,'fastai')
assert not Path('../tmp.ini').exists()

You can also pass in ConfigParser kwargs to change the behavior of how your configuration file will be parsed. For example, by default, inline comments are not handled by Config. However, if you pass in the inline_comment_prefixes with whatever your comment symbol is, you’ll overwrite this behavior.

# Create a complete example config file with comments
cfg_str = """\
[DEFAULT]
user = fastai # inline comment

# Library configuration
lib_name = fastcore

# Paths
some_path = test 

# Feature flags
some_bool = True

# Numeric settings
some_num = # missing value
"""

with open('../tmp.ini', 'w') as f:
    f.write(cfg_str)

# Now read it back to verify
try: cfg = Config('..', 'tmp.ini', inline_comment_prefixes=('#'))
finally: os.unlink('../tmp.ini')
test_eq(cfg.user,'fastai')
test_eq(cfg.some_num,'')

source

Config.get

 Config.get (k, default=None)

Keys can be accessed as attributes, items, or with get and an optional default:

test_eq(cfg.user,'fastai')
test_eq(cfg['some_path'], 'test')
test_eq(cfg.get('foo','bar'),'bar')

Extra files can be read before cfg_path/cfg_name using extra_files, in the order they appear:

with tempfile.TemporaryDirectory() as d:
    a = Config(d, 'a.ini', {'a':0,'b':0})
    b = Config(d, 'b.ini', {'a':1,'c':0})
    c = Config(d, 'c.ini', {'a':2,'d':0}, extra_files=[a.config_file,b.config_file])
    test_eq(c.d, {'a':'2','b':'0','c':'0','d':'0'})

If you pass a dict types, then the values of that dict will be used as types to instantiate all values returned. Path is a special case – in that case, the path returned will be relative to the path containing the config file (assuming the value is relative). bool types use str2bool to convert to boolean.

_types = dict(some_path=Path, some_bool=bool, some_num=int)
cfg = Config('..', 'tmp.ini', create=_d, save=False, types=_types)

test_eq(cfg.user,'fastai')
test_eq(cfg['some_path'].resolve(), (Path('..')/'test').resolve())
test_eq(cfg.get('some_num'), 3)

source

Config.find

 Config.find (cfg_name, cfg_path=None, **kwargs)

Search cfg_path and its parents to find cfg_name

You can use Config.find to search subdirectories for a config file, starting in the current path if no path is specified:

Config.find('settings.ini').repo

'fastcore'

Foundational Functions

working_directory

add_docs

docs

is_iter

coll_repr

is_bool

mask2idxs

cycle

zip_cycle

is_indexer

product

L helpers

CollBase

L

L Methods

L.__getitem__

L.__setitem__

L.unique

L.val2idx

L.groupby

L.filter

L.argwhere

L.argfirst

L.map

L.map_dict

L.zip

L.map_zip

L.zipwith

L.map_zipwith

L.itemgot

L.attrgot

L.sorted

L.split

L.range

L.concat

L.copy

L.map_first

L.setattrs

Config

save_config_file

read_config_file

Config

Config.get

Config.find

`L` helpers

`L` Methods

L.getitem

L.setitem