Network functionality

Network, HTTP, and URL functions

from fastcore.test import *
from nbdev.showdoc import *
from fastcore.nb_imports import *

urlread/urljson retrieve a URL (POSTing when data or kwargs are given), urlsave downloads to a file named from the URL, and urlopen wraps urllib with quoting, header, and data handling; on HTTP failure they raise a status-specific exception (HTTP404NotFoundError, …), each a subclass of HTTP4xxClientError or HTTP5xxServerError, with the response body included in the message. start_server/start_client make TCP — or, passing a string port, Unix-socket — connections; waitfor polls a callable until truthy with a timeout; and is_port_free/wait_port_free help coordinate with servers.

URLs

urlquote

def urlquote(
    url
):

Update url’s path with urllib.parse.quote

urlquote("https://github.com/fastai/fastai/compare/master@{1.day.ago}…master")

'https://github.com/fastai/fastai/compare/master@%7B1.day.ago%7D%E2%80%A6master'

urlquote("https://www.google.com/search?q=你好")

'https://www.google.com/search?q=%E4%BD%A0%E5%A5%BD'

urlwrap

def urlwrap(
    url, data:NoneType=None, headers:NoneType=None
):

Wrap url in a urllib Request with urlquote

HTTP4xxClientError

def HTTP4xxClientError(
    url, code, msg, hdrs, fp
):

Base class for client exceptions (code 4xx) from url* functions

HTTP5xxServerError

def HTTP5xxServerError(
    url, code, msg, hdrs, fp
):

Base class for server exceptions (code 5xx) from url* functions

urlopener

def urlopener():

Call self as a function.

urlopen

def urlopen(
    url, data:NoneType=None, headers:NoneType=None, timeout:NoneType=None, **kwargs
):

Like urllib.request.urlopen, but first urlwrap the url, and encode data

With urlopen, the body of the response will also be returned in addition to the message if there is an error:

try: urlopen('https://api.github.com/v3')
except HTTPError as e: 
    print(e.code, e.msg)
    assert 'documentation_url' in e.msg

404 Not Found
====Error Body====
{
  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest",
  "status": "404"
}

urlread

def urlread(
    url, data:NoneType=None, headers:NoneType=None, decode:bool=True, return_json:bool=False,
    return_headers:bool=False, timeout:NoneType=None, **kwargs
):

Retrieve url, using data dict or kwargs to POST if present

urljson

def urljson(
    url, data:NoneType=None, headers:NoneType=None, timeout:NoneType=None
):

Retrieve url and decode json

test_eq(urljson('https://httpbin.org/get')['headers']['User-Agent'], url_default_headers['User-Agent'])

urlclean

def urlclean(
    url
):

Remove fragment, params, and querystring from url if present

test_eq(urlclean('http://a.com/b?c=1#d'), 'http://a.com/b')

urlretrieve

def urlretrieve(
    url, filename:NoneType=None, reporthook:NoneType=None, data:NoneType=None, headers:NoneType=None,
    timeout:NoneType=None
):

Same as urllib.request.urlretrieve but also works with Request objects

urldest

def urldest(
    url, dest:NoneType=None
):

Call self as a function.

urlsave

def urlsave(
    url, dest:NoneType=None, reporthook:NoneType=None, headers:NoneType=None, timeout:NoneType=None
):

Retrieve url and save based on its name

#skip
with tempfile.TemporaryDirectory() as d: urlsave('http://www.google.com/index.html', d)

urlvalid

def urlvalid(
    x
):

Test if x is a valid URL

assert urlvalid('http://www.google.com/')
assert not urlvalid('www.google.com/')
assert not urlvalid(1)

Basic client/server

start_server

def start_server(
    port, host:NoneType=None, dgram:bool=False, reuse_addr:bool=True, n_queue:NoneType=None
):

Create a socket server on port, with optional host, of type dgram

You can create a TCP client and server pass an int as port and optional host. host defaults to your main network interface if not provided. You can create a Unix socket client and server by passing a string to port. A SOCK_STREAM socket is created by default, unless you pass dgram=True, in which case a SOCK_DGRAM socket is created. n_queue sets the listening queue size.

start_client

def start_client(
    port, host:NoneType=None, dgram:bool=False
):

Create a socket client on port, with optional host, of type dgram

tobytes

def tobytes(
    s:str
)->bytes:

Convert s into HTTP-ready bytes format

test_eq(tobytes('foo\nbar'), b'foo\r\nbar')

http_response

def http_response(
    body:NoneType=None, status:int=200, hdrs:NoneType=None, **kwargs
):

Create an HTTP-ready response, adding kwargs to hdrs

exp = b'HTTP/1.1 200 OK\r\nUser-Agent: me\r\nContent-Length: 4\r\n\r\nbody'
test_eq(http_response('body', 200, User_Agent='me'), exp)

recv_once

def recv_once(
    host:str='localhost', port:int=8000
):

Spawn a thread to receive a single HTTP request and store in d['r']

Waiting

waitfor polls a callable until it returns truthy – often needed when coordinating with servers or other processes that take time to change state. The sync and async variants share one polling loop (_waitfor), so they behave identically apart from how they sleep.

waitfor_async

async def waitfor_async(
    f, timeout:int=20, msg:NoneType=None
):

Async version of waitfor

waitfor

def waitfor(
    f, timeout:int=20, msg:NoneType=None
):

Call f every 0.1s until it returns truthy, raising TimeoutError with msg after timeout secs

A predicate that succeeds on its third call returns normally; one that never succeeds raises after timeout seconds:

vals = iter([0,0,1])
waitfor(lambda: next(vals))
test_fail(lambda: waitfor(lambda: False, timeout=0.15), contains='Timeout')

vals = iter([0,1])
await waitfor_async(lambda: next(vals))

is_port_free

def is_port_free(
    port, host:str='localhost'
):

Is port free on host?

wait_port_free_async

async def wait_port_free_async(
    port, host:str='localhost', max_wait:int=20
):

Async wait for port to be free on host

wait_port_free

def wait_port_free(
    port, host:str='localhost', max_wait:int=20
):

Wait for port to be free on host

A port held by a start_server socket is busy until the socket closes, at which point wait_port_free returns promptly. Note that start_server binds the hostname’s interface by default, while the port helpers default to localhost – so when combining them, pass an explicit host to keep both on the same interface:

s = start_server(8877, host='localhost')
assert not is_port_free(8877)
s.close()
wait_port_free(8877)
assert is_port_free(8877)
await wait_port_free_async(8877)