Quartz 4

07. Functions as First-Class Objects

10 min read

Functions in Python are First-Class Objects. Programming language researches define a “first-class object” as a program entity that can be:

  • created at runtime;
  • assigned to a variable or element in a data structure;
  • passed as an argument to a function;
  • returned as the result of a function.

Having functions as first-class objects is an essential feature of Functional Programming. However, this concept is so useful that it has been adopted by “non-functional programming” languages.

Treating a Function like an Object

Let’ prove that functions are objects in Python:

>>> def factorial(n):
... 	   """returns n!"""
... 	   return 1 if n < 2 else n * factorial(n-1)
 
>>> factorial(42)
1405006117752879898543142606244511569936384000000000
 
>>> factorial.__doc__
returns n!
 
>>> type(factorial)
<class 'function'>
  • This is a console session, so we’re creating a function at “runtime”.
  • __doc__ is one of several attributes of function objects.
  • factorial is an instance of the function class.

Now let’s take factorial function as example and we’ll show the “first-class” nature of a function object:

>>> fact = factorial
>>> fact
<function __main__.factorial(n)>
>>> fact(5)
120
>>> list(map(factorial, range(11)))
[1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]

In this example we could:

  • Line 1 - Line 4: assign it to a variable fact with an assignment operator without the parenthesis and call it through that name;
  • Line 6: pass factorial as an argument to another function (known as higher-order function), map in our case.

Higher-Order Functions

A function is a “higher-order function” if:

  • takes a function as an argument or
  • returns a function as the result.

Example with key argument in sorted() function, which let us provide a function to be applied to each item for sorting:

>>> fruits = ["strawberry", "apple", "cherry"]
>>> sorted(fruits, key=len)
['apple', 'cherry', 'strawberry']

Any one-argument function can be used as key. For example, to create a rhyme dictionary it might be useful to sort each word spelled backward:

>>> def reverse(word):
...     return word[::-1]
>>> reverse('testing')
'gnitset'
>>> sorted(fruits, key=reverse)
['apple', 'strawberry', 'cherry']

In the Functional Programming paradigm, some of the best known higher-order functions are:

  • apply (deprecated in Python 2.3 and removed in Python 3);
  • map, filter, and reduce (although better alternatives are available).

Modern Replacements for map, filter, and reduce

map andfilterfunctions

map andfilter are built-ins in Python 3, but, since they both return generators (a form of iterator), their combination is often replaced by list comprehensions and generator expressions:

>>> list(map(factorial, range(6)))
[1, 1, 2, 6, 24, 120]
 
>>> [factorial(n) for n in range(6)]
[1, 1, 2, 6, 24, 120]
 
>>> list(map(factorial, filter(lambda n: n%2, range(6))))
[1, 6, 120]
 
>>> [factorial(n) for n in range(6) if n%2]
[1, 6, 120]

reduce function

The reduce function was demoted from a built-in in Python 2 to the functools module in Python 3. Its most common use case, summation, is better served by the sum built-in:

>>> from functools import reduce
>>> from operator import add
>>> reduce(add, range(100))
4950
 
>>> sum(range(100))
4950

The common idea of sum and reduce is to apply some operation to successive items in a series, accumulating previous results, thus reducing a series of values to a single value.

Other reducing built-ins are:

  • all(iterable): returns True if there are not falsy elements in the iterable (all([]) returns True).
  • any(iterable): returns True if any element of the iterable is truthy(any([]) returns False)

To use a higher-order function, sometimes it’s convenient to create a small, one-off function, known as Anonymous Function.

Anonymous Functions

The lambda keyword creates an Anonymous Function within a Python Expression. The simple syntax of Python limits the body of the lambda functions to be pure expressions (it cannot contain other Python statements, such as while, try, etc. Assignment with = is also a statement, so it cannot occur in a lambda, but you can use the new assignment expression syntax using :=. However, if you need it, your lambda is probably too complicated and hard to read and it’s better to use a regular function using def).

The best use of anonymous functions is in the context of an argument list for a higher-order function. For example we can rewrite the previous function sorted(fruits, key=reverse) without defining the reverse function:

>>> sorted(fruits, key=lambda word: word[::-1])
['apple', 'strawberry', 'cherry']

Outside the limited context of arguments to higher-order functions, anonymous functions are rarely useful in Python.

The Nine Flavors of Callable Objects

In Python, a callable is any object that you can call using a pair of parentheses and, optionally, a series of arguments. The call operator () may be applied to other objects besides functions. To determine whether an object is callable , use the callable() built-in function.

As of Python 3.9, the data model documentation lists 9 callable types:

  • User-defined functions: Created with def statements or lambda expressions.
  • Built-in functions: Functions like len() and print().
  • Built-in methods: Methods of built-in objects, like list.append().
  • Methods: Functions defined within a class.
  • Classes: When called, they create a new instance of the class.
  • Class Instances: Instances of classes that implement the __call__ method (explanation in the next paragraph).
  • Generator Functions: Functions that yield values using the yield keyword, when called, return a generator object.
  • Native Coroutine Functions
  • Asynchronous generator Functions

User-Defined Callable Types

Any Python object that implements the __call__ instance method acts like a function. Example:

import random
 
class BingoCage:
    def __init__(self, items):
        self._items = list(items)
        random.shuffle(self._items)
 
    def pick(self):
        try:
            return self._items.pop()
        except IndexError:
            raise LookupError('pick from empty BingoCage')
        
    def __call__(self):
        return self.pick()
 
 
bingo = BingoCage(range(10))
print(bingo._items)
print(bingo()) # shortcut for bingo.pick()
print(bingo())
print(bingo())

Output:

[8, 7, 6, 1, 0, 3, 2, 9, 4, 5]
5
4
9

A class implementing __call__ is as easy way to create function-like objects that have some internal state that must be kept across invocations (in this case the internal state is represented by the remaining items in the BingoCage).

From Positional to Keyword-Only Parameters

One of the best features of Python functions is the extremely flexible parameters handling mechanism, of which the parameters unpacking symbols like * and ** are two examples. Let’s create a function:

def tag(name, *content, class_=None, **attrs):
    """Generate one or more HTML tags"""
    if class_ is not None:
        attrs['class'] = class_
    attr_pairs = (f' {attr}="{value}"' for attr, value in sorted(attrs.items()))
    attr_str = ''.join(attr_pairs)
    if content:
        elements = (f'<{name}{attr_str}>{c}</{name}>' for c in content)
        return '\n'.join(elements)
    else:
        return f'<{name}{attr_str} />'

In the terminal:

>>> tag('br')
'<br />'
 
>>> tag('p', 'hello')
'<p>hello</p>'
 
>>> print(tag('p', 'hello', 'world'))
<p>hello</p>
<p>world</p>
 
>>> tag('p', 'hello', id=33)
'<p id="33">hello</p>'
 
>>> print(tag('p', 'hello', 'world', class_='sidebar'))
<p class="sidebar">hello</p>
<p class="sidebar">world</p>
 
>>> tag(content='testing', name='img')
'<img content="testing" />'
 
>>> my_tag = {'name': 'img', 'title': 'Sunset Boulevard', 'src': 'sunset.jpg', 'class': 'framed'}
>>> tag(**my_tag)
'<img class="framed" src="sunset.jpg" title="Sunset Boulevard" [/](https://file+.vscode-resource.vscode-cdn.net/)>'
  • Line 1: a single positional argument produces an empty tag with that name.
  • Line 4-7: any number of positional arguments after the first are captured by *content as a tuple.
  • Line 11: keyword arguments not explicitly named in the tag signature are captured by attrs as a dict.
  • Line 14: the class_ parameter can only be passed as a keyword argument.
  • Line 18: the first positional argument can also be passed as a keyword.
  • Line 22: prefixing my_tag with ** passes all its items as separate arguments.

Keyword-Only Arguments are a feature of Python 3. To specify Keyword-Only Arguments when defining a function, name them after the argument prefixed with *. In our example, the class_ parameter can only be given as keyword argument because it was named after *content. If you don’t want to support positional arguments but still want keyword-only arguments, put a * by itself in the signature:

def f(a, *, b):
    return a, b
 
f(1, b=2) # output: (1, 2)
f(a=1, b=2) # output: (1, 2)
f(1, 2) # output: TypeError: f() takes 1 positional argument but 2 were given

Positional-Only Parameters

Positional-Only Arguments is a feature introduced since Python 3.8 for user-defined functions. To specify Positional-Only Arguments when defining a function, name them before / in the parameter list:

def divmod(a, b, /):
	return (a//b, a%b)
 
divmod(10,4) # output: (2, 2)
divmod(a=10, b=4) # output: TypeError: divmod() got some positional-only arguments passed as keyword arguments: 'a, b'

In our previous example, we can specify name argument to be positional-only with this signature:

def tag(name, *content, class_=None, **attrs):
	...

Packages for Functional Programming

Guido van Rossum said explicitly that he did not design Python to be a Functional Programming Language. Nonetheless, a functional coding style can be used thanks to first-class functions, pattern matching, and the support of packages like operator and functools.

The operator Module

Arithmetic Operator

Often in functional programming it is convenient to use an arithmetic operator as a function. For example, you can get the summation of a sequence of numbers without recursion with built-in sum() by sum([1, 2, 3]). There is no equivalent function for multiplication. As we have already mentioned in reduce function paragraph above, you could compute the factorial by usingreduce and mul function from operator module:

from functools import reduce
from operator import mul
 
def factorial(n):
	return reduce(mul, range(1, n+1))

instead of using a trivial function with lambda:

from functools import reduce
 
def factorial(n):
	return reduce(lambda a, b: a*b, range(1, n+1))

itemgetter and attrgetter

Another group of functions that allows us to avoid lambda consists of itemgetter and attrgetter.

A common use of itemgetter is to sort a list of tuples by the value of one field:

from operator import itemgetter
 
metro_data = [
    ('Tokyo', 'JP', 36.933, (35.689722, 139.691667)),
    ('Delhi NCR', 'IN', 21.935, (28.613889, 77.208889)),
    ('Mexico City', 'MX', 20.142, (19.433333, -99.133333)),
    ('New York-Newark', 'US', 20.104, (40.808611, -74.020386)),
    ('São Paulo', 'BR', 19.649, (-23.547778, -46.635833))
    ]
 
for city in sorted(metro_data, key=itemgetter(1)):
    print(city)

instead of using a trivial function with lambda:

for city in sorted(metro_data, key=lambda x: x[1]):
    print(city)

What is happened is that itemgetter(1) creates a function that, given a collection, returns the item at index 1.

Another common use of itemgetter is to retrieve items from collections such as lists, tuples, and dictionaries:

cc_name = itemgetter(2,0)
 
for city in metro_data:
    print(cc_name(city))

Output:

(36.933, 'Tokyo')
(21.935, 'Delhi NCR')
(20.142, 'Mexico City')
(20.104, 'New York-Newark')
(19.649, 'São Paulo')

Since itemgetter uses the [] operator, it supports not only sequences but also any class that implements __getitem__.

TODO: attrgetter explaination

Summary

The goal of this chapter was to explore the first-class nature of functions in Python. The main ideas are that you can

  • assign functions to variables
  • pass them to other functions
  • store them in data structures
  • access function attributes, allowing frameworks and tools to act on that information.

Higher-order functions, a staple of functional programming, are common in Python. The sorted, min, and max built-ins, and functools.partial are examples of commonly used higher-order functions in the language. Using map, filter, and reduce is not as common as it used to be, thanks to list comprehensions (and similar constructs like generator expressions) and the addition of reducing built-ins like sum, all, and any.

Graph View