The many uses of Python's ExitStack

by Taylor Richberger

Context managers and the ExitStack

In any garbage collected language, eventually you will need the ability to manage resources separately from the garbage collection cycle. GC is fine for memory, but other resources like file handles, database transactions, and mutexes are more costly or even incorrect to hold onto beyond their scope. The obvious solution is to just do it in a method.

1
2
3
file_lock = take_lock()
# Do things while the file is locked
file_lock.release()

This works fine, until you need to exit early, or an exception is thrown. Nearly every garbage-collected language has a solution for this (even Lua as of 5.4). In Python, that answer is context managers.

1
2
with take_lock():
    # Do things while the file is locked

Here, the lock will always be released, even if an exception is thrown. The protocol is easy to implement, so you can create your own context managers, but this quickly hits a wall when you need to handle a dynamic number of resources. You could feasibly do this with a recursive function, but the result is pretty ugly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import json
import sys
from typing import TextIO

def zip_files_as_json(filenames: list[str], files: list[TextIO]):
    if filenames:
        with open(filenames[0], 'r') as file:
            return zip_files_as_json(filenames[1:], files + [file])
    for lines in zip(*files):
        print(json.dumps(lines))

zip_files_as_json(sys.argv[1:], [])

This is the typical example of when an ExitStack saves you.

1
2
3
4
5
6
7
8
import json
import sys
from contextlib import ExitStack

with ExitStack() as stack:
    files = (stack.enter_context(open(fname)) for fname in sys.argv[1:])
    for lines in zip(*files):
        print(json.dumps(lines))

That reads much more nicely and won't cause a stack overflow, but it's really just the surface level. An ExitStack lets you do things that are otherwise difficult with plain context managers. They clean up context manager wrappers, they allow you to approximate move semantics that you'd see in RAII languages like C++ and Rust.

Robustness for context manager wrappers

Sometimes you might need to wrap a context manager in another type. Frequently, you can reach for contextlib.contextmanager to make this more tractable (putting the yield within its own context block), but often this isn't sufficient, because you need an actual type, or the context manager needs to be re-entrant, or something of the sort. Writing this is pretty clean and easy when there's only one inner context manager to worry about.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from contextlib import contextmanager
from os import PathLike
from sqlite3 import Connection, connect
from types import GeneratorType, Self


@contextmanager
def database(path: PathLike) -> GeneratorType[Connection]:
    connection = connect(path)
    try:
        yield connection
    finally:
        connection.close()

class Wrapper:
    def __init__(self, db_path: PathLike):
        self._db_path = db_path

    def __enter__(self) -> Self:
        self._db_manager = database(self._db_path)
        self._connection = self._db_manager.__enter__()
        return self
        
    def __exit__(self, *args, **kwargs):
        try:
            self._db_manager = self._db_manager.__exit__(*args, **kwargs)
        finally:
            del self._connection, self._db_manager

It's common but by no means universal for a context manager to return itself from its __enter__ method, so often each context manager involves two stored values: the context manager itself as well as its returned value.

Either way, this gets hideous when you have multiple context-managers involved.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
from contextlib import AbstractContextManager
from typing import Self


class Wrapper:
    def __init__(
        self,
        alpha: AbstractContextManager,
        beta: AbstractContextManager,
        gamma: AbstractContextManager,
    ):
        self._alpha = alpha
        self._beta = beta
        self._gamma = gamma

    def __enter__(self) -> Self:
        self._alpha_value = self._alpha.__enter__()
        try:
            self._beta_value = self._beta.__enter__()
            try:
                self._gamma_value = self._gamma.__enter__()
            except BaseException as e:
                try:
                    self._beta.__exit__(type(e), e, e.__traceback__)
                    del self._beta_value
                except:
                    pass
                raise
        except BaseException as e:
            try:
                self._alpha.__exit__(type(e), e, e.__traceback__)
                del self._alpha_value
            except:
                # We don't want to swallow the higher-level exception because it
                # might be some kind of interrupt that can't be swallowed.
                pass
            raise
        return self

    def __exit__(self, exc_type, exc, traceback):
        del self._alpha_value, self._beta_value, self._gamma_value
        reraise = False
        for manager in (self._alpha, self._beta, self._gamma):
            try:
                manager.__exit__(exc_type, exc, traceback)
            except BaseException as e:
                reraise = True
                exc_type = type(e)
                exc = e
                traceback = e.__traceback__
        if reraise:
            raise exc

Ugly to work with, and ugly to reason about. This is where ExitStack really shines.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
from contextlib import AbstractContextManager, ExitStack
from typing import Self


class Wrapper:
    def __init__(
        self,
        alpha: AbstractContextManager,
        beta: AbstractContextManager,
        gamma: AbstractContextManager,
    ):
        self._alpha = alpha
        self._beta = beta
        self._gamma = gamma

    def __enter__(self) -> Self:
        with ExitStack() as exit_stack:
            self._alpha_value = exit_stack.enter_context(self._alpha)

            def drop_alpha():
                del self._alpha_value

            exit_stack.callback(drop_alpha)
            self._beta_value = exit_stack.enter_context(self._beta)

            def drop_beta():
                del self._beta_value

            exit_stack.callback(drop_beta)
            self._gamma_value = exit_stack.enter_context(self._gamma)

            def drop_gamma():
                del self._gamma_value

            exit_stack.callback(drop_gamma)

            self._exit_stack = exit_stack.pop_all()

        return self

    def __exit__(self, exc_type, exc, traceback):
        return self._exit_stack.__exit__(exc_type, exc, traceback)

Not only does this catch and clean up on early exit, it also correctly handles the exit condition at the end, which we actually haven't done before now; returning True from an __exit__ method is supposed to suppress exceptions, and the wrapper should handle that appropriately too.

Move semantics

In Rust and C++, resource management is handled via destructors. This can tie the resource to its scope unless you have a way of lifting it out, and in Rust and C++11, that is move semantics (prior to C++11, you'd have to either design your type to handle its own moving or return a heap pointer and handle the memory manually).

This is a problem in Python context managers. Once you enter a context manager, how do you get out of its scope without closing it? There are a few options, some more palatable than others. This is effectively the same problem as writing your own context manager, with many of the same solutions. The most obvious approach is the manual approach: do not enter the context manager at all, but call __enter__ and __exit__ manually.

Another solution is to enter the context manager in a generator, yielding in the middle to interrupt it, and then re-entering it when you want to close it.

The best approach for this is still the ExitStack.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
from collections.abc import Generator
from contextlib import ExitStack, contextmanager


@contextmanager
def context[T](a: T) -> Generator[T]:
    print('entering')
    yield a
    print('exiting')

def do_something[T](a: T) -> tuple[ExitStack, T]:
    with ExitStack() as e:
        v = e.enter_context(context(a))
        # Do things here that are fallible, and might need early cleanup.
        return e.pop_all(), v

stack, value = do_something('foo')

with stack:
    print('in outer context')
    assert value == 'foo'
1
2
3
entering
in outer context
exiting

Conclusion

There are many features in the Python standard library that probably don't belong there or are weirdly designed, but the ExitStack is one feature that distinctly undersells itself in its documentation. It files away all the sharp corners and smooths all the clunkiness off python context managers, and gives you nearly all the flexibility to do whatever you might need to do with them without having to dig into all of the details manually.