The many uses of Python's ExitStack
Context managers and the ExitStack ¶
In any garbage collected language, eventually you will need the ability to manage resources separately from the garbage collection cycle. GC is fine for memory, but other resources like file handles, database transactions, and mutexes are more costly or even incorrect to hold onto beyond their scope. The obvious solution is to just do it in a method.
1 2 3 | file_lock = take_lock() # Do things while the file is locked file_lock.release() |
This works fine, until you need to exit early, or an exception is thrown. Nearly every garbage-collected language has a solution for this (even Lua as of 5.4). In Python, that answer is context managers.
1 2 | with take_lock(): # Do things while the file is locked |
Here, the lock will always be released, even if an exception is thrown. The protocol is easy to implement, so you can create your own context managers, but this quickly hits a wall when you need to handle a dynamic number of resources. You could feasibly do this with a recursive function, but the result is pretty ugly.
1 2 3 4 5 6 7 8 9 10 11 12 | import json import sys from typing import TextIO def zip_files_as_json(filenames: list[str], files: list[TextIO]): if filenames: with open(filenames[0], 'r') as file: return zip_files_as_json(filenames[1:], files + [file]) for lines in zip(*files): print(json.dumps(lines)) zip_files_as_json(sys.argv[1:], []) |
This is the typical example of when an ExitStack saves you.
1 2 3 4 5 6 7 8 | import json import sys from contextlib import ExitStack with ExitStack() as stack: files = (stack.enter_context(open(fname)) for fname in sys.argv[1:]) for lines in zip(*files): print(json.dumps(lines)) |
That reads much more nicely and won't cause a stack overflow, but it's really just the surface level. An ExitStack lets you do things that are otherwise difficult with plain context managers. They clean up context manager wrappers, they allow you to approximate move semantics that you'd see in RAII languages like C++ and Rust.
Robustness for context manager wrappers ¶
Sometimes you might need to wrap a context manager in another type. Frequently, you can reach for contextlib.contextmanager to make this more tractable (putting the yield within its own context block), but often this isn't sufficient, because you need an actual type, or the context manager needs to be re-entrant, or something of the sort. Writing this is pretty clean and easy when there's only one inner context manager to worry about.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | from contextlib import contextmanager from os import PathLike from sqlite3 import Connection, connect from types import GeneratorType, Self @contextmanager def database(path: PathLike) -> GeneratorType[Connection]: connection = connect(path) try: yield connection finally: connection.close() class Wrapper: def __init__(self, db_path: PathLike): self._db_path = db_path def __enter__(self) -> Self: self._db_manager = database(self._db_path) self._connection = self._db_manager.__enter__() return self def __exit__(self, *args, **kwargs): try: self._db_manager = self._db_manager.__exit__(*args, **kwargs) finally: del self._connection, self._db_manager |
It's common but by no means universal for a context manager to return itself from its __enter__ method, so often each context manager involves two stored values: the context manager itself as well as its returned value.
Either way, this gets hideous when you have multiple context-managers involved.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | from contextlib import AbstractContextManager from typing import Self class Wrapper: def __init__( self, alpha: AbstractContextManager, beta: AbstractContextManager, gamma: AbstractContextManager, ): self._alpha = alpha self._beta = beta self._gamma = gamma def __enter__(self) -> Self: self._alpha_value = self._alpha.__enter__() try: self._beta_value = self._beta.__enter__() try: self._gamma_value = self._gamma.__enter__() except BaseException as e: try: self._beta.__exit__(type(e), e, e.__traceback__) del self._beta_value except: pass raise except BaseException as e: try: self._alpha.__exit__(type(e), e, e.__traceback__) del self._alpha_value except: # We don't want to swallow the higher-level exception because it # might be some kind of interrupt that can't be swallowed. pass raise return self def __exit__(self, exc_type, exc, traceback): del self._alpha_value, self._beta_value, self._gamma_value reraise = False for manager in (self._alpha, self._beta, self._gamma): try: manager.__exit__(exc_type, exc, traceback) except BaseException as e: reraise = True exc_type = type(e) exc = e traceback = e.__traceback__ if reraise: raise exc |
Ugly to work with, and ugly to reason about. This is where ExitStack really shines.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | from contextlib import AbstractContextManager, ExitStack from typing import Self class Wrapper: def __init__( self, alpha: AbstractContextManager, beta: AbstractContextManager, gamma: AbstractContextManager, ): self._alpha = alpha self._beta = beta self._gamma = gamma def __enter__(self) -> Self: with ExitStack() as exit_stack: self._alpha_value = exit_stack.enter_context(self._alpha) def drop_alpha(): del self._alpha_value exit_stack.callback(drop_alpha) self._beta_value = exit_stack.enter_context(self._beta) def drop_beta(): del self._beta_value exit_stack.callback(drop_beta) self._gamma_value = exit_stack.enter_context(self._gamma) def drop_gamma(): del self._gamma_value exit_stack.callback(drop_gamma) self._exit_stack = exit_stack.pop_all() return self def __exit__(self, exc_type, exc, traceback): return self._exit_stack.__exit__(exc_type, exc, traceback) |
Not only does this catch and clean up on early exit, it also correctly handles the exit condition at the end, which we actually haven't done before now; returning True from an __exit__ method is supposed to suppress exceptions, and the wrapper should handle that appropriately too.
Move semantics ¶
In Rust and C++, resource management is handled via destructors. This can tie the resource to its scope unless you have a way of lifting it out, and in Rust and C++11, that is move semantics (prior to C++11, you'd have to either design your type to handle its own moving or return a heap pointer and handle the memory manually).
This is a problem in Python context managers. Once you enter a context manager, how do you get out of its scope without closing it? There are a few options, some more palatable than others. This is effectively the same problem as writing your own context manager, with many of the same solutions. The most obvious approach is the manual approach: do not enter the context manager at all, but call __enter__ and __exit__ manually.
Another solution is to enter the context manager in a generator, yielding in the middle to interrupt it, and then re-entering it when you want to close it.
The best approach for this is still the ExitStack.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | from collections.abc import Generator from contextlib import ExitStack, contextmanager @contextmanager def context[T](a: T) -> Generator[T]: print('entering') yield a print('exiting') def do_something[T](a: T) -> tuple[ExitStack, T]: with ExitStack() as e: v = e.enter_context(context(a)) # Do things here that are fallible, and might need early cleanup. return e.pop_all(), v stack, value = do_something('foo') with stack: print('in outer context') assert value == 'foo' |
1 2 3 | entering in outer context exiting |
Conclusion ¶
There are many features in the Python standard library that probably don't belong there or are weirdly designed, but the ExitStack is one feature that distinctly undersells itself in its documentation. It files away all the sharp corners and smooths all the clunkiness off python context managers, and gives you nearly all the flexibility to do whatever you might need to do with them without having to dig into all of the details manually.