Finish Python Dependency Hell with pip-compile-multi | by Jake Schmidt | Dec, 2022

Preserve your undertaking reproducible and your advanced Python dependencies organized

Most Python initiatives of consequence have advanced dependency administration necessities which can be inadequately addressed by widespread open-source options. Some instruments attempt to sort out the complete packaging expertise, whereas others purpose to resolve one or two slender subproblems. Regardless of the myriad options, builders nonetheless face the identical dependency administration challenges:

1. How can new customers and contributors simply and appropriately set up dependencies?
2. How do I do know my dependencies are all suitable?
3. How do I make builds deterministic and reproducible?
4. How do I guarantee my deployment artifacts use coherent and suitable dependencies?
5. How do I keep away from dependency bloat?

This publish will concentrate on answering these questions utilizing pip-compile-multi, an open-source command line instrument that extends the capabilities of the favored pip-tools to handle the wants of initiatives with advanced dependencies.

A partial resolution is to take care of a dependency lockfile, and instruments resembling poetry and pip-tools allow this. We are able to consider a lockfile nearly like a “dependency interface”: an abstraction that tells the undertaking what exterior dependencies it must perform correctly. The issue with having a single, monolithic lockfile in your whole undertaking is that, as an interface, it’s not well-segregated: to make sure compatibility, determinism, and reproducibility, each client of the code (consumer, developer, packaging system, construct artifact, deployment goal) might want to set up each single dependency the lockfile enumerates—whether or not they truly use it or not. You’ve encountered this challenge when you’ve ever struggled to separate your linting and testing libraries out of your manufacturing construct, for instance.

The ensuing dependency bloat is usually a actual challenge. Apart from unnecessarily ballooning construct instances and bundle/artifact measurement, it will increase the floor space of safety vulnerabilities in your undertaking or software.

Ideally, we might restructure our dependency interface into a number of, narrower ones—a number of lockfiles that:

• group dependencies by perform
• will be composed with one another
• will be consumed independently
• are mutually suitable

If we are able to do this, issues get simpler:

• understanding what dependencies are used the place
• packaging variants (e.g. defining pip extras)
• multi-stage builds (e.g. Docker multi-stage)

Happily, pip-compile-multi does the entire above! It’s a light-weight, pip– installable CLI constructed on high of the superb pip-tools undertaking. You merely break up your necessities.txt file into a number of pip necessities files (usually suffixed .in). Every file might include one or -r / --requirement choices, which hyperlink the information collectively as a Directed Acyclic Graph (DAG). This DAG illustration of dependencies is central to pip-compile-multi.

Instance

Let’s say your necessities.txt seems to be like this:

# necessities.txt
flake8
mypy
numpy
pandas
torch>1.12

Step one is to separate out these dependencies into useful teams. We’ll write one group to important.in and one other to dev.in. We should always now delete our necessities.txt. Our two new .in information would possibly look one thing like this, forming a easy two-node dependency DAG:

Every node is a .in file defining a dependency group. Every directed edge represents the requirement of 1 group by one other. Every node defines its personal in-edges with a number of -r / --requirement choices.

As soon as we now have this dependency DAG outlined, working pip-compile-multi will generate an equal lockfile DAG. The instrument will output a .txt pip necessities file for every .in within the DAG.

By default, the produced lockfiles shall be created in the identical listing because the .in information and mirror their names.

Autoresolution of cross-file conflicts

The killer function that separates pip-compile-multi from different lockfiles instruments resembling pip-tools is autoresolution of cross-file conflicts, simply enabled with the --autoresolve flag. In autoresolve mode, pip-compile-multi will first pre-solve for all dependencies, then use that resolution to constrain every node’s particular person resolution. This ensures every lockfile stays mutually suitable by stopping any conflicts of their transient dependencies. As a way to use autoresolution, your DAG should have precisely one supply node (be aware that the pip-compile-multi documentation, inverts the directionality of DAG edges, so they’ll consult with sink nodes after I say supply, and vice-versa).

Lockfile verification

One other helpful command is pip-compile-multi confirm, which checks that your lockfiles match what’s laid out in your .in information. This can be a easy but useful verify you’ll be able to simply incorporate into your CICD pipeline to guard towards errant dependency updates. And it’s even out there as a precommit hook!

Manage dependencies appropriately

For those who group your dependencies poorly, you’re setting your self up for failure. Attempt to outline teams based mostly on the supposed perform of the dependencies in your code: don’t put flake8 (a code linter) in a gaggle with torch (a deep studying framework).

Have a single supply node and a single sink node

I’ve discovered that issues work finest when you’ll be able to set up your most ubiquitous dependencies right into a single “core” set of dependencies that every one different nodes require (a sink node), and your whole improvement dependencies in a node that requires all others (straight or not directly) require (a supply). This sample retains your DAG comparatively easy and ensures you should utilize pip-compile-multi’s nice autoresolve function.

Allow the pip cache

Setting the --use-cache flag can drastically pace up pip-compile-multi as a result of it permits caching within the underlying calls to pip-compile.

To make issues extra clear, let’s work by way of an instance from the realm of machine studying.

A typical machine studying system may have at the very least two parts: a coaching workload that creates a mannequin on some information, and an inference server to serve mannequin predictions.

Each parts may have some widespread dependencies, resembling libraries for information processing and modeling. We are able to checklist these in a textual content file known as important.in, which is only a pip necessities file:

# necessities/important.in
pandas
torch>1.12

The coaching part may need some idiosyncratic dependencies for distributed communications, experiment monitoring, and metric computation. We’ll put these in coaching.in:

# necessities/coaching.in
-r important.in
horovod
mlflow==1.29
torchmetrics

Discover we add the -r flag, which tells pip-compile-multi that coaching.in requires the dependencies from important.in.

The inference part may have some unique dependencies for serving and monitoring, which we add to inference.in:

# necessities/inference.in
-r important.in
prometheus
torchserve

Lastly, the complete codebase shares the identical improvement toolchain. These improvement instruments, resembling linters, unit testing modules, and even pip-compile-multi itself go in dev.in:

# necessities/dev.in
-r inference.in
-r coaching.in
flake8
pip-compile-multi
pytest

Once more, discover the -r flags indicating dev.in depends upon coaching.in and inference.in. We don’t want a -r important.in as a result of coaching.in and inference.in have already got it.

Collectively, our dependency DAG seems to be like this:

Assuming our .in information are inside a listing known as necessities/, we are able to use the next command to resolve our DAG and generate lockfiles:

pip-compile-multi --autoresolve --use-cache --directory=necessities

After the command succeeds, you will note 4 new information inside necessities/: important.txt, coaching.txt, inference.txt, and dev.txt. These are our lockfiles. We are able to use them the identical means we’d use a sound necessities.txt file. Maybe we might use them to construct environment friendly Docker multi-stage picture targets: