Managing dependencies is deceptively hard.
Talk to anyone who has to manage a
I’m sure they’ll have stories.
Python is not immune to this hard problem.
For years, the community rallied around the
to manage dependencies,
but there are some subtle flaws
that make dependency handling more confusing
To fix these issues,
the Python Packaging Authority,
which is the group responsible
for many things including
pip and PyPI,
proposed a replacement for
We’re going to look
at the two file formats
to see why a
Pipfile is a better fit
for the community
in the future
and how you can get started using one.
Let’s look at
requirements.txt to see
where the flaws are.
requirements.txt file has a very primitive structure.
Here’s a sample file from the handroll project
that I work on.
Jinja2==2.8 Markdown==2.4 MarkupSafe==0.23 PyYAML==3.11 Pygments==2.1.3 Werkzeug==0.11.4 argh==0.26.1 argparse==1.2.1 blinker==1.4 docutils==0.12 mock==1.0.1 pathtools==0.1.2 textile==2.2.2 watchdog==0.8.3
The core requirement is that each line in the file specifies one dependency.
The example adds a version specifier
for each package
even though that is not required.
The file could have said
Jinja2 instead of
In that small detail,
we can begin to see weaknesses in the structure.
Which is more correct, to specify versions or not?
Specifying the version of a package is called pinning. Files that pin versions for every dependency make it possible to reproduce the environment. This quality is very valuable for operating in a production scenario.
What’s the downside?
It’s very hard to determine which packages are the direct dependencies.
For instance, handroll directly uses
MarkupSafe is only listed
because it is a dependency of a dependency.
Jinja2 depends on
MarkupSafe is a transitive dependency
The reason to include the transitive dependency
comes back to reproducing the environment.
If we only listed
it’s possible for an updated version
MarkupSafe to be installed
that could break handroll.
That leads to a bad user experience.
We’ve reached the core problem
of the older format:
requirements.txt is attempting to be two views of dependencies.
- A pinned
requirements.txtacts as a manifest to reproduce the operating environment.
- An unpinned
requirements.txtacts as the logical list of dependencies that a package depends on.
There is also a secondary problem related to the audience. If I’m a user of handroll, I only care about the dependencies that make the tool work. If I’m a developer for handroll, I also would like the tools needed for development (e.g., a linter, translation tools, upload tools for PyPI).
At this stage, conventions begin to break down
in the community.
Some projects use a
for developer-only dependencies.
Others opt for a
that contain many different files
Both are imperfect solutions.
We’re now positioned to consider what a
brings to the problem.
Pipfile handles the problems
requirements.txt does not.
It is important to note that a
Pipfile is not a novel creation.
Pipfile is a Python implementation of a system that appears
from each of those languages
that follow a similar pattern.
What traits do these systems have in common?
- Split logical dependencies and a dependency manifest into separate files.
- Separate the sections for user and developer dependencies.
Pipfile manages the logical dependencies
of a project.
When I write “logical,”
I’m referring to the dependencies
that a project directly
in its code.
One way to think about the logical dependencies
is as the set of dependencies
excluding the transitive dependencies.
Pipfile.lock is the set
including the transitive dependencies.
This file acts as the dependency manifest
to use when building an environment
for a production setting.
Pipfileis for people. The
Pipfile.lockis for computers.
Having a clear distinction between files offers a couple of benefits.
- People can read and reason about the
Pipfile. There is no need to guess if a dependency is a direct dependency of a project.
- Extra metadata can be stored in the
Pipfile.lock. The metadata can include things like
sha256checksums that help verify the integrity of a package’s content.
Users and developers
The other trait of a
Pipfile is the split
between user and developer dependencies.
Let’s look at the
a project that I converted recently to the
[[source]] url = "https://pypi.python.org/simple" verify_ssl = true [dev-packages] babel = "*" flake8 = "*" mock = "*" requests = "*" tox = "*" twine = "*" [packages] pytest = "*" "tap.py" = "*"
Pipfile uses TOML,
it can include sections
requirements.txt file could not.
The sections give a clear delineation
between user packages and developer packages.
The other dependencies do developer specific things.
mock help with test execution,
twine is for uploading the package to PyPI,
and so on.
I hope that you could have an intuition
about pytest-tap dependencies
even without my prose descriptions.
splitting things out permits regular users
to skip installing extra packages.
That’s the power
Now that we’ve covered the benefits,
how do you create a
for your own project?
Kenneth Reitz, of
a tool to manage a
pipenv helps users add and remove packages
with a virtual environment.
Rather than manipulating a virtual environment and pip directly,
you use the
and it will do the work for you.
If you come from the Ruby world,
this is very similar to
Suppose you have a project that depends on Django. You could prepare your Django project with these commands:
$ pipenv --three $ pipenv install Django $ pipenv lock
Those steps would:
- create a Python 3 virtual environment
- install Django and add it to a
- generate a
Once the files are created, you can share your work, and others should be able to recreate your environment.
Pipfile is still an emerging standard.
In spite of that,
it is very promising
and solves some problems
that arise when working
We saw how
Pipfile beats out the venerable
and we’re equipped with pipenv
Pipfiles for our projects.
I hope you learned something about Python dependencies and the brighter future that is accessible today.
Learn more about Python and other software topics
You can join my newsletter and get access to other useful topics to help you learn more about software development. I hope you'll join up!