Python venv in Practice: Create, Use and Rebuild

Installing Python packages with sudo pip install works until two projects need different versions of the same library. Or a system tool written in Python breaks because you upgraded something it depended on. A virtual environment fixes that: a private directory with its own python, its own pip and its own site-packages, one per project.

Python has shipped the venv module since 3.3, so there is nothing to install on most systems. This is how I use it every day. I tested every step with the system Python 3.6.8 on an old CentOS-family machine. I also tested with Python 3.12.14 (plus 3.11.13 for one comparison) on the same machine, and I point out where they behave differently.

Create and activate

Run venv as a module of the interpreter you want the project to use, and give it a directory name. I always call it .venv inside the project folder:

$ cd ~/myproject
$ python3.12 -m venv .venv
$ ls .venv .venv/bin
.venv:
bin
include
lib
lib64
pyvenv.cfg

.venv/bin:
activate
activate.csh
activate.fish
Activate.ps1
pip
pip3
pip3.12
python
python3
python3.12

The python files in bin/ are symlinks back to the real interpreter, and pyvenv.cfg records which one. There is no copy of Python itself in there, just a place for packages:

$ cat .venv/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.12.14
executable = /usr/bin/python3.12
command = /usr/bin/python3.12 -m venv /home/awie/myproject/.venv

On 3.6 the file has only the first three lines. Activating the environment is a matter of sourcing a script into your current shell (use activate.fish or activate.csh for those shells):

$ which python3
/usr/bin/python3
$ . .venv/bin/activate
(.venv) $ which python pip
/home/awie/myproject/.venv/bin/python
/home/awie/myproject/.venv/bin/pip
(.venv) $ python -c 'import sys; print(sys.prefix); print(sys.base_prefix)'
/home/awie/myproject/.venv
/usr

That is all activation does. It puts .venv/bin at the front of PATH, sets VIRTUAL_ENV, changes the prompt and defines a deactivate function that undoes it. Nothing special happens to Python itself. The interpreter works out from its own location that it is inside a venv, which is why sys.prefix differs from sys.base_prefix. That check is also the reliable way for a script to ask "am I in a venv?".

Note: on Debian and Ubuntu the venv module is split into a separate package. If python3 -m venv complains that ensurepip is not available, install python3-venv (or the version-specific package such as python3.12-venv) first.

Install packages and freeze them

With the environment active, pip installs into .venv and nowhere else, so no sudo is needed. The test machine for this article has no internet access, so I installed two small packages from a local folder of wheel files with --no-index --find-links. On a normal machine you would simply type pip install requests or whatever you need:

(.venv) $ pip install --no-index --find-links ../wheelhouse greet
Looking in links: ../wheelhouse
Processing /home/awie/wheelhouse/greet-1.2.0-py3-none-any.whl
Processing /home/awie/wheelhouse/colourise-0.3.1-py3-none-any.whl (from greet)
Installing collected packages: colourise, greet
Successfully installed colourise-0.3.1 greet-1.2.0

That offline method is useful by itself: it is how I install onto servers that are not allowed to reach PyPI.

pip freeze prints every installed package with its exact version, including dependencies you never asked for directly. Redirect it into requirements.txt and commit that file:

(.venv) $ pip freeze > requirements.txt
(.venv) $ cat requirements.txt
colourise==0.3.1
greet==1.2.0

Pinned versions are what you want for an application you deploy: the server gets exactly what you tested. For a library you publish, you would list looser ranges in its packaging metadata instead, but that is a different topic.

Use the venv without activating it

Activation is a convenience for interactive shells. In cron jobs, systemd units, deploy scripts and Makefiles I never activate. I call the interpreter inside the venv by its path. Python sets itself up correctly from that alone:

$ deactivate
$ which python3
/usr/bin/python3
$ .venv/bin/python -c 'import greet; print(greet.hello("world"))'
Hello, world
$ python3.12 -c 'import greet'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'greet'

The same goes for anything a package installs into bin/: .venv/bin/pip, .venv/bin/gunicorn and so on all run inside the environment. A crontab line looks like this:

*/15 * * * * cd /srv/myproject && .venv/bin/python sync.py

Running source inside a cron job or a #!/bin/sh script is a common cause of "works in my terminal, fails in cron" problems. The full path avoids all of them.

Delete and recreate, do not repair

A venv is disposable. If it gets into a strange state, or the system Python is upgraded underneath it, I do not try to fix it. I delete it and rebuild from requirements.txt:

$ rm -rf .venv
$ python3.12 -m venv .venv
$ .venv/bin/pip install --no-index --find-links ../wheelhouse -r requirements.txt
Looking in links: ../wheelhouse
Processing /home/awie/wheelhouse/colourise-0.3.1-py3-none-any.whl (from -r requirements.txt (line 1))
Processing /home/awie/wheelhouse/greet-1.2.0-py3-none-any.whl (from -r requirements.txt (line 2))
Installing collected packages: colourise, greet
Successfully installed colourise-0.3.1 greet-1.2.0
$ .venv/bin/pip list
Package   Version
--------- -------
colourise 0.3.1
greet     1.2.0
pip       23.2.1

python -m venv --clear .venv does the same cleanup in one step, leaving an empty environment with just pip.

Why the venv does not belong in Git

Add .venv/ to .gitignore from the start. My test environment with nothing but pip in it was already 15 MB and about a thousand files. Size is not the main reason, though. A venv is tied to the machine and the path where it was created:

• The python entries are symlinks to /usr/bin/python3.12 on this machine. On a server with a different Python they point at nothing.
• Packages with compiled extensions are built for this OS, CPU and Python version.
• Scripts in bin/ and the activate script have the absolute path of the venv written into them.

That last point is easy to demonstrate. I copied the whole project to another directory and asked the copied pip who it was:

$ cp -a ~/myproject ~/moved
$ cd ~/moved
$ .venv/bin/pip --version
pip 23.2.1 from /home/awie/myproject/.venv/lib64/python3.12/site-packages/pip (python 3.12)

It ran without error, but it is still using the original environment's interpreter. Delete ~/myproject and the copy breaks. Worse, while both exist, installing into the copy silently modifies the original. Moving or renaming a project folder has the same effect. This is another reason to treat the venv as something you recreate rather than move around. Commit requirements.txt, never .venv.

Differences between Python versions

The basics have not changed since Python 3.3, but a few details have:

The bundled pip is whatever version shipped with that Python release, and an old pip can fail to install modern packages. The usual first step in a fresh environment is therefore:

$ .venv/bin/python -m pip install --upgrade pip

Python 3.9 added --upgrade-deps, which does that upgrade as part of creating the venv (python3 -m venv --upgrade-deps .venv). It needs network access to PyPI, so I could not run it on the offline test machine. I confirmed what each version's --help says: 3.11 upgrades pip and setuptools, 3.12 only pip, and 3.6 rejects the option with "unrecognized arguments".

The setuptools change surprises people: code that ran python setup.py or imported pkg_resources inside a fresh 3.12 venv now fails until you pip install setuptools yourself. You may still have to support 3.6 (it has been end-of-life since December 2021, but old enterprise distributions still use it). Everything in this article apart from --upgrade-deps works there as shown.

The venv documentation covers the remaining options such as --system-site-packages and --prompt.

Related reading

• Python subprocess.run() instead of os.system()
• systemd timers as a cron replacement