Python subprocess.run() Instead of os.system()

Most of the Python admin scripts I take over from others run commands with os.system(). It is the first thing people find, and it looks like the shell they already know. It also passes the whole string to /bin/sh, gives you back an encoded wait status instead of an exit code, and has no way to capture output. subprocess.run(), available since Python 3.5, does all of that properly and is what I convert those scripts to.

The examples are written for Python 3.8 or newer and were run with Python 3.12.14. The test machine's system Python is 3.6.8, so I also ran them there and note what breaks on older versions.

What is wrong with os.system()

import os
status = os.system("ls /nonexistent")
print("status:", status)
print("exit code:", os.waitstatus_to_exitcode(status))

ls: cannot access '/nonexistent': No such file or directory
status: 512
exit code: 2

On Linux os.system() returns the raw wait status, so an exit code of 2 comes back as 512. You need os.waitstatus_to_exitcode() (3.9+) or a bit shift to decode it. The output goes directly to the terminal, a failure does not raise anything, and every call runs through the shell, which brings quoting and injection problems with it.

The basic call: a list of arguments

subprocess.run() takes the command as a list. The first item is the program, and every other item is passed to it as one argument, exactly as written:

import subprocess
result = subprocess.run(["wc", "-c", "my file.txt"])
print(result)
print("returncode:", result.returncode)

4 my file.txt
CompletedProcess(args=['wc', '-c', 'my file.txt'], returncode=0)
returncode: 0

The filename has a space in it and I did not quote it, because no shell is involved. Python executes wc directly and passes my file.txt as a single argument. The function waits for the command to finish and returns a CompletedProcess object with the arguments, the exit code in returncode, and captured output if you asked for it.

If the program does not exist at all, you get a Python exception rather than a shell's "command not found":

FileNotFoundError: [Errno 2] No such file or directory: 'no-such-tool'

shell=True and command injection

You can pass a string and shell=True, and run() will behave like os.system(): the string goes to /bin/sh -c. That is where the danger is. Suppose a filename arrives from a user, a web form or the name of an uploaded file:

import subprocess
filename = "notes.txt; echo INJECTED"
# shell=True: the string goes to /bin/sh, so ; starts a new command
subprocess.run("wc -l " + filename, shell=True)
# argument list: the whole value is one argument to wc
subprocess.run(["wc", "-l", filename])

1 notes.txt
INJECTED
wc: 'notes.txt; echo INJECTED': No such file or directory

With shell=True the semicolon ended the wc command and the shell ran whatever followed. Here that was a harmless echo. In a real attack it would be rm, curl ... | sh or anything else the web server user can run. With an argument list, the same value is just a strange filename that wc cannot find. Nothing is interpreted.

My rule is simple: use a list, and only use shell=True when I actually need shell features such as pipes, globbing or redirection, with a string I wrote myself. If outside data must go into a shell string, quote it with shlex.quote():

import shlex
import subprocess
filename = "notes.txt; echo INJECTED"
cmd = "wc -l " + shlex.quote(filename)
print(cmd, flush=True)
subprocess.run(cmd, shell=True)

wc -l 'notes.txt; echo INJECTED'
wc: 'notes.txt; echo INJECTED': No such file or directory

Most pipelines are easier to do in Python anyway: run one command, capture its output, and process the text or feed it to the next command with input= (shown below).

check=True and CalledProcessError

By default run() does not care whether the command succeeded. A non-zero exit code is simply stored in returncode and your script continues. Add check=True and a failure raises CalledProcessError, which is almost always what a script should do:

import subprocess
try:
    subprocess.run(["ls", "/nonexistent"], check=True)
except subprocess.CalledProcessError as e:
    print("command:", e.cmd)
    print("exit code:", e.returncode)

ls: cannot access '/nonexistent': No such file or directory
command: ['ls', '/nonexistent']
exit code: 2

If you do not catch it, the script stops with a traceback that ends in a clear message. It exits with status 1, which is the Python equivalent of set -e in Bash:

subprocess.CalledProcessError: Command '['ls', '/nonexistent']' returned non-zero exit status 2.

Leave check off when a non-zero exit code is an answer rather than an error. grep returns 1 for "no match", for example, and that is something you want to test, not stop on:

import subprocess
r = subprocess.run(
    ["grep", "-q", "root", "/etc/passwd"]
)
print("root found" if r.returncode == 0 else "not found")
r = subprocess.run(["grep", "-q", "nobody-here", "/etc/passwd"])
print("returncode:", r.returncode)

root found
returncode: 1

Capturing output: capture_output and text

To get a command's output into a variable, pass capture_output=True. Add text=True to get strings instead of bytes:

import subprocess
result = subprocess.run(
    ["git", "--version"],
    capture_output=True,
    text=True,
    check=True,
)
print(repr(result.stdout))
print(result.stdout.strip())

'git version 2.43.7\n'
git version 2.43.7

Note the trailing newline. Command output nearly always ends with one, so .strip() or .splitlines() is the usual next step. Without text=True you get bytes, which is right for binary data but awkward for everything else:

b'hello\n'

When the command fails, the captured standard error is attached to the exception, which makes for much better error messages than a bare exit code:

import subprocess
try:
    subprocess.run(
        ["ls", "/nonexistent"],
        capture_output=True, text=True, check=True,
    )
except subprocess.CalledProcessError as e:
    print("exit code:", e.returncode)
    print("stderr:", e.stderr.strip())

exit code: 2
stderr: ls: cannot access '/nonexistent': No such file or directory

To merge stderr into stdout, as 2>&1 would, use stdout=subprocess.PIPE, stderr=subprocess.STDOUT instead of capture_output. To discard output, pass subprocess.DEVNULL for either stream.

Sending input with input=

input= writes a string (or bytes) to the command's standard input and closes it. Here is sort fed from Python with no temporary file:

import subprocess
text = "banana\napple\ncherry\n"
r = subprocess.run(
    ["sort"], input=text, capture_output=True, text=True, check=True
)
print(r.stdout, end="")

apple
banana
cherry

Together with captured output, this is how I replace shell pipelines. The stdout of one run() becomes the input= of the next. I can inspect or filter it in between.

timeout: never wait forever

A command that hangs, such as an SSH connection to a dead host or a mount that stopped responding, will make your script hang too. timeout= takes a number of seconds. When it runs out, the child is killed and TimeoutExpired is raised:

import subprocess
try:
    subprocess.run(["sleep", "10"], timeout=2)
except subprocess.TimeoutExpired as e:
    print("timed out:", e.cmd)

timed out: ['sleep', '10']

That printed after two seconds, not ten. I put a timeout on anything that touches the network.

Python 3.6 and older

capture_output and text were added in Python 3.7. On 3.6 every example that uses them fails before the command even starts:

TypeError: __init__() got an unexpected keyword argument 'capture_output'

If you must use 3.5 or 3.6, the older form does the same thing and still works on current versions:

import subprocess
# Python 3.5/3.6 spelling of capture_output=True, text=True
r = subprocess.run(
    ["git", "--version"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    universal_newlines=True,
    check=True,
)
print(r.stdout.strip())

git version 2.43.7

I ran that on both 3.6.8 and 3.12.14 with the same result. check=True, timeout=, input= and argument lists all behave the same on 3.6 as shown above, and os.waitstatus_to_exitcode() needs 3.9.

A summary I keep in my head

• Pass a list, not a string. Leave shell=True for fixed strings that really need the shell.
• Add check=True unless a non-zero exit is an expected answer.
• Use capture_output=True, text=True when you need the output.
• Set a timeout for anything that can hang.
• Use cwd= and env= instead of cd ... && or VAR=x prefixes in a shell string.

Everything else is in the subprocess documentation, including Popen for the rare case where you need to talk to a running process.

Related reading

• Python venv in practice
• Bash strict mode: set -euo pipefail
• _execq(): running commands quietly in Bash