Subprocess

A subprocess in Python is a task that a python script delegates to the Operative system (OS).

The subprocess library allows us to execute and manage subprocesses directly from Python. That involves working with the standard input stdin, standard output stdout, and return codes.

We don’t have to install it with PIP, since it’s part of the Python standard library.

Therefore we can start using subprocesses in python just by importing the module.

import subprocess

First subprocess application

import subprocess 

subprocess.run('ls')

Pass arguments to the shell with subprocess

import subprocess

# subprocess.run('ls')  # Simple command

subprocess.run('ls -la', shell=True)

We’re running this command as a string and using the argument shell. That means we’re invoking a shell at the start of the execution of our subprocess, and the command argument is interpreted directly by the shell.

However, the use shell=True has many downsides, and the worst are the possible security leaks. You can read about them in the official documentation.

The best way to pass commands to the run function is to use a list where lst[0] is the command to call (ls in this case) and lst[n] are the arguments of that command.

If we do so, our code will look like this.

import subprocess

# subprocess.run('ls')  # Simple command

# subprocess.run('ls -la', shell=True) # Dangerous command

subprocess.run(['ls', '-la'])

If we want to store the standard output of a subprocess in a variable, we can do it by setting the argument capture_output to true.

list_of_files = subprocess.run(['ls', '-la'], capture_output=True)

print(list_of_files.stdout)

❯ python list_dir.py 
b'total 36\ndrwxr-xr-x 3 .. .. ..

Usage Examples of subprocess

Program checker

One of the main usages of this library is the ability to make simple OS operations.

For instance, a simple script that checks if a program is installed. In Linux, we can do this with the which command.

'''Program checker with subprocess'''

import subprocess

program = 'git'

process = subprocess. run(['which', program], capture_output=True, text=True)

if process.returncode == 0: 
    print(f'The program "{program}" is installed')

    print(f'The location of the binary is: {process.stdout}')
else:
    print(f'Sorry the {program} is not installed')

    print(process.stderr)

Note: In UNIX when a command is successful its status code is 0. Otherwise, something went wrong during the execution

Since we’re not using the shell=True argument, we can take the user input securely. Also, we can check if the input is a valid program with a regex pattern.

import subprocess

import re

programs = input('Separe the programs with a space: ').split()

secure_pattern = '[\w\d]'

for program in programs:

    if not re.match(secure_pattern, program):
        print("Sorry we can't check that program")

        continue

    process = subprocess. run(
        ['which', program], capture_output=True, text=True)

    if process.returncode == 0:
        print(f'The program "{program}" is installed')

        print(f'The location of the binary is: {process.stdout}')
    else:
        print(f'Sorry the {program} is not installed')

        print(process.stderr)

    print('\n')

In this case, we’re getting the programs from the user and using a regex expression that certifies the program string only includes letters and digits. We check the existence of each program with a for a loop.

Simple Grep in Python

Your friend Tom has a list of patterns in a text file and another large file in which he wants to get the number of matches for each pattern. He would spend hours running the grep command for every pattern.

Fortunately, you know how to solve this problem with Python, and you’ll help him to accomplish this task in few seconds.

import subprocess

patterns_file = 'patterns.txt'
readfile = 'romeo-full.txt'

with open(patterns_file, 'r') as f:
    for pattern in f:
        pattern = pattern.strip()

        process = subprocess.run(
            ['grep', '-c', f'{pattern}', readfile], capture_output=True, text=True)

        if int(process.stdout) == 0:
            print(
                f'The pattern "{pattern}" did not match any line of {readfile}')

            continue

        print(f'The pattern "{pattern}" matched {process.stdout.strip()} times')

Taking a look at this file, we define two variables which are the filenames we want to work with. Then we open the file that contains all the patterns and iterate over them. Next, we call a subprocess that runs a grep command with the “-c” flag (means count) and determine the output of the match with a conditional.

If you run this file (remember you can download the text files from the Github repo)

Set up a virtualenv with subprocess

One of the coolest things you can do with Python is process automation. This kind of script can save you hours of time per week.

For example, we’re going to create a setup script that creates a virtual environment for us and tries to find a requirements.txt file in the current directory to install all the dependencies.

import subprocess

from pathlib import Path


VENV_NAME = '.venv'
REQUIREMENTS = 'requirements.txt'

process1 = subprocess.run(['which', 'python3'], capture_output=True, text=True)

if process1.returncode != 0:
    raise OSError('Sorry python3 is not installed')

python_bin = process1.stdout.strip()

print(f'Python found in: {python_bin}')

process2 = subprocess.run('echo "$SHELL"', shell=True, capture_output=True, text=True)

shell_bin = process2.stdout.split('/')[-1]

create_venv = subprocess.run([python_bin, '-m', 'venv', VENV_NAME], check=True)

if create_venv.returncode == 0:
    print(f'Your venv {VENV_NAME} has been created')

pip_bin = f'{VENV_NAME}/bin/pip3'

if Path(REQUIREMENTS).exists():
    print(f'Requirements file "{REQUIREMENTS}" found')
    print('Installing requirements')
    subprocess.run([pip_bin, 'install', '-r', REQUIREMENTS])

    print('Process completed! Now activate your environment with "source .venv/bin/activate"')

else:
    print("No requirements specified ...")

In this case, we’re using multiple processes and parsing the data we need in our python script. We’re also using the pathlib library which allows us to figure it if the requirements.txt file exists.

If you run the python file you’ll get some useful messages of what’s happening with the OS.

❯ python setup.py 
Python found in: /usr/bin/python3
Your venv .venv has been created
Requirements file "requirements.txt" found
Installing requirements
Collecting asgiref==3.3.4 .......
Process completed! Now activate your environment with "source .venv/bin/activate"

Note that we get the output from the installation process because we’re not redirecting the standard output to a variable.

Run another Programming Language

We can run other programming languages with python and get the output from those files. This is possible because the subprocesses interact directly with the operative system.

For instance, let’s create a hello world program in C++ and Java. In order to execute the following file, you’ll need to install C++ and Java compilers.

helloworld.cpp

#include <iostream>

int main(){
    std::cout << "This is a hello world in C++" << std::endl;
    return 0;
}

helloworld.java

class HelloWorld{  
    public static void main(String args[]){  
     System.out.println("This is a hello world in Java");  
    }  
}  

I know this seems a lot of code compared to a simple Python one-liner, but this is just for testing purposes.

We’re going to create a Python script that runs all the C++ and Java files in a directory. To do this first we want to get a list of files depending on the file extension, and glob allows us to do it easily!

from glob import glob

# Gets files with each extension
java_files = glob('*.java')

cpp_files = glob('*.cpp')

After that, we can start using subprocesses to execute each type of file.

for file in cpp_files:
    process = subprocess.run(f'g++ {file} -o out; ./out', shell=True, capture_output=True, text=True)
    
    output = process.stdout.strip() + ' BTW this was runned by Python'

    print(output)

for file in java_files:
    without_ext = file.strip('.java')
    process = subprocess.run(f'java {file}; java {without_ext}',shell=True, capture_output=True, text=True)

    output = process.stdout.strip() + ' A Python subprocess runned this :)'
    print(output)

Copy

One little trick is to use the string function strip to modify the output and only get what we need.

Note: Be carefully to run large Java or C++ files since we’re are loading their output in memory and that could produce a memory leak.

Open external programs

We’re able to run other programs just by calling their binaries location through a subprocess.

Let’s try it out by opening brave, my preferred web browser.

import subprocess

subprocess.run('brave')

This will open a browser instance, or just another tab if you already have running the browser.

Opened browser

As with any other program that accept flags we can use them to produce the desired behavior.

import subprocess

subprocess.run(['brave', '--incognito'])

Incognito flag