Python for DevOps: infrastructure automation in practice

Why use Python for infrastructure automation?

Python is a good fit for infrastructure automation because every major cloud has a Python SDK, the main configuration and infrastructure-as-code tools speak it, and a Python script is easy to read, test and review. A shell one-liner is fine for a one-off. Once a task runs every week, touches production or needs someone else to maintain it, Python pays back.

Here is what teams typically automate with it:

• Provisioning: creating servers, databases, buckets, queues and permissions.
• Configuration: installing packages, writing config files and starting services on servers.
• Deploys: pulling a release, installing dependencies, running migrations and tests, restarting the app and checking it is healthy.
• Housekeeping: stopping idle development machines, rotating old snapshots, cleaning up unused resources and reporting costs.
• Environment setup: giving every developer the same local environment with one command.

The rest of this guide is five recipes, from a single script to a full pipeline, then a section on choosing between a Python script and a declarative tool.

Recipe 1: cloud SDK scripts with boto3

The quickest win is a script that calls your cloud provider’s API directly. On AWS that is boto3; Azure and Google Cloud publish equivalent Python libraries.

This script finds running EC2 instances tagged env=dev and, only when you pass --apply, stops them. Listing first and acting only on request is a habit worth keeping for every script that changes infrastructure.

"""Stop running EC2 instances tagged env=dev. Run with --apply to act."""
import argparse

import boto3


def running_dev_instances(ec2):
    paginator = ec2.get_paginator("describe_instances")
    pages = paginator.paginate(
        Filters=[
            {"Name": "tag:env", "Values": ["dev"]},
            {"Name": "instance-state-name", "Values": ["running"]},
        ]
    )
    return [
        instance["InstanceId"]
        for page in pages
        for reservation in page["Reservations"]
        for instance in reservation["Instances"]
    ]


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--region", default="ap-south-1")
    parser.add_argument("--apply", action="store_true", help="stop them (default: list only)")
    args = parser.parse_args()

    ec2 = boto3.client("ec2", region_name=args.region)
    ids = running_dev_instances(ec2)
    print(f"{len(ids)} running dev instance(s): {ids}")
    if ids and args.apply:
        ec2.stop_instances(InstanceIds=ids)
        print("Stop requested.")


if __name__ == "__main__":
    main()

Three details make it production-worthy: the paginator (so it still works with hundreds of instances), the tag filter (so it never touches production), and the dry run by default. Run it on a schedule from your pipeline or as a scheduled Lambda function, and development machines stop costing money overnight.

Save it as stop_dev_instances.py. You can test it without an AWS account using moto, which fakes AWS services in memory:

import boto3
from moto import mock_aws

from stop_dev_instances import running_dev_instances


@mock_aws
def test_finds_only_running_dev_instances():
    ec2 = boto3.client("ec2", region_name="ap-south-1")

    def launch(env):
        return ec2.run_instances(
            ImageId="ami-12c6146b", MinCount=1, MaxCount=1,
            TagSpecifications=[{"ResourceType": "instance", "Tags": [{"Key": "env", "Value": env}]}],
        )["Instances"][0]["InstanceId"]

    dev = launch("dev")
    launch("prod")
    assert running_dev_instances(ec2) == [dev]
    ec2.stop_instances(InstanceIds=[dev])
    assert running_dev_instances(ec2) == []

Both files were run with boto3 1.42, moto 5.1 and pytest on Python 3.11 while writing this guide.

Recipe 2: infrastructure as code in Python

Scripts are good at actions; for resources that should exist permanently (networks, databases, buckets), describe the desired state and let a tool work out the changes. Pulumi and the AWS CDK let you write that description in Python.

A Pulumi program for a private bucket for user uploads looks like this:

"""__main__.py in a Pulumi project: a private bucket for uploads."""
import pulumi
import pulumi_aws as aws

uploads = aws.s3.Bucket("uploads")
aws.s3.BucketPublicAccessBlock(
    "uploads-private",
    bucket=uploads.id,
    block_public_acls=True,
    block_public_policy=True,
    ignore_public_acls=True,
    restrict_public_buckets=True,
)
pulumi.export("uploads_bucket", uploads.bucket)

pulumi preview shows what would change, and pulumi up applies it. Because it is ordinary Python, you can use functions, loops and tests to build a staging and a production environment from the same code. Resource names and arguments change between provider versions, so check them against the Pulumi registry for the version you install.

Terraform and OpenTofu are the other common choice. They use their own declarative language (HCL), not Python, but Python often sits around them: generating variable files, running terraform plan in a pipeline, and reading terraform output -json into later steps.

Recipe 3: configuration management with Ansible

Once a server exists, something has to install packages, write configuration and start services. Ansible does this over SSH with no agent on the server. It is written in Python, runs Python on the managed hosts, and lets you write your own modules in Python when the built-in ones don’t cover a task.

A small playbook that prepares app servers:

- name: Prepare app servers
  hosts: app
  become: true
  tasks:
    - name: Install system packages
      ansible.builtin.apt:
        name: [python3-venv, nginx]
        state: present
        update_cache: true

    - name: Start nginx on boot
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true

Run it with ansible-playbook -i inventory.ini servers.yml. Most Ansible modules are idempotent: running the playbook twice leaves the server in the same state, which is exactly what you want from configuration code.

Recipe 4: deploy scripts with Fabric, tests and a health check

For apps deployed to servers you control, Fabric runs commands over SSH from a Python script. The important part is the order: the release is tested on the server before the service restarts, and a health check confirms the app came back.

"""Deploy a tagged release to one server over SSH."""
import argparse
import os

from fabric import Connection


def deploy(host, release):
    c = Connection(host)
    with c.cd("/srv/app"):
        c.run(f"git fetch --tags && git checkout {release}")
        c.run(".venv/bin/pip install -r requirements.txt")
        c.run(".venv/bin/python -m pytest -q")
    c.sudo("systemctl restart app")
    c.run("curl --fail --silent http://localhost:8000/health")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--release", required=True)
    args = parser.parse_args()
    deploy(os.environ["DEPLOY_HOST"], args.release)

Any failing command stops the script, so a broken release never reaches the restart. The script assumes SSH key access and permission to restart the service without a password prompt; set both up once on the server. For more than one server, loop over hosts and deploy to one first, so a bad release stops at a single machine.

Recipe 5: wiring it into CI/CD

Automation is most useful when nobody has to remember to run it. A pipeline runs your tests and deploys only when they pass. This GitHub Actions workflow runs when you push a version tag such as v1.4.0, tests the code, then deploys that tag:

name: deploy
on:
  push:
    tags: ['v*']

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python -m pytest -q
      - uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.DEPLOY_SSH_KEY }}
      - run: mkdir -p ~/.ssh && echo "${{ secrets.DEPLOY_KNOWN_HOSTS }}" >> ~/.ssh/known_hosts
      - run: python deploy.py --release ${{ github.ref_name }}
        env:
          DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}

The workflow loads the SSH key into an agent from the DEPLOY_SSH_KEY secret and trusts the server through a DEPLOY_KNOWN_HOSTS secret (the output of ssh-keyscan for your host). The host, key and known-hosts entry all live in the repository’s encrypted secrets, never in the code. GitLab CI, Jenkins and other runners follow the same shape: check out, install, test, deploy. Our guide, CI/CD pipelines explained, covers the stages in more depth.

When a Python script fits, and when a declarative tool fits

Use a Python script for actions and a declarative tool for things that should stay in a known state. The traits of the task decide it:

Most real setups use both: infrastructure as code for the environment, Ansible or a container platform for configuration, and Python scripts for the glue and the housekeeping. If you are deciding how to host the app in the first place, our post on cloud-native development covers that choice.

Habits that keep infrastructure automation safe

Automation repeats mistakes as reliably as it repeats good work, so a few habits matter more than any library:

• No secrets in code. Use IAM roles, environment variables from the pipeline’s secret store, or a secret manager. Rotate any key that ever reached a repository.
• Dry run by default. Scripts that change infrastructure should list what they would do unless told to act.
• Least privilege. Give each script’s role only the permissions its job needs.
• Idempotent steps. Check before you create, so a rerun after a failure doesn’t duplicate resources.
• Tests and review. Test automation code with moto or a staging account, and review it like application code.
• Pinned versions. Pin library and provider versions so an upgrade happens on purpose, not on a Monday morning.
• Logs and alerts. Every scheduled job should log what it did and alert when it fails.

How we’d set up provisioning and deploys for a client web app

For a new custom software development project, this is how we’d approach the automation, as a method: environments defined as code from the first week, a pipeline that tests and deploys every change to staging, production releases from a tagged version, and scheduled Python jobs for backup checks and housekeeping. Where Python development is already the app’s language, we’d keep the automation in the same repository as the app, reviewed the same way. How a project moves from scoping to release is on how we run projects.