Python Workflow Automation: Automate Repetitive Tasks

Every team has a task that somebody does by hand each week: download a few exports, paste them into one spreadsheet, fix the column names, total it up and email it to the same people. It takes an hour, nobody enjoys it, and it goes wrong whenever the person who knows how is on leave.

That is the kind of work Python is good at. This guide is a set of practical recipes for the most common jobs, each with code you can adapt. It ends with the part most tutorials skip: how to tell when a script has become something people rely on, and what it needs then.

What tasks can you automate with Python?

You can automate most repetitive computer work that follows clear rules and rarely needs judgement. A good test: if you could write the steps down for a new colleague and they would never need to ask you a question, a script can probably do it.

Common candidates:

• Files and folders. Renaming, sorting, archiving, converting formats, cleaning up old downloads.
• Spreadsheets and data. Combining monthly files, cleaning columns, removing duplicates, building summary tables.
• APIs. Pulling orders, leads, tickets or analytics from the services you already use, and pushing data between them.
• Websites. Collecting public information, such as notices, prices or listings, where there is no API.
• Reports and email. Generating a weekly summary and sending it to the people who need it.
• Browser and desktop tasks. Filling in forms or clicking through an app that has no API, as a last resort.

Poor candidates are tasks that change every time, need a decision a person should own, or happen so rarely that writing the script takes longer than doing the work.

Before you start: set up a clean project

Give each automation its own folder, its own virtual environment and a requirements.txt, so it keeps working when other projects change. It takes two minutes and saves hours later.

mkdir weekly-report && cd weekly-report
python3 -m venv .venv
source .venv/bin/activate        # on Windows: .venv\Scripts\activate
pip install pandas openpyxl requests beautifulsoup4
pip freeze > requirements.txt

Keep passwords and API keys out of the code. Read them from environment variables, as the recipes below do, and never commit them to version control.

Recipe 1: sort and tidy files

The safest first automation is a file tidy-up, because you can see every change and undo it. This script sorts files in your Downloads folder into subfolders by type. It runs in “dry run” mode first, printing what it would do without moving anything.

from pathlib import Path
import shutil

DOWNLOADS = Path.home() / "Downloads"

FOLDERS = {
    ".pdf": "Documents",
    ".docx": "Documents",
    ".xlsx": "Spreadsheets",
    ".csv": "Spreadsheets",
    ".jpg": "Images",
    ".png": "Images",
    ".zip": "Archives",
}


def sort_folder(folder: Path, dry_run: bool = True) -> None:
    for item in folder.iterdir():
        if not item.is_file():
            continue
        target_name = FOLDERS.get(item.suffix.lower())
        if target_name is None:
            continue
        target_dir = folder / target_name
        target = target_dir / item.name
        if target.exists():
            print(f"skip (already exists): {target}")
            continue
        print(f"{item.name} -> {target_name}/")
        if not dry_run:
            target_dir.mkdir(exist_ok=True)
            shutil.move(str(item), str(target))


if __name__ == "__main__":
    sort_folder(DOWNLOADS, dry_run=True)  # set dry_run=False once the output looks right

Two habits from this recipe are worth keeping in every script: a dry run before anything destructive, and a check that never overwrites an existing file.

Recipe 2: combine spreadsheets into one summary

Combining a folder of spreadsheets into one summary is the task that wins most teams over to automation. pandas reads each file, stacks them and totals them in a few lines.

This example assumes a folder called monthly_sales holding Excel files that share the columns region and amount. Change the folder and column names to match your files.

from pathlib import Path
import pandas as pd

files = sorted(
    f for f in Path("monthly_sales").glob("*.xlsx")
    if not f.name.startswith("~$")  # skip Excel lock files for open workbooks
)
frames = [pd.read_excel(f).assign(source_file=f.name) for f in files]
sales = pd.concat(frames, ignore_index=True)

summary = (
    sales.groupby("region", as_index=False)["amount"]
    .sum()
    .sort_values("amount", ascending=False)
)

summary.to_excel("weekly_summary.xlsx", index=False)
print(f"Combined {len(files)} files, {len(sales)} rows")

The source_file column records which file each row came from, so when a number looks wrong you can trace it back. pandas needs the openpyxl package to read and write .xlsx files. For cleaning and analysis beyond this, see our guide to data analysis with pandas.

Recipe 3: pull data from an API

Most tools you already pay for, such as a CRM, a helpdesk, an online store or an analytics service, have an API, and pulling data from it is more reliable than downloading exports by hand. The Requests library handles the HTTP part.

The endpoint, parameters and field names below are made up to show the shape. Your API’s documentation gives the real ones, including how it pages through results.

import csv
import os

import requests

API_URL = "https://api.example.com/v1/orders"


def fetch_orders(since: str) -> list:
    headers = {"Authorization": f"Bearer {os.environ['API_TOKEN']}"}
    orders, page = [], 1
    while True:
        response = requests.get(
            API_URL,
            params={"since": since, "page": page},
            headers=headers,
            timeout=30,
        )
        response.raise_for_status()
        batch = response.json()["orders"]
        if not batch:
            return orders
        orders.extend(batch)
        page += 1


def save_csv(rows: list, path: str) -> None:
    if not rows:
        return
    with open(path, "w", newline="") as f:
        fieldnames = list(dict.fromkeys(key for row in rows for key in row))
        writer = csv.DictWriter(f, fieldnames=fieldnames, restval="")
        writer.writeheader()
        writer.writerows(rows)


if __name__ == "__main__":
    save_csv(fetch_orders(since="2026-01-01"), "orders.csv")

The fieldnames line collects every key that appears in any row, in order, so a row with an extra field doesn’t crash the export. Three details matter more than they look: a timeout, so a slow server can’t hang the script forever; raise_for_status(), so an error page is never saved as data; and the token read from an environment variable, not typed into the file.

Recipe 4: collect information from a website

When a site has no API, you can read its public pages with Requests and Beautiful Soup. Check the site’s terms and its robots.txt first, keep your request rate low, and prefer an official API or data download whenever one exists.

This example lists the titles and links of notices on a page. The URL and the CSS selector are placeholders: open the page in your browser’s developer tools to find the selector that matches the items you want.

from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

URL = "https://example.com/notices"

response = requests.get(
    URL,
    headers={"User-Agent": "notice-checker (you@example.com)"},
    timeout=30,
)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
for link in soup.select("a.notice-title"):
    print(link.get_text(strip=True), urljoin(URL, link["href"]))

Pages built with JavaScript may not contain the data in their HTML at all. For those, a browser automation library such as Playwright or Selenium loads the page as a browser would. It is slower and breaks more often, so treat it as a last resort.

Scrapers break when the site changes its layout. Anything that depends on one needs a check that notices when it suddenly returns nothing.

Recipe 5: send a report by email

Python’s standard library can send email with an attachment, so the weekly summary from Recipe 2 can go out without anyone opening their inbox. Use an account set up for sending, such as an app password or an email service’s SMTP details, never your personal password in the script.

import os
import smtplib
from email.message import EmailMessage
from pathlib import Path


def build_report_email(attachment: Path, to: str) -> EmailMessage:
    msg = EmailMessage()
    msg["Subject"] = f"Weekly report: {attachment.stem}"
    msg["From"] = os.environ["REPORT_FROM"]
    msg["To"] = to
    msg.set_content("Hi,\n\nThis week's report is attached.\n\nThis email was sent automatically.")
    msg.add_attachment(
        attachment.read_bytes(),
        maintype="application",
        subtype="octet-stream",
        filename=attachment.name,
    )
    return msg


def send(msg: EmailMessage) -> None:
    host = os.environ["SMTP_HOST"]
    port = int(os.environ.get("SMTP_PORT", "587"))
    with smtplib.SMTP(host, port) as server:
        server.starttls()
        server.login(os.environ["SMTP_USER"], os.environ["SMTP_PASSWORD"])
        server.send_message(msg)


if __name__ == "__main__":
    report = Path("weekly_summary.xlsx")
    send(build_report_email(report, to="team@example.com"))

Say in the email that it was sent automatically, and send it to yourself for the first few weeks before you add anyone else.

Recipe 6: run it on a schedule, with logging and retries

A script only saves time once it runs without you. Schedule it with the tool your machine already has, and make it write a log so you can see what happened while you weren’t watching.

On macOS or Linux, use cron. Run crontab -e and add a line. This one runs the report every Monday at 7am:

0 7 * * 1 cd $HOME/weekly-report && .venv/bin/python weekly_report.py >> cron.log 2>&1

On Windows, use Task Scheduler. Create a basic task, choose the trigger, and point the action at the Python inside your virtual environment, with the script as its argument.

For scripts kept in a repository, a scheduled CI job, such as a GitHub Actions workflow with a schedule trigger, runs them without a machine of your own. Its cron times are in UTC.

Then wrap the work in logging and retries, so a brief network error doesn’t cost you a week’s report and a real failure leaves a trace:

import logging
import time

logging.basicConfig(
    filename="automation.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)


def with_retries(task, attempts=3, wait_seconds=30):
    for attempt in range(1, attempts + 1):
        try:
            result = task()
            logging.info("%s succeeded on attempt %d", getattr(task, "__name__", repr(task)), attempt)
            return result
        except Exception:
            logging.exception("%s failed on attempt %d", getattr(task, "__name__", repr(task)), attempt)
            if attempt == attempts:
                raise
            time.sleep(wait_seconds * attempt)

Call your main function through it, for example with_retries(build_weekly_report). Each failure is logged with its full error, the wait grows with each attempt, and after the last attempt the error is raised so the scheduler records a failed run.

When you have many jobs that depend on each other, a workflow orchestrator such as Apache Airflow, Prefect or Dagster manages the schedule, the order and the retries for you. Most teams don’t need one until they have dozens of jobs.

Which Python automation tools should you use?

Use the standard library first, and add a package only when it saves real work. Python ships with most of what file, email and logging tasks need, and every extra package is something to install and update.

For infrastructure work, such as servers, deployments and cloud resources, the tools and habits are different. Our guide to Python for infrastructure automation covers that side.

When should a script become a maintained system?

A script should become a maintained system when people depend on its output, when it touches money or customer data, when it runs unattended, or when a silent failure would hurt. Until then, a tidy script with a log is enough. After that, the cost of a quiet failure is higher than the cost of doing it properly.

What a maintained system adds:

• Version control. The code lives in a repository, not on one person’s laptop.
• Logging and alerts. Every run is logged, and a failure, or a run that returns suspiciously little data, sends an alert to a person.
• Retries and safe reruns. Running the job twice must not send two emails or create duplicate records.
• A review step for risky actions. Anything that sends to customers, changes prices or deletes data waits for a person to approve it.
• Tests. At least a check against a small sample file, so a change in the input format is caught before it reaches a report.
• A named owner. Someone who is told when it fails and who decides when it changes.
• Documentation. A short README: what it does, where it runs, which secrets it needs, and how to run it by hand.

Signs you have crossed the line: the script’s output goes to people outside your team, someone asks “did the report go out?”, or the person who wrote it is the only one who can fix it.

Some automations outgrow scripts entirely. When a shared spreadsheet and a script have become the way a team runs a process, it may be time to turn a spreadsheet into a web tool with proper logins, history and permissions. When a workflow needs reading documents, classifying requests or drafting replies, that is the territory of AI automation for business workflows.

Automation at scale: an example

Here is the scale automation can reach: Tailwebs built and maintains the Crime and Punishment database (crimeandpunishment.in) for the Vidhi Centre for Legal Policy: a database of 7,305 crimes across 370 central laws. A scraping engine plus AI extracts punishment data and tags law sections.

How we’d approach automating a workflow

We’d start by writing the workflow down as it happens today: every step, who does it, how long it takes, and where it goes wrong. Then we’d pick the step with the most repetition and the least judgement, automate that one first, and run the script beside the manual process until the outputs match. Only then would we add the next step, and the logging, alerts and owner from the section above. Where the work calls for a full build rather than a script, see our Python development page.

The short version

• Automate tasks that follow clear rules and repeat often. Leave judgement to people.
• Give each automation its own folder, virtual environment and requirements file, and keep secrets in environment variables.
• Start with files and spreadsheets, where mistakes are visible and easy to undo.
• Use APIs before scraping, and treat browser automation as a last resort.
• Schedule with cron, Task Scheduler or a CI job, and log every run.
• Add retries, alerts, a review step, tests and an owner once people depend on the output.