Pytest Fixtures and Parametrize for Data Pipelines

A data-cleaning or intake pipeline tends to accumulate edge cases faster than any other kind of code: a missing field, a differently formatted date, a duplicate row, a value that is technically valid but clearly wrong. Writing a new, nearly-identical test function for each one is how test files become the least-read part of a codebase. Pytest's fixtures and parametrize mark exist specifically to separate two things that get tangled together in a naive test suite: the setup a test needs, and the specific inputs it should be checked against.

Fixtures: shared setup, not shared state

A fixture is a function decorated with @pytest.fixture that returns something a test needs — a database connection, a sample dataframe, a temporary file. Any test that declares the fixture's name as an argument receives its return value, and pytest handles calling the fixture function and passing the result through:

import pytest
import pandas as pd

@pytest.fixture
def raw_loads_df():
    return pd.DataFrame({
        "load_id": ["L-1", "L-2", "L-2", "L-3"],
        "weight_lbs": [42000, None, 38500, 51000],
        "origin": ["San Jose, CA", "reno, nv", "Reno, NV", ""],
    })

def test_drops_duplicate_load_ids(raw_loads_df):
    cleaned = clean_loads(raw_loads_df)
    assert cleaned["load_id"].is_unique

The value of the fixture is not that it saves a few lines of setup — it is that every test using raw_loads_df starts from the exact same known state, and a bug in the setup itself only needs to be fixed in one place. A fixture with a narrower scope than the default (scope="function", meaning it reruns for every test) is worth reaching for when setup is genuinely expensive, such as opening a real database connection: scope="module" or scope="session" runs the fixture once and reuses the result, which matters for speed but changes the contract — tests sharing a session-scoped fixture must not mutate it in ways that would leak into the next test.

conftest.py: sharing fixtures without importing them

A fixture used by only one test file can live in that file. A fixture used across several — a sample dataframe representative of real intake data, say — belongs in conftest.py, a file pytest discovers automatically and makes available to every test in its directory without an explicit import:

# tests/conftest.py
import pytest
import pandas as pd

@pytest.fixture
def messy_freight_df():
    return pd.DataFrame({
        "load_id": ["L-1", "L-2", None, "L-4"],
        "rate": ["$1,250.00", "980", None, "not a number"],
        "pickup_date": ["2026-08-01", "08/02/2026", "2026-08-03", ""],
    })

Any test file in the same directory tree can now declare messy_freight_df as an argument with no import statement, which keeps test files focused on assertions rather than on repeated setup boilerplate. The tradeoff is discoverability — a fixture used by name with no visible import can be harder to trace back to its definition — so a fixture only worth sharing across files should actually be shared; a fixture only one file needs is easier to understand living next to the test that uses it.

Parametrize: one test function, many cases

@pytest.mark.parametrize runs the same test body against a list of input-and-expected-output pairs, so ten related edge cases become ten reported test results without ten near-identical function definitions:

@pytest.mark.parametrize(
    "raw_weight, expected",
    [
        ("42000", 42000.0),
        ("42,000 lbs", 42000.0),
        ("42000.5", 42000.5),
        (None, None),
        ("", None),
        ("heavy", None),
    ],
    ids=["plain", "with-commas-and-unit", "decimal", "none", "empty-string", "non-numeric"],
)
def test_parse_weight(raw_weight, expected):
    assert parse_weight(raw_weight) == expected

The ids argument is easy to skip and worth not skipping: without it, a failing case shows up in the test report as test_parse_weight[case3], and finding out what case 3 actually was means going back to the source. With descriptive ids, the failure reads as test_parse_weight[non-numeric], which is most of the debugging already done before opening the file. As a rule of thumb: reach for parametrize when the thing varying between tests is the data flowing through one code path, and reach for a fixture when the thing varying is a resource or piece of shared setup that several tests need access to. When both are true at once — the same shared setup, exercised with several different inputs — a parametrized fixture combines them, running the fixture itself once per parameter value rather than once per test.

Parametrized fixtures: testing multiple implementations the same way

A parametrized fixture is worth the extra layer specifically when several different components need to satisfy the same test suite — for example, testing that both a CSV-backed store and a database-backed store handle the same set of operations identically:

@pytest.fixture(params=["csv_store", "sqlite_store"])
def load_store(request, tmp_path):
    if request.param == "csv_store":
        return CsvLoadStore(tmp_path / "loads.csv")
    return SqliteLoadStore(tmp_path / "loads.db")

def test_store_roundtrips_a_load(load_store):
    load_store.save({"load_id": "L-1", "weight_lbs": 42000})
    assert load_store.get("L-1")["weight_lbs"] == 42000

Every test that takes load_store as an argument now runs twice — once per backend — without being written twice. This catches the class of bug where two implementations of the same interface quietly diverge on an edge case, which a test suite written against only one implementation would never surface. It is the same principle as testing a client against both a real and a mocked version of a dependency, just applied to storage.

Where this earns its complexity, and where it does not

Stacking two or three parametrize decorators multiplies test counts — two decorators of five cases each produce twenty-five tests, which is a reasonable trade for genuine coverage of independent variables. The same stacking with two decorators of ten cases each produces a hundred tests that mostly assert the same thing with cosmetic input differences, which is a sign to collapse the cases into one explicit table of the combinations that actually matter rather than taking the full cross product. A parametrized case should also keep to one logical assertion; a test that branches heavily on its own parameter to decide what to assert is really several tests wearing one decorator, and splitting it back out usually makes failures easier to read, not harder.

Fixtures and parametrize also do not replace the kind of state-level assertion covered in the idempotency-key testing example — a parametrized test that only checks a return value can still miss a duplicate row written as a side effect. The two techniques answer different questions: parametrize answers "does this hold across many inputs," and asserting on stored state answers "did the code actually do what the return value implies." A thorough pipeline test suite needs both.

A short checklist

Is setup that more than one test needs pulled into a fixture, rather than repeated inline?
Do fixtures shared across files live in conftest.py, and fixtures used by one file stay local to it?
Does every parametrized case have a descriptive id, so a failure is legible from the test name alone?
Are stacked parametrize decorators producing meaningful coverage, or just a large number of near-duplicate cases?
Where the pipeline mutates state, does at least one test assert on that state directly rather than only on a return value?

This is the same discipline behind the API integration contract tests written for third-party calls — different layer of the system, same underlying goal of making a test suite's failures point directly at the actual problem instead of requiring a rerun with print statements added.