Systems &
Infrastructure

What this week covers

You've now seen both engines. This week is the skills that apply to working inside them — and the part nobody teaches you in a class, because it only matters when code runs without a human watching.

Two halves. First, system design: how a new component plugs into an existing engine, where its boundaries sit, and which tradeoffs are different because the system is live. Second, running it: where the code executes, how it gets scheduled, and how it gets credentials without you pasting them into a file.

By Friday you should be able to:

• Describe the interface between a component you're adding and the system it plugs into — and say what crosses that line.
• Name the three tradeoffs that only matter because the system is live: latency, failure handling, data freshness.
• Read one of our Airflow DAGs and say what runs, when, and what happens if it fails.
• Get an API key into running code with no credential in the repository.

Part 1 — System Design

Designing at the component level

You will almost never build a system from scratch here. You'll add a component to a system that already runs: a new fetcher, a new cleaner, a new strategy, a new risk check. The design question is never "how would I build this?" It's "where does this plug in, and what does it promise?"

A well-built system has already answered that question for you. It has an abstract base class per role, and adding a capability means implementing one — a new vendor, a new destination, a new strategy. Week 5 showed the shape: find the role your work belongs to, implement that interface, and register it in config. If you find yourself editing the controller, stop and check whether you've picked the wrong seam.

Whatever you add, it answers four questions:

Interfaces and boundaries

An interface is the promise your component makes to everything downstream. It isn't the code — it's the shape of what crosses the line, and it's the part you can't change casually once something depends on it.

The biggest interface you'll meet here is a table:

producer  ──writes──▶  a table  ──reads──▶  consumer

The schema is the contract. Integrating through shared storage is common and sensible — it lets systems in different languages and different repositories evolve separately. It's also a database boundary, which behaves differently from an interface inside one codebase. There, a function signature is compiler-enforced: change it and the build breaks. Across a table, the agreement is the schema, and neither side's compiler can see the other.

So the guarantees are maintained deliberately:

• Narrow. The fewer columns and assumptions that cross, the less breaks when either side changes.
• Explicit about time. Every interface in a trading system carries a timestamp and a meaning for that timestamp. "The daily bar for the 14th" — is that the session that opened on the 14th or closed on it? Write it down.
• Stable under replay. Re-run your component over the same historical window and you get the same answer. If it depends on "now," it isn't replayable, and it can't be backtested.

Three tradeoffs that are different because it's live

General software engineering has opinions about all three. A live trading system pushes each one somewhere unusual — that's what makes this the QD-specific part of the job.

latest = data_access.get_latest_date_for(schema, table)

if latest is None or (today - latest) > MAX_STALENESS:
    raise StaleDataError(f"{table}: newest row is {latest}, expected {today}")

Part 2 — Deployment & Runtime

Where code actually runs

Your laptop is not the runtime. This sounds obvious and is the source of roughly half of all "but it worked for me" incidents.

Our stack runs on AWS EC2, in containers, 24/7. Code that runs on your machine has your environment variables, your Python version, your packages, your network access, your clock, and your filesystem. The server has none of those.

Three properties the runtime has that your laptop doesn't:

• It's always on. That's the point — your laptop being closed can't be why a table is stale.
• It's small and shared. Genuinely small — our scheduler host has run on well under a gigabyte of RAM while carrying the scheduler, the web UI, and every pipeline. Memory is a real constraint here, and it shows up in real design decisions (below).
• Nobody is watching it. There's no terminal. If your code prints something and nothing captures it, that output never existed.

Scheduling: code that runs without you

The canonical QD job: "pull yesterday's data every morning before we need it." Nobody types that command. It's scheduled.

We do this two different ways, and the difference is instructive.

Airflow — the data pipeline

Airflow is a scheduler. That's the one-sentence version, and it's most of what you need this week.

You describe your job in a Python file: what to run, how often, what to do if it fails. Airflow's scheduler reads that file, fires the job on schedule, records whether each run succeeded, retries what it's told to retry, and gives you a web UI showing green and red squares per day. That's the deal — you write the work, Airflow decides when it happens and remembers what happened.

Three words you'll see constantly:

• DAG — the Python file describing a schedulable job. It stands for "directed acyclic graph," because tasks can depend on each other and cycles would never finish. All three of ours have exactly one task, so the graph part isn't doing much work here yet.
• Operator — one unit of work. Ours are all PythonOperator: "call this function."
• Task instance — one operator on one date. This is the thing that succeeds or fails, and the square you see in the UI.

We run one DAG per data pipeline. Each fires early in the morning, a few minutes apart, and does one thing: run that pipeline's Orchestrator against its config. A DAG per pipeline, a task per DAG.

Here's the shape of one, lightly simplified. It's shorter than you'd expect:

local_tz = pendulum.timezone("America/New_York")

default_args = {
    "owner": "airflow",
    "depends_on_past": False,
    "email_on_failure": True,
    "email_on_retry": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
}

def run_pipeline(**kwargs):
    import asyncio                                    # ← note: imports inside the function
    from utils.dynamic_loader import load_config
    from src.orchestrator import Orchestrator

    config = load_config(CONFIG_PATH)
    orchestrator = Orchestrator(config=config)
    asyncio.run(orchestrator.run())

with DAG(
    "data_pipeline_dag",
    default_args=default_args,
    schedule_interval="15 7 * * 1-5",   # weekdays 7:15 AM ET
    start_date=datetime(2024, 12, 1, tzinfo=local_tz),
    catchup=False,
    max_active_runs=1,
    tags=["data_pipeline"],
) as dag:
    PythonOperator(task_id="run_pipeline", python_callable=run_pipeline)

Six details in that file are load-bearing. Read them in order:

• schedule_interval is cron syntax. Five fields — minute, hour, day, month, weekday. 15 7 * * 1-5 is "7:15, Monday through Friday." 1-5 and not * because markets are closed on weekends, and a run that finds no data is at best noise.
• America/New_York, not UTC. The schedule is pinned to the market's timezone, so 7:15 AM ET stays 7:15 AM ET through daylight saving. This is a deliberate choice and the right one for market data: the thing we care about — the exchange session — moves with New York, so our schedule should too. Had we written it in UTC, the job would silently drift an hour twice a year relative to the open.
• catchup=False. Airflow's default is the opposite: turn it on and a DAG with an old start_date will, on first deploy, try to run every day since — hundreds of runs, at once, on the smallest machine we own. catchup=False says "only run from now on." Backfills are a deliberate act, not a side effect of deploying.
• max_active_runs=1. Never two copies of this pipeline at once. If yesterday's run is somehow still going, today's waits.
• retries: 1 + retry_delay: 5 minutes. One retry, five minutes later. This is the "transient failure" bet — a vendor blip resolves in five minutes; a real bug doesn't, and burning ten retries on it just delays the email.
• email_on_failure: True. The whole alerting story. There's no dashboard and no pager — if a DAG fails, an email goes out, and someone has to read it.

The imports inside the function

That's the detail most worth understanding, because it's a real constraint leaving a fingerprint on real code.

Airflow's scheduler re-parses every DAG file, constantly — that's how it notices changes. Parsing means executing everything at module level. So a top-level import pandas means pandas gets imported over and over, forever, just to work out what time it is.

On a small host, that's enough to blow Airflow's DAG-import timeout — at which point Airflow decides the DAG is broken and stops scheduling it. Your pipeline doesn't crash. It silently never runs.

Hence the pattern: keep the DAG file's module level trivially cheap, and defer the heavy imports into the task function, which only executes when the task actually runs. The comment in our DAGs says so outright.

cron — the trade engine

Not everything here runs under Airflow. A job that fires once a day can just as well run on plain cron:

30 9 * * *   sh /app/scripts/run_live_trend.sh >> /var/log/cron.log 2>&1

One line, one job, once a day at 09:30. Note what you give up versus Airflow: no run history, no retry policy, no UI, no failure email. And note the two details that still matter:

• >> ... 2>&1 — send stdout and stderr to a log file. Without 2>&1, the error messages — the only part you actually want — go nowhere. Cron's default is to email them to a local mailbox nobody reads.
• Cron has no PATH. Not your shell's, anyway. This is the single most common cron failure: a bare python that works in your terminal and doesn't exist in cron's environment. Use absolute paths for everything.

Why have both? Airflow earns its keep when you have several jobs, want retries, and benefit from run history. Cron is fine when one line is genuinely all you need — reaching for a scheduler to run a single daily binary is more machinery than problem. Match the tool to the job rather than standardizing for its own sake.

Ordering between jobs

When one job's output is another's input, something has to sequence them. The blunt approach is a time gap: run the producer early, the consumer hours later, and rely on the slack. It's simple and it usually works.

What it costs is worth naming. A gap is not a dependency — the consumer doesn't know the producer succeeded, it just reads whatever is there and trusts it. Overrun the gap and the consumer runs on yesterday's data with no complaint from anyone. The alternative is an explicit dependency, where the consumer waits for a real signal. Either is defensible; drifting into the first without noticing you chose it is not.

Make it safe to run twice

Schedulers retry. Humans re-run things by hand. A job that corrupts data when it runs twice is a job that will corrupt data.

The property you want is idempotency — running it again produces the same end state, not a doubled one. For a database write, that's usually one clause:

-- Not this: a re-run silently doubles rows
INSERT INTO some_table (...) VALUES (...);

-- This: a re-run changes nothing
INSERT INTO some_table (...) VALUES (...)
ON CONFLICT DO NOTHING;

Idempotency is what makes "just re-run it" a safe first response to a failure — and that matters more than it sounds, because it's the difference between a 3am incident you can act on and one you have to reason about first.

Know what the clause means, though: DO NOTHING skips rows that already exist, so a re-run fills gaps but never overwrites. DO UPDATE would be the opposite choice. Which one you want depends on whether a later fetch of the same day should be treated as a correction or as a duplicate.

Secrets and API keys

Your code needs vendor API keys and the database password. None of them go in the repository — not in a config file, not in a notebook, not in a docstring, not "temporarily."

Why this is absolute: git history is permanent. A key committed and deleted in the next commit is still in the history, still on every clone, and still on the remote. Removing it means rewriting history for everyone. And a leaked vendor key is a bill someone else runs up on the fund's subscription.

The mechanism is a .env file that is never committed. What is committed is .env.template — the shape, with placeholders and nothing real:

# .env.template  (committed — placeholders only)

VENDOR_API_KEY=vendor-api-key

DB_HOST=db-host
DB_PORT=db-port
DB_USER=db-user
DB_PASSWORD=db-password
DB_NAME=db-name

You copy it to .env, fill in real values, and .env is gitignored. Code reads them from the environment:

from dotenv import load_dotenv
import os

load_dotenv()
key = os.environ["VENDOR_API_KEY"]      # KeyError immediately if unset

The template is doing real work here, and it's a pattern worth reusing. It documents which variables the app needs without containing a single secret — so a new engineer can set up in minutes, and the list of required config never drifts out of date, because the app fails loudly the moment one is missing.

Use os.environ["KEY"], not os.environ.get("KEY"). The bracket form fails immediately and loudly when the variable is missing. .get() returns None, which surfaces later as a confusing auth error — or worse, as a request that quietly returns no rows.

Common mistakes