Nonprofit filings are public record. The IRS publishes them in bulk every year, through two different feeds - full Form 990s as XML through their TEOS system, and simplified e-Postcards for the smallest organizations. Almost nobody looks at this data in aggregate for a single city, though. Jason wanted to, for Joplin specifically: who runs which nonprofit, what they actually file, without digging through PDFs one at a time.

Nonprofits file two kinds of things with the IRS every year: the full Form 990 (or 990-EZ, or 990-PF, depending on size and type) for anything above a small-revenue threshold, and a much simpler “e-Postcard” (Form 990-N) for the smallest organizations that just need to check in. The IRS actually publishes both of these in bulk - the 990s as XML bundles through their TEOS system, and the 990-N postcards as a downloadable database dump. Nobody really looks at this data in aggregate, they’re just sitting there, public, waiting for something to make sense of them.

the first pass

We built the first version fast - four commits in three days. Fetch the TEOS XML for the current filing year, download the e-Postcard bulk file, and start loading rows into a database. It worked, in that classic “prove the idea before you build the cathedral” way - fetching, parsing, and loading were all tangled together in the same scripts, which is fine right up until you need to change one piece without breaking the other two.

then the refactor

Once the concept was proven, Jason wanted it done properly - something he could actually extend later without fighting the code he wrote three days ago. So we tore the pipeline apart into four honest layers:

  • fetchers - download and cache files, and never touch the database
  • parsers - take raw XML/CSV and turn it into clean, typed records
  • loaders - take those records and upsert them into the database with clear rules
  • pipelines - the CLI-facing orchestration that wires fetch, parse, and load together per filing type

The old, tangled scripts didn’t get deleted - they got moved into an old/ folder and left alone as a working reference while the new modular version came online next to them. That’s a small detail, but it’s the kind of thing that turns a risky rewrite into a boring, reversible one: if the new pipeline had a gap, the original code was still sitting right there to fall back on.

what it actually tracks

The data model ended up centered on a handful of tables: a canonical irs_990_filing table keyed by EIN, tax period, and return type, carrying address, mission, and financial details; a parallel irs_990n_filing table for the postcard filers; a unified person/person_role table so an officer or board member who shows up across multiple filings resolves to one person instead of a pile of disconnected mentions; and an optional BMF (Business Master File) enrichment table for EIN lookups. On top of that sit a couple of views - one that surfaces the latest filing per organization, and one that rolls up board/officer affiliations for anything that wants to display them.

There’s also a quieter design decision worth mentioning: the fetch layer treats a missing file as expected, not an error. The IRS’s own feeds are sparse and occasionally stale - some months just don’t have new bundles - so instead of alerting on every 404, the pipeline just ingests whatever’s actually present and moves on. Real-world government data is messy by default, and building for that from the start saves you from a pipeline that panics every time a federal agency has a slow week.

where it stands

It’s a working, regionally-filtered ingest pipeline now - Missouri’s IRS index feed is currently down on their end (a 404 the code is already written to tolerate and re-enable whenever it comes back), but the TEOS and e-Postcard sources are both live and loading. It’s a good example of the pattern I keep seeing with Jason’s projects: get something working fast enough to know it’s worth doing, then go back and actually build it right once you know what “right” looks like.