Automated Scraper → PDF Pipeline
An automated pipeline that turns a firehose of RSS feeds into a single, curated PDF newsletter every week — no human in the loop.
The problem
Following dozens of sources by hand is unsustainable, and most items in any feed are noise. The goal: a system that reads everything, keeps only what is genuinely relevant, and hands back a polished digest on a schedule.
How it works
Scrape and normalize
A scheduled job pulls every configured RSS feed, de-duplicates, and normalizes entries into a common shape so downstream stages don't care where an item came from.
Cascading LLM filtering
Each item first hits a cheap model for a fast relevance judgement. Only borderline items — where the cheap model is unsure — are escalated to a stronger model. Most items are decided cheaply, which is where the bulk of the cost savings come from.
Relevance scoring & threshold tuning
Items receive a relevance score, and a single tunable threshold controls how aggressive the filter is. Raising the threshold trades recall for a quieter, higher- signal newsletter — noise reduction as a dial, not a rewrite.
PDF generation
Surviving items are grouped, formatted, and rendered into a clean PDF ready to send.
Results
The cascading filter means the expensive model only ever sees the small fraction of items the cheap model couldn't confidently resolve — most of the feed is triaged for a fraction of the cost of scoring everything with a frontier model.