AI-Powered Large-Scale Extraction Pipeline for Legal Professionals
An async pipeline that crawled 1.3M web pages across 1,500 unrelated law firms — no shared platform, no shared structure — to build a structured database of attorney profiles and contact information that matched a specific litigation field.
The brief
A legal research startup reached out to me with a CSV of roughly 1,500 law firms — two columns, name and website. They needed a high-precision, targeted contact list of litigation associates — not partners, not corporate or tax attorneys. Each firm ran its own site, with its own navigation and its own bot defenses, and there was no shared data source between any of them. The scope was specific about what counted:
Identify associates with an "Associate" title. Verify litigation focus by confirming at least one of the following on the profile page: "litigation," "dispute(s)," "trial(s)," or "resolution." Exclude non-litigation associates — corporate, tax, real estate, etc.
The constraint that shaped everything downstream wasn't the scraping. It was the ratio of budget to scale:
- Fixed budget
- Short timeline
- 1,500 different websites
That meant the pipeline couldn't treat every page the same way. Every firm's sitemap holds thousands of irrelevant links — news, media, forms, and other boilerplate. A much smaller handful are genuinely a litigation associate's bio. The system has to tell those apart on its own, and only spend real money — browser rendering, LLM calls — on the pages that actually need it.
The procedure
Every firm moves through six stages. The first three are cheap and run on nearly every page a firm's sitemap contains. The pipeline only pays for the last two on pages that survive everything before them.
What each stage actually removed
Three quarters of what looked right by keyword and title alone wasn't.
Bar width scales with volume — each stage keeps about a fifth of the one before it.
| Stage | Count | Mechanism | Cost |
|---|---|---|---|
| Discovered | 1.3M URLs | Sitemap / robots.txt walk | Free |
| Fetch-worthy | 500K URLs | URL structure — excluded paths, numeric-ID slugs, length | Free |
| LLM candidates | 120K pages | Keyword match + first-pass LLM on the page title | ~$4 (Groq) |
| Verified leads | 15K leads | Second-pass LLM reads keyword context, judges intent | ~$2 (DeepSeek) |
What "verified" means for that last row:
- Profiles are filtered to only those with the "Associate" title
- Bios are scanned for litigation-related keywords (litigation, dispute(s), trial(s), resolution)
- Emails are validated using the client-provided validation script
- When a page had multiple emails, an LLM-powered parser extracted names and titles to match the correct email to the correct attorney
Here's a real example of that gap, anonymized to remove firm and client identifying details:
| Page title | Keyword match | Model verdict | Outcome |
|---|---|---|---|
| Firm Secures Motion for Summary Judgment in Injury Case | Yes | "Title does not contain a person’s name and appears to be a news article." | Rejected |
| Insurance Solutions — Practice Overview | Yes | "Title does not clearly contain a person’s name and appears to be a listing page." | Rejected |
| Appellate Team Secures Unanimous Reversal in New York | Yes | "The title does not clearly contain a person’s name, it seems to be a general page about the law firm." | Rejected |
| Jamie Witter | Yes | "Single attorney profile." | Kept |
Three of these four pages mention litigation. One of them is a litigation associate. A keyword can't close that gap on its own — something has to actually read the page.
Rulings
On fetching — one adaptive fetcher, not a manual choice. HTTP first, always. AdaptiveFetcher escalates to a real browser only on an actual block signal — 401/403/429, a 5xx, a short body, or a bot-challenge marker. A clean 404 stays a 404; a browser wouldn't change that.
On cost — no page reaches the model without a keyword match first. A page needs "associate" and a litigation term before it's worth an LLM call. That single check is what kept 500K fetched pages from turning into 500K model calls.
On classification — two passes, sized to what they need to see. Pass one screens title and candidate emails on every keyword-filtered page. Pass two re-fetches only the flagged pages and reads the litigation-keyword context, falling back to full text only when it fits the token budget.
On persistence — the ORM stays synchronous, on purpose. Peewee isn't async. Every database call from the pipeline runs through asyncio.to_thread(...) instead of adopting a second, async-native ORM just to match.
On failure — kill the process, don’t lose the work. Firm and page status write to the database via upsert() as the pipeline runs. Anything already done or is_processed gets skipped on the next run — restarting resumes, it doesn't start over.
Challenges
Access Restrictions
Some websites employ anti-bot measures or access restrictions, requiring specialized handling.
Unstructured Directories
Certain firms lack predictable URL patterns or structured directories, slowing down profile discovery.
Multiple Emails Per Page
Matching the correct email to the correct associate required an LLM-based parsing solution.
Inconsistent Layouts
Law firm websites vary widely in structure, making standardization challenging.
The data model
Three tables. Firm tracks one row per site. Page tracks one row per sitemap URL and everything the cheap first pass learned about it. AttorneyProfile is the high-precision result — a strict subset of Page rows that survived the second pass.
| Table | Field | What it captures |
|---|---|---|
| Firm | firm_host | homepage URL, used as the primary key |
| Firm | status | pending → in_progress → done / failed |
| Page | has_associate_litigation | result of the keyword pre-filter |
| Page | is_attorney_page | first-pass LLM verdict |
| Page | restricted_access | true if every fetch attempt hit 401/403 |
| AttorneyProfile | is_associate / is_litigation_related | second-pass verdicts, stored independently |
| AttorneyProfile | field | Litigation / Corporate / Tax / Real Estate / Employment / Unknown |
Tests run against a temp-file SQLite database, not :memory: — the pipeline dispatches ORM calls through a worker thread, and SQLite's in-memory database is private to the connection that created it.
Stack
Need a large-scale extraction pipeline?
Let's talk about turning unstructured web data into a structured database.