I spent the last quarter building a lead generation pipeline that scrapes dozens of niche directories, enriches every record with LLM-powered inference, and pushes the result straight into our CRM. This is the story of what worked, what broke, and how you can replicate the whole thing without a data science team.
The Problem I Was Trying to Solve§
Our sales team was drowning in manual research. We sell middleware to B2B SaaS companies, and the ideal outbound lead is a startup with 20–200 employees, recent engineering hires, and a tech stack that includes Kafka or RabbitMQ. Finding those companies requires visiting Crunchbase, AngelList, G2Crowd, and half a dozen niche directories. Then we had to cross-reference job postings and GitHub activity to guess whether they were actively building distributed systems.
The manual process took roughly 20 minutes per qualified lead, and we only needed 50 new leads a week. That's 16 hours of repetitive clicking, copying, and pasting. I knew we could automate the scraping and enrichment, but every off-the-shelf tool I tried either had brittle selectors or no way to reason about unstructured text. A competitor's scraped list would give me a company name and a website URL, but not the nuanced "they're using RabbitMQ and hiring Python devs" signal that actually matters.
I wanted a pipeline that could: ingest raw HTML from multiple sources, extract structured fields (name, domain, size, location), infer intent signals from job descriptions and blog posts, and merge everything into a clean record. That sounds like a job for an LLM. So I built it.
Tools and Setup§
I used a stack that prioritizes developer ergonomics and low cost. Everything runs as Python scripts orchestrated by Cursor — the AI-native IDE that made me 3x faster at writing glue code. The scraper itself is built with Scrapy and BeautifulSoup, deployed on a small EC2 instance. For proxies, I used rotating residential IPs from BrightData, though I'll be honest: this was the most expensive part.
The LLM layer is split across two providers. For high-volume extraction, I use DeepSeek's API — it's incredibly cheap (roughly $0.14 per million input tokens at the time of writing) and handles JSON mode reliably. For complex reasoning tasks like "Does this company seem to be going through a growth spurt?" I use Claude via the Anthropic API, because its instruction-following is more nuanced.
For discovery, I lean on Perplexity to find new source directories and to validate whether a particular niche forum or job board allows scraping. I also use Cursor's built-in chat to refactor my regex patterns and to debug Scrapy selector errors — that alone saved me hours.
Step-by-Step: What I Actually Did§
1. Source discovery and feasibility check. I started with three sources: a popular SaaS directory with clean HTML, a job board that lists engineering roles, and a public GitHub API endpoint for repos tagged with "Kafka" or "RabbitMQ." For each source, I checked the robots.txt and terms of service. The job board explicitly barred scraping, so I switched to their official API. The SaaS directory allowed scraping of public company profile data.
2. Build the scraper. I wrote a Scrapy spider that crawls listing pages, extracts all company profile URLs, then fetches each profile page. I used BeautifulSoup to pull raw HTML and strip it down to a clean text blob. The key was preserving the page structure — I didn't want a wall of text because LLMs understand sections better when headers are kept.
3. Batch extraction with DeepSeek. Each profile blob was sent to DeepSeek with a strict JSON schema. The model returned fields like company_name, website, employee_count, location, description_summary, and tech_signals (an array of strings). I used a batching strategy: instead of sending one page per API call, I concatenated three profiles into a single prompt, separating them with ---RECORD--- delimiters. This cut costs by 2.5x and didn't hurt accuracy.
4. Enrichment with Claude. Once I had the base record, I ran a second pass with Claude to answer more subjective questions: "Based on the job postings and recent press, does this company appear to be actively expanding its engineering team?" and "What messaging angle would resonate with them?" Claude's answers were long-form, so I asked for a 1–2 sentence response. This became the personalized icebreaker for our sales emails.
5. Deduplication and CRM push. I maintained a JSON file of already-seen domains. For new leads, I used HubSpot's API to create a contact and a linked company. A simple Python loop ran the entire pipeline nightly, and a Slack webhook notified me of new leads.
Code Samples / Prompts Used§
Here's the core extraction function I used with DeepSeek:
import requests
import json
import re
def extract_lead_with_deepseek(profile_text: str, api_key: str) -> dict:
"""Extract structured lead data from raw profile HTML using DeepSeek."""
url = "https://api.deepseek.com/chat/completions"
prompt = f"""Extract the following fields from the text between <PROFILE> tags. Output JSON only.
Fields:
- company_name: string
- website: string
- employee_count: integer (best guess)
- location: string
- description_summary: string (max 50 words)
- tech_signals: array of strings (e.g. "Kafka", "Python", "AWS")
If a field is not found, use null. Do not invent data.
<PROFILE>
{profile_text}
</PROFILE>
"""
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
payload = {
"model": "deepseek-chat",
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.1,
"response_format": {"type": "json_object"}
}
resp = requests.post(url, json=payload, headers=headers, timeout=30)
content = resp.json()["choices"][0]["message"]["content"]
return json.loads(content)
# Example usage
# with open("profile.html", "r") as f:
# html = f.read()
# print(extract_lead_with_deepseek(html, "your-api-key"))The prompt for Claude's enrichment was simple but effective:
Based on the following company profile, write a single sentence that explains why they might need a better event-streaming solution. Focus on their current pain points or recent growth. Do not mention the source.
Profile: {compiled_profile}
Answer:I wrapped this in a retry loop that catches context_length_exceeded errors by trimming the profile to the first 2,000 characters. That handled over 90% of failures.
What Worked Well§
The combination of a cheap extraction LLM (DeepSeek) and a smart reasoning LLM (Claude) was a home run. DeepSeek's JSON mode gave me perfectly structured data 92% of the time. When it failed, it was usually because the profile text was too short or ambiguous — not because the model was confused. I was able to process 1,000 profiles for about $1.20 using DeepSeek. Claude enrichment cost about $0.30 per 100 leads, still negligible compared to the value of a single qualified meeting.
Batching three records per prompt significantly reduced API costs. I initially tried one profile per call, and the token overhead of the system prompt was killing me. Batching lowered overhead by 60%.
The deduplication layer was straightforward. Storing domain hashes in a local JSON file worked fine for our scale. I never hit a case of duplicate leads in the CRM, because I checked website domain before creating a contact.
What Failed and Why§
Anti-bot protections were the #1 failure point. The SaaS directory started serving CAPTCHAs after my scraper hit 200 requests in a minute. I had to implement rate limiting (2 requests per second) and rotate user agents. That was easy. The hard part was when the job board's API began returning 403 errors for unrecognized TLDs. I had to add a proxy rotation service, which added latency and cost.
LLM hallucination on missing data. DeepSeek would occasionally invent an employee count or a tech signal even when the profile had no mention. For example, it inferred "AWS" from a generic phrase like "cloud-based," which is technically wrong. I fixed this by prompting: "If a field is not explicitly present, leave it null." That helped, but not completely. I had to write a validator that rejected records where company_name was null or website didn't match a regex. That eliminated 95% of bad records.
Context window issues with Claude. A few profiles had 15,000 characters of raw text, and I was feeding them verbatim to Claude. That blew through the context window and caused expensive retries. I fixed it by truncating to the first 2,000 characters, but I likely lost important signals in the long tail. I'm now experimenting with extracting bullet points from DeepSeek before passing to Claude, which reduces the token count by 80%.
Scrapy selectors break when sites update their HTML. This is the oldest problem in web scraping, and it plagued me twice during the project. I had to write a monitoring script that checks for a known CSS selector on a random page every hour and alerts me if it's missing. That gave me enough time to fix the spider before the nightly run failed silently.
Results and Takeaways§
After three weeks of iterations, I had a stable pipeline that collects 300 new leads per night, enriches them with intent signals, and pushes the whole list to our CRM by 8 AM. Our SDRs now spend their mornings contacting qualified leads instead of searching for them. The conversion rate on these LLM-enriched leads is 2.4x higher than the previous non-enriched list, because the personalization line actually references a specific tech choice or hiring trend.
Here are the key takeaways:
- Use two LLMs, not one. Pair a cheap, high-volume extractor (DeepSeek) with a stronger reasoning model (Claude) to keep costs down and quality high.
- Always validate LLM output. Set up regex and null checks to catch hallucinations before they enter your CRM.
- Batching is your friend. Send 3–5 records per prompt to reduce token overhead, and always include clear delimiters.
- Plan for site changes. Scrapers are not fire-and-forget; add health checks and alerts to catch breakage early.
Try It Yourself§
You don't need a massive budget or a dev team to replicate this. Start with a single source you know well, like a directory or an open API. Write a simple scraper that saves raw HTML to disk. Then use the DeepSeek API with the code above to extract structured fields. Add one enrichment step with Claude. Feed the output to a spreadsheet or CRM.
If you want to go further, here are three directions to explore:
- Use embeddings to match leads to your ideal customer profile. Generate a vector for each lead's description and compare it to your ICP description using cosine similarity.
- Add a review summarizer. Pull G2 or Trustpilot reviews and have an LLM generate a short "what customers love/hate" summary.
- Build a feedback loop. Have your SDRs mark replies as positive, negative, or neutral, then feed that back into the LLM prompt so it learns to pre-filter bad leads.
I've open-sourced a minimal version of the scraper on GitHub — you can find it by searching "agentic-lead-scraper" on my profile. Drop a star if it saves you time. And if you've built something similar, I'd genuinely love to hear what worked for you.
