The Problem I Was Trying to Solve§

Six months ago, I was managing a content site with 800+ articles. Every week, I'd export sitemaps, run Screaming Frog crawls, cross-reference GA4 data, and manually audit which pages needed updates. My competitor was publishing 30+ articles per week while I was stuck in spreadsheet hell. The real killer wasn't the writing—it was the operational overhead of content auditing, competitive gap analysis, and internal link optimization. I needed a way to automate these workflows so I could focus on strategic decisions.

Agentic SEO meant building autonomous agents that could crawl my site, analyze competitors, identify linking opportunities, and inject internal links—all without me babysitting every step. The core insight: most SEO tasks follow deterministic rules, and combining LLM reasoning with scripted execution can eliminate 80% of manual work.

Tools and Setup§

I used three main tools for this pipeline:

  • **DeepSeek** (via API) as the primary LLM agent for reasoning and content generation—its 128K context window let me feed entire competitor articles and get back structured analysis.
  • **Claude** (Sonnet 3.5) for validating internal link opportunities because its factual grounding is superior for determining semantic relevance.
  • **Perplexity** as a research assistant to find trending topics and keyword gaps.
  • **Cursor** as the IDE for writing and running the Python scripts that orchestrated the agents.
  • Screaming Frog (headless mode) for initial crawl data, but I replaced it later with a custom crawler using requests and BeautifulSoup to avoid licensing limits.
[Loading prompt card for DeepSeek Chat...]
[Loading prompt card for Claude...]
[Loading prompt card for Perplexity AI...]

I ran everything on a cheap VPS (2 vCPU, 4GB RAM) with a cron job triggering the pipeline weekly. The stack: Python 3.11, SQLite for storing page metadata, and a local LLM (Ollama with Mistral) for fallback when APIs were down.

Step-by-Step: What I Actually Did§

Phase 1: Automated Content Audit

First, I built a script that:

  1. Crawled my sitemap (XML) and extracted all URLs.
  2. For each URL, fetched the page content via requests, stripped HTML tags, and extracted meta description, word count, and headings.
  3. Sent the content to DeepSeek with a prompt to evaluate: "Rate this page from 1-10 on SEO quality (title, meta, headings, content depth, keyword usage). Suggest specific improvements."
  4. Stored the scores and suggestions in SQLite, then generated a CSV report sorted by priority (lowest scores first).

The agent could process ~50 pages per minute (API rate limited). It identified 34 pages with scores below 5, most of which were outdated or thin content.

Phase 2: Competitor Gap Analysis

I used Perplexity to fetch the top 10 ranking competitor articles for my target keywords. Then fed each competitor URL to a headless browser (Playwright) to capture full text. I sent the text to DeepSeek with this instruction: "List all subtopics covered in this article that our site doesn't cover. Format as a JSON array of objects with 'subtopic', 'competitor_url', 'potential_value' (high/medium/low)."

The agent returned structured gaps. I aggregated these into a priority list of new content to create, and also identified 15 existing pages that needed expansion to compete.

Phase 3: Internal Link Injection

This was the trickiest. I wanted an autonomous agent that would:

  1. Read every page on my site (stored as chunks in SQLite).
  2. For each page, find all outgoing internal links.
  3. Identify semantically related content using embeddings (sentence-transformers all-MiniLM-L6-v2).
  4. If a page had fewer than 3 internal outbound links and there was a highly related page not currently linked, inject a contextual link into the page content.

I used Claude to review each proposed link: "Given the source page content and the target page title and summary, would adding a link to [target] be natural and helpful? Only approve if the link adds value and fits the context." Claude approved about 70% of suggestions, rejecting those that were forced or spammy.

Code Samples / Prompts Used§

Here's the core function for content auditing using DeepSeek:

import requests
import json

def audit_page(title, meta, content, headings):
    prompt = f"""Rate the following page from 1-10 on SEO quality.
Consider: title relevance, meta description, heading structure, content depth, keyword usage.
Output ONLY JSON: {{"score": int, "improvements": [str]}}

Title: {title}
Meta: {meta}
Headings: {headings}
Content sample (first 2000 chars): {content[:2000]}
"""
    response = requests.post(
        "https://api.deepseek.com/v1/chat/completions",
        headers={"Authorization": "Bearer YOUR_KEY"},
        json={
            "model": "deepseek-chat",
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0,
            "max_tokens": 200
        }
    )
    data = response.json()
    return json.loads(data["choices"][0]["message"]["content"])

And the prompt for competitor gap extraction I used with DeepSeek:

You are an SEO analyst. Below is a competitor article (full text). Compare it to our site's coverage of the same topic. Our site covers: [list of our related article titles]. Identify any subtopics in the competitor article that we have not covered. Return a JSON array: [{"subtopic":"...", "competitor_url":"...", "potential_value":"high|medium|low"}]. Only include subtopics with clear content gap. Be strict.

For internal link injection, I used Claude to validate:

def validate_link(source_text, target_title, target_summary):
    prompt = f"""I want to add an internal link from the source page to a target page.
Source page text (first 1000 chars): {source_text[:1000]}
Target page title: {target_title}
Target summary: {target_summary}

Would adding a link to the target page be natural and beneficial? Answer only 'yes' or 'no' and explain briefly."""
    # call Claude API...

What Worked Well§

  • Content audit scores were surprisingly accurate. DeepSeek correctly flagged thin pages (e.g., product pages with 200 words) and missing meta descriptions. The CSV report let me hand off actionable tasks to junior writers.
  • Competitor gap analysis saved me 5 hours per week. Instead of manually reading 10 competitor articles, I got a structured list of gaps. I created 8 new articles based on high-value gaps, and 3 of them now rank in top 10 for target keywords.
  • Internal link injection increased average internal links per page from 2.1 to 4.7. Claude's validation prevented linking unrelated pages (e.g., linking "dog food" to "credit cards"). Bounce rate on pages with new links dropped 12%.
  • The cron job ran flawlessly for 2 months without intervention. Only failed when DeepSeek API was down (3 times). I added a fallback to local Mistral which handled basic audits.

What Failed and Why§

  • Automated link injection on live pages was too aggressive. Early versions inserted links into the first paragraph, which looked unnatural. I fixed it by only injecting after the third paragraph and only if Claude approved.
  • Embedding-based similarity for linking often matched irrelevant pages because of keyword overlap (e.g., "apple fruit" vs "Apple company"). I added a filter: the target page must share at least 2 of the source page's top 5 keywords (extracted via TF-IDF).
  • Perplexity competitor research sometimes returned outdated or low-quality articles. I added a date filter (only articles published in last 6 months) and validated URLs against Ahrefs domain rating via manual review.
  • The initial crawl tool (Screaming Frog in headless mode) was too slow and required a GUI. Switching to a custom requests crawler with concurrent workers sped up crawling by 10x.

Results and Takeaways§

After 8 weeks:

  • Content audit time dropped from 4 hours to 15 minutes per site.
  • Competitor gap analysis: identified 42 actionable subtopics; 12 articles were created or updated; 8 reached page 1 for medium-competition keywords.
  • Internal links increased by 125% across the site. Pages with 4+ internal links saw a 9% increase in organic traffic on average.
  • Overall, I reclaimed ~10 hours per week that I reinvested into link building and content strategy.

Key Takeaways:

  • Agentic SEO doesn't replace judgment; it automates grunt work. Use LLMs for evaluation, but always validate with a separate model (e.g., DeepSeek for audit, Claude for linking) to reduce hallucination.
  • Start with a small batch (e.g., 10 pages) before running on the entire site. Tune thresholds for semantic similarity and link density.
  • Embedding-based linking needs keyword filters to avoid false positives. Combine cosine similarity with keyword overlap.
  • The infrastructure cost is minimal ($10/month VPS + API calls ~$20/month) compared to the time savings. Invest in fault tolerance (fallback models, retry logic).

Try It Yourself§

  1. Fork the template repository: https://github.com/llmdb/agentic-seo-template (includes crawl, audit, gap analysis, and link injection scripts).
  2. Set up API keys for DeepSeek and Claude. Run pip install -r requirements.txt.
  3. Edit config.py to point to your sitemap URL and a list of competitor URLs (text file).
  4. Run python pipeline.py — it will first crawl, then audit, then gap analysis, then link injection (dry-run by default).
  5. Inspect the output CSV files in the reports/ folder. To enable actual link injection, set DRY_RUN=False in config.

Start small: pick your top 50 pages and 5 competitor articles. Expect the full pipeline to take ~30 minutes for that scope. Once comfortable, scale to your entire site.

Agentic SEO isn't about replacing yourself—it's about amplifying your capacity. The agents handle the boring, repeatable parts; you handle the creative strategy. That's where the real leverage lives.