The Problem I Was Trying to Solve§
I own a legacy Python service that has grown to 200,000 lines of code. It has no docstrings, no comments, and a README that says "This service handles the main business logic" — which was true three years ago. When I onboarded two new engineers, they spent their first two weeks reading code, guessing at dependencies, and asking me basic questions like "What does process_webhook do?" I needed to fix the documentation gap, but not with a wiki nobody would maintain.
Manual documentation was a dead end. Any doc I wrote would be obsolete by the next commit. I wanted a pipeline that could read the code itself, extract its structure, and use an LLM to generate human-readable prose. By parsing the code into an Abstract Syntax Tree (AST), I could feed the LLM a compact, structured representation of every class, function, and parameter. This article is a case study of how I built that pipeline, what worked, and what fell over.
Tools and Setup (auto-link injection fires here)§
I started with Python's built-in ast module because the codebase was pure Python. But I quickly realized I also needed to capture comments and whitespace, and Python's ast drops those. I moved to tree-sitter along with tree-sitter-python. Tree-sitter gives me a syntax tree that preserves comments and source offsets, which lets me associate docstrings with the correct node. For LLM calls, I used OpenAI's GPT-4 to prototype, then switched to Claude — specifically the claude-sonnet-4 model — because it followed my prompt's formatting rules more reliably. I also ran a batch with DeepSeek to test if a cheaper model could produce acceptable outputs. It couldn't, and I'll explain why.
My setup was a simple virtualenv with a stack: tree-sitter, tree-sitter-python, tiktoken, openai, claude, pydantic, and typer for the CLI. I wrote most of the extraction script in Cursor, which autocompleted the traversal boilerplate. For prompt engineering, I used Perplexity to find research on avoiding hallucinated code references in LLM-generated docs. That research led me to include strict "do not invent" rules. The entire pipeline ran on a 32GB MacBook Pro, but you could run it on a Raspberry Pi with any LLM API.
Step-by-Step: What I Actually Did§
The first step was building a file walker. I used os.walk to collect every .py file, excluding venv, tests, __pycache__, and build directories. For each file, I read the source and parsed it into a tree-sitter CST. Why CST instead of AST? Because tree-sitter gives me the source byte ranges for every identifier, which makes it trivial to slice out the original signature string. That string becomes a key part of the prompt.
Next, I wrote an extractor that walked the CST and produced a JSON object for each module. The schema included the module path, imports, classes, functions, global variables, and any existing docstrings. For each function I captured its full signature, parameter names with defaults, return annotations, decorators, and a complexity score. I used a simple cyclomatic complexity counter on the CST by counting if, for, while, and case nodes. This score gave the LLM a hint about which functions were risky and needed more cautious documentation.
Then came token budgeting. I set a maximum of 4,000 tokens per LLM call. A module's JSON might be 2,000 to 6,000 tokens depending on size. I wrote a recursive splitter that split large modules at class boundaries first. Only if a class was still too big did it split at function boundaries. The splitter output multiple JSON chunks, each with a header indicating the module path and the chunk index. This semantic chunking was crucial — it kept related methods together and gave the LLM coherent context.
The generation loop was straightforward. For each chunk, I filled in a prompt template with the JSON and sent it to the LLM with a temperature of 0.2. I collected the returned Markdown and appended it to a docs file under docs/modules/<module_path>.md. After every module was generated, I ran a second pass that sent the list of module summaries to the LLM and asked it to generate an index README with a module dependency map. That output became the foundation of our team's wiki.
Code Samples / Prompts Used§
Here's the core extraction function I used to walk tree-sitter's CST and pull out function signatures. It's simplified but captures the pattern:
import tree_sitter
from tree_sitter import Language, Parser
PY_LANGUAGE = Language('build/my-languages.so', 'python')
parser = Parser()
parser.set_language(PY_LANGUAGE)
def extract_functions(source_code):
tree = parser.parse(source_code.encode())
functions = []
def traverse(node):
if node.type == 'function_definition':
# Get the name and parameters from the CST
name_node = node.child_by_field_name('name')
params_node = node.child_by_field_name('parameters')
# Slice the original source to get a readable signature
signature = source_code[name_node.start_byte:params_node.end_byte]
functions.append({
'signature': signature,
'byte_range': (node.start_byte, node.end_byte)
})
for child in node.children:
traverse(child)
traverse(tree.root_node)
return functionsThis gave me a clean list of signatures. I then attached existing docstrings by looking for a string node in the function's body after the parameter list. That became the JSON that went into the prompt.
Here's the exact prompt template I used with Claude. It contains explicit rules to prevent the LLM from inventing code:
You are a senior technical writer documenting a Python codebase. You will receive a JSON object that represents a single module. Your task is to write Markdown documentation for this module. The target audience is an engineering team with intermediate Python knowledge.
Rules:
- Do NOT modify or invent code. Only describe what is in the JSON.
- For every function, state its purpose, parameters, return value, and any side effects.
- For classes, document class-level attributes and each method.
- If a field is empty, say "Not documented".
- Highlight any complexity score above 10 as a potential risk.
- Use headers: "## Function Index", then "### functionName" for each function.
Input JSON:
{module_json}In the script, {module_json} was replaced with the serialized module JSON. I used zero-shot — no few-shot examples — because I didn't want to spend tokens on examples. It worked unexpectedly well.
What Worked Well§
The pipeline produced surprisingly accurate docs for individual modules. Because the JSON contained descriptive function names and parameter names, the LLM could infer intent. For example, a function named compute_risk_score with parameters account_id, amount, and customer_country was documented as "Computes a risk score for a transaction using account history, amount, and country." That's a legitimate technical summary, not a hallucination. I measured factual accuracy by manually reviewing four modules; roughly 80% of the generated sentences matched what I knew from reading the code.
The semantic chunking by class boundaries was a clear win. When I first tried fixed-size token splitting, I got bizarre outputs where a function would be split across two docs and the LLM would misname it. Once I switched to splitting on CST boundaries, those errors disappeared. The LLM also handled complexity hints well — functions with a high complexity score got extra caveats like "This function has multiple branches; test carefully." That surprised me and made the docs actually useful.
Another pleasant surprise: the generated docs surfaced a real bug. For a function process_refund, the LLM wrote "this function returns True if the refund was approved, False otherwise." My manual reading confirmed that, but I also noticed the return value was None on one error path. The LLM had marked the function as returning Optional[bool]. That bug had been there for months.
What Failed and Why§
The first major failure was cross-module references. When a module imported a database connection object or a utility function from another module, the LLM would invent a description for it because only the module JSON was in context. It wrote things like "this function uses the central database connection" — which was true, but it also said "the database connection is initialized in this module" — false. I fixed the symptom by post-processing the generated Markdown and replacing known symbol names with relative links, but the prose still contained speculative statements. The real fix would require injecting a global symbol table into the prompt, but that would blow up the token budget.
The second failure was deeply nested logic. Functions with loops inside conditional branches elicited prose that was correct but unreadable. The LLM wrote long paragraphs instead of concise notes. I tried to control this by adding a rule "use bullet points for each separate operation," but that only worked for simple functions. For complex ones, it still wrote run-on sentences. I couldn't solve this with prompt tweaks; I suspect it needs a fine-tuned model or an auxiliary summarization pass.
DeepSeek was a flop. I ran the exact same prompt through DeepSeek's deepseek-chat model. The output was significantly shorter and often omitted functions. It also occasionally added trailing text like "This is an AI-generated document" and broke the Markdown table of contents. I know DeepSeek is cost-effective, but for this task, the quality gap was too large. I kept using Claude for the final pass, and DeepSeek only for initial drafts that I never published.
Finally, I failed to integrate the pipeline into CI. I ran it manually, and by the next sprint, my generated docs were two commits behind. The concept of "living documentation" only works if it regenerates automatically. That's a process failure, not an LLM failure, but it matters because users could no longer trust the docs after a week.
Results and Takeaways§
In one weekend, I processed 134 Python files and generated 1,200 pages of Markdown. The docs went into a docs/modules directory and were served by our internal MkDocs site. I manually reviewed four modules. About 80% of the generated prose required no edits. The AI's biggest weakness was cross-module reasoning, and its biggest strength was explaining isolated functions. Two new engineers used the docs as their primary reference for their first week. One said he stopped opening source files entirely. That feedback convinced me the approach was worth the effort.
The core lesson: an AST is the perfect intermediary between code and an LLM. It strips away syntax noise, keeps structure, and lets the LLM focus on meaning. But the LLM should never see the entire codebase. Chunk it on semantic boundaries, and give it strict rules about what it can and cannot add. When I gave the model the full source file, it started reproducing code instead of writing docs. The JSON representation sidesteps that trap.
The takeaway I want people to remember: auto-generated docs are only useful if you regenerate them before they become stale. Use a CI trigger. Also, always use a single-source-of-truth format like JSON extracted from the AST; that gives you a reversible, testable documentation pipeline. It's not a silver bullet, but it's a massive improvement over no docs.
Try It Yourself§
If you want to try this, start small. Pick a directory with three or four Python modules that you understand well. Write a script to extract the AST JSON using the snippet above, then paste the JSON into Claude or GPT-4 with the prompt template. Evaluate whether the output matches what you know. Iterate on the prompt until you get something usable. I promise it will take less than an hour to get your first useful doc.
Once the pattern works, scale up. Add more languages by swapping in tree-sitter grammars. Add a global symbol table to reduce hallucinated references. Most importantly, add a GitHub Action that regenerates the docs on every push and commits the result. Without that automation, you'll end up with a beautiful but quickly rotting knowledge base. I made that exact mistake so you don't have to.
