Building DeepSeekCodeGen: why "generates code" is the wrong bar

Most AI coding demos end at the same place: a snippet appears, someone nods, the tab closes. Nobody checks if it runs. That’s the gap I wanted DeepSeekCodeGen to sit in — not “generates code” but given a real repository and a real issue, produce a patch that passes its own tests before a human ever sees it.

That sentence is the whole design spec. Everything else in the project is in service of making it true, or being honest about where it isn’t yet.

The problem with “just call an LLM”

Ask a model to fix a bug and it will confidently rewrite a function it’s never seen the caller of. It doesn’t know your retry policy lives in config/settings.py, that three other files import the function you’re about to change, or that your test suite has an idempotency check that will catch the naive version of “add retries.” A model with no repository context produces a snippet. A model with the right repository context, forced through a test loop before you see the output, produces something closer to a pull request.

So the interesting engineering problems aren’t in the generation step at all. They’re upstream and downstream of it:

• Upstream: how do you find the handful of files, out of a repo that might have thousands, that actually matter for this issue — without blowing the context window or missing the config file that isn’t textually related to the bug report?
• Downstream: how do you know the patch is actually correct, not just plausible-looking? And when it isn’t, how do you recover without a human re-prompting from scratch?

The pipeline

GitHub Repository
       │
       ▼
Repository Indexer → AST Analysis → Dependency Graph
       │
       ▼
Semantic + Lexical Search → Context Builder (token-budgeted)
       │
       ▼
DeepSeek Coder → Generated Patch (unified diff)
       │
       ▼
Sandbox (isolated container) → Tests / Lint / Build
       │
       ├── pass → Review + Confidence Score
       └── fail → feed failure back to the model → repair loop (bounded)

A few decisions in here are worth explaining, because they weren’t the default choice:

Unified diffs, not full-file rewrites. Asking a model to regenerate an entire file is asking it to also silently reproduce (or subtly break) everything it didn’t need to touch. A diff forces the model to reason about the delta, and it applies with tooling everyone already trusts (git apply) instead of a bespoke file-replace step.

Context is graph-aware, not just similarity-aware. If you only do semantic or lexical search, you find files that talk about the issue. You miss the config file that defines the retry setting but never mentions “retry” in prose. So the context builder pulls in anything one hop away in the import graph from a real hit — the file that imports the thing you’re about to change, or the thing it imports. That’s the difference between “the model can see the function” and “the model can see the function and the two places that would break if it changed the signature.”

The repair loop is bounded and reverts on failure. A patch that fails sandbox checks gets git apply -R’d immediately, not left half-applied while the model tries again. Each failed attempt feeds the actual test/lint output back to the model as the next prompt — “here’s what broke” is a much better signal than “try again.” It’s capped at a fixed number of iterations, because an unbounded loop is just a slower way to burn API credits without converging.

The sandbox has no network access. If a generated patch tries to reach out to the internet — install something unexpected, phone home, whatever — it fails loudly in a way that won’t quietly work in one environment and silently break in another.

What’s real and what’s scaffolded

This is the part most portfolio write-ups skip, and it’s the part that actually matters if anyone reads the code:

Working, end to end: cloning and caching a repo, tree-sitter parsing for Python/JS/TS/Go, the file-level dependency graph, lexical search, the token-budgeted context assembler, the DeepSeek patch/repair prompts and diff extraction, git apply with path-traversal rejection, and — the part that makes the whole claim checkable — a Docker sandbox that actually runs pytest/ruff/mypy against the patched checkout, with zero network access.

Deliberately stubbed, and marked as such in the code, not hidden: semantic search has a real pgvector schema and query, but embed_text() raises NotImplementedError on purpose rather than returning a fake vector that would look like it works while contributing nothing. JS and Go sandbox command sets are empty entries in a dict, waiting to be filled in the same shape as the Python one. The evaluation benchmark harness runs and scores correctly — against a sample set of three toy cases, which is a harness demo, not a benchmark result.

A number that looks precise but isn’t: the confidence score is 1.0 - 0.2 × (iterations - 1) when checks pass, 0.0 when they don’t. It correlates with “how much repair this needed,” which is a real and useful signal. It is not a calibrated probability of correctness, and the architecture doc says so in those words, because the alternative — letting a clean-looking number imply more rigor than it has — is the exact failure mode this whole project is trying to avoid at the code layer. No reason to reintroduce it at the presentation layer.

Why this is the more interesting interview story

“I built a thing that calls an LLM” is not a story. “I built a pipeline that retrieves the right context under a hard token budget, generates a diff instead of a file rewrite, proves the diff works in an isolated sandbox before anyone sees it, and recovers from failure by feeding the model its own test output” is an actual systems design conversation — and one where every claim points at a piece of code you can open, not a result you have to take on faith.

The honest next steps are exactly the two gaps above: wire a real embedding model behind semantic search, and replace the three toy benchmark cases with real closed-issue/merged-PR pairs (the SWE-bench approach) so the pass-rate number means something. Everything else is “designed for, not yet filled in” — a normal state for a project at this stage, and a much stronger position to defend than pretending the gaps aren’t there.

---

Code: DeepSeekCodeGen/ — see README.md for the working-vs-scaffolded breakdown and docs/architecture.md for the full list of known gaps. A VS Code extension (extensions/vscode/) wraps the same API for in-editor use.