BM25 before grep

A coding agent enters an unfamiliar codebase with words from the issue: where does this thing handle hidden files and gitignore? The repository may describe the same behaviour with different local terms. Each failed grep costs a tool call and some context before the real work begins. In the agent tools I build, those opening minutes are expensive.

The day code-search v0.1.0 shipped, I first ran a query against its own repository that I knew would fail: respect gitignore hidden files.

code-search search --mode plain --output json "respect gitignore hidden files" .

{
  "total_matches": 0,
  "results": []
}

Zero matches. No file contains that phrase, though the code that answers it is in the repo, under words the agent hasn’t learned yet.

I built code-search to shorten that opening. Its ranked mode uses BM25 term statistics without a model or server to surface the repository’s own nouns. Exact searches can then use those terms. Here is the same query in ranked mode:

code-search search --mode semantic --output json --max-results 5 "respect gitignore hidden files" .

{
  "total_matches": 5,
  "results": [
    {
      "path": "./src/cli.rs",
      "line_number": 77,
      "column": 1,
      "line_content": "    /// Include hidden files",
      "score": 16.053624543873784
    },
    {
      "path": "./src/cli.rs",
      "line_number": 81,
      "column": 1,
      "line_content": "    /// Don't respect .gitignore",
      "score": 16.053624543873784
    },
    {
      "path": "./src/cli.rs",
      "line_number": 104,
      "column": 1,
      "line_content": "    /// Include hidden files",
      "score": 16.053624543873784
    },
    {
      "path": "./src/cli.rs",
      "line_number": 108,
      "column": 1,
      "line_content": "    /// Don't respect .gitignore",
      "score": 16.053624543873784
    },
    {
      "path": "./src/cli.rs",
      "line_number": 79,
      "column": 1,
      "line_content": "    pub hidden: bool,",
      "score": 8.026812271936892
    }
  ]
}

The query returns five hits in cli.rs. Four form pairs: the documentation comments at lines 77 and 81 reappear almost verbatim at 104 and 108. cli.rs contains two argument structs, one for search and one for init, and each defines hidden and no_ignore with the same comments. The ranked pass surfaced both implementations because the query terms repeat with the behaviour.

The agent now has the local terms hidden, .gitignore, and cli.rs. The field name no_ignore appears only after opening the file, but the agent knows which file and lines to inspect. Exact search is appropriate from here.

The flag is called --mode semantic. That oversells it. Under the hood it’s BM25 term counting: tokenize the query and the files, store term frequencies, compute inverse document frequency, score each document, pick the best lines inside the top files. No model is involved at any point. Embeddings MCP servers exist for that; this is the lane where nothing leaves your machine. The index is a JSON file at .code-search/bm25_index.json (term frequencies per document, IDFs, average document length, document count, file modification times), so when a file ranks well, the reason is sitting in a file you can open. It only re-indexes what changed, and code-search init pre-builds it so the first agent call doesn’t pay the full cost.

This does not compete with ripgrep. Andrew Gallant wrote ripgrep and memchr, the SIMD byte-search crate used by plain mode over memory-mapped files. The README benchmarks show code-search taking 219 ms on React’s 6,800-plus files while rg takes 194 ms. On a Maven tree of more than 9,400 files, the comparison is 450 ms against 300 ms. Use ripgrep when you know the exact string. Ranked mode helps before you do.

The usual caller is an agent whose result goes directly into the next prompt, so output must be bounded and parseable. JSON mode and --max-results serve that requirement. The repository’s SKILL.md tells agents to use ranked mode with --files-only to locate the relevant directory, plain search after finding an identifier, and regex for usage sites. Its query advice follows the underlying term counts. It marks “how to connect to database” as less effective and “database connection pool management” as good.

Everything here is lexical. Different words for the same design will not meet in the index, and a stale index will mislead until rebuilt with code-search init --force. Both limitations will eventually matter; this is version 0.1.0. Plain and regex search remain the fallback. The failed query above was staged, because I have not yet recorded this problem damaging a real run.

In the session above, ranked mode took one call. The next query, --mode plain "pub hidden", went directly to cli.rs:79. The repository contains nine Rust files under src/. Try it on a codebase you do not know.