BM25 before grep

Every codebase develops a private dialect. An issue says respect gitignore; the implementation calls its switch no_ignore. A person can browse around until the local name becomes obvious, while a coding agent pays for every wrong guess with another tool call and another slice of its context window.

Once the identifier is known, grep is exactly the right instrument. rg no_ignore will find definitions and uses with absurd speed. The awkward part comes a minute earlier, when the agent understands the question but has not yet learned how this repository talks about it.

I built code-search for that minute. It uses BM25 as a local orientation pass, returns a few promising files and lines, and then gets out of the way so exact search can do what it does best.

The same question, asked two ways

I tried a deliberately awkward query against the tool’s own repository: respect gitignore hidden files. Plain mode searches for the phrase and quite correctly finds nothing.

code-search search --mode plain --output json \
  "respect gitignore hidden files" .

{
  "total_matches": 0,
  "results": []
}

The ranked mode receives exactly the same words:

code-search search --mode semantic --output json --max-results 5 \
  "respect gitignore hidden files" .

src/cli.rs:77   /// Include hidden files
src/cli.rs:81   /// Don't respect .gitignore
src/cli.rs:104  /// Include hidden files
src/cli.rs:108  /// Don't respect .gitignore
src/cli.rs:79   pub hidden: bool,

Four of the five results are duplicated comments in two argument structures. BM25 has performed no conceptual magic; it noticed that the query terms were unusually concentrated in cli.rs, even though they never appeared as one exact phrase. Opening that file reveals the name the agent was missing, no_ignore, and the next search can return to plain mode.

The ranked pass only has to carry the agent from the vocabulary of the request to the vocabulary of the implementation; understanding the codebase can wait for the tools and reasoning that follow.

semantic is an optimistic name

Version 0.1.0 calls the mode semantic, which promises more than the implementation delivers. There is no model, embedding service or vector database. The implementation tokenises the query and files, stores term frequencies, calculates inverse document frequency, scores documents with BM25, and chooses representative lines from the highest-ranked files.

Its index lives at .code-search/bm25_index.json, containing term frequencies, IDFs, average document length, document count and modification times. Changed files can be updated incrementally, and the reason a file ranked well remains inspectable in an ordinary local file. Nothing from the repository leaves the machine.

Lexical ranking carries familiar limitations. Two different vocabularies for the same design may never meet. Generated code and repeated comments can dominate a result. An index which has gone stale stays misleading until code-search init --force rebuilds it. Plain and regular-expression searches remain the dependable fallback once the agent has a useful name.

Ranked mode is for orientation, not a replacement for ripgrep once the identifier is known. Plain mode uses memchr over memory-mapped files; when I know the string, I use the simpler instrument.

The usual reader of code-search output is another agent, so --max-results bounds what enters the next prompt and JSON avoids wasting context on terminal decoration. The repository’s SKILL.md teaches the sequence rather than prescribing one search mode for everything: ranking to find the neighbourhood, plain search after discovering an identifier, regular expressions for structured usage sites.

On that staged query, one ranking call found the relevant file. The next command, a plain search for pub hidden, landed on cli.rs:79. By then grep had what it needed: the repository’s own words.