CHABOT

LLM-mediated discovery

6 minute read 1,342 words

Developer product research increasingly starts in ChatGPT, Claude, Perplexity, or Gemini rather than moving directly from search results to product websites and documentation.

This adds another route through which public documentation, examples, and third-party accounts affect product evaluation.

What changed

Three changes occurred between late 2022 and mid-2026:

  1. ChatGPT’s launch (November 2022) and the rapid maturation of Claude, Perplexity, and Gemini into competent technical-research assistants.
  2. Stack Overflow’s decline in question volume. Question volume fell roughly 95% from its 2014 to 2017 peak by late 2025, while developers increasingly consulted AI assistants. See Stack Overflow.
  3. The integration of LLMs into IDEs. GitHub Copilot, Cursor, Windsurf, Claude Code, JetBrains AI Assistant, and Replit AI Agent made the AI assistant the developer’s first surface, often before they ever opened a browser.

By mid-2026, one common research sequence for choosing between two API products was:

  1. Ask ChatGPT or Claude: “What’s the difference between X and Y for use case Z?”
  2. Optionally ask the AI to draft a quickstart with one of them.
  3. Open the actual product’s website only after AI has given a recommendation.

The evidence

Several sources triangulate the shift:

  • Stack Overflow’s own declines. The 2023 SO Developer Survey marked the first measurable adoption of AI tools among professional developers; subsequent surveys show further consolidation. By the 2025 survey, the majority of professional developers report using AI coding tools regularly.
  • Faros AI and DX research (2024 to 2026) notes that developer sentiment about AI productivity is consistently positive while telemetry shows mixed results. Productivity and discovery are separate questions: a developer who already has an assistant open can use it for product research even when generated code still needs correction.
  • AngelHack’s Developer Relations in 2026 essay describes the LLM as “the new first-touch user” of developer products. Their argument: “Gaps in your docs don’t just frustrate developers anymore. They create gaps in AI-assisted adoption before a human ever enters the picture.”
  • Anecdotal evidence from DevRel professionals across companies in 2025 to 2026: the most consistent observation is that “developers arrive at our signup having already been told what to do by an AI assistant.” The integration steps they follow are often generated by the AI from your public docs, not read by the human from your docs directly.

AI-mediated research is unlikely to become less common. The practical question is whether a product’s public corpus gives assistants enough accurate material to work from.

How AI assistants produce answers about a product

AI assistants use several inputs:

1. Training data

Training data starts with the public web as it existed when the model was trained. For most major models, this is a snapshot from a particular point in time, refreshed periodically. Its influence is durable but slow to change.

Helpful inputs:

  • Long-form authoritative content: blog posts, tutorials, documentation that’s been on the web for years.
  • Citations and inbound links from authoritative sources.
  • Wikipedia and similar reference sites.
  • Books, conference papers, podcast transcripts.

Failure modes:

  • Stale or contradictory content.
  • Content that fails to use canonical terms.
  • Absence from authoritative reference sources.

2. Real-time retrieval

Real-time retrieval is what AI assistants pull during a query. ChatGPT and Claude both have web-browsing modes; Perplexity is built around live retrieval; Gemini integrates Google’s retrieval infrastructure.

Useful signals:

  • Fresh, dated content.
  • Clear semantic structure (headings, schema markup).
  • Canonical URLs that survive site reorganisations.
  • Authority signals (citations, mentions in trusted sources).
  • High-quality answers in places AI assistants prefer (Reddit, Stack Overflow archive, official docs, GitHub READMEs).

Weak signals:

  • Blocked AI crawlers (deliberate or accidental robots.txt settings).
  • Heavily client-side-rendered content (some AI crawlers can’t execute JS).
  • Login-walled or paywalled docs.

3. User feedback loops

Vendors disclose little about whether recent aggregate user activity affects answers. Do not treat this as a controllable DevRel input without product-specific evidence.

4. Live data sources

Live retrieval may use Bing for ChatGPT search, Google for Gemini, or separate indexes for Claude and Perplexity. Whether those systems index the product’s domain affects what an assistant can retrieve; see GEO and AEO for DevRel.

DevRel work affected by mediated discovery

If you are a DevRel team, you should:

Audit how AI assistants currently describe your product

Spend an hour with ChatGPT, Claude, Perplexity, and Gemini. Ask each:

  • “What is [your product]?”
  • “How does [your product] compare to [competitor]?”
  • “Write a quickstart for [your product].”
  • “What are the gotchas of [your product]?”
  • “Why would I choose [your product] over [alternatives]?”

Note errors, omissions, and outdated claims. Note whether the AI cites your product at all when asked about your category. Note the sentiment of the answer.

Repeated quarterly, this audit gives the team a concrete list of errors and omissions to prioritise.

Maintain the first-party sources assistants retrieve

Assistants may retrieve official documentation, blog posts, and open-source READMEs when answering product questions. Ambiguous or contradictory first-party material can therefore produce inaccurate instructions or inconsistent comparisons.

Concrete tactics:

  • Use canonical, consistent product names.
  • State key facts (compatibility, pricing, region availability) explicitly and repeatedly.
  • Date your content (dateModified in schema).
  • Maintain a single canonical “What is [your product]?” page that AI assistants are likely to extract from.

Monitor relevant third-party sources

In 2026 these include:

  • Reddit. A source of developer accounts and comparisons.
  • Hacker News. Technical discussion and launch commentary.
  • Stack Overflow (the archive, even though traffic is down): established technical answers.
  • GitHub. README files, code, issues, and discussions.
  • dev.to and Hashnode. Developer-authored technical content.
  • Substack and authoritative newsletters. For industry analysis.
  • YouTube transcripts. Captioning matters for AI consumption.

DevRel can contribute technical corrections, examples, and attributable expertise in these channels without treating independent communities as placement inventory.

Provide explicit AI-readable surfaces

llms.txt, MCP servers, and OpenAPI-driven reference docs. See llms.txt and MCP for DevRel.

Track AI-mediated signups separately

Use onboarding surveys, referrer analysis, or customer interviews to identify developers who discovered the product through an assistant or used an agent for the first integration. Track the cohort separately before deciding whether its onboarding or retention differs from search-mediated users.

Developers who do not use AI assistants

Continue supporting developers who avoid or cannot use AI assistants, including:

  • Senior engineers at conservative enterprises.
  • Some sectors with regulatory friction (finance, healthcare, defence).
  • Developers with strong ideological commitments to non-AI workflows.
  • Developers in regions with limited AI-assistant adoption.

DevRel should not optimise only for AI-mediated discovery. Google search, conference attendance, and peer recommendation still serve these developers. The dual-audience approach (Humans and agents) keeps those human-facing surfaces in the plan while adding agent-facing ones.

Risks and limits

Unattributed summaries. AI assistants may answer without linking. Referral analytics therefore miss some product discovery that occurred through an assistant.

Sentiment and behavioural data measure different things. A developer may report confidence in an AI-assisted choice while product telemetry records only the later visit or signup. Neither source reveals every document or recommendation that contributed.

Training-data lag. If your product was renamed, rebranded, or changed direction in the last twelve months, AI assistants may still describe the old version. Refresh cycles vary by model. Update the canonical product page, documentation, metadata, and redirects, then test live retrieval separately from model memory.

Variation across answers. AI answers vary between sessions, models, and prompt phrasings. Record repeated errors across a defined prompt set rather than drawing conclusions from one response.

Practical implication

A DevRel team cannot control every first impression produced by an AI assistant. It can maintain the public material those assistants retrieve: authoritative, well-structured, well-dated writing about what the product is and how to use it. Regular audits can then show where that material still produces inaccurate or incomplete answers.

See also