Why your AI should cite its sources and say “I don't know”

Short answer: An AI that answers confidently from nothing is worse than no AI, because somebody acts on the answer. Two properties fix that, and neither is a prompt: every answer is grounded in documents you own and carries citations you can open to the exact cited passage, and below a relevance floor the AI declines and says what it searched and where. Crew Orbit does both over a scoped knowledge library, and reports knowledge as unavailable rather than degrading quietly when it cannot search.

The failure mode is familiar to anyone who has rolled an assistant out to a team. Someone asks which service owns billing. The model does not know, but it knows the shape of an answer to that kind of question, so it produces one: a plausible service name, a confident sentence, no hedge. A week later a developer has built against it. The cost was never the wrong sentence. The cost was that the sentence was indistinguishable from a right one.

Why is a confident wrong answer worse than no answer?

Because a missing answer routes correctly and a wrong answer does not. When the AI says nothing, the person asks a colleague, checks the repo, or opens the ADR. That is the normal, slightly annoying path that produces a correct result. When the AI answers, that path closes. The question is marked resolved.

That is why the calibration matters more than the accuracy rate. A tool that is right 95% of the time and silent the rest is usable. A tool that is right 95% of the time and confidently wrong the rest forces every user to independently verify all 100%, which is more work than not having the tool. Teams sense this quickly. They do not file a bug. They just stop asking.

What does it actually mean for an AI to cite its sources?

“Cites its sources” is used loosely enough to be worthless. A footnote naming a document you cannot open is decoration. A useful citation answers a reviewer's question: which passage, in which document, and how sure are you?

In Crew Orbit, an answer's citation markers render as inline chips. Each source is one row carrying the document title, the kind of source it is, the scope it lives in, and a qualitative confidence: high, medium, or low. Opening a chip opens a drawer at the cited chunk itself, with its neighbouring text so you can judge whether the surrounding paragraph changes the meaning, plus a link through to the document in the library. If the underlying document has since been deleted, the row says so instead of offering a dead link.

Two deliberate choices are worth naming, because both are the kind of detail that separates a real implementation from a demo:

• Confidence is qualitative, not a number. Raw relevance scores never reach the interface. A cosine similarity of 0.71 tells a reader nothing actionable and invites false precision. “Medium confidence, here is the passage, read it” tells them exactly what to do.
• The cited passage comes with its neighbours. Retrieval works on chunks, and a chunk boundary can cut a qualifying sentence in half. Showing the surrounding text is how a reader catches that.

Honest abstention is the real feature

Citations get the attention, but abstention is what makes a team keep using the tool. It is also the part almost nobody builds, because it looks like a worse product in a demo.

Crew Orbit's retrieval ends in a relevance floor. Below it, the answer is a decline, and the decline is informative: it states what was searched and in which scope, then offers the two things that actually fix the situation. Add the missing knowledge, if the person is permitted to. Or re-run the same question across the whole workspace instead of one project. A refusal that names its own boundary is a navigable result. A refusal that just says “I could not find anything” is a dead end.

One distinction inside that matters more than it sounds. An empty library is named as empty, not reported as a failed match. Those are different problems with different fixes, and conflating them is how new workspaces conclude the search is broken on day one. “Nothing relevant was found in 340 documents” means rephrase or add a document. “This library has no documents yet” means go upload the ADRs. Same UI, opposite next action.

Abstention is recorded, too. When retrieval returned nothing and no tool ran, the decline is persisted on the stored assistant message alongside what was searched and the scope label. So an abstention is a reviewable event later, not just a sentence that scrolled away.

Worth being precise about one thing: empty retrieval does not end the turn. The model still runs, and it can still use its read tools (listing work items, for example), because “no document matched” is not the same as “there is nothing useful I can do.” Abstention is a decision the model reaches, not a hard stop the system imposes before it thinks.

Why not just use a bigger context window?

This is the reasonable objection, and it deserves a real answer rather than a dismissal. Context windows are large and getting larger. Why not paste the wiki in and skip retrieval entirely?

Because a context window solves a capacity problem, and the problems here are attribution, authorization, and refusal.

• Attribution. If forty documents are in the prompt, the answer is still one paragraph of prose with no pointer back. A reviewer who wants to verify a claim has to re-read the forty documents. That is exactly the work you bought the tool to avoid. Retrieval produces a record of which passages were used. Stuffing does not.
• Authorization. Everything in the prompt is visible to whoever gets the answer. A knowledge library that scopes documents workspace-wide or per-project, and additionally filters retrieval by what the asking person is allowed to see, cannot be replicated by a big prompt. The moment you paste, scope is gone. In Crew Orbit a project conversation draws on workspace-wide documents plus that project's. A workspace-wide conversation draws on workspace documents plus the projects the caller can actually read, computed server-side, with excluded projects disclosed rather than silently dropped.
• Refusal. A relevance floor is a defined threshold. If the whole corpus is in the prompt, there is no threshold. There is only a model doing its best with everything, which is precisely the condition under which it invents. Grounding gives abstention something to be measured against.

Grounding and a large window are also not in competition. Retrieval decides what deserves to be in the window and keeps a receipt. The window decides how much fits.

Five sources, one store

A knowledge base that only accepts uploaded PDFs will be out of date within a quarter, because the material a team actually reasons from is not shaped like PDFs. Crew Orbit feeds one indexed store from five source types:

• Uploaded files: specs, ADRs, contracts. Multiple files can be staged and ingested in a batch, with per-plan size limits and quotas enforced before anything is stored.
• Pasted text: the decision that lives in someone's notes and was never a document.
• URLs: fetched server-side, with egress guarded.
• Repository documentation: product cards and README/doc digests synced from connected repositories. This is the one that answers “which service owns billing?” without anyone writing a wiki page, and its citations carry the repository and the file path, so a reader lands on the actual file. Syncs can be previewed before they run, refreshed when stale, and removing a repository drops its synced documents.
• Saved conversations: a person selects the turns in a discussion that mattered, the AI drafts a note, a human edits and confirms it. Nothing is auto-captured. Saved notes link back to the room or session they came from, and deleting the note never touches the conversation.

All five are one lifecycle: searchable, citable, deletable. Deleting a document removes it from search and purges its chunks. Reorganizing the library into folders and tags is metadata only. It never re-runs ingest and never breaks a citation.

What are the honest limits?

Grounded retrieval has a hard dependency, and pretending otherwise would undercut the whole argument of this post.

Embeddings require an OpenAI credential from your workspace credential store, the same one used for AI assist. There is no alternate embedder. Without that credential, knowledge is reported as unavailable, with a clear state in the interface, rather than quietly falling back to ungrounded answers. That is the correct behaviour, but it is a prerequisite you should know before you plan a rollout. The chat model itself is separate and can be OpenAI or Anthropic. Ingest has visible statuses, and a document with no extractable text layer fails with a reason a person can act on, rather than sitting in a permanent pending state.

There is also a scale ceiling. Retrieval is designed around a workspace-scale corpus, not a company-wide document warehouse, and the chunk ceiling applies on every plan tier. For an engineering organization's specs, ADRs, repository digests, and saved decisions, that is a comfortable fit. If your intent is to index every document your company has ever produced, ask about limits before you commit.

How to evaluate this in a tool you are considering

You can test all of it in about ten minutes, and the tests are more revealing than any feature list:

• Ask a question you know is not covered by the indexed material. A good tool declines and tells you what it searched. A bad one answers.
• Ask before uploading anything. Does it distinguish “library is empty” from “no match”?
• Click a citation. Does it open the passage, or does it just name a file?
• Ask as a user who cannot see one project. Does retrieval respect that, and does it tell you something was excluded?
• Delete a cited document, then re-open the old answer. Does the citation admit it, or dangle?

A tool that passes those five is one your team can trust by default and verify by exception. That is the whole point: the value is not that the AI is always right, it is that you can always tell.

Where this fits

Grounding is about what the AI knows. It pairs with the separate question of what the AI is allowed to do. See how to give an agent read access without write access. For the review-side argument, read how to review AI-generated code without a black box, and for the visibility case, why visibility matters most in AI code generation.

If you want AI answers your team can check rather than hope about, start at crew-orbit.com.