Prompt drift: why ten developers get ten different AI outputs

Short answer: Ten developers prompting ten different ways produce ten different house styles, and reviewers pay the bill. Writing the conventions down in a wiki does not fix it, because a wiki is documentation, not a control. What fixes it is guidance that lives in the platform and applies on every AI turn without anyone remembering to paste it: rules that are always on, named commands the team invokes instead of pasted prompt blobs, and visible attribution showing which rules actually shaped an answer.

What prompt drift actually is

Prompt drift is the slow divergence of AI output quality and style across a team, caused by the fact that each person's prompt is private. Nobody is doing anything wrong. One developer always asks for tests. Another always says "keep it simple". A third pastes the same 400-word context block they wrote in February and has never updated it. A fourth joined last month and does not know the block exists.

The output is technically fine each time. It is the variance that costs money. Reviewers stop being able to predict what an AI-assisted change will look like, so they read everything from scratch. Terminology diverges in the same repository. Half the code says ticket, half says workItem, and the AI cheerfully follows whichever one it saw last. Six months later you have a codebase that reads like it was written by four companies, because in prompt terms it was.

This is the same class of problem as inconsistent AI-assisted delivery generally (see why individual AI assistants are not a team workflow), but prompt drift is the narrow, cheap-to-fix part of it. You do not need to change how anyone codes. You need to change where the standards live.

Why writing the standards down did not work

Every team that notices prompt drift tries the same first fix: a page called "How we prompt". It usually contains good advice. It almost never changes behaviour, for three reasons.

• It is opt-in at the worst possible moment. Following the doc requires remembering it exists while you are mid-thought about something else. That is the moment when people are least likely to context-switch.
• It has no enforcement surface. Nothing checks whether the AI followed it. Nothing even records that it was supposed to.
• It rots invisibly. Prompt advice ages fast. A doc nobody reads is also a doc nobody corrects, so the version people half-remember is the version from whenever they last read it.

A standard that depends on human recall on every turn is not a standard. It is a hope with a URL.

Rules: standards that apply without anyone remembering them

In Crew Orbit, reusable instruction is called guidance: admin-authored markdown, configured in Settings under AI Assist → Guidance at organization and project scope. Guidance in the Rules group is injected on every AI turn in scope. Nobody invokes it. Nobody can forget it.

The clearest example is vocabulary, and it ships seeded as a rule called vocabulary: always say work item, never ticket. Read as a wiki line, that is pedantry. Read as an enforced rule, it is the difference between a product where the AI, the backlog, the docs, and the team all use one noun, and a product where every artefact has to be mentally translated. The rule is short, it is boring, and because it is always on it is actually true.

The same mechanism carries the standards that are not boring: which layer owns validation, what "done" means for a story, that every proposed change names the affected project, that security-relevant answers must state their assumptions. Anything you would otherwise re-type is a candidate.

One property matters more than it sounds: guidance behaves identically on every chat surface. A direct Ask session, @crew in a collaboration room, and @crew in a work-item comment all resolve the same guidance. You are not maintaining three sets of standards for three places the AI shows up.

Commands: a team prompt library you invoke by name

The other half of the problem is the long prompt that only one person has. Somebody on the team has worked out a genuinely good prompt for turning a rough idea into a story with acceptance criteria. It lives in their notes app. When they are on holiday, the team does it badly.

Guidance in the Commands group is markdown authored once by an admin and invoked in the composer as /slug. That is the whole interaction. Type /refine-story, add the rough idea, send. The command body is the prompt. What you typed after it is the input. Instead of a shared document of prompts that people copy and mutate, the team has a short list of named ones that all resolve to the same current version.

Because an empty / menu is a dead feature, four commands ship seeded:

• /refine-story: turn a rough idea into a structured user story with acceptance criteria
• /bug-triage: summarize a bug report and propose severity, owner area, and next step
• /standup: summarize recent work-item activity as a short standup update
• vocabulary: the always-on house vocabulary rule described above

Seeded guidance carries a template version and can be reset to the latest platform version, so improvements to the starters reach you without you having to diff them by hand. Your own edits are yours. The reset is explicit.

Why there is deliberately no template language

Command bodies contain instructions. They contain no $ARGUMENTS, no {{input}}, no placeholders of any kind. Whatever the user types after the command is the input, and that is the entire contract.

This is a deliberate design decision, and it is worth understanding why it is a feature rather than a missing one. The moment a prompt library gains a template language, it also gains: escaping rules, a validation surface, a class of runtime errors that only appear for certain inputs, and a small dialect that every author has to learn before they can contribute. Teams end up debugging their prompt templates instead of improving their standards. Non-engineers stop contributing, because now authoring a command is programming.

With no template language, a product manager can write a useful command in plain English on their first attempt and it cannot fail to render. Bodies are capped at 8,000 characters and stored inline, which keeps them off the slow path of a chat turn. That cap is also a useful forcing function: a rule you cannot express in 8,000 characters is probably several rules.

The org sets the floor: a project overrides one rule, not the whole set

Central standards fail in the other direction too. If the only options are "the org rule applies" or "this project opts out entirely", every project with one genuine exception forks the whole set, and within a year nobody knows what the actual baseline is.

Guidance resolves as a slug overlay: System → Org → Project. The narrowest scope wins, per slug. A project that needs different review emphasis overrides the review rule by reusing its slug and inherits everything else untouched. The overridden row is visibly badged as overridden in settings, so the override is a fact somebody can see rather than a surprise somebody discovers. Disabled rows never resolve at all.

There is one more scoping property worth knowing: a workspace-scope Ask session resolves organization guidance only. Project-specific vocabulary does not leak into an answer that spans the whole workspace. If you have thought about how orgs, projects, and permissions should be structured, this is the same instinct applied to instructions.

A large library without paying for every rule on every turn

Always-on rules have an obvious cost: every one of them enters every turn. A team that writes forty rules has quietly made every question more expensive and diluted the ones that matter. The usual advice ("keep the library small") is really just asking you to under-document.

There is a middle ground. A rule can be registered so that only its catalog line (the slug and a one-line description) enters the turn, and the model pulls the full body through a read-only load step when the description suggests it is relevant. A forty-rule library then costs forty short lines instead of forty full bodies, and the DynamoDB conventions load when the question is about DynamoDB.

That is not free and it is not magic: the model has to judge relevance from the description, so descriptions carry real weight. But it changes the shape of the trade-off. You can document the long tail without taxing every question, and per-turn injection stays budget-capped either way. Per-turn prompt and completion token usage is recorded on the assistant message, so the cost is observable rather than inferred.

Applied vs cited: seeing which rules shaped the answer

Invisible guidance is a new kind of problem. If the AI's behaviour is being shaped by instructions the user never sees, "why did it answer that way?" becomes unanswerable, and the feature starts to feel like magic. In an engineering organization that is a synonym for untrustworthy.

Crew Orbit reports guidance in two layers, in a [G#] namespace that parallels how knowledge citations work:

• Applied: the server's snapshot of everything that entered the turn: slug, display name, activation, and scope. It is a record, not a re-render. The bodies themselves are never printed back into the thread.
• Cited: the [G#] markers the model emitted because a specific piece of guidance materially shaped that answer.

The split exists because always-on style rules would otherwise spam every answer with chips. They stay in applied. Only guidance that genuinely changed the answer gets cited. Applied-but-uncited rows fold behind an "N more applied" control, so the full record is one click away without dominating the thread. And when a turn pauses for an approval or an inline input, resuming replays that same snapshot rather than re-resolving settings mid-turn. So an answer is never shaped by two different versions of your standards.

The practical effect: guidance becomes auditable. When a reviewer disagrees with how the AI framed something, the conversation is about a specific rule at a specific scope, and the fix is an edit somebody can make.

Guidance shapes behaviour: skills teach your stack

These two are easy to confuse, and keeping them separate is what stops either from becoming a dumping ground.

Guidance is about how the AI behaves when it talks: vocabulary, tone, what a good answer includes, which command does what. Its consumers are the chat surfaces: Ask Crew Orbit, room @crew, comment @crew. It is selected by activation: always on, invoked by name, or loaded on demand.

Skills are the run-side sibling: attachable markdown playbooks that teach AI runs your stack. Eighteen platform skills ship seeded (java, react, spring-boot, dynamodb, security, api-design, testing-pyramid and others), and org- and project-scoped skills merge over them, so a run applies your conventions without anyone restating them. Skills are attached to work items and materialized into the run workspace for the active AI provider.

Be precise about the boundary, because it is a real one: guidance is a control on the chat surfaces today. It is not the mechanism by which your standards reach a delivery run. That is what skills are for. Two mechanisms, two consumers, no overlap.

What to standardize first

You do not need forty rules. Start with the three that reviewers keep repeating:

• Vocabulary. One noun per concept. It is the cheapest rule to write and the one whose absence is most visible six months later.
• What a complete answer contains. If your reviewers always ask "what about the migration?" or "which project is this?", that is a rule, not a review habit.
• The one prompt somebody is good at. Turn it into a command with a slug. Then it survives their holiday.

Everything after that is a judgement call about cost and clarity, and you will make it better with attribution data in front of you than by guessing now.

Standardize how your team uses AI

Prompt drift is not a discipline problem and it does not respond to more documentation. It responds to putting the standards where the prompt already goes. If that is the gap on your team, start at crew-orbit.com.

Related reading: the same guidance layer carries the privilege switch. See how to give an agent read access without write access. Then the AI coding mistakes that cost teams the most and how to review AI-generated code without a black box.