How to gather data requirements that survive the build

A regional service planner opens her dashboard. It flags thirty orders at risk because a part might not reach the warehouse in time.

She opens a spreadsheet, subtracts reservations the warehouse management system (WMS) counted as available, swaps in substitute parts the system doesn’t recognize, and rebuilds the picture by hand to find that the dashboard was only 50% accurate.

So what happened?

The data engineer who shipped it had fulfilled the documented requirements: joins were clean, tests passed, and the refresh ran on time.

But the business requirements had several major logical gaps: which ETA to trust, whether reserved stock counted as available, and other business decisions the data pipeline had to make.

The data coding agent filled those gaps with assumptions instead. Requirements elicitation is the work of finding and resolving those missing business rules before implementation begins.

Start requirements gathering with the decision, not the requested output

The service-order example above shows why a request for a dashboard is not yet a set of data requirements. It tells the data team what the stakeholder wants to see, but not all the business decisions the data pipeline must encode to produce the right answer.

Start requirements gathering with a different question: What decision will someone make differently because this data exists?

For the regional planner, the answer is whether to expedite a shipment, substitute a part, move the appointment, or leave the job alone.

Before planning a data model, make the request answerable at the level where the pipeline must make choices:

• Decision and consumer: who acts on the output, and what action can they take?
• Grain: what does one row represent, and which fields make it unique?
• Business event: what creates or changes the fact?
• Time rule: which clock assigns the record to a period or determines whether it is late?
• Source authority: which system owns each attribute, status, and date?
• Exceptions: which records are excluded, quarantined, or treated differently?
• History: should later changes restate prior results or only affect the current view?
• Validation: which known result or behavior will prove the output is useful and correct?

A mockup shows where a value will appear, but a spec explains how that value earns its place there.

Gather evidence before asking stakeholders to reconstruct the business

Requirements gathering begins with what the company has recorded: dashboards, tickets, source documentation, spreadsheets, meeting transcripts, SQL, dbt models, tests, incident notes, and the configuration of the operational systems themselves.

Production code belongs in that evidence set. It tells you what the pipeline does today, though it cannot tell you whether that behavior was approved or emerged as a workaround.

Data profiling makes this evidence more useful. Broad questions force a subject matter expert to reconstruct years of operating knowledge from memory. A profile lets you arrive with a specific anomaly and ask for a decision.

Look for evidence that exposes the choices hidden inside the request:

• values represented differently across systems
• identifiers that are duplicated, reused, or mapped inconsistently
• timestamps that could each plausibly define the same business event
• manual adjustments that do not exist in source systems
• records that repeatedly fall outside the happy path
• differences between current pipeline output and a result the business trusts

Instead of asking, “How does this process work?”, you can ask why a specific set of records was excluded, which source should win when values conflict, or whether a manual adjustment represents an approved rule.

Requirements gathering finds candidate requirements and contradictions. Requirements elicitation turns those gaps into approved decisions.

Choose requirements elicitation techniques based on the gap you need to resolve

The best requirements elicitation technique is the one that produces a rule, owner, exception, or validation target. Data teams get the most value when they choose the method based on the uncertainty in front of them.

Turn stakeholder answers into requirements the code can execute

Requirements get muddy when a team collapses evidence, opinion, approval, and uncertainty into the same sentence. Keep four states separate:

• Observed behavior: what a source, spreadsheet, report, or model currently does.
• Stated rule: what a stakeholder says should happen.
• Approved rule: what the accountable owner has confirmed for a defined use case and effective period.
• Open decision: a conflict or gap that still needs an owner and resolution.

This separation protects the engineer from treating current behavior as policy. The WMS may report gross on-hand inventory, the planning workbook may subtract active reservations, and a warehouse lead may say that expired reservations should be ignored after 24 hours. Those are three pieces of evidence. The requirement emerges when the accountable operations owner approves the rule, scope, and effective date.

Grain, time, source precedence, and history are where plausible requirements break

Many data requirements documents look complete while leaving the hardest semantics implicit. Those omissions surface later as duplicate facts, unstable joins, unexplained restatements, and dashboards that answer different versions of the same question.

Make the grain explicit

Name the business event represented by one row and the fields that make it unique. “Service order part” may still be ambiguous if a part can be split across sites, fulfilled by a substitute, or allocated in several reservation events. If the requested output uses a different grain from the source, document the aggregation boundary and which attributes remain valid after aggregation.

The grain should also determine your tests. If the declared grain is service order, part, site, and scheduled date, uniqueness at that key is a requirement, not a cleanup task for the implementation.

Name the clock the business uses

Created, promised, scheduled, committed, shipped, received, and updated timestamps answer different questions. Capture the event, timezone, cutoff, and late-arrival behavior. If the visit moves after the risk state was calculated, decide whether the team needs an as-of view, a current view, or both.

Time semantics also shape incremental processing. A transfer ETA that changes three times in a day may require change capture and a short recomputation window. A backdated warehouse receipt may require a wider replay strategy if historical service-level metrics must be restated.

Assign source authority by attribute and use case

A company rarely has one source of truth for an entire entity. The field-service platform may own the current appointment, the ERP may own the work-order requirement, the WMS may own physical inventory, and the procurement system may own the committed vendor date. Record authority at the attribute level, including precedence, fallback, and owner.

This prevents “use the ERP” from becoming a vague instruction that breaks as soon as two systems represent the same concept differently.

Decide how history should behave

Operational truth changes. Appointments move, reservations expire, ETAs slip, substitutions are approved, and warehouses correct receipts. Decide whether the product should answer “What is at risk now?”, “What did we believe was at risk yesterday?”, or both. That choice determines whether you need effective-dated mappings, event history, snapshots, or a simpler current-state model.

Edge cases tell you what model to build

The happy path rarely tells you enough to design the right data model. Ask about the cases that produce manual corrections, exceptions, disputed numbers, or special handling.

Focus on edge cases that force an implementation decision:

• Partial states: can a business event be partly complete, partly valid, or partly fulfilled?
• Many-to-many relationships: can one record map to several valid alternatives, categories, or owners?
• Conflicting sources: what happens when two systems provide different values for the same attribute?
• Late or backdated changes: should prior results be restated when new information arrives?
• Changing identifiers: how should renamed, merged, reused, or superseded entities be handled?
• Manual overrides: does an override represent a one-off exception, an approval, or a business rule that should become part of the model?
• Unknown or invalid states: should incomplete records be excluded, quarantined, surfaced explicitly, or allowed through?
• Historical changes: does the business need the current truth, the truth as it was known at the time, or both?

These answers affect grain, relationship design, source precedence, history, incremental processing, exception handling, and validation.

If an edge-case discussion does not change the specification, implementation plan, or test suite, the requirement is probably still too vague.

Reconcile during discovery so acceptance becomes executable

A trusted output is one of the strongest requirements artifacts a data team has.

Test proposed logic against a report, curated list, or other result the business trusts, and reconcile at the smallest slice that can isolate a rule. Every unexplained variance can expose missing context, such as a grain problem, incorrect time rule, or undocumented manual override.

Keep a reconciliation log of expected and actual results, the suspected rule, supporting evidence, owner, resolution, and resulting test so acceptance becomes something the pipeline can keep proving.

Business acceptance criteria should be executable wherever possible:

• Keys are unique at the declared grain.
• Known records exercise each important inclusion, exclusion, mapping, and temporal rule.
• Closed-period outputs match an approved benchmark within a named tolerance.
• Late arrivals and source corrections produce the approved historical behavior.
• New unmapped values and broken assumptions fail visibly rather than entering a default bucket.
• Every published fact can be traced to its source records and requirement rule.

Freshness, referential integrity, and compilation checks still matter. They tell you whether the implementation is healthy. Reconciliation targets and rule-level tests tell you whether it implements the business decision the team approved.

Requirements traceability keeps business decisions from disappearing into SQL

The line of SQL that excludes quarantined stock rarely explains why the rule exists, who approved it, or when it became valid. Give material requirements stable identifiers and record the owner, evidence, approval, effective period, affected models, and validation criteria, with links back to the source that supports the decision.

Requirements traceability makes change safer because you can see which models, tests, metrics, alerts, or semantic-layer objects inherit a rule when it changes. Data contracts and semantic layers can preserve and enforce approved decisions, but they should not be asked to make those business decisions in the first place.

Discovery is done when every material ambiguity is resolved or owned

Requirements work can expand forever. The useful finish line is a build packet that makes remaining uncertainty visible and prevents the developer from settling it by accident.

Before implementation begins, confirm that:

• The business outcome, consumer, and decision supported by the data are clear.
• Grain, keys, entities, relationships, and aggregation boundaries are explicit.
• Event time, timezone, effective dating, and restatement behavior are defined.
• Source ownership and precedence are defined by attribute and use case.
• Mappings, calculations, inclusions, exclusions, and material exceptions are implementable.
• Historical coverage, backfills, late arrivals, and schema-change behavior are understood.
• Trusted outputs or known records provide business-level validation targets.
• Each approved rule has an owner and evidence.
• Each unresolved decision has a named owner and a disposition.
• Any deliberate assumption is labeled, scoped, and accepted.

A material decision can remain open only if the affected work is excluded, blocked, or proceeds under an explicitly approved assumption. Handing the ambiguity to an engineer or agent is not a disposition.

A coding agent should inherit decisions, evidence, and stop conditions

A coding agent can inspect schemas, profile columns, follow lineage, generate dbt models, compile the project, and add structural tests. Project metadata gives it strong context about what exists. When two credible sources conflict, that context cannot reveal which business interpretation the organization intends.

Give the agent a specification that names the grain, source authority, mappings, calculations, edge-case behavior, open decisions, and validation targets. Include the evidence behind important rules and stop conditions for missing sources or unresolved conflicts. The agent should have room to choose implementation details inside those constraints, while business meaning remains owned by the people accountable for the process.

That separation matters as implementation gets faster. A clean, well-tested project can encode the wrong rule with impressive efficiency. With spec-driven development, the team can review what “correct” means before the coding agent implements it.

From scattered evidence to an agent-ready specification

Autopilot Spec helps compress the work between the initial request and the build. It reads the material where requirements live, including meeting transcripts, dashboards, documentation, spreadsheets, and source-system exports. It identifies conflicting definitions and missing decisions, captures approved resolutions, generates validation criteria, and packages the result for the coding agent your team uses.

The goal is to preserve the judgment data engineers bring to discovery while removing the manual work of reconstructing the same business context for every pipeline.

Give your coding agent the business requirements, approved logic, and validation criteria it needs before the build. Try Autopilot Spec.