What is a spec? Spec-driven development for data teams

A coding agent can build a data pipeline, generate its own tests, and pass every single one of them. But what happens when the tests are based on the same incorrect business assumptions as the code?

As coding agents become more capable, data developers need a reliable way to establish business requirements and rules before implementation begins. A spec provides this foundation, and spec-driven development turns it into a repeatable way of working.

What is a spec? Definition for data teams

A spec is a structured, reviewable blueprint for a data pipeline, grounded in business discovery. It captures:

• Business requirements for a specific dashboard, report, or other request from business stakeholders, including the questions the finished data pipeline must answer
• The relevant source systems, business entities, relationships, and transformation rules
• Definitions, calculations, exceptions, and decisions approved by business stakeholders
• Conflicts, gaps, and unanswered questions requiring human intervention
• Validation criteria, including trusted figures or expected behaviors against which the output can be checked

In data engineering, a spec reaches upstream and connects business requirements to technical implementation. It gives data developers a common reference for reviewing decisions and coding agents a defined outcome to build toward.

When a data developer is asked to build a data pipeline for a particular reporting outcome, the business may have been using the relevant definitions, classifications, and processes for years. The developer needs to discover and understand those rules before translating them into data models.

A spec can contain detailed modeling guidance, but it doesn't have to prescribe every SQL expression or physical design choice. Developers and coding agents still need room to investigate sources and determine how best to implement the approved requirements.

More generally, in software engineering, a specification, or spec, describes a system's expected behavior, constraints, and conditions for acceptance. It might take the form of a feature specification, an API contract, or a technical design.

With the emergence of AI coding agents, specifications are taking on a more prominent role.

A useful spec for data pipelines begins with business discovery

In enterprise data, business requirements rarely arrive as a neat, comprehensive document.

The relevant knowledge may be spread across meeting transcripts, dashboard screenshots, spreadsheets, operating procedures, and the experience of people who have managed a process for years. Much of this material is readily available to data teams, albeit difficult to collect and consolidate across different formats and departments.

It is much more difficult to then turn sources of information into complete, reliable business requirements in a format easily understood by coding agents.

In the client project we ran at Maxa, the requested sales and pull-through reporting depended on two ERP systems, a field-service CRM, and a spreadsheet containing sales targets.

While a coding agent could inspect the source tables, identify keys, and propose joins, it struggles to determine how the systems contributed to the client's reporting requirements.

A technically sound model built from one system could still miss essential business activity recorded elsewhere.

This is the business discovery process Autopilot Spec is designed to support. Data developers can provide existing business materials, and Autopilot Spec organizes the evidence into requirements, identifies contradictions or missing rules, and surfaces questions for the appropriate stakeholders.

A useful spec, therefore, encodes business knowledge just as much as data engineering best practices.

By documenting business decisions and their sources, specs also preserve business knowledge for future developers, coding agents, and changes to the data pipeline.

What is spec-driven development?

Spec-driven development (SDD) is a development methodology in which a specification guides implementation and validation throughout the development process. It involves defining an intended outcome, establishing business requirements, resolving important decisions, and agreeing on how correctness will be evaluated.

Coding agents can then plan, implement, validate, and test against an explicit specification.

How spec-driven development differs when applied to data engineering

Many data engineering projects begin with an expected end result: an existing report to reproduce, specific questions to answer, and so on. The transformation logic still needs to be designed, but the underlying business rules often predate the project. They may be undocumented or disputed, and establishing which rules apply is part of the business discovery and requirements gathering process.

With coding agents, it's possible to get quite far into implementation without going through this process.

Spec-driven development brings more of this discovery and decision-making into the planning stage. Developers establish the intended outputs, document the relevant business rules, and identify unresolved requirements before implementation begins. This reduces the likelihood of spending multiple development cycles correcting assumptions that could have been clarified earlier.

Implementation remains iterative. Profiling source data, testing models, and comparing outputs against trusted figures can reveal further requirements or challenge an existing interpretation. These findings can then inform revisions to the spec, keeping business requirements and technical implementation aligned as development progresses.

We've explored the broader rationale in The case for spec-driven development in analytics engineering.

A spec delivers the business context that improves coding agent accuracy

Enterprise reporting requirements often span multiple systems, departments, and business processes.

Different teams may contribute to the same workflow, maintain separate records, or use conflicting definitions. Establishing what a data pipeline should produce requires understanding how these processes fit together, resolving disagreements, and confirming which business rules are authoritative.

These decisions directly affect the accuracy of the implementation, and technical correctness alone won't reveal logical errors if the tests are based on incomplete business requirements.

A spec captures the results of business discovery as explicit requirements, approved rules, source mappings, and validation criteria. Coding agents can use the spec, which carries business context, domain expertise, and technical best practices, to implement the intended logic, while data teams can evaluate the resulting data pipeline against the business requirements it was designed to satisfy.

Where specs fit in the emerging agentic data stack

Coding agents can now draw on more context from existing data tools.

For example, dbt Wizard can use project metadata such as lineage, tests, and model health, while semantic layers make governed business definitions available to agents.

These capabilities help coding agents understand existing data infrastructure. A spec provides the requirements for what a new data pipeline should produce.

Other approaches exist; a semantic layer can make approved definitions accessible. But specs work upstream of semantic layers and the like; establishing which definition a particular reporting requirement calls for, document stakeholder approval, and identify the validation criteria the implementation must satisfy.

Ultimately, spec-driven development gives data developers a way to apply their judgment where it has the greatest leverage: understanding what the organization needs before committing its requirements to code.

With a clear specification, coding agents can move quickly and with fewer iterations, while data teams retain a reliable basis for deciding whether the resulting data pipeline delivers what the business intended.