I tested a DIY spec against Maxa Autopilot Spec

Every source system is built for the people who use it. A technician closes a work order in the field service app, an accountant posts an invoice in the ERP, and a salesperson updates an opportunity in the CRM.

Each system stores what its users need to do their jobs, and very little else.

A data model built from these source systems, therefore, needs the added business context that exists in complex and scattered forms outside the systems. Capturing that context is usually the hardest part of the job.

So I wanted to test if a capable data practitioner could use a coding agent to build their own business context layer. And if they could, how would the resulting data pipeline compare to one built using Maxa Autopilot Spec?

With the latest generation of coding models becoming increasingly capable, I wanted to test how far they could go beyond implementation. Claude Code can inspect source systems, reason through unfamiliar schemas, and build data pipelines. Could it also turn business requirements into a spec reliable enough to guide its own work?

I work at Maxa, so I have an obvious interest in the answer. But I also wanted to give the DIY or do-it-from-scratch approach a fair shot. If we couldn't demonstrate a meaningful difference in the results produced by a spec generated by a coding agent and one produced by Autopilot Spec, we needed to understand why.

So I decided to build the same data pipeline using three approaches: no spec, a DIY spec created by Claude Code, and Maxa Autopilot Spec. Then I'd compare the results against an existing client implementation.

Read my colleague’s article on why spec-driven development for analytics engineering, so you have the basis for why spec in the first place.

I chose to run the experiment on a dataset I hadn't worked with before

A client provides HVAC, plumbing, and other services to commercial, industrial, and institutional customers.

I chose this client for my experiment partly because I knew relatively little about them. I hadn't built their existing production data pipeline, so I didn't have the benefit of knowing their business rules from previous implementation work.

The assignment was to build a sales and pull-through data pipeline using information from multiple systems:

• COINS ERP, one of the client's enterprise systems
• Sage ERP, another source of operational and financial records
• ServiceTrade CRM, the client's field service system
• A client-managed spreadsheet containing sales goals and budget information

The three application systems had data loaded into Snowflake, while the spreadsheet provided additional business context.

To make my task slightly easier, there was an existing project to compare against. The client had signed off on a data model, and we had 17 questions the resulting pipeline needed to answer. These gave me a practical basis for evaluating coverage and business correctness.

I also wanted to use a capable coding agent. Earlier experiments with other models had shown me how much the model itself could influence the result. For this comparison, I used Claude Code with Opus 5.5 across all three builds.

The same coding agent, with varied company knowledge

The experiment begins!

I kept the technical setup consistent. Each run had access to the same raw client materials and warehouse data, along with two internal agent skills which are based on publicly available repos from dbt and Snowflake:

• dbt-standards: Our modeling conventions, based on dbt documentation, covering model organization, naming, macros, testing, and documentation.
• snowflake-cli: Command-line access to Snowflake, allowing the agent to explore tables, profile source data, execute queries, and validate results.

I disabled Maxa-specific development skills and used a conventional medallion architecture, with bronze, silver, and gold layers, alongside dimensional modeling.

The difference was how each agent received its business requirements

• In the no-spec run, Claude Code read the client materials and started building directly.
• In the DIY spec run, I gave Claude Code a single prompt to create its own business context layer from the available material. It then used its specification to build the pipeline.
• In the Autopilot Spec run, the agent received a specification Autopilot Spec generated from the client's business workshops and meeting transcripts.

The objective was to see how much business context each approach could capture and carry into implementation.

The no-spec build passed every test and missed the assignment

I'd reused a prompt from another project and accidentally left in the phrase "operations use case." All three runs received the same wording, but without a spec, Claude Code took it literally and built an operations reporting data pipeline when the client needed sales reporting.

• In about 35 minutes, it produced clean, documented models
• All its checks passed, and its order counts matched the source system
• But it calculated revenue from a field that was empty on most orders, overstated sales conversions, and left out the field service system and the client's sales targets
• It answered none of the client's 17 test questions

The agent had built a technically sound solution without understanding the client's requirements. Both spec-driven runs managed to focus on sales reporting despite my mistake in the prompt, making DIY versus Autopilot Spec the more useful comparison.

DIY built a useful spec, but missed much of the business scope

I was encouraged by parts of the DIY result.

Claude Code produced a reasonable specification and built models addressing the correct reporting subject. It was particularly good at technical details such as identifying source tables, keys, and edge cases. It even implemented one revenue rule correctly that the Autopilot Spec build missed.

For a developer working directly with source data, this kind of precision is valuable. The agent could inspect the actual source systems and document how particular tables and fields should be used.

But when I compared the completed builds against the client's approved data model and business questions, the differences were substantial.

The DIY run covered 48% of the required model, compared with 74% using Autopilot Spec. It also incorporated just one of the three warehouse source systems.

I also had to intervene five times during the DIY build, compared with just once in the Autopilot Spec run.

The Autopilot Spec build incorporated all three and could answer 16 of the 17 client questions. It even included a tie-out to the general ledger, which wasn't part of the client's existing reference model.

These counts describe what happened during the experiment, rather than establishing a general measure of engineering effort.

The broader coverage was more significant to me than the intervention count.

A DIY specification can describe one part of the business in considerable detail and support that part well. The difficulty is knowing which other parts are missing. A developer might finish a sales model, only to discover later that the reporting requirements depend on field service activity or another source the agent never incorporated.

At that point, correcting the data pipeline involves more than refining your code. You have to revisit the requirements, identify missing relationships, potentially speak with stakeholders again, and extend the model.

The specifications gave Claude Code very different instructions

When I compared the specifications themselves, I could see why the results differed. Autopilot Spec generated a specification fully aligned with the business, while the DIY spec described the data.

Autopilot Spec's flagged gaps and contradictions in business requirements that the DIY approach completely missed, flagging critical gaps to be bridged before the agent started the build.

What I'd change if I repeated the experiment

I'm careful about drawing sweeping conclusions from this experiment.

These published results come from one client and one run per approach. A different prompt, another coding agent, or a more experienced developer spending additional time on the DIY specification could produce different results.

I also couldn't guarantee perfect isolation between coding sessions. I cleared sessions and caches and explicitly instructed the agent not to inspect previous pull requests, but I would want stronger technical controls in a more rigorous evaluation.

If I ran it again, I'd repeat each approach several times, test additional clients, and use a more structured evaluation framework.

I'd particularly want to improve how we measure accuracy. A single incorrect revenue calculation can create a substantial financial discrepancy, while numerous small technical differences might have little effect on the final numbers. A simple accuracy percentage doesn't communicate those differences particularly well.

Despite the limitations of my experiment, I learned a lot from it.

What I learned from the experiment

I wouldn't dismiss DIY-ing your own spec, but it's hard to be systematic and rigorous when stitching together inputs from multiple stakeholders and source systems in a large-scale enterprise setting.

That's where I saw the value with Autopilot Spec: it captured a much fuller picture of the business requirements and gave the coding agent a stronger foundation to build from.

Claude Code handled the technical details well and even got a business rule right that the Autopilot Spec build missed. But it overlooked major business requirements that Autopilot Spec gave the agent a much fuller picture of.

If I ran this experiment again, I'd start with a solid spec, let the coding agent work through the source data, and validate against the client's figures. I'd also pay more attention to keeping the spec current as new decisions emerge.

The experiment showed me how much business context can improve a build when it's captured in a spec the coding agent can work from. It also left me thinking about how we keep that spec current as the business evolves.

If you’d like to run a few experiments for yourself, try Autopilot Spec for free.