Context engineering for data engineers using coding agents

Shopify records when a fulfillment is created, when it's delivered, and the tracking events in between. One order can also have multiple fulfillments.

Ask a coding agent for a daily shipment report and it must choose a timestamp: fulfillment creation, carrier pickup, or delivery. Which event does the operations team intend to count?

The agent can inspect the schema, write valid code, and pass its tests without knowing which event the operations team intends to measure. The API describes what the fields mean, but it can't make the business decision. (Source: Shopify's Fulfillment API documentation)

Context engineering addresses the information an AI agent needs to perform a task accurately. For data pipeline development, it involves establishing trustworthy technical and business knowledge, then delivering it when the agent needs it.

What is context engineering?

Context engineering is the practice of selecting, structuring, and maintaining the information an AI agent receives at each step of a task, so it can reason and act using relevant, trustworthy knowledge.

A model processes information within a context window. For a data engineering agent, that context can include much more than the prompt it receives.

Four main categories contribute to an agent's context:

• Instructions: System prompts, project rules, and task-specific directions.
• Knowledge: Documentation, schemas, definitions, and retrieved data.
• Tools and their outputs: Available actions, such as SQL execution, and their results.
• Memory and conversation history: Prior decisions and information carried across steps.

Retrieval-augmented generation (RAG) is one technique for supplying relevant information from external sources rather than placing an entire knowledge base in the prompt. The Model Context Protocol (MCP) provides a standard way for AI applications to connect to tools, resources, and other external capabilities.

How context engineering fits into the agentic data stack

Semantic layers, knowledge graphs, ontologies, catalogs, contracts, context layers, and specifications can all contribute to an agent's knowledge. They overlap but serve different purposes.

A semantic layer defines governed metrics and relationships. A context layer may assemble knowledge from semantic models, catalogs, documents, and live systems for the task at hand. Their boundaries vary by platform.

We've explored those distinctions in greater depth in our article on semantic layers and context layers.

In practice, these technologies increasingly overlap. A platform might combine governed metrics, business definitions, and knowledge inferred from existing queries or dashboards to give agents a richer understanding of enterprise data. Databricks' Genie Ontology is one example of this approach.

Even when a platform brings these sources together, data teams still need to decide which definitions to trust, resolve gaps, and make the approved knowledge available to each agent.

What context engineering looks like for data engineers

For data engineers, context engineering involves making source schemas, lineage, modeling conventions, business definitions, and validation targets available to coding agents as they build data pipelines.

A data team also needs to establish which sources are authoritative and flag missing or conflicting business rules before an agent turns them into transformation logic.

Context engineering vs prompt engineering

Prompt engineering shapes instructions, while context engineering determines the additional knowledge available when the agent follows those instructions. For example:

Build a dbt model showing daily inventory availability.

The agent needs to know whether reserved inventory counts as available, which system owns stock status, and how corrections affect historical reporting.

A clearer prompt helps, but the agent still needs those business rules to build the right model.

As Anthropic explained in September 2025, the challenge becomes more significant with agents that perform multiple tool calls and work across long-running tasks. Context must be maintained and refined as the agent progresses.

The four techniques of context engineering

In July 2025, LangChain organized context engineering into four broad techniques:

• Write: Preserve decisions and requirements outside the active context window.
• Select: Retrieve what the current task needs, such as stock-status definitions for an inventory model.
• Compress: Summarize information without losing constraints or decisions.
• Isolate: Separate tasks into distinct contexts, passing findings between agents.

Context for agents querying data and building data pipelines

Context for agents consuming enterprise data

An analytics agent answering a retention question needs table relationships, metric definitions, dimensions, and approved business terminology. A semantic layer and catalog can help it select data and generate the appropriate query.

In June 2026, dbt Labs described several practical context engineering patterns, including documenting metric exclusions, model grain, lineage, and data freshness.

Context for agents building data pipelines

A data engineering agent has a different responsibility. It must translate requirements into transformations, models, tests, and other implementation artifacts.

Agents also need the rules behind a new model. Those rules may be scattered across ERP configurations, dashboards, spreadsheets, and stakeholder knowledge.

A CRM account might represent an entire corporate group, while the ERP tracks each subsidiary as a separate billing entity. Before joining those records, the data team needs to map the relationships and agree on the grain of the new data pipeline.

An agent querying existing data can use the definitions and mappings the data team has put in place. But when a coding agent is building a new data pipeline, those definitions might still be missing, contradictory, or waiting for someone to make a decision.

Four categories of context a coding agent needs

A useful context package brings together four categories of information.

Even a well-organized context package can't supply a business rule the team hasn't agreed on.

When a coding agent encounters an open rule, a plausible assumption can become transformation logic, tests, and downstream metrics. Material decisions need an owner and a stop condition before implementation.

How to prepare and deliver context to coding agents

Gather the evidence and establish authority

Start with the use cases the data pipeline must support. Identify the models, source systems, business processes, and expected outputs involved.

Gather existing code, source-system documentation, spreadsheets, dashboards, transcripts, and trusted outputs. Confirm missing decisions with people accountable for the business process.

Source documentation and process documents can tell you how a system works or how a team operates, but you'll still need to confirm which business rules the new data pipeline should follow.

For material decisions, capture the rule, its supporting evidence, the responsible owner, its approval status, and any relevant effective period.

Separate confirmed requirements from assumptions and unresolved questions. If two sources conflict, record the disagreement rather than selecting whichever definition seems more convenient.

For a more detailed approach to gathering and validating business requirements, see our guide on how to gather data requirements.

Turn approved requirements into an agent-readable specification

Giving a coding agent a pile of documents leaves it to sort through conflicting statements and decide which requirements to follow.

Returning to the Shopify example, here's an illustrative context package for a daily delivery model.

Example: structured context for a daily delivery model

Illustrative project requirements based on documented Shopify fields.

The stop condition lets the agent prepare staging models without choosing a delivery timestamp or reporting timezone on its own.

A full data pipeline specification may also contain source precedence, transformation rules, entity relationships, mappings, exclusions, history behavior, acceptance tolerances, and references to supporting evidence.

Include only requirements that affect implementation or validation.

Choose how context reaches the agent

Project instructions and rules files establish engineering conventions. Files such as CLAUDE.md, AGENTS.md, and Cursor project rules can describe repository structure, preferred modeling patterns, naming conventions, commands, and testing procedures.

Reusable agent skills package procedures for source inspection, modeling, and testing.

Project-specific specifications carry approved definitions, business rules, and validation targets for the current build.

Retrieval systems bring in relevant documentation without loading the entire collection.

Tool integrations and MCP servers let agents inspect schemas, query samples, and execute checks, subject to permissions.

A project rule might tell the agent to load the relevant spec, retrieve documentation when necessary, and run validation before proposing changes.

Keep reusable engineering instructions separate from the business rules for each data pipeline. Otherwise, a rule intended for one build could be applied to another.

We've covered the distinction in detail in A coding agent skill is not a substitute for business context.

Managing context throughout agentic data pipeline development

A data engineering agent may inspect dozens of files, execute queries, retrieve documentation, and revise models over several sessions. Meanwhile, stakeholders can clarify requirements or change business rules.

Context rot and the limits of large context windows

Context rot describes the deterioration in an AI model's ability to use information reliably as its input context grows, even when the information fits within the supported context window.

In July 2025, Chroma published a study of context rot across 18 language models. The researchers found that performance on controlled retrieval and related tasks became less reliable as input length increased. Distracting information also affected how successfully models identified relevant content.

For data teams, loading every transcript and outdated glossary can introduce irrelevant or contradictory definitions. Several related risks are worth distinguishing:

• Stale context: Outdated rules or system configurations.
• Context clashes: Competing definitions of the same concept.
• Context poisoning: Incorrect or malicious content reused as reliable evidence.

Shortening the context won't fix outdated rules, conflicting definitions, or missing decisions.

Compress context without losing decisions

Compaction summarizes a long agent session so work can continue in a shorter context window. It can also lose decisions and constraints that determine what gets built.

A particularly relevant example comes from an October 2026 experiment published by dbt Labs.

The team compared four methods of supplying context to an AI skill that produced customer-call summaries. Across 80 runs, their engineered-summary approach reduced average cost by 48.5% and runtime by 43.2% compared with retrieving full transcripts directly.

The results also revealed a trade-off: on the decisions criterion, the engineered-summary variant scored 0/20, compared with 16/20 for the full-transcript control.

The experiment tested customer-call summaries rather than data pipeline generation. It shows why teams should check what compaction preserves, alongside any token savings.

For a coding agent, a compressed project summary should preserve material decisions, their status, exceptions, supporting evidence, and unresolved questions. A shorter summary that drops the reporting grain or source-authority rule can be more damaging than a longer context package.

Keep context scoped, current, and reviewable

Update project specifications when requirements change. Keep reviewed specifications under version control alongside the code. When a rule changes, record who approved it, when it takes effect, and which models need rebuilding or retesting.

How to evaluate whether context engineering is effective

A data pipeline can compile, respect data types, satisfy uniqueness constraints, and still implement the wrong business rule. Tests derived from the same mistaken interpretation may also pass.

dbt, for example, supports both generic data tests and custom SQL assertions. Those tests are useful, but the team still needs to agree on the business rule each assertion should enforce. (Source: dbt data testing documentation)

Evaluate the agent's decisions and the resulting data

• Source and definition selection. Did the agent use the correct systems, relationships, time fields, and approved definitions?
• Business-rule fidelity. Did the resulting transformations implement the specified grain, calculations, exclusions, mappings, and exception handling?
• Handling of uncertainty. Did the agent flag unresolved decisions rather than inventing rules?
• Independent reconciliation. Does the output reproduce expected results established outside the generated data pipeline? Compare relevant totals, record-level examples, and business slices against trusted reference data, using agreed tolerances.
• Maintainability and change behavior. Can reviewers connect transformations to approved requirements and identify models affected by a changed rule?

Generating an expected total from the same transformation logic merely checks that the calculation agrees with itself.

Build a repeatable context evaluation

Choose a representative data pipeline task with documented requirements and an independently established expected result. Run the same data engineering agent under comparable conditions with different context packages, holding the model, tools, and task instructions constant as far as practical.

Score the resulting outputs against a fixed rubric. Useful measures include source selection, rule adherence, unresolved-decision handling, reconciliation variance, number of correction cycles, and token consumption.

Include edge cases such as late-arriving records, duplicate events, and backdated corrections, which reveal whether the agent followed the intended rules.

For a stronger evaluation, repeat tasks across several runs and review the results without relying exclusively on the agent's own explanation of its work.

Context engineering with Autopilot Spec

Maxa Autopilot Spec runs a business discovery process to create specs for agentic data pipeline development.

Autopilot Spec is an agentic spec builder for data developers using data engineering agents to build agentic data pipelines. It turns scattered business requirements into a structured, reviewable specification that captures the logic and decisions behind a particular build.

The workflow begins with business discovery. Autopilot Spec works with materials such as meeting transcripts, documents, spreadsheets, and dashboard screenshots. It draws on knowledge of source systems, business processes, and data modeling to identify requirements and investigate gaps.

Rather than waiting for an incorrect assumption to surface during evaluation, Autopilot Spec examines those materials for conflicting rules, missing details, and decisions requiring human input. It asks targeted follow-up questions and captures the answers before finalizing the specification.

Autopilot Spec packages the specification in agent-readable Markdown for use with coding agents such as Claude Code, Cursor, Codex, Copilot, Snowflake CoCo, and dbt Wizard. An agent can then build against reviewed requirements instead of interpreting conflicting or incomplete source material on its own.

For developers, the benefit extends beyond the initial handoff. Clearer requirements can reduce repeated stakeholder questions, SQL and dbt rewrites, and prompting cycles. When business rules change, the specification helps developers identify which models and tests may need updating.

Autopilot Spec complements data engineering agents, semantic layers, and warehouses by supplying reviewed, project-specific business context.

Explore Maxa Autopilot Spec to turn complex business context into an agent-ready specification.