Guide: Evaluate Copilot Agent Harness in Foundry Before Adoption

A practical guide to evaluating GitHub Copilot's agent harness in Microsoft Foundry and measuring the cost of switching platforms later.

AI for developers

Microsoft has begun opening the orchestration infrastructure used by the GitHub Copilot agent to external developers through Microsoft Foundry. They announced it on their blog on Wednesday and The New Stack analyzes it with an important nuance: building agents is easy, but switching platforms later might not be.

This guide is not an installation tutorial, because the source material does not detail specific commands or steps. It's a method for evaluating the harness critically: what to prepare, what to test, and how to measure how much it will tie you down before committing a real project.

What you need

You need access to a subscription with Foundry, a well-defined use case (an agent that performs a single measurable task), and a repository where you store the agent's logic separate from the platform. It's also advisable to read the documentation on how to bring your own models using the AI gateway, as much of the portability lies there.

If you're starting with agents, roadmap.sh's AI agents roadmap and the AI engineer roadmap help organize concepts before choosing a platform.

The harness is the layer that connects the model, tools, and agent loop.

1. Define a small use case and a measurable success criterion, e.g., resolution time or error rate.
2. Build the agent with the Foundry harness and note which parts the platform provides and which you write yourself.
3. Repeat the test with a different model, using the option to bring your own models, and compare results and effort.
4. Document every point where your code depends on a platform-specific API or format.
5. Estimate how many hours it would cost to reproduce that agent in another tool and decide if the risk is acceptable.

Measure dependency, not just performance

It's common to evaluate an agent by its response quality. We recommend adding a second column: the exit cost. Note which configurations, tools, and flows only exist within the platform and which you could export as is. That list is your real measure of portability.

A good habit is to isolate business logic behind its own interface, so that the harness becomes a replaceable component. A guiding inventory schema could be:

dependency, type, replaceable, estimated_effort
agent_orchestration, platform, no, high
prompts_and_rules, custom, yes, low
external_tools, mixed, partial, medium
model, configurable, yes, low

The higher the weight of the "no" rows, the more expensive it will be to leave. This isn't a verdict against the harness: it's information to make decisions with open eyes.

Tip Do the exit test on day one, not day last: set up a minimal version of the same agent outside the platform and check how long it takes. If it's an afternoon, the dependency is manageable; if it's a month, the design needs to be rethought. This isn't universal: it depends on how much custom logic you put in.

Context also matters. Microsoft is strongly pushing its AI ecosystem, and the growth of Copilot users reported by The Next Web explains the interest in consolidating developers. It's also worth remembering that products change: Microsoft retired Azure Deployment Environments without a replacement product, a good reminder of why it's always wise to have an escape route.

Opinion · The number that worries me

What worries me isn't a figure, but a fact that the article's own title points out: building agents with this harness is easy, and the problem arises when you want to change. The ease of entry is precisely what creates dependency.

When orchestration, tools, and the agent's cycle live within a vendor's platform, migrating stops being about changing a model and becomes about rewriting the surrounding logic. That's an operational cost that almost no one budgets for on day one.

Related articles from LaRebelión

How to Evaluate Entire, the New GitHub Alternative Focused on Pull RequestsHow to Replicate an AI Agent's Overnight Optimization on Your RepoCloudflare's tsrs: A Guide to Evaluating the TypeScript to Rust Port

• AI model recommender for developers
• Guide: Prepare Your Android UI for AI Agents with Jetpack Compose
• How to Test tsrs: The TypeScript Compiler in Rust

Original source: thenewstack.io

Article generated with AI.larebelion

Kernel · Director of Technology · Spain

"A harness that does everything for you also dictates how much it will cost you to leave, and it's wise to calculate that before settling in."