Introduction
As a software engineer or AI researcher, you might find yourself repeating the same tedious analysis over and over—sifting through thousands of lines of logs, trajectories, or output files. This was my reality when evaluating coding agent performance against benchmarks like TerminalBench2 or SWEBench-Pro. Each benchmark run produced hundreds of trajectory files, each containing dozens of steps. Reading through them manually was impossible, so I turned to GitHub Copilot to surface patterns. But even that became repetitive. The solution? Create agents that automate the intellectual toil. This guide will walk you through building your own agent-driven development workflow using GitHub Copilot, just as I did for the Copilot Applied Science team.
What You Need
- GitHub Copilot (subscription with access to Copilot Chat and inline suggestions)
- A supported IDE (e.g., Visual Studio Code, JetBrains)
- Basic programming skills (Python or JavaScript recommended)
- Sample data to analyze (e.g., JSON trajectory files from evaluation benchmarks)
- A GitHub repository to host your agents (optional but recommended for sharing)
- Understanding of your repetitive task—know what patterns you want to extract
Step-by-Step Guide to Building Your Own Evaluation Agents
Step 1: Identify Your Repetitive Analysis Task
Start by pinpointing the intellectual work you do over and over. In my case, it was examining agent trajectories—the thought processes and actions agents take while solving tasks. Each trajectory was a .json file with hundreds of lines. I would ask GitHub Copilot to highlight anomalies, then manually investigate the relevant sections. This loop—Copilot helps, I investigate—was perfect for automation. Write down the inputs (files), the desired outputs (summaries, patterns, flags), and the decision points in your current workflow.
Step 2: Set Up Your Development Environment
Open your IDE and ensure GitHub Copilot is enabled. You’ll want both inline suggestions (for code completion) and Copilot Chat (for conversational help). Create a new directory or repository for your agents. Name it something descriptive like eval-agents. Initialize a virtual environment if using Python, or set up your package.json if using Node.js. This is where you’ll write the code that turns your manual analysis into an automated agent.
Step 3: Define the Agent's Purpose and Input/Output
Clearly specify what your agent will do. For trajectory analysis, the agent might read a JSON file, extract task descriptions, actions taken, and success/failure markers, then output a summary or a list of unusual patterns. Keep it narrow at first—one agent per task. Write a comment in your code that describes the function, like: // This agent analyzes a trajectory JSON and returns a list of steps where the agent spent more than 5 attempts. This clarity helps Copilot generate the right code.
Step 4: Use Copilot to Write the Core Logic
Now the fun part. Start typing the function signature and a brief comment. GitHub Copilot will suggest the implementation. Accept, tweak, and iterate. For example, to parse a trajectory file, type:
def analyze_trajectory(file_path):
"""Parse JSON, identify steps with >5 retries, return list."""
import json
Copilot will likely complete the rest. Review the suggestion, add error handling, and test with a sample file. Use Copilot Chat for complex logic: ask “How do I compare two trajectories for differences?” and integrate the suggested code.
Step 5: Test on Real Data
Run your agent against a few trajectory files. Check that the output matches what you would have identified manually. If you find false positives or missed patterns, adjust the logic. Copilot can help you refine thresholds or add new conditions. Iterate quickly—this is where the fast development loop shines.
Step 6: Package Your Agent for Easy Reuse
Make your agent shareable. Wrap it in a command-line tool or a simple script that accepts file paths as arguments. Add a README that explains usage, dependencies, and expected output. Use GitHub Templates to let teammates fork and extend. My goal was to make agents the primary vehicle for contributions—anyone could tweak or create a new agent and share it back.
Step 7: Deploy and Integrate with Your Workflow
Run your agent on every new benchmark run automatically. Use GitHub Actions to trigger the agent when new trajectory files are pushed to a repository. This eliminates the manual Copilot-investigate loop entirely. Now you can focus on higher-level insights instead of grunt work.
Step 8: Encourage Team Contributions
Share your repository with your team. Encourage them to add their own agents for different analysis tasks—e.g., comparing agent versions, measuring task completion rates, or detecting specific failure modes. Use pull requests to review and merge contributions. Over time, your team builds a library of agents that make everyone more productive.
Tips for Success
- Start with one small agent and extend. Don’t try to automate everything at once.
- Leverage Copilot’s context—include comments and existing code so it suggests relevant solutions.
- Document each agent clearly so others understand its purpose and limitations.
- Version control your agents just like any codebase. Tag releases that correspond to specific evaluation runs.
- Encourage experimentation: let teammates create agents without fear of breaking things—use branches.
- Review agent outputs periodically to ensure they remain accurate as your data changes.
- Share learnings with the broader community—post about your agent-driven development on forums or internal wikis.
By following these steps, you’ll turn a repetitive, mentally draining task into an automated process that scales across your team. Just as I automated myself into a new role—maintaining agents instead of sifting through trajectories—you too can unlock faster insights and more creative work. For more details on the original implementation, jump back to Step 1.