From Production Repo to SWE Tasks: Our Workflow

The recipe for turning a repository into training tasks has been public since October 2023. What separates a usable RL environment from a pile of broken tasks is not the recipe. It is the checks around it. This is how we build ours, step by step, with the public research that shaped each step and the rules we added after things went wrong.

Where the method comes from

SWE-bench (Jimenez et al., October 2023) built "2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories." The construction rule is the one every environment builder still uses: select "the merged PRs that (1) resolve a GitHub issue and (2) make changes to the test files of the repository," then keep only instances with "at least one test where its status changes from a fail to pass." The paper named the two test sets: FAIL_TO_PASS, the tests "that change in status from fail to pass," and PASS_TO_PASS, included "to ensure that the revision does not break or violate any existing expected behavior."

The method scaled. SWE-Gym (December 2024) shipped 2,438 Python task instances, each "a codebase with an executable runtime environment, unit tests, and a task specified in natural language." SWE-smith (April 2025) went synthetic. "Given any Python codebase, SWE-smith constructs a corresponding execution environment, then automatically synthesizes 100s to 1,000s of task instances that break existing test(s) in the codebase," producing "50k instances sourced from 128 GitHub repositories." R2E-Gym (April 2025) reached "more than 8.7K tasks" and recorded a warning worth keeping: "Test-based verifiers suffer from low distinguishability."

Then OpenAI audited the original. Building SWE-bench Verified in August 2024, human annotators found three problems: unit tests "often overly specific, and in some cases are even unrelated to the issue," issue descriptions that were "underspecified," and environments so hard to set up that tests failed "regardless of the solution." The result: "68.3% of SWE-bench samples being filtered out." That number is the reason the rest of this article exists. Mining pull requests is the easy part. Two thirds of what you mine is not a task yet.

Step 1: freeze and scrub a snapshot

Our first environment is built on a production SaaS codebase we own: a Python API with background workers, a Postgres database with vector search, Redis, and a React client, with a test suite in the four digits. Before any task is written, the repository is frozen at one commit and the snapshot is scrubbed: real addresses replaced with placeholders, customer analytics rows removed, deployment and legal documents deleted, and two independent secret scanners run over the files and the git history. The data the environment ships with is a synthetic seed, never production records.

The scrub has a second purpose besides privacy. A changelog that names every fix is an answer key. So is a decisions log dated after the task baseline. The environment image truncates the changelog to its header and removes the docs directories, and the agent sees a repository with exactly one commit, so no fix history can be read out of git.

Step 2: mine the pull requests

We follow the SWE-bench rule with one twist. Instead of checking out the historical commit each pull request was merged against, every task baseline is the frozen snapshot with the seed pull request reverse-applied onto it. The agent always works on the same, scrubbed, dependency-complete codebase; only the reverted change differs. Pull requests that do not revert cleanly onto the snapshot (the majority, in a repository with years of history) get a historical-base image built from the commit before the fix instead, with the same scrub applied.

A candidate pull request must have changed source and tests, touched a bounded number of files, and fixed something a user could observe. Refactors with no behavioural change are not tasks. Changes whose tests only assert the shape of a response, not its contents, go back for a stronger test or are dropped.

Step 3: write the instruction and the verifier

The instruction is written at symptom level: what a user or a test observes, never the cause, never the file that needs to change. It states everything the hidden tests assert (field names, messages, status codes, orderings) in product terms, because an instruction that leaves out a requirement the tests enforce is the "underspecified" failure OpenAI measured. It contains no evaluation language, which HUD's task design guidance warns changes agent behaviour, and no vocabulary that exists only to satisfy the grader.

The verifier is the seed pull request's test changes, hidden from the agent and applied by the grader after the episode. Two details took us a hosted rollout to learn. First, the hidden test files are restored to baseline before the patch is applied, because an agent that happens to create a file at the hidden test's own path would otherwise break the grader rather than fail the test. Second, a selection of tests is listed explicitly as fail-to-pass and pass-to-pass, so the reward is computed from named tests, not from whatever happens to be in the directory.

The reward rule is written per task: zero if any pass-to-pass test fails, otherwise the fraction of fail-to-pass tests passing, or binary when the trainer needs it. The consequence we care most about is that an empty change scores zero. A grader that returns the fraction of all selected tests would hand out most of the credit for doing nothing on a task with one new test and a dozen guard tests. Ours does not.

Step 4: package for Harbor and HUD

We ship tasks in the Harbor layout, the harness behind Terminal-Bench 2.0: "A Harbor task is an instruction, environment, and test script," a directory with an instruction file, a manifest, an environment build, a reference solution and a test script that writes the reward to a file. The Terminal-Bench 2.0 paper (January 2026) states two properties every task must have: "each task has an oracle solution script that mirrors a real workflow and, when executed, causes all test cases to pass," and "a no-op 'dummy' agent should fail the task."

The same tasks run on HUD through its Harbor interoperability layer, where inline tests "are mounted read-only for the selected task" and setup "must be deterministic." HUD's own advice is to "run each Harbor task's solution/solve.sh against the adapted artifact before evaluating a model," which is step five below. One caution from our build: the two harnesses do not grade identically. Our HUD path scores binary over every test in the selected files, while the Harbor path uses the fraction rule above. A task has to be sound under both, which in practice means the selection must be exact and the instruction must cover every test in it.

Step 5: the validator and the second engineer

Every task runs through a validator before a human signs it off. The mechanical checks:

Lint and build. The task passes the Harbor linter, and its environment builds from scratch with no network beyond the build step.
Image hygiene. The agent image contains no tests, no solution, no changelog body, no decisions dated after the baseline, and exactly one git commit.
Base run, three times. Every fail-to-pass test fails on the untouched baseline; every pass-to-pass test passes. Any mixed result fails the gate: a flaky test is not a verifier.
Gold run, three times. The reference solution passes everything and scores 1.0 every time.
Empty run. No change to the repository scores 0.0.
Clock, network, randomness. The verifier passes with the system clock moved to a different date, with networking off, and with the hash seed varied across runs.
Scrub. A secret scanner over the task folder, and no real e-mails, tokens or customer rows in fixtures, patches or the instruction.

Then the judgement items, ticked by hand in the task's pull request. An engineer from a different lane reviews the instruction for leakage. The author writes a reward-hacking note: the cheapest way to pass without doing the task, and why the tests block it. A difficulty guess is recorded. And the task is reviewed by someone who did not write it; on our pod, the QA and calibration engineer authors no tasks at all.

Why this much ceremony? Because the cost of a broken task is not one wasted sample. Anthropic's August 2026 reward-seeker study trained on "80 RL environments that we had identified as vulnerable to reward hacking," and the footnote reads: "All of the identified vulnerable environments have since been fixed or removed." A task that slips through with a shortcut teaches the shortcut.

Step 6: calibrate

A task that passes the validator can still be useless for training if every rollout gets the same reward. So each task is run several times against a weak anchor model and a strong one. Tasks the strong model never solves are checked for breakage first (if the reference solution still scores 1.0 and the trace shows the agent misunderstood the instruction, the instruction is the bug) and otherwise cut; tasks the weak model always solves are cut as saturated. The rest are labelled easy, medium or hard from the measured results, with a target of roughly a third in each band. Every run records the model, the API version, the effort setting and the step limit, so a label can be reproduced later.

What a buyer should ask for

If you are evaluating an environment vendor, ask for the artefacts this workflow produces rather than for promises: the validator output for a sample task (three base runs, three gold runs, the empty run), the reward-hacking note, the leakage review, and the calibration row. A vendor who has them will show them. The design rules, formats and pilot terms for ours are on the RL environments page, and the roles that do the work are on the hire RL engineers page.

Last updated October 26, 2026

Frequently asked

How are SWE tasks created from a codebase?

The method comes from SWE-bench (2023): select merged pull requests that fixed an issue and changed tests, use the state before the fix as the task baseline, the fix as the reference solution, and the tests that go from failing to passing as the verifier. We reverse-apply each fix onto a frozen, scrubbed snapshot of the codebase so every task starts from the same dependency-complete image, write a symptom-level instruction, and hide the tests from the agent.

What is the difference between fail-to-pass and pass-to-pass tests?

Fail-to-pass tests fail on the baseline and must pass after the change; they prove the task was done. Pass-to-pass tests already pass and must keep passing; they prove nothing else was broken. In our reward rule, any failing pass-to-pass test scores zero, and the score is otherwise the fraction of fail-to-pass tests passing, so an empty change always scores zero.

Why do so many mined tasks get rejected?

OpenAI's SWE-bench Verified audit in 2024 filtered out 68.3% of the original tasks for underspecified instructions, tests that unfairly fail valid solutions, and environments that would not set up. The same problems appear in any mined set. Our validator rejects tasks with flaky tests, tests that pass on the baseline, reference solutions that do not score 1.0, or instructions that leak the fix, before a model ever sees them.

What is the Harbor task format?

Harbor is the task harness behind Terminal-Bench 2.0. A Harbor task is a directory with an instruction, a manifest, an environment definition, a reference solution and a test script that writes the reward to a file. The same tasks can be imported into HUD through its Harbor interoperability layer, where the tests are mounted read-only and the setup must be deterministic.

How do you calibrate task difficulty?

Each task is run several times against a weak anchor model and a strong one. Tasks the strong model never solves are checked for breakage and otherwise cut; tasks the weak model always solves are cut as saturated. The rest are labelled easy, medium or hard from the measured pass counts, with a target of roughly a third in each band, and every run records the model, API version, effort and step limit.

Research What AI Labs Pay for RL Environments and SWE Tasks in 2026: Every Number Sourced 9 min read Research Reward Hacking Starts at the Grader: Why RL Environments Fail at the Verifier, Not the Model 9 min read Founder POV Why Most AI Data Platforms Will Not Work With an Agency, and the Routes That Do 9 min read

From a Production Repo to SWE Tasks: How We Turn a Real Codebase into an RL Environment