Journal of Transfer and Generalization · Vol. 1 No. 1 (2026)
Original publication: November 1, 2026
Alexander Collins — Poland; Blake Reynolds — Hungary; Evan Anderson — Croatia
Bounded execution of reviewed public benchmark functions. No coding assistant was trained or asked to generate a repair.
A benchmark test suite can expose a buggy program while a small retained subset misses it. On twenty-six QuixBugs tasks passing a declared reference-execution and oracle-compatibility gate, a uniformly chosen single provided test detected the captured buggy implementation with mean probability 0.690. Keeping three tests raised that probability to 0.927, an average gain of 0.237. These probabilities are calculated exactly from executed test outcomes rather than estimated from repeated random subsets. Five other JSON-backed tasks were excluded by the gate, including numerical-oracle and execution-budget cases. The study concerns test retention under a fixed outcome convention; it neither estimates repair success nor treats a runtime cap as proof of nontermination.
Program-repair evaluation often summarizes whether a candidate passes a supplied suite. The summary depends on which tests are retained. A small subset can accept the original buggy implementation even when the full suite detects it. That observation does not require generating a new repair: the public buggy and corrected programs already provide a controlled way to ask how much detection evidence a reduced test budget retains.
Our research question is how much uniformly retaining three provided tests rather than one increases the chance of detecting a captured buggy implementation. We separate the execution evidence from the sampling calculation. First, each available JSON case is run on reviewed benchmark functions under a fixed execution policy. Then the resulting pass/fail pattern supplies an exact probability for each retained-test budget. The procedure involves no learned test selector or repeated random-subset approximation.
The preference-learning setting in Tao and Kamata, 2026 makes the quality of evaluation signals relevant: a comparison between coding outputs needs to specify what counts as successful behavior. Our experiment does not reproduce that fine-tuning method or create pairwise training preferences. It examines a more elementary signal, namely whether a reduced subset of existing tests still rejects the original faulty code. Its findings cannot establish how an assistant would respond to richer tests or what it would learn from their outcomes.
The resource introduced by Lin et al., 2017 contains short algorithmic programs with captured buggy and corrected versions. It was designed as a multilingual repair benchmark. We analyze only its Python functions with line-delimited JSON input/output cases. Broader repair resources such as Jimenez et al., 2023, the agent setting in Yang et al., 2024, and the approach in Xia et al., 2024 involve repository-level context and a different form of evaluation. They are relevant context, not additional tasks included in our sample.
The programming-task collection in Austin et al., 2021 and evaluation issues addressed by Liu et al., 2023 also illustrate that a test count is not a complete description of the evidence tests supply. A case may be redundant with others, exercise a special boundary, or reject an output that another semantic convention would accept. Our exact retention probabilities account for the observed number of rejecting tests, while leaving semantic adequacy and untested behavior outside the measured endpoint.
Code-model systems in Chen et al., 2021, Le et al., 2022, Fried et al., 2022, Guo et al., 2020, and Li et al., 2023 establish relevant settings in which executable feedback can matter. None is executed here. No model-generated candidate enters the analysis, so an observed detection probability must not be labeled an assistant accuracy or repair pass rate. Keeping that distinction explicit allows a test-budget study to inform evaluation design without borrowing achievements from an unrun model comparison.
The captured official source archive contains forty main benchmark programs. Thirty-one have JSON-backed tasks under the interface used here; the remaining graph-oriented tasks are outside this design. Exact archive paths distinguish python_programs from correct_python_programs. Suffix matching would be unsafe because the corrected-directory name contains the buggy-directory suffix. We therefore select both files by their complete paths and preserve the archive hash defining the analyzed version.
The functions are parsed and screened before execution. Module contents are limited to function definitions, reviewed imports, and descriptive string literals. The available builtins exclude file access and dynamic evaluation. Reviewed standard helpers for heaps, character alphabets, and counting are supplied directly, and the repository's general test driver is not run. These controls support a bounded experiment on simple functions rather than executing a downloaded project's unrestricted initialization path.
Every call receives a copied version of its JSON arguments so that a mutating implementation cannot alter inputs supplied to a subsequent call. Generator outputs are materialized; tuple and list representations are normalized to a common sequence form. Numeric comparisons use a tolerance of 1e-10. These are evaluation conventions, not discoveries about semantic equivalence. An algorithm with several admissible outputs can require a richer oracle than exact agreement with one recorded output.
A trace counts executed lines in the reviewed benchmark source. Each call has a limit of 100,000 such lines. Hitting the cap is recorded separately from a wrong output or another exception. This operational boundary prevents an unbounded computation from blocking the study, but does not prove that the function would never terminate. Recursion-limit exceptions are likewise recorded as actual execution outcomes under the current interpreter, rather than silently combined with ordinary numerical mismatches.
Before calculating a task's retention probability, its corrected implementation must pass every captured JSON oracle within the same execution policy. This gate avoids using a suite as unquestioned evidence when the declared evaluator does not accept the supplied reference. It also creates a restricted sample. The gate can exclude computationally expensive tasks even if their algorithms are semantically correct, and it can exclude tasks whose numeric outputs satisfy a broader specification while differing from recorded values.
Of the thirty-one JSON-backed tasks, twenty-six pass the gate. The excluded tasks are knapsack, Levenshtein distance, longest common subsequence, possible change, and square root. The first four have at least one corrected call reaching the line limit. Square root has two corrected-output disagreements under the strict numerical oracle convention. We do not interpret those two disagreements as proof that the corrected algorithm violates every valid square-root specification.
Across all thirty-one tasks, 242 JSON cases are executed on both captured versions. Corrected calls pass 231 cases, reach the line budget in nine, and disagree numerically with two recorded outputs. The eligible twenty-six tasks contribute 198 cases. All their corrected calls pass by construction. The exclusion rule is reported alongside the result so that an apparently favorable test-budget summary is not mistaken for evidence covering every original benchmark task.
Repository-data methods in Bird et al., 2009 and empirical reporting principles in Kitchenham et al., 2002 motivate recording the extracted unit and transformations. Barr et al., 2015 distinguishes an executable oracle from a complete correctness criterion, while Rothermel et al., 2001 addresses ordering tests for regression testing. Our fixed retention probabilities use neither an optimized ordering nor a representative sample of production defects. Their unit is a JSON-backed short task passing the declared evaluator gate, with expected-output conventions that must remain visible.
For an eligible program, the execution record identifies how many provided cases reject its captured buggy implementation. Rejection includes wrong output, an exception, or exceeding the declared line budget. We consider uniform sampling without replacement from that program's own provided cases. The probability that a retained subset detects the bug is one minus the probability that every selected case belongs to the passing set.
The one-test probability is consequently the observed rejecting fraction. For three tests, the calculation uses the corresponding combinatorial counts of all-passing subsets and all possible subsets. Every eligible task has at least three provided cases, so the same budget is feasible throughout. We calculate the ratio directly rather than simulating many random subsets. There is no Monte Carlo standard error attached to the resulting finite-suite probability.
This exact calculation does not make the test-generation process random. The original suite was curated for its benchmark purpose, and a deployment test selector need not sample uniformly. Our probability belongs to a declared retention policy, conditional on the captured suite and evaluator. It cannot be interpreted as the probability that three arbitrary future tests will detect an arbitrary real-world bug. Such a claim would require an additional model of how tests and defects arise.
Figure 1. Reference eligibility and bounded buggy-program outcomes feed exact one-test and three-test detection probabilities.
We give every eligible program equal weight when averaging its one-test probability, three-test probability, and their difference. This prevents a program with more supplied cases from dominating the summary merely because its suite is larger. The equal-program mean asks how a randomly selected task from this fixed eligible set behaves under the declared within-task retention policy. Pooling every test across programs would instead answer a case-weighted question.
An intercept-only ordinary-least-squares regression summarizes the program-level gain. HC3 covariance is used, following the heteroskedasticity considerations in White, 1980, with a normal-reference interval. There are twenty-six observations. The exact finite-corpus mean is already known from the execution table; the interval is meaningful only under a hypothetical interpretation of programs as draws from a broader task population. It is not uncertainty from repeating the same combinatorial calculation.
Dependence perspectives in Cameron and Miller, 2015 caution that programs can share algorithm families or suite-design practices even when their source files differ. No reliable independent family-cluster sample is established here. A small set of clusters invented after seeing the result would not solve that problem. We preserve the simple HC3 summary and describe its interpretation limit instead of claiming a fully population-representative uncertainty calculation.
Table 1. Equal-program difference between exact three-test and one-test detection probabilities.
| Model | Term | Estimate (SE) | 95% CI | N |
|---|---|---|---|---|
| M1 | Mean outcome/contrast | 0.2370 (0.0337) | [0.1710, 0.3029] | 26 |
| M1: fit | R-squared = 0.0000 | - | - | 26 |
Model labels: M1 = Mean three-minus-one test detection probability.
The mean one-test detection probability is 0.68956. With three tests it is 0.92651. The mean gain is 0.23696, with standard error 0.03367 and interval 0.17096 to 0.30295 under the recorded summary model. Expressed as percentage points, the average gain is approximately 23.7. The intercept fit has effectively zero R-squared because it estimates a mean gain rather than explaining variation among programs.
All twenty-six eligible full suites detect their captured buggy implementation. Nevertheless, three-test detection remains below one on average. A reduced suite can therefore accept the original faulty version even in a resource whose complete eligible suite rejects every such version. This finding concerns loss of observed rejecting evidence under retention. It does not imply that every larger suite is adequate, that every added test supplies unique coverage, or that a passing repair is semantically correct.
The execution record distinguishes rejection types rather than presenting all failures as wrong answers. Across the full thirty-one-task execution, captured buggy calls include 105 wrong outputs, twenty-three line-budget events, twenty-two recursion-limit exceptions, eight index errors, four type errors, and nine value errors; seventy-one pass. Those totals describe the broader executed interface, while the probability regression uses only the eligible subset. Mixing those denominators would give an incorrect account of the reported average gain.
Scoring ideas in Gneiting and Raftery, 2007 clarify the role of the endpoint: the object is a probability of rejection under a selected retention rule, not a calibrated confidence that code is correct. A candidate that passes one retained suite still has untested behavior. Measurement issues in Carroll et al., 2006 apply to the evaluator itself, including its declared container normalization, numeric tolerance, and execution limits. The reference gate exposes some such limitations without proving that every remaining oracle is complete.
Validation concerns in Roberts et al., 2017 and selection-bias reasoning in Varma and Simon, 2006 become relevant if retention policies or evaluator thresholds are tuned to these same outcomes. The current study fixes one and three tests and the line budget. It does not select a budget maximizing a favorable reported gain. A future policy trained to prioritize tests would need separate tasks for policy development and evaluation, and should retain its own reference-compatibility accounting.
Missing-data principles in Little and Rubin, 2002 also apply to tasks absent from this interface or excluded by the gate. The excluded computationally expensive functions can differ systematically from retained functions in how tests expose faults. Treating their absence as random would be unwarranted. The result therefore remains conditional on twenty-six eligible tasks, not a complete benchmark comparison disguised by a smaller denominator.
Influence reasoning in Cook, 1977 could identify programs with unusually large gains, but they are not deleted here. Guidance in Wasserstein and Lazar, 2016 and concerns in Ioannidis, 2005 support reporting the actual magnitude and eligibility limits. Broader exploration over many budgets would introduce multiplicity issues addressed by Benjamini and Hochberg, 1995 and Benjamini and Yekutieli, 2001. The present exact budget contrast does not eliminate those concerns for a future search over test-selection strategies.
The experiment demonstrates that supplied-suite size and supplied-suite adequacy are distinct empirical objects. Under one transparent retention policy, three tests preserve much more of the suite's observed rejection ability than one, while still leaving a detectable evidence gap relative to the complete suite. That conclusion can guide evaluation reporting without assuming any particular repair system. It is especially useful when a study is tempted to describe a small retained suite only by its number of cases.
Documentation following Gebru et al., 2021 records exact source paths, input copying, output normalization, reference eligibility, runtime bounds, rejection categories, and the probability calculation. Wilkinson et al., 2016 motivates preserving both the case execution table and task summaries. Human review remains necessary for oracle semantics and generalization beyond these tasks. The article provides executed evidence about a test-retention decision, while leaving actual assistant improvement, repair correctness, and industrial defect coverage to designs that observe those outcomes directly.