Temporal Systems and Forecasting · Vol. 1 No. 1 (2026)
Original publication: November 4, 2026
Mark Scott — Slovakia; Oliver Young — Malta; Ryan King — Cyprus
Exploratory internal chronological split of SWE-bench Lite. The outcome is reference-patch line count; this is not a coding-agent evaluation or an official benchmark score.
Can older issue records tell us how large later repairs will be? A four-parameter proxy fitted to 105 SWE-bench Lite pull requests through 2019 barely improved on a constant when applied to 195 later tasks. Mean absolute error in log edit size fell by 0.0062, with a repository-clustered interval crossing zero. A later-window bias contrast was also uncertain and rested on six repositories observed in both windows. The result limits the practical role of these inexpensive descriptors without establishing that newer coding tasks are intrinsically harder.
A coding assistant is expected to change an appropriate amount of code, not merely produce a plausible continuation. Before evaluating that behavior, researchers often need a simpler description of the repair task: how much changed, where it changed, and whether the information available beforehand could have anticipated its size. An issue's length or formatting is inexpensive to obtain, but inexpensive metadata may provide little information about the resulting patch.
The question examined here is whether an issue-record proxy fitted to older pull requests remains informative about the line-count size of later reference repairs. We analyze SWE-bench Lite Jimenez et al., 2023, using its public task records Princeton NLP, 2026 as an observational source. The outcome is the logarithm of one plus added and removed non-test patch lines. It describes the reference artifact, not the assistant's behavior, semantic completeness, reviewer satisfaction, or the minimum change necessary to solve a problem.
Tao and Kamata, 2026 study fine-tuning coding assistants from pairwise preferences about what to change. That research makes the definition and availability of change-related information especially relevant. Our analysis does not implement its training procedure or construct human preference labels. It examines a more modest ingredient in a possible evaluation workflow: whether three readily encoded record descriptors provide a useful chronological baseline for reference edit size.
Two propositions guide the empirical design. A metadata proxy should show an advantage over a declared constant reference on later records before it is treated as useful for planning. In addition, apparent temporal changes should be separated from changes in the repository composition of those records. Neither proposition presumes that performance must deteriorate over time. Weak improvement, inconsistent yearly errors, and uncertain period contrasts remain admissible results.
Mining a software repository requires attention to how records are created and selected Bird et al., 2009. A benchmark task pairs an issue with a reference solution under benchmark-specific eligibility rules; it is not a random observation of all developer work. Misclassification can affect software prediction studies Herzig et al., 2013, and the expectations surrounding code review extend beyond the final text of a patch Bacchelli and Bird, 2013. These considerations constrain what line-count forecasting can establish.
Lines changed are also a restricted definition of edit scope. Structural complexity McCabe, 1976 concerns properties that a line count does not directly encode. A short edit can have a large behavioral effect, while a long edit can contain repetitive changes. Test-based reference support has its own limits, as emphasized by the oracle problem Barr et al., 2015. We therefore retain a literal, reproducible count without relabeling it semantic difficulty, repair correctness, or a developer's preferred amount of change.
Other public coding resources provide different empirical units. Model evaluation on coding problems Chen et al., 2021, crowd-sourced programming tasks Austin et al., 2021, paired buggy and corrected functions Lin et al., 2017, and more extensive tests for generated code Liu et al., 2023 cannot simply be substituted for dated repository repairs. Those resources motivate broader future comparisons, but none supplies additional observations to the current regression. The chronological evidence comes only from the captured Lite records.
The local snapshot contains 300 complete task records from twelve repositories. We read the three acquired JSON pages and require unique instance identifiers and no truncated cells. Source documentation defines created_at as the creation date of the solution pull request. It is not the original issue-opening time, the date a developer first recognized the problem, or the execution date of an evaluated coding model. The distinction is essential to the meaning of temporal transport.
We fit the proxy on the 105 tasks dated through 2019 and evaluate it on the 195 tasks dated from 2020 onward. The fitting period begins on 20 September 2012 and ends on 31 December 2019. Later records extend from 1 January 2020 to 17 July 2023. All records originate in the benchmark's public test split; our internal division does not create an official train-test benchmark protocol or establish that a pretrained model never encountered these repositories.
The older subset contains seven repositories and the later subset eleven. Five later repositories are absent from fitting: Matplotlib, Seaborn, Flask, Pylint, and Sphinx. The proxy therefore contains no repository categorical predictor requiring arbitrary assignment of an unseen level. We keep all later observations in the primary loss comparison, rather than discarding repositories that make transport harder. The later-period composition is part of the practical scope of that comparison.
Predictors are log one plus the issue's regex word count, an indicator for a triple-backtick fence, and an indicator for nonempty hints. The source defines hints as issue comments before the solution pull request's first commit. Such comments need not be available when the issue first opens. Consequently, this is a retrospective descriptor of the pull-request task record, not a validated prediction service operating at issue creation. Empty hints are treated as an observed field state, not proof that no useful discussion existed.
An ordinary least-squares fit relates log edit size to the three predictors and an intercept. There is no hyperparameter search, repository-specific coefficient, model-generated patch, or test-patch predictor. Its fitted coefficients are frozen before computing later losses. Chronological evaluation principles Tashman, 2000 motivate keeping fitting and assessment periods distinct, while dependence-aware validation Roberts et al., 2017 explains why the later task count alone should not determine inferential precision.
The constant reference is the older sample's mean log edit size. It corresponds to an intercept-only squared-loss fit. It is not claimed to be the absolute-loss-optimal training median. The point-forecast literature Gneiting, 2011 makes this alignment between a fitting objective and a scoring objective consequential. We deliberately retain the declared constant and evaluate both predictions under the same later absolute error on the log scale, without replacing the reference after seeing the outcome.
For each later task, the primary contrast subtracts the constant's absolute log error from the metadata proxy's error. A negative value favors the proxy. Model 1 estimates the equal-task average contrast for all 195 later tasks. Repository-clustered covariance uses eleven groups, small-sample correction, and a t reference with ten degrees of freedom Cameron and Miller, 2015. Robust covariance methods White, 1980 address variation in residual scale, but they do not make eleven repositories a representative population of software projects.
The secondary question concerns signed error, defined as predicted minus actual log edit size. We contrast 2022-2023 with 2020-2021 after repository effects. Only six repositories appear in both windows. Model 2 therefore uses their 173 tasks, retaining those with direct within-repository temporal comparison and excluding twenty-two tasks only from this model. It uses six repository clusters and t5 intervals. The loss contrast and period contrast intentionally have different denominators and answer different questions.
Figure 1. Older pull-request records fit a frozen metadata proxy; later gold edit counts define loss contrasts, while six bridging repositories support a separate period comparison.
Time-series cross-validation results Bergmeir et al., 2018 do not automatically validate this repository sample or permit random mixing of its eras. Similarly, selection-bias concerns in model evaluation Varma and Simon, 2006 favor keeping the simple formula fixed rather than searching many metadata combinations on later tasks. Figure 1 makes the separation explicit: later gold patches score the already fitted proxy and never become predictor inputs.
The older fit has R-squared 0.014706. Its intercept is 2.243283; coefficients are -0.077501 for log issue words, 0.039911 for a code fence, and 0.190870 for nonempty hints. These values offer little in-sample explanation of reference edit-size variation. The negative length coefficient is recorded without a narrative that longer issues cause smaller repairs. It is a conditional coefficient in one selected older subset.
Across the 195 later tasks, mean absolute log error is 0.648835 for the proxy and 0.655064 for the constant. The mean difference is -0.006229, with clustered standard error 0.003654 and interval -0.014370 to 0.001912. The proxy's point estimate is slightly better, but the interval crosses zero. That is insufficient evidence for a dependable planning benefit and is not an equivalence finding or proof that all issue metadata are uninformative.
Table 1. Later-period error contrasts from frozen predictions. M1 is an intercept-only mean of paired absolute-log-error differences, N195 with eleven repository clusters and t10. M2 regresses signed log error on a 2022-2023 indicator and repository effects, N173 in six repositories present in both windows, with six clusters and t5. Only the period coefficient is displayed for M2; no era-causal interpretation is intended.
| Model | Term | Estimate (SE) | 95% CI | N |
|---|---|---|---|---|
| M1 | Mean outcome/contrast | -0.0062 (0.0037) | [-0.0144, 0.0019] | 195 |
| M1: fit | R-squared = 0.0000 | - | - | 195 |
| M2 | 2022-2023 minus 2020-2021 window | 0.2310 (0.1102) | [-0.0522, 0.5141] | 173 |
| M2: fit | R-squared = 0.0945 | - | - | 173 |
Model labels: M1 = Held-period issue-proxy minus constant absolute log-scope error; M2 = Later-window signed error with repository composition.
The repository-adjusted later-window signed-error coefficient is 0.230967, with standard error 0.110157 and interval -0.052200 to 0.514133. Model 2 explains 0.094487 of signed-error variation in its bridging subset. The interval includes zero and allows appreciable changes in either inferential direction around the estimate. In particular, an eleven-repository count from Model 1 cannot be reused to make this six-repository comparison appear more precise.
Overlap is uneven even among those six projects. Django contributes fifty-seven earlier-window and thirty-one later-window tasks, while SymPy contributes twenty-four and nine. Matplotlib contributes just one earlier-window task against twenty-two later-window tasks. Thus the adjusted contrast is identified by a highly unbalanced set of within-project comparisons. A repository indicator removes a level difference; it does not manufacture comparable examples of every task type at both dates. The saved project-by-window counts make that limitation inspectable rather than hiding it behind the pooled task total.
The date division also does not describe continuous observation of a software system. There are no repeated measurements of the same unresolved issue as it ages, and no record of a coding assistant's predictions made in each historical year. Each later prediction is produced now by the same frozen descriptive fit. This construction can assess chronological separation of captured task records, but it cannot establish a historical degradation trajectory for a deployed model or infer changes in the source repositories' overall engineering productivity.
Later-year sample sizes are sixty-six in 2020, forty-two in 2021, fifty-seven in 2022, and thirty in 2023. Proxy absolute log errors are 0.694113, 0.521369, 0.641374, and 0.741855 respectively. Constant-reference errors are 0.685942, 0.528861, 0.666866, and 0.741394. The proxy is worse in the first and last years and better in the two middle years. This sequence does not form a monotone deterioration pattern.
Signed errors similarly change direction: -0.250094 in 2020, 0.140182 in 2021, 0.104214 in 2022, and -0.231421 in 2023. These values are on the transformed edit-count scale, not numbers of missing code lines. Average gold log scope also varies across years. We avoid exponentiating an average signed log error and calling it the average raw-line error, because nonlinear transformations do not preserve those means.
The distinction between absolute and squared error is substantive Willmott and Matsuura, 2005, and the circumstances favoring one summary over another deserve explicit consideration Hodson, 2022. Here absolute log error emphasizes proportional-scale discrepancies while reducing domination by very large patches. That choice does not establish the loss function a developer would use. A deployment study would need a separately justified cost of underestimating versus overestimating an edit budget.
The result gives a narrow answer: these three descriptors, under this older fitting sample and declared scoring rule, provide only a small and uncertain gain over a constant. It does not rule out richer representations of the issue, repository state, dependency graph, or intended behavioral change. Nor does it show that training a coding assistant on preferences fails. A line-count proxy and a preference-trained repair policy optimize different objects.
Snapshot completeness also differs from complete information. Missing-data reasoning Little and Rubin, 2002 helps distinguish an empty captured hints field from an unobserved development process. Measurement-error analysis Carroll et al., 2006 reinforces that a noisy descriptor can have weak predictive association even when the underlying task property matters. Neither consideration licenses filling unavailable discussions or treating an unmeasured semantic task complexity as if it were an observed covariate.
A calendar association cannot identify the cause of a shift. Repository practices, benchmark selection, issue mix, and repair conventions can all change alongside dates. Causal-graph reasoning Greenland et al., 1999 clarifies why a later-window indicator is not an intervention. Work on predictive uncertainty under distribution shift Ovadia et al., 2019 motivates further stress testing, but it does not convert this metadata-only exercise into evidence about uncertainty calibration of a large coding model.
Large individual losses could shape a small-sample comparison, making influence a reasonable follow-up concern Cook, 1977. We have not removed a tail or selected a different fitting formula after inspecting later outcomes. The displayed mean, year summaries, and full retained records remain available for subsequent investigation. The uncertainty should be read with the context-first interpretation encouraged by Wasserstein and Lazar, 2016, rather than converted into a claim based on proximity to a significance threshold.
An unhelpful proxy can add maintenance without solving the planning problem, connecting this result to technical debt in learning systems Sculley et al., 2015. Production-failure diagnosis by distributed tracing, the topic of Yan, 2026, similarly makes the observational context relevant, but it does not validate our issue-date or edit-size proxy. A prospective extension would need to record information available at the actual planning decision and separate repositories during validation. Dataset documentation Gebru et al., 2021 should therefore state timestamp meanings and benchmark filters rather than treating every dated record as a commensurate operational event.
An older issue-record fit transferred with a very small average advantage over a constant in later reference edit sizes. The uncertainty, nonmonotone yearly errors, and limited repository overlap prevent a strong temporal-performance claim. This evidence favors treating cheap issue descriptors as a transparent baseline to be challenged, rather than as an established measure of what a coding assistant should change.