Explorations · Essay · AI & evidence

AI Is Not a Static Treatment

Why an estimate of the effect of AI depends on which AI, used how, and for how long.

In early 2025 the research group METR ran an experiment that surprised nearly everyone. Sixteen experienced open-source developers worked through 246 tasks, each randomly assigned to allow AI tools or forbid them. The developers expected AI to cut their time by 24 percent. With AI allowed, tasks took 19 percent longer.

That August METR began a follow-up with newer tools and more developers. For the ten who returned, the estimated time saving was 18 percent, though the uncertainty interval ran from 38 percent less time to 9 percent more. METR warned that it did not trust the new number. Developers were declining to participate, or withholding tasks, because they did not want to work without AI. Pay was lower too, and concurrent agents made time harder to measure.

The follow-up interests me as much as the original result. The tools had changed, and so had the people and tasks the study could reach. We usually ask what the effect of AI is, on productivity or learning, as if AI were one fixed thing a person either uses or does not. What happens when none of that holds still?

Start with the tool, which is more than the model. Bastani and colleagues gave Turkish high-school students GPT-4 through two interfaces. One worked like ordinary ChatGPT and the other gave hints instead of answers. Once the tool was removed, students who had used the first scored 17 percent below the control group; those who had used the second showed almost no loss.

Changing tools, changing histories and biased comparisons are different problems. Randomizing access, with valid follow-up, can still estimate the effect of offering the tool when people adapt their use. But it does not automatically answer what sustained use would do, or what next year’s tool will do. The harder problem arises when earlier use changes something that influences both later use and the outcome.

Consider eight hundred students taking two months of algebra, with a midterm and a final exam. In month one, half use an AI tool, assigned at random. In month two anyone can choose it, and students who struggled on the midterm are more likely to do so. In this constructed example, the true effect of use in both months is known: ten points lower on the final than use in neither. That is an illustration, not an estimate of AI’s effect on real students.

The obvious comparison is students who used AI in both months against students who never did. It gives a loss of fifteen points, which is too large. The month-two users were mostly students who had struggled, and many would have done worse on the final anyway.

So I regress the final score on both months of use and the midterm. The loss shrinks to seven points. Now it is too small. The midterm was itself affected by month-one use, so holding it fixed removes part of the damage I want to measure. It also changes who is compared. Among students with the same midterm result, month-one AI users tend, in this example, to have stronger study habits, which also help on the final.

The midterm is a result of past AI use and a cause of future use. Epidemiologists call this treatment-confounder feedback, and explain why conventional adjustment can fail. Here, leaving the midterm out is wrong. Simply putting it into the regression is also wrong.

Same data, three answers Points lost on the final exam after two months of AI use, in an invented class of 800 students. Comparing students who used AI in both months with students who never did gives 15.2 points. Controlling for the midterm gives 6.9 points. The true effect, built into the data, is 10.0 points. Same data, three answers Points lost on the final exam after two months of AI use. Invented data, 800 students. Used AI in both months against never used it: 15.2 points (too large) Both months against never the obvious comparison: too large 15.2 The same comparison, controlling for the midterm: 6.9 points (too small) Controlling for the midterm the usual fix: too small 6.9 True effect of two months of AI use: 10.0 points True effect known, because the data are built 10.0
Figure: 800 invented students. The first comparison uses 450 always-users and never-users; the regression uses all 800. feedback_example.do reproduces the calculations.

What, then, do common designs tell us? An experiment with a single session and an immediate test measures a short-run effect. A study following adopters and non-adopters over time needs assumptions about how their outcomes would otherwise have changed. Even when those hold, adoption bundles together later use, adaptation and model upgrades.

Strömberg, Lei and Wu’s panel of Chinese secondary students, a working paper, shows that this bundle moves. The estimated cost of five months of use was about 25 percent of the average exam score when measured in early 2023 and about 16 percent by June 2025. The authors report that a changing mix of students does not explain the decline. The tools may have improved, or students and teachers may have adapted. The number describes adopting then, with those tools, used that way.

Better evidence starts by naming the question: which intervention, for how long, compared with what. Epidemiologists call this specifying the target trial, and Arbour and colleagues have argued for applying it to AI systems. Record the model and interface. Distinguish access from actual use. Keep a no-AI arm while one is still possible. Follow the same people, with unaided tests weeks or months later; re-randomizing access can compare different access histories.

There are also methods built for feedback. In the classroom, the g-formula recovers the ten-point loss by averaging final-score predictions over the midterm results each month-one treatment would produce. That exact recovery is constructed. In real data, it requires the relevant confounders to be measured, enough comparable people following each treatment history, and appropriate models.

The first METR result remains evidence about a particular experiment. The follow-up shows how difficult preserving a comparable experiment became; its estimate remains uncertain. Parikh and colleagues have argued, about AI and health, that exposure to AI varies over time and comes in many versions. I think the same holds for productivity and learning. A useful average effect does not require AI to have one universal effect. It requires a clear question: which AI, used how, for how long, by whom, and compared with what. The last part may be the hardest. If people will no longer work without the tool, who remains in the comparison?

This is a shorter version of longer work I have been doing to understand how the effect of AI can be measured when the tools, and the ways people use them, keep changing.

Sources

  1. Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity. arXiv. arxiv.org/abs/2507.09089
  2. Becker, J., Rush, N., Cunningham, T., Rein, D., & Mahamud, K. (2026, February 24). We are changing our developer productivity experiment design. METR. metr.org
  3. Bastani, H., Bastani, O., Sungu, A., et al. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. PNAS, 122(26), e2422633122. doi.org/10.1073/pnas.2422633122
  4. Hernán, M. A., & Robins, J. M. (2020). Causal Inference: What If. Chapman & Hall/CRC. Chapter 20. miguelhernan.org/whatifbook
  5. Robins, J. M., Hernán, M. A., & Brumback, B. (2000). Marginal structural models and causal inference in epidemiology. Epidemiology, 11(5), 550–560. doi.org/10.1097/00001648-200009000-00011
  6. Strömberg, D., Lei, V., & Wu, Y. (2026). The generative AI learning penalty: Evidence from Chinese secondary education. SSRN working paper. ssrn.com/abstract=6868618
  7. Hernán, M. A., & Robins, J. M. (2016). Using big data to emulate a target trial when a randomized trial is not available. American Journal of Epidemiology, 183(8), 758–764. doi.org/10.1093/aje/kwv254
  8. Arbour, D., Bojinov, I., Feller, A., & Ni, T. (2026). Toward causal field evaluations of AI systems. Harvard Data Science Review, 8(2). doi.org/10.1162/99608f92.7d74e33e
  9. Naimi, A. I., Cole, S. R., & Kennedy, E. H. (2017). An introduction to g methods. International Journal of Epidemiology, 46(2), 756–762. doi.org/10.1093/ije/dyw323
  10. Parikh, H., McCormick, T., Johnson, E., Hickey, L., Ranney, M., & Mukherjee, B. (2026). The epidemiology of artificial intelligence. arXiv. arxiv.org/abs/2604.14086

AI disclosure: I wrote this essay with Claude (Anthropic). The question is mine: whether the effect of AI can be estimated as if AI were a fixed treatment. Claude was used to search for relevant literature and assist in drafting and editing the article. The classroom data was simulated and analysed with the help of Claude. Codex (OpenAI) helped fact-check and revise the draft. The judgments in the piece, and any errors, are mine.

The classroom example uses invented data. The unadjusted comparison uses 250 always-users and 200 never-users. The adjusted estimate sums the two AI coefficients from a regression on both months of use and the midterm, using all 800 students. The g-formula standardizes the final-score means over the midterm distribution induced by each month-one treatment. It returns 56.8 for always-use and 66.8 for never-use, a ten-point loss.

← Back to Explorations