We Can’t Know Our A.I. Future if We Don’t Study It

Every few months, the public is offered a new prophecy about artificial intelligence. It will cure disease or destroy the job market. It will supercharge education or hollow out the capacity to think. It will make governments more efficient or make democracy ungovernable. What almost all of these predictions share is a striking absence of evidence.

That absence is not inevitable. It is a choice u2014 one made by companies that release powerful systems without meaningful outside scrutiny, by policymakers who legislate faster than they learn, and by a research ecosystem that has been given only fragmentary access to the data it would need to answer the questions that matter most.

The questions we are not answering

Consider how little is settled about technologies now used by hundreds of millions of people. Does routine reliance on chatbots improve or degrade students’ reasoning? What happens to the quality of medical decisions when clinicians consult an A.I. assistant? Which categories of work are actually being automated, and which are simply being reorganized? Do conversational systems meaningfully shift users’ political views, purchasing behavior or emotional well-being over months and years, not minutes?

These are empirical questions. They can be studied with the ordinary tools of social and behavioral science: randomized trials, longitudinal panels, audits, natural experiments, careful measurement. What they cannot be answered by is anecdote, marketing copy or the intuitions of people with a financial stake in the answer.

Why the research isn’t happening

Part of the problem is structural. The most consequential systems are proprietary. Their training data, usage logs and internal evaluations sit behind corporate walls. Independent researchers who want to study real-world effects often must rely on simulated conditions or scraped fragments, and those who probe too aggressively can find themselves in violation of terms of service.

Part of the problem is speed. Models are updated, deprecated and replaced on timelines far shorter than a peer-reviewed study. A finding about last year’s system may be obsolete before it is published u2014 a genuine methodological challenge, but not an excuse for giving up. Medicine studies treatments that evolve. Economists study economies that change underfoot. The answer is better methods, not fewer.

And part of the problem is funding. Enormous sums flow toward building these systems. Comparatively little flows toward understanding what they do once released.

What would help

A serious research agenda would require a few unglamorous things. Legally protected access for vetted independent researchers, with privacy safeguards, so that studying widely deployed systems is not a liability. Sustained public funding for work that no company has an incentive to commission. Standardized reporting, so that claims about capability and harm can be compared rather than merely asserted. And a norm u2014 borrowed from pharmaceuticals and aviation u2014 that deployment at scale carries an obligation to monitor outcomes.

None of this settles the underlying political arguments about how A.I. should be governed. Reasonable people will continue to disagree about risk tolerance, competition and speed. But those disagreements would be far more productive if they were arguments about evidence rather than about competing visions.

The honest position today is uncertainty. That is not a comfortable thing to admit in a debate dominated by confident forecasts. It is, however, the necessary starting point. We are conducting an unprecedented experiment on a global population. The least we can do is collect the data. Read More