ChatGPT got a D on my last labor economics midterm. You could respond, “AI is overrated now, but amazing results are coming soon,” but I’m skeptical. I take base rates very seriously, and the base rate for new techs grossly overpromising and underdelivering is at least 95%. Kahneman’s maxim that “Nothing is as important as you think it is, while you’re thinking about it” is relevant as well. To my eyes, AI fandom looks more like a mass hysteria than a sensible reaction. While AI could be the exception that proves the rules, I doubt it.
Matthew Barnett, who researches trends related to AI at Epoch AI, thinks I’m wrong - and he’s confident enough to bet. Here are the terms we’ve worked out.
By January 30, 2029, Bryan Caplan will give his six most recent midterm exams to an AI selected by Matthew Barnett. The AI will be instructed to take those exams.
Bryan will then grade the AI's work, as if it were one of his students. The AI will be allowed to do each exam only once.
If the AI gets an A on at least 5 of out 6 of those exams using same grading scale as his students, then Bryan owes Matthew $500. Otherwise, Matthew owes Bryan $500. For the purpose of this bet, an A- counts as an A.
Matthew will prepay the $500 in January 2023; the preceding terms have been pre-adjusted to compensate Matthew for expected inflation.
If Matthew suspects that an exam was flawed or grading was unfair, he can appeal to Alex Tabarrok, or another economist agreed upon by both parties, who has final authority to exclude an exam from the pool and replace it with Bryan’s most recent preceding midterm.
If more than four exams in total are excluded, the bet is called off and Matthew receives his $500 back.
If either party is unable to comply with the terms due to death or incapacity, Bryan’s heirs keep the $500.
Matthew has already paid me, so the bet is on!
The real question is whether the AI’s exam performance means anything at all. Studies show very little overlap between what AIs do in school and the skills they actually need on the job.
If your goal is to actually identify breakthrough technologies even slightly ahead of the curve, then I don't think it's helpful to apply base rates, for this exact reason. You will always predict “no”, you will be right 95+% of the time, and you will miss every transformative technology until it's too obvious to ignore.
I think AI is on a strong trajectory to be extremely useful, but I'm not sure I would take this bet. “Passing exams” is not an economically useful function (except to students who want to cheat?) and it's not clear to me that AI will be engineered or optimized for this. If you picked something with a clear economic value, like generating marketing copy or writing scripts for TV and movies, I would be much more likely to take the bet.