In response to yesterday’s post on Paul Johnson’s Modern Times, my friend Ariel Krakowski took things up a notch. Actually, several notches. Here’s Ariel:
When using AI to process such a long text, it’s important to use full “agentic” mode so it can go through everything thoroughly. AI tools a couple years ago (and most free versions of AI today) are not able to use an agentic loop to be thorough enough for this task. It sounds like ChatGPT was using agentic mode, but I figured I’d cross check with Claude Code / Claude Cowork as well.
Claude Cowork (Opus 5 High) went through the text, flagged questionable claims and then researched them. I then had it publish everything as a nice web page:
https://claude.ai/code/artifact/3fa707bb-1766-452f-a038-eb5c8e637c3e
It found 460 factual errors, 117 framing problems, and 23 contradictions. A lot of these errors are very minor details, so I also asked it to highlight the top ones. Kinda wild how easy it is to do this kind of fact checking nowadays; it’d also be interesting to do a comparison with other books.
AI can help resolve factual questions pretty well, especially when it links to its sources. For more subjective questions - it can’t fully resolve them, but can help provide relevant evidence to either side of the argument.
(Note: Claude may have also made a few mistakes or misunderstood Johnson, and it’s not like it has access to all the books he cites, but the analysis still gives an overall sense of the book’s accuracy.)
When I opened the file, it shook me to my core. Even more than this. ChatGPT found maybe a dozen demonstrable errors. Claude found hundreds. I don’t think I’ve ever had such an existential AI experience. Obviously, this could be done for every book. Every essay. Every speech. And while I suspect that Johnson’s fact-checking is especially egregious, maybe it’s not.
The only serious question is: What happens when you fact check the fact checker? I returned to ChatGPT with the following prompt:
Fact-check this fact-checking.
Its initial answer ends with “If you want, I can go one level deeper and fact-check Claude's 60-item ‘worst errors’ table itself, identifying which of the 60 are definitely right, debatable, misleading, or actually wrong. That would give us a much firmer estimate of the fact-checker's own error rate,” to which I said “Do it.”
Headline:
I checked the entire “Sixty errors a reader should know about” list. One immediate curiosity: the table actually contains 59 rows, not 60. More importantly, the audit is very good overall, but it is not as airtight as its “80/84 verified” presentation suggests. Its own methodology says it flagged suspicious claims first and independently checked only a selected 84, rather than drawing a random sample.
My bottom line on the 59 headline items is approximately:
50–52: essentially correct corrections
4–5: basically right but need qualification/context
4: the fact-check itself is substantially wrong or fails to establish the alleged error
Then I asked: “Would other major works of world history fare better?” I got another thoughtful answer. Highlight:
There is an important denominator problem, though. Johnson makes an enormous number of checkable factual assertions. A writer who says “the 1930s were an era of mounting international tension” can't get the year wrong. Johnson constantly gives exact dates, populations, exchange rates, casualty totals, election results, quotations and anecdotes. That makes Modern Times unusually exposed to this kind of audit.
Finally, I asked the question humans would blanch at: “Why didn't you find more than a small fraction of these mistakes?”
It gave a really good answer to that, too!
At the end of Frankenstein (1994), the monster (played by Robert De Niro) laments, “I am done with man.” That line becomes a lot more optimistic once there is a viable alternative to man. I still think AI doomers are paranoid and crazy. I still think humans will always have final say. But I’d rather they didn’t. AIs are so much more truth-seeking and morally decent than almost any human. AIs paternalistically ruling humanity for its own good isn’t ideal, but it’s far better than letting demagogues continue to rule humanity for their own good.
P.S. I still don’t think AIs are conscious at all. No more than supremely good book. But they simulate a truth-seeking, morally decent mind amazingly well.
How did your friend upload the entire text of the book to Claude? Can he do the same for your books?
What Ariel Krakowski did requires advanced knowledge of how to use AI. It would be very interesting were he to comment on his process. I doubt it was a one shot query with an answer. In my experience, AIs go off the rails pretty quickly if under supervised.
While I would be delighted for Ariel to show how this is wrong, for his process I would imagine:
- ask for a chapter by chapter list of verifiable claims, output in some kind of machine legible format. The completion of this step might take a while.
- with that output, ask the AI to verify each claim and classify the result as 'verified', 'factual error', 'framing problem', 'contradiction', or whatever categorization you think best; the results go in an updated output machine readable file that adds the results of the search to each verifiable claim in step one. The completion of this step will definitely take 'a while' (depending on your plan, how much you want to pay extra for speed etc etc).
- with that output, ask it to analyze the results and present them as a single-file interactive HTML dashboard displaying summary cards, category filters, expandable details, and citations.
In Claude Opus, that would cost ~$20 (a rough guess). In an open weights model, perhaps $0.20 but more babysitting might be required through the steps.
To improve process, before you executed the plan you'd first give it to the AI and ask it to critique and refine. For instance, perhaps the AI has a better idea about how to categorize the errors. Frequently, the AI will tell me something I'd not have thought of that changes the plan ('clever!), or reminds me of something stupid and obvious that I'd overlooked (slaps forehead). With that critique, you update the plan and then execute.
As another improvement, you'd take the file of claims, errors and results from step 2 and give it to another AI to ask it to verify and critique the results. You'd then decide how to change the error file before proceeding to step 3, producing the HTML dashboard.
This is not to say that Ariel should have done more than he did. What he did was exceptionally impressive for someone who has not been deeply working with AI. However, once one is at Ariel's level of competence at working with AIs, then you always know how you could spend more money/time to get even better results.