Archivists Turn to LLMs to Decipher Handwriting at Scale
Archivists are increasingly using general-purpose AI models to transcribe handwritten documents at scale, a task that once required paleography training, custom software or weeks of work. The author describes feeding bell hooks' journal pages to ChatGPT at a Berea College archive, then profiles researchers who have tested LLMs on archival handwriting. In one study, large language models outperformed the specialized software Transkribus on accuracy, speed and cost, turning previously hidden collections into searchable records.
Key Takeaways
- When I sat down with bell hooks' personal journals at an archive at Berea College in Kentucky, I expected an intimate peek into her private thoughts, her voice before the editing.
- Yann LeCun , who later went on to win the Turing Award for his contributions to deep learning, published landmark work on handwritten digit recognition in the 1980s that showed what was possible in narrow, controlled settings.
- But with no index and no standardization, finding an individual pensioner meant going through files at random.
The records were written by hundreds of different clerks, officers, and administrators, which ruled out the standard workaround of training a specialized model to recognize one person's handwriting.
- On documents it had not been trained on, Transkribus had character error rates of around 8 percent.
Humphries' best LLM-based approach pushed that below 2 percent, while completing the work 50 times as fast and at roughly 1/50th the cost.
- The practical consequences are already unfolding.
Stats & Key Facts
- #A professor of history and coordinator of the applied generative AI program at Wilfrid Laurier University in Waterloo, Ontario, had digitized 10 million pages of World War I pension records in Canada.
- #On a corpus of 50 English-language letters, legal records, and diary entries dating from the 18th and 19th centuries, large language models (LLMs) outperformed Transkribus , the specialized handwriting recognition software used by more than 150 major universities and archives, on accuracy, speed, and cost.
When I sat down with bell hooks' personal journals at an archive at Berea College in Kentucky, I expected an intimate peek into her private thoughts, her voice before the editing. What I got instead was frustration. Her handwriting was dense cursive, all loops that looked identical to my eye, and there were years of journals to go through.
I found myself photographing pages and feeding them to ChatGPT just to read what she'd written. My tool of choice worked well, and it turns out I'm not the first person in an archive to have figured this out. Getting computers to reliably read human handwriting, in all its variations, has challenged AI researchers since the earliest days of the field.
Researchers in the 1960s predicted machines would soon simply devour handwritten text; instead the problem spawned decades of specialized research and entire commercial industries. Yann LeCun , who later went on to win the Turing Award for his contributions to deep learning, published landmark work on handwritten digit recognition in the 1980s that showed what was possible in narrow, controlled settings. Real archives were another matter.
For more details please read the original article at IEEE Spectrum AI.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.