October 5, 2026 · 7 min read
SNEWPapers Inaugural Blog Post!
Contents
What is SNEWPapers?
Hello, World! I'm Brett, the creator and principal engineer behind SNEWPapers.
At SNEWPapers we are building the most accurate extraction pipeline for historical newspapers that the world has ever seen. A Rosetta Stone of sorts, for an elusive dataset. To prove it, we are in the process of extracting the entirety of the Library of Congress' Chronicling America data. You can follow our progress here. We are also building the most advanced AI Super Intelligence tools in town (and probably the first ever for this data) for you to dive in. Here's how to connect ChatGPT, Gemini, Claude or Grok to our data.
Why does this matter?
You may or may not be aware that there are more than a billion images of historical newspapers out there in the U.S. alone, or that hundreds of millions of dollars have been spent on preserving and digitizing these images. About 25 million of these images are in the Library of Congress' Chronicling America collection, funded mostly by NEH grants to individual States through the National Digital Newspaper Program.
In 2026 as we're in the dawn of the Super Intelligence era, you'd probably assume that someone has figured out how to extract the content of these newspapers?
You would be wrong, by-and-large, nobody knows what's in them.
It's quite a tricky problem actually. Many others have tried and many have fallen short, especially at large corpus and time scale. Chronicling America's data is perfect as a proving ground for any technology claiming to have solved this problem, as its 25 million pages spans roughly 230 years and has every conceivable layout, simple to complex, as well as every conceivable scan and/or preservation quality, from pristine to completly unusuable.
What will we be posting in our Substack over time?
Our primary objectives are:
- to show examples of interesting content that can be found by connecting your AI tools to SNEWPapers, like our Decade by Decade series
- examples of deep-dives on a variety of interesting topics from the 1730s to the 1960s (corpus range), hopefully inspiring others to dive in as well and share with the rest of us, for example 1793's yellow fever epidemic, in real time
- demonstrations of the types of research that would be virtually impossible without SNEWPapers
A little bit more about SNEWPapers
It took more than 4,000 hours of research and development to figure out how to go from the raw images below, to the pretty segmented picture beneath it:
What you're looking at there, is a color coded representation of the correct segmentation of the entire issue into a taxonomy of 33 "types", e.g. "domestic news", "editorial", "obituary", "filler", "poem", with thousands of sub-types (and in some cases themes) enabling unprecedented faceting. The components which span columns and pages are linked together correctly. Each component is transcribed and touched up by AI to fix OCR errors, but most importantly, we've inverted the unit-of-retrieval most common with this data from "find me a page with this word on it", to "find me individual components, even if they span pages, matching semantically and/or by keywords."
To clarify what I mean by "inverted the unit-of-retrieval", if you go to the Library of Congress to search, you can look for words on a page. The LoC and many other newspaper archives use very old technology that scans an entire page and finds words on it. It looks like this. The text is often garbled and innacurate, they don't know if it's part of an ad, story, editorial or anything like that, or whether is spans columns or pages. It's kind of like searching for a needle in a haystack. SNEWPapers deciphers the complex layouts, finds the individual components, and returns those to you. In SNEWPapers, you would just have a normal chat and ask something like, "find me every article you can about the Lincoln assassination and conspiracy", or "what have you got about the 1793 yellow fever outbreak?". You'll get a list of articles with summaries (and full text if you want) and citations.
Correct segmentation at the component level is what makes all this possible. This is just the tip of the iceberg though: We've designed our system explicitly to be used by AI Tools like ChatGPT, Gemini, Claude and Grok. Here's how to connect your favorite AI tool in 5 minutes. This enables not only hardcore researchers, but ordinary people like you and I to analyze vast amounts of data at an orders-of-magnitude reduction in time.
As far as we know, nothing else exists at this fidelity (component segmentation and text accuracy) and scale. You don't need to take my word for it though, connect your AI to our data, ask it what the SNEWPapers connector is capable of, investigate a particular topic, then ask your AI (which was trained on most of the internet), if it's ever seen anything like this.
Somebody else must have tried? Prior Art
I'll focus mainly on another segmentation effort as that's the #1 objective behind SNEWPapers.
The most notable prior segmentation attempt on this corpus is the 2023 Harvard project American Stories. If you like stats and deep dives and want to see how SNEWPapers stacks up against it, a much more extensive comparison can be found here.
The data in American Stories is not available to the common consumer. You need to know how to code and navigate vast amounts of data to even begin to analyze their results. But don't worry, we've done it for you.
Excerpt from their Abstract
Existing full text datasets of U.S. public domain newspapers do not recognize the often complex layouts of newspaper scans, and as a result the digitized content scrambles texts from articles, headlines, captions, advertisements, and other layout regions. OCR quality can also be low.
Excerpt from the paper
Chronicling America does not recognize oftentimes complex newspaper layouts, and so digitized texts are provided at the page level, often scrambling headlines, articles, advertisements, captions, and other content regions together. Because a non-trivial share of the underlying scans are illegible, incoherent texts are prevalent, with illegibility varying across space and time
American Stories was in pre-llm land, and uses something called convolutional neural networks (CNN), which is essentially visual pattern matching for page regions. They primarily attempted to use this to look for articles, headlines, bylines, advertisements and pictures.
A picture and some data might explain it better. Here's a nice 1920s issue, well preserved and formatted with a distinct layout and nicely sized headlines and bylines. It might be slightly difficult to understand, but the American Stories one is color coded for mostly headline and article (body), and SNEWPapers is color coded by content type of the component (more colors). Dotted lines are reading order of the component, solid lines are connections between pages (only SNEWPapers does this). If you look at the Decembrists story on the bottom of page one that continues on to page four, this is all one component. American Stories has it as ten separate articles.
A comparison of the text is useful as well:
American Stories, just the first bit, it's a long story...
WE could not expect a prOleta :, " ridn class ideology, ace could not demand 1t from the people Of those days. The working class only began to form in the days Of the Deccmbrists and therefore there could be no proletdridn class party. =. The proietdridt must and docs appreciate the Decem- brists toho dared to Iedrc their privileged class, toho sacrificed their Iires IN those days tChen the prolctdriat did not get erist." C. ZinorieU. "Historg Of the Russian Communist Pdrty.D ON July 10, 1926 in the courtyard of ~ Petropavlovsk fortress IN. Lenin- grad (then st. Petersburg) were hanged p. /. Pestel, X. F. Rylyeyev, p. G. Kakhovsky, M. A. Bestuzhev- Ryumin, and s. M. Muryavyev-Apos- toll
SNEWPapers, also just the first bit (and demo of page connections)
Decembrists Put to Death in Czarist Russia 100 Years Ago
"WE could not expect a proletarian class ideology, we could not demand it from the people of those days. . . The working class only began to form in the days of the Decembrists and therefore there could be no proletarian class party... The proletariat must and does appreciate the Decembrists who dared to leave their privileged class, who sacrificed their lives in those days when the proletariat did not yet exist."
โG. Zinoviev, "History of the Russian Communist Party."
On July 13, 1826 in the courtyard of Petropavlovsk fortress in Leningrad (then St. Petersburg) were hanged P. I. Pestel, K. F. Rylyeyev, P. G. Kakhovsky, M. A. Bestuzhev-Ryumin, and S. M. Muravyev-Apostol.
....
we could see quite well their faces. They were all calm, but their (Continued on page 4.)
Decembrists Put to Death in Russia in 1825
(Continued from page 1)
faces were serious as if they were thinking about a very important matter.
....
The issue above, is well formatted and has distinctive headlines and things which the CNN vision model can latch on to. It's worth keeping in mind however, that Chronicling America is mostly centered around 1800s content and lots of it is not structured that well. Those nice structures are roughly speaking found from about 1900 onward. Prior to that, ads in particular mostly just look like text and in general it's often the case that components have no extremely well defined headlines, bylines etc.. Here's a more typical 1800s example where the page structures are harder to detect:
Many pages look exactly like this one. This should make it a little more clear: Pattern recognition alone is entirely insufficient for tackling this data. Giant blobs of text get lumped together, multiple unrelated components end up in the same "article", the newspaper dateline gets attributed to an article, and so on and so forth.