SNEWPapers vs American Stories

What's this about?

You may or may not be aware that there are more than a billion images of historical newspapers out there in the U.S. alone, or that hundreds of millions of dollars have been spent on preserving and digitizing these images. About 25 million of these images are in the Library of Congress' Chronicling America collection, funded mostly by NEH grants through the National Digital Newspaper Program.

In 2026 as we're in the dawn of the AI Super Intelligence era, you'd probably assume that someone has figured out how to extract the content of these newspapers?

You would be wrong, by-and-large, nobody knows what's in them

This might sound straightforward, and you might wonder how it isn't a solved problem. I can assure you it's quite tricky, and that many others have tried and many have fallen short, especially at large corpus and time scale. Chronicling America's 25 million pages spans roughly 230 years and has every conceivable simple to complex layout, with scan and/or preservation quality ranging from pristine to unusuable.

This post is about two attempts at deciphering this massive dataset. SNEWPapers was created by yours truly, and we'll compare it now against another attempt, a project called American Stories.

By the end of reading this you'll see why we believe there to be over 100 million segmentation errors in their dataset.

Want to check our work? The comparison reports for all 18 issues are online: browse them here, or download them all (a 207 MB zip) to look through offline.

What is SNEWPapers?

At SNEWPapers we are building the most accurate extraction pipeline for historical newspapers that the world has ever seen. To prove it, we are in the process of extracting the entirety of the Library of Congress' Chronicling America data. You can follow our progress here. We are also building the most advanced AI Super Intelligence Research Assistant (and possibly the first ever) in town for you to dive into the data. Here's how to connect ChatGPT, Gemini, Claude or Grok to our data.

When we say most accurate, what we mean is the most correctly segmented and highest text quality. Correctly segmented means we have identified individual components on a page or across pages in an issue as being one thing. Nobody has crunched it quite like SNEWPapers.

What is American Stories?

American Stories is a 2023 Harvard project led by Melissa Dell that attempted to accurately segment Chronicling America. You can find the website here and the PDF here.

In their own words:

This study develops a novel, deep learning pipeline for extracting full article texts from newspaper images and applies it to the nearly 20 million scans in Library of Congress’s public domain Chronicling America collection. The pipeline includes layout detection, legibility classification, custom OCR, and association of article texts spanning multiple bounding boxes.

Why didn't we just use American Stories as our foundation?

The #1 overarching goal of SNEWPapers is correct segmentation and accurate transcription. We did look at their data early on in our process, and determined that there were a multitude of errors, to the point where attempting to correct them would've been more work than building from the ground up.

American Stories Process

At its core, American Stories is a pattern matching approach, they use computer vision models to identify things which look like headlines, bylines, articles, captions and pictures, classifying their legibility as well, while also attempting to avoid things that look like ads. Their pipeline includes layout detection, legibility classification, custom OCR and association of article text spanning multiple bounding box detections.

Things they do (or try to do)

• process pages individually
• not process ads
• avoid tables, as the technology at the time wasn't very good
• not process newspaper headers/mastheads
• classified detections is legible and illegible (they usually don't OCR extract what they deem illegible)
• associate headlines with articles by vertical overlap

This last one is very important as it's the source of an extremely large number of errors. Their reasoning for this is as follows:

Rule-based methods perform less well for articles spanning multiple columns or pages, and language understanding is required. However, we find on a labeled sample that only around 3.8% of articles span multiple article bounding boxes, and only 0.2% of articles span multiple pages, meaning there is little scope for gains from neural methods. (Articles spanning multiple columns and pages become more common after the Chronicling America period, as the price of paper falls and font size increases.)

We'll come back to this in a moment.

SNEWPapers Process

We also rely on pattern matching this initial stages of our pipeline, although we favor a combination of document layout analysis, bespoke vision models and heuristics as our initial structural estimator over the one-shot region detections that the Harvard folks use for classifying page components. More importantly, we utilize LLMs liberally across our pipeline. We believe that accurate segmentation is virtually impossible without them, especially when considering the issue as a whole rather than individual pages.

What we try to do:

• process issues as a whole, rather than individual pages
• go beyond the generic clases headline and article targetted by American Stories, labelling components into a vast taxonomy of top level classes, e.g. domestic/foreign news, editorial, letter to the editor, story, literary, poem and a couple dozen more. What do we actually have in SNEWPapers?
• segment ads as well, although we don't (for now) serve them from the API
• process tables, usually quite well
• ignore mastheads
• OCR all text, and touch it up with LLMs, while also collecting quality metrics to warn about low quality extracts
• gather an enormous amount of metrics and statistics at the component level, page level, issue level and corpus level
• use LLMs for many purposes, including as a feedback loop for human-reading order assumptions, segmentation, labelling and more

A closer look at some assumptions

I'd like to return to this statement from American Stories:

Rule-based methods perform less well for articles spanning multiple columns or pages, and language understanding is required. However, we find on a labeled sample that only around 3.8% of articles span multiple article bounding boxes, and only 0.2% of articles span multiple pages, meaning there is little scope for gains from neural methods. (Articles spanning multiple columns and pages become more common after the Chronicling America period, as the price of paper falls and font size increases.)

Firstly, "3.8% of articles span multiple article bounding boxes". There is a common error where the prediction box itself contains multiple articles, a partial article or 1+ articles with additional partial component text. We'll quantify these errors in a moment.

Because SNEWPapers labels things by type rather than just generic "article", it's worth looking at some stats by type. American Stories doesn't associate articles with boxes side-by-side or across pages, so all of these are lost connections. In general, someone going through this data is far more likely to want to look at things with longer, interesting narratives like news, editorials, letter to editor, literary and others. Missing those connections means the components are incomplete. So how prevalent are multiple columns and multiple pages?

Percent of text-heavy components that span 2+ columns by decade

Percent of text-heavy components that span 2+ pages by decade

So those 3.8% and 0.2% numbers, perhaps when looked at from a corpus wide assumption based on a small sample, belie the fact that with their approach, the most interesting content, and A LOT of it, is being split apart into multiple articles that would all need to be stitched back together, assuming the articles were all on the same page, segmented properly to begin with, all were detected and all were marked legible, which they often are not. Connections across pages would be even more difficult, if not impossible, with a purely pattern matching approach.

For example, pattern matching is never going to make these connections across columns and pages:

Deep Dive

Now we can get to the juicy bits. What I've done is selected an example issue from each decade where SNEWPapers and American Stories overlap, 18 decades from the 1770s to the 1950s (American Stories has no data for the 1780s). If you'd like to follow along by looking at the data in depth yourself, you can go here, as I'll be referencing them as I explain what was measured and why. If you're reading this and you're comfortable with Python coding, there's also an open-source GitHub repo where you can launch a Jupyter notebook (notebooks/compare.ipynb), connect to our public API, search for something, find the issue, then compare it to American Stories. We're hoping that in that way, we won't be stigmatized for "cherry picking", as you're welcome to compare anything you like. In fact, we chose nice clean looking issues, one per decade, on the same Library of Congress scans American Stories used, so both approaches should do well on them. There's plenty of far messier data worth comparing as well.

We could have gone into our database and grabbed the things we exclude from the public API like ads, subscription terms, page furniture etc.., but we didn't want the results to look any different than what someone who uses the open source tools against our API would be able to produce, so the following is reproducible by anyone who wants to dive that deep.

Here are some top level aggregate stats (across all 18 issues or 72 pages) to whet your appetite. You won't understand them yet, but it should give you an indication of the magnitude of the errors in American Stories:

American Stories (AS) errors found

Something that might look odd at first when you look at the large comparison images are lots of unsegmented space. American Stories attempts to exclude ads (but does include quite a few accidentally). SNEWPapers excluded ads, table of contents (rare to see), colophon, images (and their captions), page furniture (section titles, newspaper tagline etc...) from the public API, as it is component based. Zooming in on unsegmented space, you'll likely notice it's (hopefully) all ads.

Interpreting the score card

We'll dive into a few issues to demonstrate how each count is tallied, so you know how to read the reports. Let's start with the first one.

The Massachusetts Spy 1770-12-17
Scorecard or if you downloaded the zip, it's snewpapers-vs-american-stories/sn83021193_1770-12-17_ed-1/index.html
Comparison Image or if you downloaded the zip, it's snewpapers-vs-american-stories/overlays/sn83021193_1770-12-17_ed-1_verdicts.webp

How the comparison and adjudication work

Every comparison starts from a SNEWPapers component and asks what American Stories did with the same part of the page. Both sides are put on the same scan, and the American Stories boxes are cut along our component's edges, so we know which of their articles hold which part of our text. From there each component lands in one of a handful of cases.

Many cases need no judgement at all. If both systems drew the same article, that's an "=". If American Stories has no box over our component, or boxed it but marked it illegible and read nothing, that's measured from the boxes and the American Stories data alone.

Where the two systems disagree, we ask an LLM. We used a cheap general purpose model DeepSeek V4.1 Flash. It is given the newspaper, the date, the page, and the OCR text only, no images, and it is never told which system is which. There are four questions, one per situation, and you can read the exact prompts here (in the repo, snewpapers_tools/prompts/):

• One or several? When one system has a passage as one component and the other has it as several, the model sees the passage as numbered pieces and is told "one system believes this passage is one component, the other believes it is N components". It answers "one", "several" or "mixed".
• Does this text belong? When American Stories attaches text outside our component, or puts a line of our component in another article, the model is asked whether that text belongs to the component: "belongs", "separate" or "furniture".
• What kind of text is this? For text only American Stories has: an advertisement, subscription terms, page furniture, or other content. If it's one of these, SNEWPapers correctly didn't serve this, if not, we missed it and that's a mark against us.
• Is this one item? When American Stories marked part of our component illegible, its pieces don't show the text, so the model is shown only our component and asked whether it is one printed item. A yes counts for SNEWPapers, anything else counts for nobody, never for American Stories.

The first two questions are asked twice, with the two claims stated in opposite orders. If the answer changes with the order, the verdict is "unstable" and counts for nobody. The prompts also say that segmentation systems occasionally leave out page furniture (titles, datelines, section headings, jump lines like "Continued on page 3"), and to judge by the overall content rather than tiny fragments. If the model thinks the text in question is only furniture, it counts for nobody either.

The prompts are deliberately general. They know nothing about SNEWPapers' own, much more detailed, segmentation rules, which is why American Stories wins a handful of judgement calls (for example, several datelined dispatches under one heading, which we serve as separate components and the model reads as one story). We left those in. The whole benchmark of 18 issues cost a few cents to adjudicate.

If you're looking at the scorecard, you'll see aggregate stats at the top as well as component by component decisions:

• a batch with an icon for the group / type of comparison that was done
• the judgement, e.g. match, snewpapers right (llm adjudication), AS only etc...
• clippings compared side by side
• text compared side by side with diffs
• llm responses / reasoning where appropriate
• you can find things in the image with the component keys like p3-1 (just ctrl+f to find it)
• text quality comparisons based on how many words are real words

Some of the metrics at the top and what they'll look like in the image:

Same article on both sides (=) - This means SNEWPapers and American Stories segmented the component the same. In The Massachusetts Spy, page 3, you'll see one.

One snewpapers component, several AS articles - This is when SNEWPapers has one component, and American Stories has it as multiple articles. In The Massachusetts Spy, page 1 & 2, there are two examples side by side. The second one runs from page 1 onto page 2. American Stories doesn't attempt to link pages, so a story that continues on another page is always at least two of their articles judged incorrect. An impartial LLM adjudicates which is correct. You'll see green checks where SNEWPapers won this one, and you'll see the "x" markings on the American Stories articles at the top (x's not shown in image below).

Several snewpapers components, one AS article - This is the case where American Stories has included several distinct components inside a single article. In The Massachusetts Spy, page 2 of the comparison image, you'll see examples like these from the bottom left. On the American Stories row the article is tagged "4 items in 1", and each of the four counts as an item American Stories merged.

One snewpapers component, one AS article running on into other text - In this case, we have one component, and the AS one has additional text beyond that. It's easier to see it than to explain it. The Massachusetts Spy doesn't have one, so this example is from the Alexandria Advertiser of November 4, 1802: our component is the Ten Dollars Reward notice, and the American Stories article carries on into the "Just received and for Sale" advertisement below it. The model is asked whether the extra text belongs to the notice; here it said "separate" both times, so SNEWPapers is right and the American Stories article counts as running on into other items.

AS text attached outside a snewpapers component - This one has no badge of its own in the line by line, but basically when AS attaches text outside SNEWPapers, we send it to an adjudicator to ask if it's a valid part of the component. In the following case from page 3 near the bottom, a bit of text about a plausibly or at least subject matter related thing exists in AS, so the adjudicator asks if it's related, once with the SNEWPapers example first and again with AS first. In this case it's on the fence about it so nobody wins. You'll see these in the line-by-line in the scorecard as a blue-edged box under the summary that reads "American Stories also attaches: «…» (outside this component; AS article #N)". It has the verdict badge (e.g. "snewpapers right" or "unstable") and the model's two answers.

A line of a snewpapers component in another AS article - This one has no badge of its own in the line by line, and can be a bit subtle. This example is from another issue, The Gazette of Portland, Maine, June 11, 1798. Here's a picture of p3-26 and p3-27. In p3-26 you can see the small "May 30." at the bottom of the Lodge notice. Then in p3-27, American Stories has put that line at the top of the next article. Sometimes it's more obvious, but this is what it means. You'll see these in the line-by-line as a blue-edged box that reads "American Stories puts part of this component in another article: «…» (inside this component; AS article #N)". It has the verdict badge and both of the model's answers. (The "unstable" on p3-27 itself is a different question, whether the two items below are one notice or two.)

The rest of the rows are about text only one side has. The examples are from a few different issues, since no single issue has them all.

snewpapers component, AS has no text, no AS box there (an AS error) - American Stories drew no box at all over one of our components, so the text simply isn't in their data. Very often these are notices and small items that look like ads to a layout model, like this runaway notice, which their ad filter most likely dropped. On the comparison image our component is tagged "SNEW ONLY". It's decided from the boxes alone, nothing is asked.

snewpapers component, AS has no text, AS marked it illegible (an AS error) - American Stories did draw a box here, but its legibility classifier marked it illegible, so it was never read and the box has no text. We read the same print without trouble. Our component is tagged "AS ILLEGIBLE", and the American Stories box gets a grey "ILLEGIBLE" tag. The Workmen's Advocate of 1886 is full of these.

snewpapers component, AS has text for under 90% of it, the rest in no AS box (an AS error) - American Stories has some of the component, but no box over the rest. We measure it row by row: how much of our component's height has American Stories text beside it. Under 90% is partial. In this notice, American Stories has only the two signature lines and the date, about a fifth of it. Our component is tagged "AS PARTIAL" and the American Stories article gets an "x".

snewpapers component, AS has text for under 90% of it, the rest marked illegible (an AS error) - The same, but the missing part sits under a box American Stories marked illegible. The most common shape is a headline American Stories could read above a body it couldn't, as here; the scorecard badge says "AS has the headline only (body marked illegible)".

AS text in no snewpapers component: ads, terms or page furniture (left out on purpose) - American Stories has text where we have no component, and the model says it's an advertisement, subscription terms, or page furniture such as a section heading. Those are things SNEWPapers leaves out of the API on purpose, so they count for nobody. The comparison image tags them "NOT SERVED". Here it's a column of patent medicine ads in The Daily Argus of January 20, 1875: SNEWPapers segments ads but doesn't serve them from the API, while American Stories kept them (i.e. their ad detector failed).

… of which ads (AS says it leaves ads out) - The subset of those the model called advertisements. American Stories says it doesn't OCR ads, so every one of these is an ad it meant to leave out and didn't. It's counted on its own line, not as a segmentation error.

AS text in no snewpapers component: other content snewpapers does not serve - Text only American Stories has that the model says is something other than an ad, terms or furniture. It's tagged "AS ONLY". Some of these are things SNEWPapers doesn't serve from the API for other reasons, like this photo caption (we don't serve contextual image searches yet or their captions, and American Stories says it leaves captions out too). Some are OCR so garbled the model can't tell what they are.

AS text in no snewpapers component: too short to ask - Under 20 characters, a heading or a stray scrap. There's nothing to judge, so it isn't asked and counts for nobody.

AS boxes in no snewpapers component with no text at all (AS read nothing) - Whole American Stories articles, headline and body, with no text at all, usually because every box was marked illegible. Here it's a column of ads American Stories boxed but couldn't read. The comparison image tags them "NO TEXT", and they count for nobody.

AS text in no snewpapers component: no verdict - The model didn't return a valid answer. It's rare, it didn't happen at all in these 18 issues, and it counts for nobody.

The rest of the scorecard

The first two tables in each scorecard are the cases above, with a count for each. The first one, "Where the two segmentations agree or differ", has a column per outcome: how many times the case came up, how many the model decided for SNEWPapers, for American Stories, for "neither" (both wrong), "unsure" (unstable, or no valid answer), and "page furniture, not counted". Over all 18 issues:

And "Where only one side has text":

The other three tables summarize what all of that adds up to.

What the decisions judged counts articles and components rather than decisions. One decision can be about many of them: if American Stories split one of our stories into 15 articles and the model says it's one story, that's one decision, one SNEWPapers component judged right, and 15 American Stories articles judged wrong. Each article or component is counted once per side, however many decisions it was in. This is the table at the top of this post.

American Stories errors found counts the errors behind those decisions, with the number of decisions next to them where there is one. An American Stories article holding 7 of our items, decided for SNEWPapers, is 1 decision and 7 items merged. A story split into 15 articles is 1 decision and 15 fragment articles. The rows with no decision count (partials, components and pages American Stories has no text for, duplicated text) come straight from the boxes and the American Stories data, without asking the model. The last three rows are data problems rather than segmentation errors: text American Stories has twice (some of their boxes sit inside other boxes, and both were read), ads it kept, and boxes it marked illegible that we read.

Text quality compares the OCR itself, on the same components on both sides: the share of each side's words (3 or more letters) found in an English word list, allowing for the long s that early type prints and OCR reads as "f" ("neceſſity" read as "neceffity" still counts). The table also gives the share of American Stories' own boxed area it marked illegible and didn't read: 2.2% across these issues, and 35% in the Workmen's Advocate of 1886. More on text quality below.

Text quality

Segmentation is only half of it. If the words inside a correctly segmented article are garbled, it's still hard to search and hard to read. Over the 18 issues, SNEWPapers' text is 99.3% real words (146,372 of 147,392) against American Stories' 94.9% (135,217 of 142,452).

The gap isn't even across time. American Stories does well on later papers, where the type is cleaner and the scans are better, and much worse on the oldest ones:

SNEWPapers stays between 98% and 99.8% in every decade. American Stories is around 86% for the 18th century and the first years of the 19th, and only reaches 97% to 98% from the 1890s on.

This measure is generous to both sides: a misread that happens to be another real word still counts as real, and garbled OCR produces more of those, so the real gap is likely bigger than these numbers. The worst issue for American Stories was the Alexandria Advertiser of November 4, 1802, and the worst article in it that both systems segmented the same way is a warning about counterfeit dollars, p3-11 in that issue's scorecard. American Stories' text is 66.7% real words, SNEWPapers' 98.6%.

CONCORD, IN. HI Oaober II.
COUN IERFEIF DOLLARS.

1be bnbHc are eautionfd agamA talins

e1aa!eyfH doAlo,s, qcbic5 eflax bSs,e 5een
Sle1tyulJ erealatfam The viici;aity yC.f.
.a,7va and DeryfeN, IN this aate. Try
are d"'ed 1877 appear 10 be literally
7yeA executed; but On a er;iicalexamiiuu-
tim, are toaud to BE- di9st<nt ty0m tie n1,
d'llar IF Tie /ame dAte !; Tby may be ay--
coVered BY the jozo7vmg marks, qis-76e
Q seer the n in the bad dJlIay, mteytstes
qgf'b it, qvbicb in the good daAlny dses not.
III. ngr is ratber moye p,eJsara nud 5eek'd
at tie us9rl, than the ygood o4r, avbieb
n3psors more 5luat ar rouud,ng. There are
bxt f7vo leeyss brJeatd above tie kead,
in tie tad dal,ar, The thru appearing
./ame-0bat ike dot, uot toaebing auy b0rt
9t0 tie hat; zvb,'eas IL tbe goed aolar
three lea.vs gre p}ui4ly IA be /sra. A (c,nfch
bet7vern The ends :ftbs nbJau, beisaO tie
5azO, Is dIc5vered On The bad asllar; but
aone az tie 2oaa. The A at lbs end eftbs
7pord VIIRA en- The rigbf hand pillar,
an@ Tie P in HISPAN, uealy tauch eatb
ofber; 705erras, IN tie good do4la,, tdy
arr rogEdsra5A liFa1cr apart; the Jane
nay BE- o9/etvr@ IL rslpe17 to !be p in
PLUS On tbe I.fi pillar, and ibe r OH or
ny The elgc;; tbr djqauce 6et7veen The
5iil0rs in tbt bad 4Aar beIzg .grsatertbau
that Of the gOod. itb boped the tey5ns
1o5o b.ve beeu d,rliaea BY taking ks/e
coAvterfrjts, will NE tbeir b,R euaeavouys
u b4ve the Peyptyators bro4gbt to jyfiee.

Conclusion

Returning to the aggregate table above for the 18 issues or 72 pages, but this time with a share so we have a denominator.

Whether it be fragments, incorrect merged text, wrong boundaries, missing text, illegibile text that extracts fine or any other reason, about 30% of American Stories articles are segmented incorrectly, and 1.4% of SNEWPapers is segmented incorrectly, although those 16 come from 4 adjudication decisions, and 11 from just one decision in the 1890s Daily New Dominion p3-15, where our actual segmenter is instructed to break each local news stories into a component). The 2 from 1860s Vermont Phoenix p3-16 are death notices which we also segment separately on purpose. The 2 from 1920s Hyder Weekly Herald p1-16 and the 1 from 1910s Tonopah Daily Bonanza are legitimately questionable.

Across 20 million pages, if these ratios are representative of the corpus at large, that means there are roughly 107,000,000 segmentation errors latent in American Stories, and as I demonstrated in the decade by decade multi-column and multi-page stats, most of those errors are likely within the most interesting content, long-form prose. This also doesn't account for vast amounts of unaccounted for space that was flagged as non-content and not extracted.

Rather than trying to put humpty dumpty back together again, we forged our own path.

If our adjusted number for SNEWPapers is representative, then there will probably be around 830,000 segmentation errors in SNEWPapers when we get to 20 million pages. Because we gather so many stats, for the most part, we are actually already aware of where the worst quality components are, and we're building a variety of processes based on metric pattern matching to go in and fix things surgically (i.e. a component re-run rather than a page/issue re-run) based on the specific type of quality issue. Given the nature of the data, perfection isn't the goal, a lot of it is exceptionally messier than the examples displayed here, but we do strive to get as close as we can, chipping away at the pile day by day.