Pulls clean text out of documents at speed. A multi-threaded extraction engine that turns a mountain of PDFs and attachments into plain, searchable text — and feeds the rest of the pipeline.
Screenshot — pending
PDF Extraction is a production-grade, multi-threaded engine for pulling clean plain text out of PDFs and other documents. It exists to process large volumes of document attachments quickly and reliably, feeding a downstream pipeline — in particular, the attachment text and search index behind Penelope, the email archive. When there are tens of thousands of files to read, the difference between fast-and-correct and slow-and-lossy is the whole job.
It uses two-phase processing with content-hash deduplication and smart caching — identical files are recognised and never re-extracted. At its core is a single extractor class offering a clean programmatic interface, wrapped by parallel batch scripts with configurable concurrency, batch size, and limits. Database commits are batched, and a graceful interrupt handler saves partial progress if a run is stopped midway.
It is a Node.js package, MIT-licensed, built on the poppler tool-chain for PDF text and standard utilities for archives, and it handles a wide spread of formats — PDF, Word, plain text, HTML, CSV, JSON, and XML. The reported numbers are striking: on the order of 357 files per second on a 24-core machine, a 99.6% success rate on PDFs, and content-hash deduplication saving between six and fifty percent of the work, all with live progress, estimated-time, and cache-hit reporting. The benchmarks come from a real dataset of more than twenty-one thousand attachments.