Project updates
Changelog
Recent work on EPS, listed from newest to oldest.
Document size filtering
The document search UI and REST API now support page-count range filters, letting visitors narrow results from zero pages up to any maximum document size.
Multi-source document filtering
Search results can now be filtered across one or more document sources, including DOJ, CourtListener, and DocumentCloud records, without leaving the document viewer.
Detailed project changelog
The public changelog was expanded into a curated technical timeline so visitors can understand how EPS has evolved without exposing internal development metadata.
Frontend request limits
The document search UI now caps result requests at 100 documents, matching the REST API limit and preventing oversized browser requests.
SQLite to MySQL migration hardening
The migration script now inserts OCR-heavy text through prepared row-level statements, preventing arbitrary document text from breaking generated SQL batches.
Filename search
A filename search mode was added to the document viewer, backed by a metadata-only REST endpoint for quickly finding records by file name.
Largest corpus refresh so far
The compressed SQLite corpus was refreshed to 4,437 documents and 112,000 pages, incorporating newly collected CourtListener and DocumentCloud material.
Contact access and inquiry form
A dedicated contact page and floating contact widget were added so visitors can reach the project maintainers from the site.
REST API startup orchestration groundwork
Internal route registration and connection-pool management were reorganized to make the REST API easier to start from configuration rather than scattered hardcoded setup.
Expanded crawler subjects
Crawler search subjects grew from 44 to 71 terms, adding estate, banking, staff, and witness-related topics while improving multi-word query behavior.
Corpus refresh after keyword expansion
The corpus was refreshed to 4,397 documents and 108,372 pages after broadening search coverage and pruning false positives.
Read-only API response caching
The nginx layer now caches read-only corpus responses, reducing repeated load on the C++ API and database for common list, document, and search requests.
Compressed JSON API responses
JSON responses are gzip-compressed through the reverse proxy, which is especially useful for text-heavy document and search payloads.
Safer pagination parameters
REST API list and search parameters are now validated and clamped, preventing oversized result sets and malformed query values from disrupting requests.
Bounded search snippets
Search results now fetch a bounded text window for snippets instead of transferring full document text for every hit.
Metadata-only document listing
The document list endpoint now returns metadata without full OCR text, making list views lighter while keeping full text available through detail endpoints.
UTF-8 safe snippets
Search snippet generation was adjusted to avoid cutting through multibyte characters, preventing invalid JSON from strict parsers.
Corpus cleanup refresh
The corpus was refreshed to about 3,815 documents and 67,113 pages after removing use-of-force false positives from the collected set.
Connection pool manager groundwork
REST API infrastructure work began on managing multiple connection pools, laying groundwork for broader database support.
OCR memory usage fix
PDF ingestion now processes pages in smaller windows at a lower default DPI and worker count, reducing peak memory use while keeping OCR practical for very large documents.
Corpus snapshot restored
The compressed corpus snapshot was regenerated from the current database, producing 3,787 documents and 66,042 pages after a crawl and ingest pass.
REST API package linkage fix
Build and helper scripts were adjusted so the REST API package links correctly against MySQL client headers and libraries.
Reusable REST API package layout
The REST API core was reorganized toward a shared-library package with public headers, CMake package metadata, and pkg-config support.
Documentation refresh
The root, frontend, and environment documentation were rewritten to better explain the project, local setup, and container command gateway.
Site metadata and crawler guidance
Site metadata, robots output, sitemap configuration, manifest data, and LLM-oriented API guidance were added for clearer discovery and automated access.
Runtime REST API configuration
The frontend gained a runtime configuration endpoint so deployments can supply the REST API origin without relying only on a baked browser bundle value.
Document visualizer deployment
The search page moved from raw JSON output toward a usable OCR document viewer with result lists, metadata, snippets, and page navigation.
Footer and legal content cleanup
Dead frontend links were removed, and the terms, privacy policy, and legal notice were rewritten around the current EPS document-search purpose.
Container command gateway
A container-oriented command bridge and whitelist were added so an LLM assistant can perform approved project operations through controlled host-side commands.
REST API beta deployment setup
The REST API received beta deployment configuration, environment examples, nginx reverse-proxy configuration, install script updates, and CORS-related startup options.
MySQL credential handling
Database credentials moved out of helper scripts and into environment variables, keeping local defaults documented while making production configuration safer.
Database configuration naming cleanup
REST API database configuration fields were renamed and rewired so host, database, port, and password settings match their actual runtime meaning.
SQLite corpus connected to the REST API
A MySQL schema and migration path were added so the scan manager corpus can be copied from SQLite into the database served by the C++ REST API.
Crawler pipeline commands
The scan manager wrapper gained migrate and pipeline commands, making crawl, ingest, and database refresh flows easier to run in order.
DOJ crawler hardening
The crawler gained stronger session handling, browser-state persistence, stealth support, challenge detection, and faster cookie-backed downloads.
Frontend build cleanup
Obsolete frontend pages, unused auth pieces, and orphaned components were removed so production builds could complete reliably.
Frontend deployment cleanup
The frontend was trimmed for deployment by removing unused API routes, stale user/order/image components, and legacy generated output.
Beta build controls
CMake flags, helper commands, and install-script updates were added to support beta builds of the REST API.
CORS and configuration loader
The REST API gained CORS handling plus a YAML-like configuration loader for server and MySQL settings.
Next.js frontend foundation
The initial TypeScript frontend was added with shared layout components, service pages, theming support, and a first search page shell.
Court document REST routes
The REST API gained the first court-document repository and route implementation for document search and retrieval.
Multi-source corpus pipeline
The scan manager expanded into a multi-source crawler and OCR ingest pipeline with SQLite FTS5 search, covering DOJ, CourtListener, and DocumentCloud sources.
Initial compressed corpus snapshot
A compressed SQLite database snapshot was added so the document corpus could be shared and restored without publishing the raw database file.
DOJ disclosure crawler
The crawler was rewritten to walk the structured DOJ Epstein disclosure index directly, collecting document links from disclosure sections instead of relying on blocked search pages.
Crawler query and token configuration
Crawler query generation, YAML token templates, and scan-manager guidance were expanded to support OCR extraction and document parsing experiments.
Matrix setup script groundwork
Early infrastructure work added a Matrix setup script and nginx reverse-proxy setup notes that informed later deployment tooling.
Initial EPS application foundation
The first substantial project structure landed: REST API scaffolding, scan manager utilities, database repository examples, helper scripts, and early documentation.
