The web & document layer
for AI agents
Any URL, document, email, archive or source file in — clean Markdown or structured JSON out. One self-hosted Rust binary with a CLI, Firecrawl & Jina Reader APIs, an MCP server, and packages for Python, Node.js, Go, Java, C#, Ruby, PHP and more.
$ pagesift scrape https://example.com # Markdown to stdout $ pagesift scrape report.pdf # PDF, Office, e-book, email… $ pagesift crawl https://docs.rs --limit 500 # whole sites, one file per page $ pagesift mcp # MCP server for your agent
Measured, not promised
Strongest result per domain from the published 1.1 benchmark report — the same numbers as the project README, with methods and sources in the full report.
Latency per page — lower is better
Throughput — higher is better
Peak memory per worker — lower is better
Benchmarks measured for PageSift 1.1; xberg 1.3.9 measured 2026-11-04. Full methods: docs/benchmarks.md · Wild-crawl coverage 0.740 vs Crawl4AI 0.529 (polite, 40 sites × 60 pages).
One pipeline, every input
The request pipeline from the architecture docs: entry points feed one engine, policy and network guards run before any fetch, a render decision picks the cheapest path that adds content, and every source lands in the same outputs.
What it can do
Six verbs cover the API surface — the same operations in the CLI, the HTTP APIs and the MCP tools.
scrape · Clean content
Main-content detection and Markdown that keeps headings, lists, tables, code and math. Renders JavaScript only when that adds content.
crawl · Whole sites
Sitemaps, query-focused crawls that stop when pages stop helping, trap handling, robots.txt and per-host limits. WARC output on every crawl.
map · Site maps
Discover the URLs of a site before you spend budget fetching them, then feed the list into crawl, batch or extract.
batch · Many URLs, one job
Batch scraping with job persistence, durable webhooks and per-owner isolation. Poll or cancel jobs from the API or MCP.
monitor · Change monitoring
Scheduled checks that separate real changes from dates, counters and ads, with webhooks when something actually moved.
agent · Research runs
POST /v2/agent answers a question from the web with your own model — and checks each field of the answer against the page it came from.
Documents & OCR
Own Rust readers for PDF, Office, e-books, email, archives and databases — including damaged files and CJK fonts. Scanned pages go through PP-OCRv6, display formulas through PP-FormulaNet, on CPU or NVIDIA GPU.
Structured extraction
JSON by CSS selectors that are relocated when a site changes, or by a schema your model writes once and PageSift caches. Browser actions — clicks, typing, scrolling, kept sessions — when a page needs them.
Safety & honest results
Private-network and DNS-rebinding guards. A partial page, a truncated file or a skipped scan returns a warning code instead of looking complete.
Almost anything goes in
One call handles the whole pile — no separate converters to wire up.
Web pages
URLs & HTML, rendered only when it adds content
Including scanned pages, via OCR
Office
Word, Excel, PowerPoint — current & legacy — plus OpenDocument, RTF, HWP
E-books
EPUB, MOBI, FB2
.eml, .mbox, .msg — attachments recurse
Archives
zip, tar, gz, bz2, xz, 7z — and WARC in & out
Data files
CSV, SQLite, dBASE, Markdown & text formats
Source code
373 source-code & config languages, split by function & class
Drop-in compatible, everywhere
Already have Firecrawl or Jina Reader code? Change the API URL and keep going. Building an agent? The MCP server speaks the same operations.
HTTP APIs
Firecrawl v1/v2 and Jina Reader (/r/ and /s/) endpoints on the same self-hosted binary — plus a file-upload API for local documents.
# Firecrawl API curl localhost:3002/v2/scrape \ -H 'Authorization: Bearer sk-test' \ -d '{"url":"https://example.com","formats":["markdown"]}' # Jina Reader curl localhost:3002/r/https://example.com
MCP server — stdio or HTTP
Ten tools for agents: the eight everyday operations, plus the agent pair when a model is configured. Attach with one command:
claude mcp add pagesift -- pagesift mcp
Packages & bindings
Native packages for the big three, a Rust crate, and a C ABI that carries the engine into a dozen more languages. LangChain and LlamaIndex integrations included.
Platforms & runtime
Linux x86_64 / aarch64 (glibc ≥ 2.28 and musl), macOS arm64, Windows x86_64, and a Docker image. Nothing to install for HTTP, PDF and Office; a browser for JavaScript pages, an NVIDIA GPU for OCR and your own model for /v2/agent are found at run time.
Running in a minute
Pick your surface — they all run the same engine.
| Surface | Install | First call |
|---|---|---|
| CLI | cargo install pagesift --locked | pagesift scrape https://example.com |
| Python | pip install pagesift | pagesift.scrape("https://example.com").markdown |
| Node.js | npm install @pagesift/node | (await new Scraper().scrape("https://example.com")).markdown |
| Server | docker run -p 3002:3002 ghcr.io/yolow92/pagesift:1.2.2 | curl localhost:3002/r/https://example.com |
| MCP | — | claude mcp add pagesift -- pagesift mcp |
Access-barrier recovery, opt-in
- Off by default. Enable with
[itna] enabled— nothing changes until you ask. - Classified, then bounded. Barriers are classified first; a blocked or challenged page gets at most one browser fallback, sharing one network budget across redirects and sub-requests.
- Never escalates the wrong cases. Rate limits (429) and login walls never trigger the browser path.
- Isolated sessions. Sessions are leased per owner, origin, egress and fingerprint; HTTP cookies reach the browser only through an exact-origin handoff.
- Humans stay in the loop. Interactive CAPTCHAs are reported as needing a human — never solved.
What's new in 1.2.2
- Crawl, batch and extract jobs are restricted to their owner.
- Search and extract page attempts now count against the daily quota.
- Cached pages are checked against robots.txt before reuse.
- Unsupported snapshot and base64-image options are rejected instead of silently ignored.
Good fit — and when to pick something else
Feeding pages and documents to an LLM, a RAG index or an agent; self-hosting a Firecrawl-style service with no per-page fees; mixed input on small machines. Pick something else for standalone image files, a managed cloud service, or if AGPL does not work for you and a commercial license is not an option.
Open source, commercially licensable
Free to self-host under AGPL — with a commercial route for proprietary deployments.
AGPL-3.0
Use, modify and self-host PageSift under the GNU AGPL v3. The full source is on GitHub.
Commercial license
For proprietary linking or private deployment without AGPL obligations, a commercial license is available.