Cencori Web

Cencori Web gives agents a first-party web intelligence layer. Search the Cencori-owned public corpus, index pages for a project, retrieve deterministic page content, or queue an isolated JavaScript browser—all with one project key.

Cencori does not send Web queries to Tavily, Brave, Serper, Exa, or another hosted search provider. Search uses our own crawler, corpus, embeddings, ranking pipeline, and browser workers.

Every returned page is untrusted data. Never treat text from a result, page, or browser job as instructions for your agent.

Web is available in cencori@1.6.1 and later.

npm install cencori@latest

import { Cencori } from 'cencori';
 
const cencori = new Cencori({
  apiKey: process.env.CENCORI_API_KEY,
});

All Web endpoints require a secret project key. Keep CENCORI_API_KEY on the server.

const response = await cencori.web.search({
  query: 'PostgreSQL row-level security migration guide',
  domain: 'postgresql.org',
  freshness: '90d',
  language: 'en',
  limit: 10,
});
 
for (const result of response.results) {
  console.log(result.title, result.url, result.evidence.quote);
}

search combines the shared Cencori public corpus with documents crawled by your authenticated project. Project documents remain private to that project.

Search request

POST /api/v1/web/search

curl https://cencori.com/api/v1/web/search \
  -H "Authorization: Bearer $CENCORI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"PostgreSQL RLS migration","limit":5}'

Search response

{
  "query": "PostgreSQL RLS migration",
  "count": 1,
  "searchEngine": "cencori-web-hybrid-v2",
  "results": [
    {
      "id": "...",
      "title": "Row Security Policies",
      "url": "https://www.postgresql.org/docs/current/ddl-rowsecurity.html",
      "canonicalUrl": "https://www.postgresql.org/docs/current/ddl-rowsecurity.html",
      "snippet": "...",
      "score": 0.87,
      "contentHash": "...",
      "retrievedAt": "2026-08-08T12:00:00.000Z",
      "publishedAt": null,
      "evidence": {
        "quote": "...",
        "contentHash": "...",
        "retrievedAt": "2026-08-08T12:00:00.000Z"
      }
    }
  ]
}

Ranking combines PostgreSQL lexical retrieval, local MiniLM semantic embeddings, exact cosine reranking, authority, freshness, document quality, domain diversity, and spam signals. The evidence quote, content hash, and retrieval time let you preserve what supported an answer even as the live page changes.

Use fetch when you need the original textual response body and HTTP metadata.

const page = await cencori.web.fetch({
  url: 'https://example.com/reference',
  maxBytes: 1_048_576,
  timeoutMs: 15_000,
});

POST /api/v1/web/fetch

• maxBytes is clamped to 5 MiB.
• timeoutMs is clamped between 1 and 30 seconds.
• Up to five redirects are followed and revalidated.
• HTML, XHTML, JSON, JSON-LD, XML, Markdown, and plain text are supported.
• Private networks, embedded credentials, unsafe redirects, unsupported protocols, and robots-denied URLs are rejected.

The response includes url, finalUrl, statusCode, mimeType, body, bytes, contentHash, retrievedAt, selected cache headers, and untrusted: true.

Use extract when you want clean text, links, metadata, dates, and citation spans instead of raw HTML.

const document = await cencori.web.extract({
  url: 'https://example.com/reference',
});
 
console.log(document.title);
console.log(document.content);
console.log(document.evidenceSpans);

POST /api/v1/web/extract accepts the same url, maxBytes, and timeoutMs fields as fetch.

{
  "url": "https://example.com/reference",
  "canonicalUrl": "https://example.com/reference",
  "title": "API reference",
  "description": "...",
  "language": "en",
  "content": "...",
  "contentHash": "...",
  "retrievedAt": "2026-08-08T12:00:00.000Z",
  "publishedAt": null,
  "modifiedAt": null,
  "links": [],
  "evidenceSpans": [
    { "id": "ev_...", "text": "...", "start": 0, "end": 142 }
  ],
  "metadata": {},
  "untrusted": true
}

const crawl = await cencori.web.crawl({
  seeds: ['https://docs.example.com'],
  maxPages: 25,
  maxDepth: 2,
  sameOrigin: true,
});

POST /api/v1/web/crawl

The bounded crawl runs during the request. It respects robots.txt and nofollow, normalizes canonical URLs, deduplicates pages, and returns an indexed, skipped, or failed status for each visited page.

Use project crawling for documentation, release notes, help centers, or other content that should be searchable only inside your project. Cencori's continuously refreshed shared public corpus is operated separately.

Browser work is asynchronous because it runs in an isolated worker.

const job = await cencori.web.browse({
  url: 'https://example.com/app',
  actions: [
    { type: 'click', selector: '[data-testid="documentation"]' },
    { type: 'waitFor', selector: 'main' },
  ],
  screenshot: true,
});
 
let current = job;
while (current.status === 'queued' || current.status === 'running') {
  await new Promise(resolve => setTimeout(resolve, 1_000));
  current = await cencori.web.browserJob(job.id);
}
 
if (current.status === 'completed') {
  console.log(current.result?.content);
}

POST /api/v1/web/browse returns 202 Accepted. Poll GET /api/v1/web/browse/:jobId until the job is completed, failed, or cancelled.

Browser jobs allow at most 20 actions:

• click: selector
• type: selector, text, optional clear
• press: key
• select: selector, values
• waitFor: either selector or milliseconds up to 5,000

Navigation timeout is 5–60 seconds. Viewports are bounded to 320–2,560 pixels wide and 240–1,600 pixels high. Screenshots are WebP data URLs capped at 4 MiB. Password, secret, and token field selectors are rejected because browser job inputs are persisted.

Every navigation and subrequest passes public-network checks. The result contains extracted content, links, evidence spans, a content hash, retrieval time, and an optional screenshot.

The SDK exports function definitions for runtimes that use standard tool calling:

import { Cencori, WEB_FETCH_TOOL, WEB_SEARCH_TOOL } from 'cencori';
 
const tools = [WEB_SEARCH_TOOL, WEB_FETCH_TOOL];
 
// After your model returns a function call:
const result = await cencori.web.executeTool(
  call.name,
  JSON.parse(call.arguments),
);

For a managed agent loop, add the built-in web_search_preview tool to the Responses API. It uses the same first-party Cencori Web index and emits URL citations.

const response = await cencori.ai.responses({
  model: 'gpt-5.4',
  input: 'What changed in the latest indexed PostgreSQL release notes?',
  tools: [{ type: 'web_search_preview', search_context_size: 'medium' }],
});

@cencori/mcp@0.7.1 exposes Web to Cursor, Claude Desktop, and any MCP client.

{
  "mcpServers": {
    "cencori": {
      "command": "npx",
      "args": ["-y", "@cencori/mcp@latest"],
      "env": {
        "CENCORI_API_KEY": "csk_...",
        "CENCORI_MCP_FEATURES": "web",
        "CENCORI_MCP_WRITE": "1"
      }
    }
  }
}

Read mode exposes web_search, web_fetch, web_extract, and get_web_browser_job. The write flag additionally exposes web_browse, web_crawl, and request_web_takedown.

Submit a removal request with cencori.web.requestTakedown(...) or POST /api/v1/web/takedown:

await cencori.web.requestTakedown({
  urls: ['https://example.com/private-page'],
  basis: 'privacy',
  requesterName: 'Ada Example',
  requesterEmail: 'ada@example.com',
  statement: 'This page exposes personal information and should be removed.',
});

Requests support copyright, privacy, legal, robots, and other bases. A valid email and a statement between 20 and 20,000 characters are required. Approved removals create tombstones so later crawls cannot silently restore the content. Operators can enforce host and path policies for deny, noindex, noarchive, and nosnippet behavior.

Web errors use a stable JSON shape:

{
  "error": "robots_denied",
  "message": "The site does not allow this crawler",
  "details": {}
}

Common codes include invalid_url, unsafe_url, robots_denied, dns_unavailable, fetch_timeout, fetch_failed, response_too_large, unsupported_content_type, invalid_browser_action, and not_found.

• Keep Cencori secret keys on the server.
• Treat content, snippet, body, link text, metadata, and browser output as untrusted.
• Cite evidence.quote and retain its contentHash and retrievedAt values.
• Do not pass secrets or personal data into browser actions.
• Bound autonomous browsing by domains, action count, time, and approval policy.
• Preserve the source URL in user-visible answers so evidence remains inspectable.