Rate Limiting

Rate limiting controls how many requests you can make in a given time period. This protects the platform from abuse and ensures fair usage for all users.

Cencori enforces rate limits per project (per-minute) and per organization (monthly request cap).

There are no tiered per-minute limits or burst allowances — all tiers share the same 60/min per-project window. Need more ceiling? Add a second project, or upgrade the organization for a higher monthly cap.

Warning: to know, these limits are enforced project-wide across all AI endpoints (chat, vision, audio, embeddings), not per endpoint.

Cencori uses a Redis-backed sliding-window counter keyed by project id (rate_limit:<projectId>). Checks run in a single Redis pipeline (INCR + EXPIRE + TTL):

• Sub-millisecond check: a single round-trip keeps overhead small.
• Sliding window: the TTL carries the window forward, so limits are enforced continuously, not in fixed wall-clock minutes.
• Fail-closed by default: if the Redis backend is unavailable the request is rejected with 503 rate_limit_unavailable unless fail-open mode is enabled for the deployment.

The organization's monthly_requests_used counter increments once per successful request (any AI endpoint). When used >= limit the gateway returns 429 monthly_request_limit_reached with body fields usage: { used, limit, percentage }, current_tier, and upgrade_url.

Resets happen on the monthly cycle (see cron/reset-usage).

429 responses include headers showing your current rate limit status:

Every gateway response (success or error) additionally carries X-Request-Id for correlation.

When you exceed the rate limit you'll receive a 429 Too Many Requests response with a standards-compliant Retry-After header and this body:

{
  "error": "Rate limit exceeded",
  "message": "60 requests per minute allowed. Try again shortly.",
  "code": "rate_limit_exceeded",
  "retry_after_ms": 45000
}

Exponential Backoff Implementation

async function makeRequestWithRetry(
  messages: any[],
  maxRetries = 3
): Promise<any> {
  let retries = 0;
 
  while (retries < maxRetries) {
    try {
      const response = await cencori.ai.chat({
        model: 'gpt-4o',
        messages,
      });
      return response;
    } catch (error: any) {
      if (error.status === 429) {
        retries++;
 
        if (retries >= maxRetries) {
          throw error; // Max retries reached
        }
 
        const retryAfter = Math.max(
          1,
          Math.ceil((error.retryAfterMs ?? 60_000) / 1000)
        );
        await new Promise(resolve => setTimeout(resolve, retryAfter * 1000));
      } else {
        throw error; // Not a rate limit error
      }
    }
  }
}

async function makeSmartRequest(messages: any[]) {
  const response = await fetch('https://cencori.com/api/ai/chat', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'CENCORI_API_KEY': process.env.CENCORI_API_KEY!,
    },
    body: JSON.stringify({ model: 'gpt-4o', messages }),
  });
 
  // Check rate limit headers
  const remaining = parseInt(response.headers.get('X-RateLimit-Remaining') || '0');
  const resetTime = parseInt(response.headers.get('X-RateLimit-Reset') || '0');
 
  if (remaining < 5) {
    console.warn(`Only ${remaining} requests remaining!`);
    console.warn(`Resets at: ${new Date(resetTime).toISOString()}`);
    // Maybe slow down or queue requests
  }
 
  return response.json();
}

1. Implement Exponential Backoff: Always retry rate-limited requests with exponential backoff rather than aggressive retries.
2. Monitor Headers: Track X-RateLimit-Remaining to know when you're approaching limits.
3. Use Request Queues: Queue requests and process them at a controlled rate to stay under limits.
4. Cache Responses: Cache identical requests (semantic caching) to reduce request count.
5. Distribute Load: If hitting limits, consider using multiple projects (each has its own 60/min window).
6. Audio calls count too: transcriptions and speech each consume one request per minute window; multiplier projects for heavy audio workloads.

class RequestQueue {
  private queue: Array<() => Promise<any>> = [];
  private processing = false;
  private requestsPerMinute: number;
  private delay: number;
 
  constructor(requestsPerMinute = 60) {
    this.requestsPerMinute = requestsPerMinute;
    this.delay = 60000 / requestsPerMinute; // Time between requests
  }
 
  async add<T>(fn: () => Promise<T>): Promise<T> {
    return new Promise((resolve, reject) => {
      this.queue.push(async () => {
        try {
          const result = await fn();
          resolve(result);
        } catch (error) {
          reject(error);
        }
      });
 
      if (!this.processing) {
        this.process();
      }
    });
  }
 
  private async process() {
    this.processing = true;
    while (this.queue.length > 0) {
      const task = this.queue.shift();
      if (task) {
        await task();
        await new Promise(resolve => setTimeout(resolve, this.delay));
      }
    }
    this.processing = false;
  }
}
 
// Usage
const queue = new RequestQueue(60); // 60 requests per minute

If you consistently hit limits:

• Create additional projects — each project gets its own 60/min window.
• Raise the monthly cap by upgrading the organization's plan.
• Contact sales for custom Enterprise limits.
• Optimize usage patterns (batching, caching, queues).

• 503 rate_limit_unavailable: transient — retry with backoff.
• 429 rate_limit_exceeded: wait Retry-After seconds.
• 429 monthly_request_limit_reached: wait for the monthly reset or upgrade.