Building an AI Gateway is easy when you have 10 users. It gets interesting when you have 10,000.

At Cencori, we recently hit a distinct inflection point where our Postgres-backed rate limiter started competing for resources with core application logic. Every API request triggered a database read/write transaction just to check if the user was allowed to make that request.

Latency crept up. Database CPU spiked. It was time to decouple control plane logic from our data layer.

Here is how we re-architected the Cencori AI Gateway for scale using Upstash Redis.

The Problem with Database Rate Limiting

Our initial implementation was simple:

Codetext

// The old way (Postgres)
const { count } = await supabase
  .from('requests')
  .select('*', { count: 'exact' })
  .eq('user_id', userId)
  .gte('created_at', oneMinuteAgo);
 
if (count > limit) throw new Error('Rate limit exceeded');

This works perfectly for low volume. But at scale, "count all rows from last minute" becomes an expensive query. Worse, it's strictly consistent, meaning the database locks rows, creating contention.

We needed something fast, atomic, and ephemeral.

Enter Redis-Based Token Buckets

We migrated to a Sliding Window approach using Redis. Instead of counting rows, we use atomic counters with automatic expiration.

Codetext

// The new way (Redis)
const key = `rate_limit:${projectId}`;
const requests = await redis.incr(key);
 
if (requests === 1) {
  await redis.expire(key, 60); // Window resets in 60s
}
 
if (requests > limit) throw new Error('Rate limit exceeded');

The result?

Latency: Reduced from ~15ms to < 1ms per check.
Database Load: Zero. The auth/rate-limit layer no longer touches Postgres.
Precision: Atomic INCR operations guarantee exact enforcement even with high concurrency.

Conclusion

By moving transient rate-limit state to the edge with Redis, we've significantly increased the headroom of the Cencori Gateway.

Infrastructure is effectively "invisible" until it breaks. These changes ensure Cencori stays invisible, fast, and reliable as you scale from MVP to IPO.