Rate Limiting & Throttling for ChatGPT Apps
Rate limiting and throttling are critical security mechanisms for ChatGPT apps deployed to the OpenAI App Store. Without proper controls, your MCP server can be overwhelmed by abusive users, automated bots, or legitimate traffic spikes that exhaust your infrastructure resources. Rate limiting prevents abuse by restricting the number of requests a client can make within a time window, while throttling controls the rate at which requests are processed. Together, these mechanisms ensure fair usage across all users, protect your backend services from overload, and maintain consistent performance even during peak demand. This guide covers production-ready rate limiting algorithms, distributed implementations with Redis, HTTP response handling best practices, throttling strategies, and monitoring approaches that have been battle-tested in high-traffic ChatGPT applications serving millions of requests daily.
Rate Limiting Algorithms
Choosing the right rate limiting algorithm depends on your ChatGPT app's traffic patterns, user behavior, and infrastructure constraints. The token bucket algorithm allows bursts of traffic while maintaining an average rate limit—perfect for ChatGPT apps where users may send multiple related messages in quick succession. Each user gets a bucket with a fixed capacity that refills at a constant rate; requests consume tokens, and when the bucket is empty, requests are rejected. The leaky bucket algorithm processes requests at a constant rate regardless of input spikes, smoothing traffic to your backend—ideal for protecting database connections or external API calls with strict rate limits.
The fixed window algorithm counts requests within fixed time periods (e.g., 100 requests per minute), resetting the counter at window boundaries. It's simple to implement but suffers from edge case issues: a user could make 100 requests at 12:59:59 and another 100 at 13:00:01, effectively doubling the rate limit. The sliding window algorithm solves this by calculating the rate over a rolling time window, providing smoother rate limiting without boundary effects. For most ChatGPT apps, a sliding window with token bucket characteristics offers the best balance—it allows legitimate bursts while preventing sustained abuse.
Hybrid approaches combine multiple algorithms: you might use token bucket for per-user limits (allowing conversational bursts) and fixed window for global limits (protecting infrastructure). Consider your app's concurrency model: synchronous MCP servers benefit from in-memory algorithms, while distributed deployments require coordination through Redis or DynamoDB. The algorithm should also account for different limit tiers—free users might get 10 requests per minute, while paid subscribers get 100—with separate tracking per pricing tier.
Here's a production-ready token bucket implementation with TypeScript:
// Token Bucket Rate Limiter with TypeScript
// Supports per-user limits, burst allowance, and automatic refill
interface TokenBucket {
capacity: number; // Maximum tokens
tokens: number; // Current available tokens
refillRate: number; // Tokens added per second
lastRefill: number; // Timestamp of last refill
}
interface RateLimitConfig {
capacity: number; // Max burst size
refillRate: number; // Tokens per second
refillInterval: number; // Milliseconds between refills
}
class TokenBucketRateLimiter {
private buckets: Map<string, TokenBucket> = new Map();
private config: RateLimitConfig;
constructor(config: RateLimitConfig) {
this.config = config;
// Cleanup stale buckets every 5 minutes
setInterval(() => this.cleanup(), 300000);
}
/**
* Check if request is allowed and consume token if available
*/
async allowRequest(userId: string, cost: number = 1): Promise<{
allowed: boolean;
remainingTokens: number;
retryAfter?: number;
}> {
const bucket = this.getOrCreateBucket(userId);
this.refillBucket(bucket);
if (bucket.tokens >= cost) {
bucket.tokens -= cost;
return {
allowed: true,
remainingTokens: Math.floor(bucket.tokens)
};
}
// Calculate retry-after in seconds
const tokensNeeded = cost - bucket.tokens;
const retryAfter = Math.ceil(tokensNeeded / this.config.refillRate);
return {
allowed: false,
remainingTokens: Math.floor(bucket.tokens),
retryAfter
};
}
/**
* Get or create bucket for user
*/
private getOrCreateBucket(userId: string): TokenBucket {
let bucket = this.buckets.get(userId);
if (!bucket) {
bucket = {
capacity: this.config.capacity,
tokens: this.config.capacity,
refillRate: this.config.refillRate,
lastRefill: Date.now()
};
this.buckets.set(userId, bucket);
}
return bucket;
}
/**
* Refill bucket based on elapsed time
*/
private refillBucket(bucket: TokenBucket): void {
const now = Date.now();
const elapsed = (now - bucket.lastRefill) / 1000; // Convert to seconds
const tokensToAdd = elapsed * bucket.refillRate;
if (tokensToAdd > 0) {
bucket.tokens = Math.min(
bucket.capacity,
bucket.tokens + tokensToAdd
);
bucket.lastRefill = now;
}
}
/**
* Get current bucket state for monitoring
*/
getBucketState(userId: string): {
tokens: number;
capacity: number;
percentage: number;
} | null {
const bucket = this.buckets.get(userId);
if (!bucket) return null;
this.refillBucket(bucket);
return {
tokens: Math.floor(bucket.tokens),
capacity: bucket.capacity,
percentage: Math.round((bucket.tokens / bucket.capacity) * 100)
};
}
/**
* Cleanup buckets not used in last hour
*/
private cleanup(): void {
const oneHourAgo = Date.now() - 3600000;
for (const [userId, bucket] of this.buckets.entries()) {
if (bucket.lastRefill < oneHourAgo) {
this.buckets.delete(userId);
}
}
}
/**
* Reset bucket for specific user (admin operation)
*/
resetBucket(userId: string): void {
this.buckets.delete(userId);
}
/**
* Get total number of tracked users
*/
getActiveUsers(): number {
return this.buckets.size;
}
}
// Usage example
const rateLimiter = new TokenBucketRateLimiter({
capacity: 100, // Allow burst of 100 requests
refillRate: 10, // Refill at 10 tokens/second (600/minute)
refillInterval: 100 // Check every 100ms
});
// Check if request allowed
const result = await rateLimiter.allowRequest('user-123', 1);
if (!result.allowed) {
console.log(`Rate limited. Retry after ${result.retryAfter} seconds`);
}
Implementation Strategies
For single-server ChatGPT apps, in-memory rate limiting is simple and performant—the token bucket implementation above stores state in a JavaScript Map with automatic cleanup. However, distributed deployments require shared state across multiple MCP server instances to prevent users from bypassing limits by hitting different servers. Redis is the industry standard for distributed rate limiting, offering atomic operations, TTL-based expiration, and microsecond latency that won't impact your request processing time.
The key challenge in distributed rate limiting is race conditions: two servers might simultaneously read a user's remaining quota as 1, both allow the request, and increment the counter to 2, exceeding the limit. Redis solves this with Lua scripts that execute atomically—the entire rate limit check and counter update happens as a single operation. The Redis INCR command is atomic by default, but complex operations like sliding window calculations require Lua scripting to maintain consistency.
Per-user limits are essential for ChatGPT apps with different subscription tiers. Store rate limit configurations in your database keyed by user ID or subscription level, and cache them in Redis for fast lookups. Use Redis key namespacing like ratelimit:user:{userId}:{algorithm} to organize counters by user and algorithm type. Set appropriate TTLs on Redis keys to automatically expire old data—for a 1-minute fixed window, set TTL to 60 seconds so the counter resets without manual intervention.
For multi-tenancy scenarios where multiple ChatGPT apps run on shared infrastructure, implement hierarchical rate limiting: global limits protect your infrastructure (e.g., 10,000 req/sec across all apps), per-app limits prevent one app from monopolizing resources (e.g., 1,000 req/sec per app), and per-user limits within each app ensure fair usage (e.g., 10 req/min per user). Evaluate limits in order from most specific to least specific, rejecting requests that exceed any limit.
Here's a production-ready Redis-based distributed rate limiter:
// Redis Distributed Rate Limiter with Sliding Window
// Supports distributed deployments, atomic operations, and automatic expiration
import { createClient, RedisClientType } from 'redis';
interface RedisRateLimitConfig {
windowSize: number; // Window size in seconds
maxRequests: number; // Max requests per window
keyPrefix?: string; // Redis key prefix
}
class RedisRateLimiter {
private redis: RedisClientType;
private config: RedisRateLimitConfig;
private luaScript: string;
constructor(
redisClient: RedisClientType,
config: RedisRateLimitConfig
) {
this.redis = redisClient;
this.config = {
keyPrefix: 'ratelimit',
...config
};
// Lua script for atomic sliding window rate limiting
this.luaScript = `
local key = KEYS[1]
local window = tonumber(ARGV[1])
local max_requests = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local window_start = now - window
-- Remove old entries outside window
redis.call('ZREMRANGEBYSCORE', key, '-inf', window_start)
-- Count requests in current window
local count = redis.call('ZCARD', key)
if count < max_requests then
-- Add current request
redis.call('ZADD', key, now, now .. ':' .. math.random())
redis.call('EXPIRE', key, window)
return {1, max_requests - count - 1}
else
-- Get oldest request timestamp for retry-after calculation
local oldest = redis.call('ZRANGE', key, 0, 0, 'WITHSCORES')
local retry_after = window - (now - tonumber(oldest[2]))
return {0, 0, retry_after}
end
`;
}
/**
* Check if request is allowed using sliding window algorithm
*/
async allowRequest(
userId: string,
customLimits?: Partial<RedisRateLimitConfig>
): Promise<{
allowed: boolean;
remaining: number;
retryAfter?: number;
resetAt?: number;
}> {
const config = { ...this.config, ...customLimits };
const key = `${config.keyPrefix}:${userId}`;
const now = Date.now() / 1000; // Convert to seconds
try {
const result = await this.redis.eval(this.luaScript, {
keys: [key],
arguments: [
config.windowSize.toString(),
config.maxRequests.toString(),
now.toString()
]
}) as number[];
const allowed = result[0] === 1;
const remaining = result[1];
const retryAfter = result[2];
return {
allowed,
remaining,
retryAfter: retryAfter > 0 ? Math.ceil(retryAfter) : undefined,
resetAt: Math.floor(now + config.windowSize)
};
} catch (error) {
console.error('Redis rate limit error:', error);
// Fail open: allow request if Redis is unavailable
return { allowed: true, remaining: 0 };
}
}
/**
* Get current usage for user without consuming quota
*/
async getUsage(userId: string): Promise<{
count: number;
limit: number;
remaining: number;
resetAt: number;
}> {
const key = `${this.config.keyPrefix}:${userId}`;
const now = Date.now() / 1000;
const windowStart = now - this.config.windowSize;
try {
// Remove old entries
await this.redis.zRemRangeByScore(key, '-inf', windowStart);
// Count current usage
const count = await this.redis.zCard(key);
const ttl = await this.redis.ttl(key);
return {
count,
limit: this.config.maxRequests,
remaining: Math.max(0, this.config.maxRequests - count),
resetAt: ttl > 0 ? Math.floor(now + ttl) : Math.floor(now + this.config.windowSize)
};
} catch (error) {
console.error('Redis usage check error:', error);
return {
count: 0,
limit: this.config.maxRequests,
remaining: this.config.maxRequests,
resetAt: Math.floor(now + this.config.windowSize)
};
}
}
/**
* Reset rate limit for user (admin operation)
*/
async resetLimit(userId: string): Promise<void> {
const key = `${this.config.keyPrefix}:${userId}`;
await this.redis.del(key);
}
/**
* Get all active rate-limited users
*/
async getActiveUsers(): Promise<string[]> {
const pattern = `${this.config.keyPrefix}:*`;
const keys = await this.redis.keys(pattern);
return keys.map(key => key.replace(`${this.config.keyPrefix}:`, ''));
}
}
// Usage example
const redis = createClient({ url: 'redis://localhost:6379' });
await redis.connect();
const rateLimiter = new RedisRateLimiter(redis, {
windowSize: 60, // 60-second window
maxRequests: 100, // 100 requests per minute
keyPrefix: 'chatgpt-ratelimit'
});
// Check rate limit with custom limits for premium users
const isPremium = user.subscription === 'premium';
const result = await rateLimiter.allowRequest(user.id,
isPremium ? { maxRequests: 1000 } : undefined
);
if (!result.allowed) {
console.log(`Rate limited. Retry after ${result.retryAfter}s`);
}
HTTP Response Handling
When a ChatGPT app request exceeds rate limits, returning proper HTTP responses ensures API clients can gracefully handle the rejection and retry appropriately. The standard status code for rate limiting is 429 Too Many Requests, which signals to clients (including the ChatGPT model) that they should slow down and retry later. Include the Retry-After header with the number of seconds until the limit resets—this allows clients to implement exponential backoff without repeatedly hitting your rate-limited endpoint.
Standard rate limit headers provide transparency about quota consumption: X-RateLimit-Limit shows the maximum requests allowed, X-RateLimit-Remaining shows remaining quota, and X-RateLimit-Reset shows the Unix timestamp when the limit resets. These headers should be included in every response (not just 429s) so clients can proactively slow down before hitting limits. The OpenAI ChatGPT client respects these headers and will automatically manage request pacing.
For debugging and monitoring, include additional headers like X-RateLimit-Policy describing the limit type (e.g., "100 per 1 minute") and X-RateLimit-Scope indicating what's being limited (e.g., "per-user", "per-ip", "global"). During development, a X-RateLimit-Debug header can expose internal state like current token bucket levels, but never expose this in production as it could help attackers optimize abuse patterns.
Error response bodies should be JSON with a clear error message, error code for programmatic handling, and links to documentation about rate limits. Avoid generic messages like "Too many requests"—instead, specify "You've exceeded the 100 requests per minute limit for free accounts. Upgrade to Premium for 1,000 requests per minute: [link]". This converts frustration into a sales opportunity while providing clarity about the limitation.
Here's a production-ready Express middleware for rate limit responses:
// Express Middleware for Rate Limiting with Proper HTTP Headers
// Supports multiple rate limit tiers and comprehensive response headers
import { Request, Response, NextFunction } from 'express';
import { RedisRateLimiter } from './redis-rate-limiter';
interface UserContext {
id: string;
subscription: 'free' | 'starter' | 'professional' | 'business';
email: string;
}
interface RateLimitTier {
maxRequests: number;
windowSize: number;
name: string;
}
const RATE_LIMIT_TIERS: Record<string, RateLimitTier> = {
free: { maxRequests: 10, windowSize: 60, name: 'Free' },
starter: { maxRequests: 100, windowSize: 60, name: 'Starter' },
professional: { maxRequests: 1000, windowSize: 60, name: 'Professional' },
business: { maxRequests: 5000, windowSize: 60, name: 'Business' }
};
export function createRateLimitMiddleware(
rateLimiter: RedisRateLimiter
) {
return async (
req: Request,
res: Response,
next: NextFunction
): Promise<void> => {
try {
// Extract user context from auth middleware
const user = (req as any).user as UserContext;
if (!user) {
res.status(401).json({ error: 'Authentication required' });
return;
}
// Get rate limit tier for user
const tier = RATE_LIMIT_TIERS[user.subscription] || RATE_LIMIT_TIERS.free;
// Check rate limit
const result = await rateLimiter.allowRequest(user.id, {
maxRequests: tier.maxRequests,
windowSize: tier.windowSize
});
// Add rate limit headers to all responses
res.setHeader('X-RateLimit-Limit', tier.maxRequests.toString());
res.setHeader('X-RateLimit-Remaining', result.remaining.toString());
res.setHeader('X-RateLimit-Reset', result.resetAt?.toString() || '');
res.setHeader('X-RateLimit-Policy', `${tier.maxRequests} per ${tier.windowSize} seconds`);
res.setHeader('X-RateLimit-Scope', 'per-user');
if (!result.allowed) {
// Set Retry-After header
if (result.retryAfter) {
res.setHeader('Retry-After', result.retryAfter.toString());
}
// Return 429 with upgrade information
res.status(429).json({
error: 'rate_limit_exceeded',
message: `You've exceeded the ${tier.maxRequests} requests per ${tier.windowSize} seconds limit for ${tier.name} accounts.`,
details: {
limit: tier.maxRequests,
windowSize: tier.windowSize,
retryAfter: result.retryAfter,
resetAt: result.resetAt,
currentTier: tier.name
},
upgrade: {
available: user.subscription !== 'business',
message: 'Upgrade your plan for higher rate limits',
url: 'https://makeaihq.com/pricing',
nextTier: getNextTier(user.subscription)
}
});
return;
}
// Request allowed, proceed to next middleware
next();
} catch (error) {
console.error('Rate limit middleware error:', error);
// Fail open: allow request if rate limiter fails
next();
}
};
}
/**
* Get next available tier for upgrade suggestions
*/
function getNextTier(current: string): RateLimitTier | null {
const tiers = ['free', 'starter', 'professional', 'business'];
const currentIndex = tiers.indexOf(current);
const nextTierName = tiers[currentIndex + 1];
return nextTierName ? RATE_LIMIT_TIERS[nextTierName] : null;
}
// Usage example
import express from 'express';
import { createClient } from 'redis';
const app = express();
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();
const rateLimiter = new RedisRateLimiter(redis, {
windowSize: 60,
maxRequests: 100,
keyPrefix: 'chatgpt-app'
});
// Apply rate limiting to MCP endpoints
app.use('/mcp', createRateLimitMiddleware(rateLimiter));
Throttling Strategies
While rate limiting rejects excess requests, throttling delays them to smooth traffic and protect backend resources. For ChatGPT apps, throttling is particularly useful when calling external APIs with strict rate limits (like database connections, payment processors, or AI model APIs) that can't be scaled instantly. Instead of failing requests when upstream services are slow, queue them and process at a sustainable rate.
Request queuing with priority levels ensures critical operations (like authentication or payment processing) are handled before non-critical ones (like analytics). Implement a priority queue where each user's requests are ordered by priority and timestamp, with higher-priority requests processed first. Use a concurrency limiter to control how many requests are processed simultaneously—for example, limit to 10 concurrent database queries regardless of how many requests are queued.
Backpressure mechanisms signal to clients when your system is overloaded, encouraging them to slow down before hitting hard rate limits. For MCP servers, return a Retry-After header with increasing values as queue depth grows: if the queue has 10 items, suggest retrying in 1 second; at 100 items, suggest 10 seconds. This creates a feedback loop where clients naturally throttle themselves based on system load.
Circuit breakers prevent cascading failures when downstream services are unavailable. If your ChatGPT app calls an external API that starts failing, the circuit breaker "opens" after a threshold of failures and immediately rejects new requests for a cooldown period instead of waiting for timeouts. This prevents queue buildup and allows the downstream service to recover. After the cooldown, the circuit breaker enters a "half-open" state where a few test requests are allowed through—if they succeed, the circuit closes and normal operation resumes.
Here's a production-ready request queue manager with throttling:
// Request Queue Manager with Priority and Throttling
// Supports concurrent request limits, priority queuing, and backpressure
interface QueuedRequest<T> {
id: string;
priority: number; // Higher = more important
timestamp: number;
execute: () => Promise<T>;
resolve: (value: T) => void;
reject: (error: Error) => void;
}
interface ThrottleConfig {
maxConcurrent: number; // Max concurrent executions
maxQueueSize: number; // Max queued requests
requestTimeout: number; // Timeout per request (ms)
}
class RequestThrottler<T = any> {
private queue: QueuedRequest<T>[] = [];
private activeRequests = 0;
private config: ThrottleConfig;
private processing = false;
constructor(config: ThrottleConfig) {
this.config = config;
}
/**
* Throttle a request with priority
*/
async throttle(
execute: () => Promise<T>,
priority: number = 0
): Promise<T> {
return new Promise((resolve, reject) => {
// Check queue size for backpressure
if (this.queue.length >= this.config.maxQueueSize) {
reject(new Error(
`Queue full (${this.queue.length}/${this.config.maxQueueSize}). ` +
`Retry after ${this.getBackpressureDelay()}s`
));
return;
}
// Add to priority queue
const request: QueuedRequest<T> = {
id: this.generateId(),
priority,
timestamp: Date.now(),
execute,
resolve,
reject
};
this.queue.push(request);
this.sortQueue();
this.processQueue();
});
}
/**
* Sort queue by priority (descending) then timestamp (ascending)
*/
private sortQueue(): void {
this.queue.sort((a, b) => {
if (a.priority !== b.priority) {
return b.priority - a.priority; // Higher priority first
}
return a.timestamp - b.timestamp; // Older requests first
});
}
/**
* Process queued requests with concurrency control
*/
private async processQueue(): Promise<void> {
if (this.processing) return;
this.processing = true;
while (this.queue.length > 0 && this.activeRequests < this.config.maxConcurrent) {
const request = this.queue.shift();
if (!request) break;
this.activeRequests++;
this.executeRequest(request)
.finally(() => {
this.activeRequests--;
this.processQueue(); // Process next item
});
}
this.processing = false;
}
/**
* Execute single request with timeout
*/
private async executeRequest(request: QueuedRequest<T>): Promise<void> {
const timeoutPromise = new Promise<never>((_, reject) => {
setTimeout(() => {
reject(new Error(`Request timeout after ${this.config.requestTimeout}ms`));
}, this.config.requestTimeout);
});
try {
const result = await Promise.race([
request.execute(),
timeoutPromise
]);
request.resolve(result);
} catch (error) {
request.reject(error as Error);
}
}
/**
* Calculate backpressure delay based on queue depth
*/
private getBackpressureDelay(): number {
const queueDepth = this.queue.length;
const percentage = queueDepth / this.config.maxQueueSize;
// Exponential backoff: 1s at 50% full, 10s at 90% full
return Math.ceil(Math.pow(percentage * 10, 2));
}
/**
* Get current queue statistics
*/
getStats(): {
queueSize: number;
activeRequests: number;
backpressureDelay: number;
utilizationPercent: number;
} {
return {
queueSize: this.queue.length,
activeRequests: this.activeRequests,
backpressureDelay: this.getBackpressureDelay(),
utilizationPercent: Math.round(
(this.activeRequests / this.config.maxConcurrent) * 100
)
};
}
/**
* Clear all queued requests
*/
clear(): void {
this.queue.forEach(req => {
req.reject(new Error('Queue cleared'));
});
this.queue = [];
}
private generateId(): string {
return `${Date.now()}-${Math.random().toString(36).substr(2, 9)}`;
}
}
// Usage example
const throttler = new RequestThrottler({
maxConcurrent: 10, // Process 10 requests at a time
maxQueueSize: 1000, // Queue up to 1000 requests
requestTimeout: 30000 // 30-second timeout per request
});
// High-priority request (authentication)
try {
const authResult = await throttler.throttle(
() => authenticateUser(token),
priority: 10 // High priority
);
} catch (error) {
console.error('Auth failed:', error);
}
// Low-priority request (analytics)
await throttler.throttle(
() => logAnalytics(event),
priority: 1 // Low priority
);
// Monitor queue stats
const stats = throttler.getStats();
if (stats.utilizationPercent > 80) {
console.warn('High system load:', stats);
}
Monitoring & Alerting
Effective rate limiting requires continuous monitoring to detect abuse patterns, optimize limits, and plan capacity. Track per-endpoint metrics to identify which MCP tools are most frequently rate-limited—if 90% of rate limit hits are on a single expensive operation, consider adding caching or increasing limits specifically for that endpoint. Monitor per-user metrics to identify outliers: users hitting rate limits constantly may be using inefficient clients, while users never approaching limits may indicate your limits are too conservative.
Set up real-time alerts for anomalous patterns: sudden spikes in rate limit rejections may indicate a DDoS attack, while gradual increases suggest organic growth requiring infrastructure scaling. Alert when global rate limit utilization exceeds 80%—this gives you time to scale before hitting hard limits. For ChatGPT apps, monitor the distribution of rate limit usage across subscription tiers: if free users consistently hit limits while paid users don't, your pricing tiers may be well-calibrated.
Capacity planning uses historical rate limit data to forecast infrastructure needs. If your Professional tier users average 400 requests per minute against a 1,000 limit, you have 2.5x headroom before needing to optimize or upgrade infrastructure. Track the 95th percentile of usage rather than averages—averages hide spikes that can cause outages. Use time-series databases like Prometheus or InfluxDB to store rate limit metrics with tags for user, endpoint, tier, and outcome (allowed vs rejected).
Integrate rate limit metrics with your observability stack: send events to DataDog, New Relic, or Grafana Cloud for correlation with application performance metrics. Dashboard visualizations should show rate limit hit rates over time, broken down by user tier and endpoint. During incidents, this data helps distinguish between legitimate traffic spikes and abuse. For ChatGPT apps specifically, correlate rate limits with OpenAI's app analytics to understand how rate limiting affects user engagement.
Here's a production-ready abuse detection and monitoring system:
// Rate Limit Monitoring and Abuse Detection System
// Tracks metrics, detects anomalies, and generates alerts
import { EventEmitter } from 'events';
interface RateLimitEvent {
userId: string;
endpoint: string;
allowed: boolean;
remaining: number;
timestamp: number;
tier: string;
}
interface AbusePattern {
userId: string;
severity: 'low' | 'medium' | 'high';
pattern: string;
evidence: string[];
firstSeen: number;
lastSeen: number;
occurrences: number;
}
class RateLimitMonitor extends EventEmitter {
private events: RateLimitEvent[] = [];
private abusePatterns: Map<string, AbusePattern> = new Map();
private readonly MAX_EVENTS = 100000; // Keep last 100k events
/**
* Record rate limit event
*/
recordEvent(event: RateLimitEvent): void {
this.events.push(event);
// Trim old events
if (this.events.length > this.MAX_EVENTS) {
this.events = this.events.slice(-this.MAX_EVENTS);
}
// Check for abuse patterns
if (!event.allowed) {
this.checkAbusePatterns(event);
}
// Emit event for external monitoring systems
this.emit('ratelimit', event);
}
/**
* Detect potential abuse patterns
*/
private checkAbusePatterns(event: RateLimitEvent): void {
const userId = event.userId;
const recentEvents = this.getUserEvents(userId, 300000); // Last 5 minutes
// Pattern 1: Sustained high-frequency requests
const rejectRate = recentEvents.filter(e => !e.allowed).length / recentEvents.length;
if (recentEvents.length > 100 && rejectRate > 0.8) {
this.recordAbusePattern({
userId,
severity: 'high',
pattern: 'sustained_high_frequency',
evidence: [
`${recentEvents.length} requests in 5 minutes`,
`${Math.round(rejectRate * 100)}% rejected`
]
});
}
// Pattern 2: Rapid retries without backoff
const retryIntervals = this.getRetryIntervals(userId);
const hasBackoff = retryIntervals.some(interval => interval > 1000);
if (retryIntervals.length > 10 && !hasBackoff) {
this.recordAbusePattern({
userId,
severity: 'medium',
pattern: 'no_backoff',
evidence: [
`${retryIntervals.length} rapid retries`,
`Average interval: ${Math.round(retryIntervals.reduce((a, b) => a + b, 0) / retryIntervals.length)}ms`
]
});
}
// Pattern 3: Endpoint scanning (hitting many different endpoints)
const uniqueEndpoints = new Set(recentEvents.map(e => e.endpoint));
if (uniqueEndpoints.size > 20 && rejectRate > 0.5) {
this.recordAbusePattern({
userId,
severity: 'high',
pattern: 'endpoint_scanning',
evidence: [
`${uniqueEndpoints.size} unique endpoints`,
`Appears to be scanning for vulnerabilities`
]
});
}
}
/**
* Record or update abuse pattern
*/
private recordAbusePattern(pattern: Omit<AbusePattern, 'firstSeen' | 'lastSeen' | 'occurrences'>): void {
const key = `${pattern.userId}:${pattern.pattern}`;
const existing = this.abusePatterns.get(key);
if (existing) {
existing.lastSeen = Date.now();
existing.occurrences++;
existing.evidence = pattern.evidence; // Update with latest evidence
} else {
this.abusePatterns.set(key, {
...pattern,
firstSeen: Date.now(),
lastSeen: Date.now(),
occurrences: 1
});
}
// Emit abuse alert
this.emit('abuse_detected', this.abusePatterns.get(key));
}
/**
* Get events for specific user within time window
*/
private getUserEvents(userId: string, windowMs: number): RateLimitEvent[] {
const cutoff = Date.now() - windowMs;
return this.events.filter(e =>
e.userId === userId && e.timestamp > cutoff
);
}
/**
* Calculate retry intervals for user
*/
private getRetryIntervals(userId: string): number[] {
const events = this.getUserEvents(userId, 60000); // Last minute
const intervals: number[] = [];
for (let i = 1; i < events.length; i++) {
intervals.push(events[i].timestamp - events[i - 1].timestamp);
}
return intervals;
}
/**
* Get abuse patterns for user or all users
*/
getAbusePatterns(userId?: string): AbusePattern[] {
const patterns = Array.from(this.abusePatterns.values());
return userId
? patterns.filter(p => p.userId === userId)
: patterns;
}
/**
* Get statistics for monitoring dashboard
*/
getStats(timeWindowMs: number = 3600000): {
totalRequests: number;
allowedRequests: number;
rejectedRequests: number;
rejectRate: number;
topEndpoints: Array<{ endpoint: string; count: number }>;
topUsers: Array<{ userId: string; count: number }>;
abuseCount: number;
} {
const cutoff = Date.now() - timeWindowMs;
const recentEvents = this.events.filter(e => e.timestamp > cutoff);
const endpointCounts = new Map<string, number>();
const userCounts = new Map<string, number>();
recentEvents.forEach(event => {
endpointCounts.set(
event.endpoint,
(endpointCounts.get(event.endpoint) || 0) + 1
);
userCounts.set(
event.userId,
(userCounts.get(event.userId) || 0) + 1
);
});
const allowed = recentEvents.filter(e => e.allowed).length;
const rejected = recentEvents.length - allowed;
return {
totalRequests: recentEvents.length,
allowedRequests: allowed,
rejectedRequests: rejected,
rejectRate: recentEvents.length > 0 ? rejected / recentEvents.length : 0,
topEndpoints: Array.from(endpointCounts.entries())
.map(([endpoint, count]) => ({ endpoint, count }))
.sort((a, b) => b.count - a.count)
.slice(0, 10),
topUsers: Array.from(userCounts.entries())
.map(([userId, count]) => ({ userId, count }))
.sort((a, b) => b.count - a.count)
.slice(0, 10),
abuseCount: this.abusePatterns.size
};
}
/**
* Clear old abuse patterns (cleanup task)
*/
cleanupAbusePatterns(maxAgeMs: number = 86400000): void {
const cutoff = Date.now() - maxAgeMs;
for (const [key, pattern] of this.abusePatterns.entries()) {
if (pattern.lastSeen < cutoff) {
this.abusePatterns.delete(key);
}
}
}
}
// Usage example
const monitor = new RateLimitMonitor();
// Record rate limit events
monitor.recordEvent({
userId: 'user-123',
endpoint: '/mcp/tools/search',
allowed: false,
remaining: 0,
timestamp: Date.now(),
tier: 'free'
});
// Listen for abuse alerts
monitor.on('abuse_detected', (pattern) => {
console.error('Abuse detected:', pattern);
if (pattern.severity === 'high') {
// Send alert to security team
// Consider automatic IP blocking
}
});
// Generate monitoring dashboard data
const stats = monitor.getStats(3600000); // Last hour
console.log('Rate limit stats:', stats);
// Cleanup old patterns daily
setInterval(() => monitor.cleanupAbusePatterns(), 86400000);
Conclusion
Rate limiting and throttling are foundational security mechanisms for production ChatGPT apps, protecting your infrastructure from abuse while ensuring fair resource distribution across users. The token bucket algorithm provides the best balance for conversational applications, allowing natural burst patterns while preventing sustained abuse. For distributed deployments, Redis-based rate limiting with Lua scripts ensures atomic operations and consistency across multiple MCP server instances. Proper HTTP response handling with 429 status codes, Retry-After headers, and comprehensive rate limit headers enables clients to gracefully handle limits and implement appropriate backoff strategies.
Throttling strategies like request queuing with priority levels and backpressure mechanisms ensure your app degrades gracefully under load rather than failing catastrophically. Continuous monitoring with abuse pattern detection helps you optimize limits, identify infrastructure bottlenecks, and detect malicious activity before it impacts legitimate users. By implementing these production-tested patterns, your ChatGPT app will scale reliably from hundreds to millions of requests while maintaining consistent performance and security.
Ready to build production-grade ChatGPT apps with enterprise security? MakeAIHQ automatically implements rate limiting, throttling, and abuse detection in every generated MCP server—no configuration required. Our AI-powered platform generates production-ready ChatGPT apps with distributed rate limiting, Redis integration, and comprehensive monitoring built in. Start your free trial today and deploy ChatGPT apps with enterprise-grade security in under 48 hours.
Related Resources
- ChatGPT App Security Hardening Guide - Complete security implementation guide
- MCP Server Rate Limiting Implementation - Specific rate limiting patterns for MCP servers
- Redis Caching Patterns for ChatGPT Apps - Redis performance optimization
- DDoS Protection for ChatGPT Apps - Advanced traffic filtering
External References
- IETF Rate Limit Headers Specification - Standard HTTP rate limit headers
- Redis Rate Limiting Patterns - Official Redis rate limiting documentation
- API Rate Limiting Best Practices - Industry best practices for API rate limiting