Architecture

System Overview

Helicone is built as a modular, scalable system with six core components:

┌─────────────────────────────────────────────────────────┐
│                      User / Browser                      │
└────────────┬────────────────────────────────────────────┘
             │
             │ HTTPS
             │
┌────────────▼─────────────────────────────────────────────┐
│                    Web (Next.js)                          │
│  • Dashboard UI                                           │
│  • Request visualization                                  │
│  • Analytics and reports                                  │
└────────────┬──────────────────────────────┬──────────────┘
             │                              │
             │ API Calls                    │ Auth
             │                              │
┌────────────▼──────────────────────────────▼──────────────┐
│                    Jawn (Express + Tsoa)                  │
│  • REST API server                                        │
│  • Request logging                                        │
│  • Prompt management                                      │
│  • User/org management                                    │
└─┬────────┬────────┬────────┬────────┬────────────────────┘
  │        │        │        │        │
  │        │        │        │        └─────────┐
  │        │        │        │                  │
┌─▼────┐ ┌▼────────▼┐ ┌─────▼──────┐ ┌────────▼──────┐
│Worker│ │PostgreSQL│ │ ClickHouse │ │     MinIO     │
│      │ │          │ │            │ │  (S3 Storage) │
│Proxy │ │ App DB   │ │Analytics DB│ │               │
│      │ │          │ │            │ │               │
└──┬───┘ └──────────┘ └────────────┘ └───────────────┘
   │
   │ Proxied requests
   │
┌──▼──────────────────────────────────────────────────────┐
│              LLM Providers (OpenAI, etc.)               │
└─────────────────────────────────────────────────────────┘

Core Components

Web (Next.js)

Technology: Next.js 14, React, TailwindCSS Purpose: Frontend dashboard for visualizing and managing your LLM observability data. Key Features:

Request logs and trace visualization
Cost and latency analytics
Prompt management UI
Session and agent tracing
User and organization management
Evaluation and scoring interfaces

Configuration:

# Environment variables
NEXT_PUBLIC_HELICONE_JAWN_SERVICE=http://localhost:8585
BETTER_AUTH_SECRET=your-secret-key
DATABASE_URL=postgresql://...

Default Port: 3000

Jawn (Backend API)

Technology: Node.js, Express, TypeScript, Tsoa (OpenAPI) Purpose: Main backend API server that handles all business logic and data operations. Responsibilities:

REST API endpoints for the web dashboard
Request and response logging
Prompt versioning and deployment
User authentication and authorization
Analytics aggregation
Webhook management
Dataset creation and management

Configuration:

# Core settings
PORT=8585
NODE_ENV=production

# Databases
SUPABASE_DATABASE_URL=postgresql://...
CLICKHOUSE_HOST=http://clickhouse:8123

# Object storage
S3_ENDPOINT=http://minio:9000
S3_BUCKET_NAME=request-response-storage
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin

# Auth
BETTER_AUTH_SECRET=your-secret-key

Default Port: 8585 Key Endpoints:

/v1/request/* - Request logging and retrieval
/v1/prompt/* - Prompt management
/v1/user/* - User and organization management
/v1/analytics/* - Analytics queries

Worker (Cloudflare Workers / Node.js)

Technology: TypeScript, running on Node.js (self-hosted) or Cloudflare Workers (cloud) Purpose: High-performance proxy that intercepts LLM requests, logs them, and forwards to providers. Worker Types:

OpenAI Proxy (Port 8787)
- Proxies requests to OpenAI API
- Logs requests to ClickHouse
- Stores bodies in MinIO
Anthropic Proxy (Port 8790)
- Proxies requests to Anthropic API
- Handles streaming responses
Gateway API (Port 8789)
- Unified gateway for multiple providers
- Intelligent routing and fallbacks
Helicone API (Port 8788)
- Internal API worker
- Async logging endpoints

Configuration:

WORKER_TYPE=OPENAI_PROXY
WORKER_PORT=8787
CLICKHOUSE_HOST=http://clickhouse:8123
VALHALLA_URL=http://jawn:8585
S3_ENDPOINT=http://minio:9000

PostgreSQL (Application Database)

Technology: PostgreSQL 17 Purpose: Stores application data including users, organizations, API keys, prompts, and metadata. Key Tables:

auth.users - User accounts
public.organization - Organizations
public.helicone_api_keys - API keys
public.prompts - Prompt versions
public.request - Request metadata (references ClickHouse)
public.properties - Custom properties
public.feedback - User feedback and ratings

Configuration:

POSTGRES_HOST=db
POSTGRES_PORT=5432
POSTGRES_DB=helicone_test
POSTGRES_USER=postgres
POSTGRES_PASSWORD=password

Default Port: 5432 (mapped to 54388 in docker-compose)

ClickHouse (Analytics Database)

Technology: ClickHouse 24.3 Purpose: High-performance columnar database for storing and analyzing request logs at scale. Key Tables:

default.request_response_versioned - Main request/response log
default.request_response_rmt - Replicated table for HA
default.properties_v2 - Request properties
default.score_value - Evaluation scores
default.cache_hits - Cache performance data

Why ClickHouse?

Handles billions of rows efficiently
Fast aggregations for analytics
Excellent compression (10x+ vs PostgreSQL)
Optimized for time-series data

Configuration:

CLICKHOUSE_HOST=http://clickhouse:8123
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=

Default Ports:

HTTP: 8123 (mapped to 18123)
Native: 9000 (mapped to 19000)

MinIO (Object Storage)

Technology: MinIO (S3-compatible) Purpose: Stores large request and response bodies separately from metadata. Buckets:

request-response-storage - Full request/response bodies
prompt-body-storage - Prompt templates and versions
hql-store - HQL query results

Why Separate Storage?

Request/response bodies can be very large (images, documents, etc.)
Storing in database would bloat tables and slow queries
S3-compatible storage is cheap and scalable
Bodies are referenced by ID in ClickHouse

Configuration:

MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
S3_ENDPOINT=http://minio:9000
S3_BUCKET_NAME=request-response-storage

Default Ports:

API: 9000
Console: 9001

Request Flow

1. Proxied Request Flow

When an LLM request goes through Helicone’s proxy:

┌──────────┐
│  Client  │
└────┬─────┘
     │ 1. Request with Helicone headers
     │    POST http://localhost:8787/v1/chat/completions
     │    Headers: Helicone-Auth, Helicone-Property-*, etc.
     │
┌────▼────────────────────────────────────────────────┐
│  Worker (OpenAI Proxy)                              │
│  1. Validate API key (check PostgreSQL via Jawn)    │
│  2. Extract metadata and properties                 │
│  3. Forward to OpenAI                               │
└────┬─────────────────┬────────────────────────────┬─┘
     │                 │                            │
     │ 4. Log request  │ 5. Store body             │ 3. Forward
     │                 │                            │
┌────▼──────────┐ ┌────▼──────┐            ┌───────▼──────┐
│  ClickHouse   │ │   MinIO   │            │   OpenAI     │
│  (metadata)   │ │  (bodies) │            │     API      │
└───────────────┘ └───────────┘            └───────┬──────┘
                                                    │
                                                    │ 6. Response
┌───────────────────────────────────────────────────▼──┐
│  Worker                                              │
│  7. Log response metadata to ClickHouse              │
│  8. Store response body to MinIO                     │
│  9. Return response to client                        │
└──────────────────────────────────────────────────────┘

2. Dashboard Query Flow

When viewing requests in the dashboard:

┌──────────┐
│ Browser  │
└────┬─────┘
     │ 1. GET /requests?page=1
     │
┌────▼─────────────────────────────────────┐
│  Web (Next.js)                            │
│  2. Server-side render or API call        │
└────┬──────────────────────────────────────┘
     │ 3. GET /v1/request/query
     │
┌────▼──────────────────────────────────────┐
│  Jawn API                                  │
│  4. Build ClickHouse query                 │
│  5. Execute query                          │
└────┬───────────────────┬───────────────────┘
     │                   │
     │ 6. Query metadata │ 7. Fetch bodies (if needed)
     │                   │
┌────▼──────────┐   ┌────▼──────┐
│  ClickHouse   │   │   MinIO   │
│               │   │           │
└────┬──────────┘   └────┬──────┘
     │                   │
     │ 8. Return data    │
     │                   │
┌────▼───────────────────▼───────────┐
│  Jawn API                           │
│  9. Combine metadata + bodies       │
│  10. Return JSON response           │
└────┬────────────────────────────────┘
     │
┌────▼─────────────────────────────────┐
│  Web                                  │
│  11. Render in UI                     │
└───────────────────────────────────────┘

Data Storage Strategy

What Goes Where?

Data Type	Storage	Reason
User accounts	PostgreSQL	Relational, requires ACID
API keys	PostgreSQL	Needs auth lookups
Prompt versions	PostgreSQL	Small, relational
Request metadata	ClickHouse	Fast analytics, billions of rows
Request/response bodies	MinIO	Large blobs, cheap storage
Properties	ClickHouse	Time-series, analytics
Scores/evaluations	ClickHouse	Time-series, analytics

Data Retention

┌─────────────────────────────────────────────────────┐
│ PostgreSQL (Keep Forever)                           │
│ • User accounts                                     │
│ • Organizations                                     │
│ • API keys                                          │
│ • Prompt templates                                  │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│ ClickHouse (Configurable Retention)                 │
│ • Request logs: 90 days default                     │
│ • Can configure TTL per table                       │
│ • Aggregations stored longer                        │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│ MinIO (Lifecycle Policies)                          │
│ • Request bodies: 90 days default                   │
│ • Transition to cold storage after 30 days          │
│ • Configurable per bucket                           │
└─────────────────────────────────────────────────────┘

High Availability Architecture

For production deployments:

┌─────────────────────────────────────────────────────┐
│              Load Balancer / Ingress                 │
└─────┬────────────────────────────────────────┬──────┘
      │                                        │
┌─────▼─────┐                           ┌─────▼─────┐
│  Web (1)  │                           │  Jawn (1) │
│  Web (2)  │◄──────────────────────────┤  Jawn (2) │
│  Web (3)  │    Session Sharing         │  Jawn (3) │
└───────────┘                           └─────┬─────┘
                                              │
                                              │
               ┌──────────────────────────────┼────────────┐
               │                              │            │
        ┌──────▼────────┐            ┌────────▼─────┐ ┌───▼────┐
        │  PostgreSQL   │            │  ClickHouse  │ │ MinIO  │
        │               │            │              │ │        │
        │ Primary + 2   │            │ 3-node       │ │4-node  │
        │ Read Replicas │            │ cluster      │ │cluster │
        └───────────────┘            └──────────────┘ └────────┘

Security Architecture

Authentication Flow

User Authentication (Better Auth)
- Email/password via PostgreSQL
- Session tokens stored in cookies
- OAuth providers (Google, GitHub)
API Key Authentication
- API keys stored in PostgreSQL
- Validated on each proxy request
- Scoped to organizations
Service-to-Service
- Internal services trust each other
- Network isolation in production
- Service accounts for workers

Data Security

┌─────────────────────────────────────────────────────┐
│  Encryption at Rest                                  │
│  • PostgreSQL: Encrypted volumes                     │
│  • ClickHouse: Encrypted volumes                     │
│  • MinIO: SSE (Server-Side Encryption)               │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Encryption in Transit                               │
│  • TLS 1.3 for all external traffic                  │
│  • Internal: Can use mTLS in production              │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│  Access Control                                      │
│  • Row-level security in PostgreSQL                  │
│  • Organization-scoped queries                       │
│  • API key permissions                               │
└─────────────────────────────────────────────────────┘

Scalability

Horizontal Scaling

Stateless Services (can scale infinitely):

Web: Add more replicas behind load balancer
Jawn: Add more replicas behind load balancer
Workers: Add more worker instances

Stateful Services (require clustering):

PostgreSQL: Primary + read replicas
ClickHouse: Sharded cluster with replication
MinIO: Distributed mode with erasure coding

Vertical Scaling

Recommended resource allocation:

Service	CPU	Memory	Storage
Web	0.5-1 core	1-2GB	Minimal
Jawn	1-2 cores	2-4GB	Minimal
Worker	0.5-1 core	512MB-1GB	Minimal
PostgreSQL	2-4 cores	4-8GB	100GB+
ClickHouse	4-8 cores	8-16GB	500GB+
MinIO	1-2 cores	2-4GB	1TB+

Monitoring

Health Checks

All services expose health endpoints:

# Web
curl http://localhost:3000/api/health

# Jawn
curl http://localhost:8585/v1/public/healthcheck

# ClickHouse
curl http://localhost:8123/ping

# PostgreSQL
pg_isready -h localhost -p 5432

# MinIO
curl http://localhost:9000/minio/health/live

Metrics

Exposed Prometheus metrics:

HTTP request latency (p50, p95, p99)
Request volume per endpoint
Error rates
Database connection pool stats
Queue depths
Cache hit rates

Get Started

AI Gateway

Observability

Prompt Management

Evaluation & Testing

Features

Self-Hosting

Integrations

System Overview

Core Components

Web (Next.js)

Jawn (Backend API)

Worker (Cloudflare Workers / Node.js)

PostgreSQL (Application Database)

ClickHouse (Analytics Database)

MinIO (Object Storage)

Request Flow

1. Proxied Request Flow

2. Dashboard Query Flow

Data Storage Strategy

What Goes Where?

Data Retention

High Availability Architecture

Security Architecture

Authentication Flow

Data Security

Scalability

Horizontal Scaling

Vertical Scaling

Monitoring

Health Checks

Metrics

Next Steps

Docker Deployment

Kubernetes

Get Started

AI Gateway

Observability

Prompt Management

Evaluation & Testing

Features

Self-Hosting

Integrations

Documentation Index

​System Overview

​Core Components

​Web (Next.js)

​Jawn (Backend API)

​Worker (Cloudflare Workers / Node.js)

​PostgreSQL (Application Database)

​ClickHouse (Analytics Database)

​MinIO (Object Storage)

​Request Flow

​1. Proxied Request Flow

​2. Dashboard Query Flow

​Data Storage Strategy

​What Goes Where?

​Data Retention

​High Availability Architecture

​Security Architecture

​Authentication Flow

​Data Security

​Scalability

​Horizontal Scaling

​Vertical Scaling

​Monitoring

​Health Checks

​Metrics

​Next Steps

Docker Deployment

Kubernetes

System Overview

Core Components

Web (Next.js)

Jawn (Backend API)

Worker (Cloudflare Workers / Node.js)

PostgreSQL (Application Database)

ClickHouse (Analytics Database)

MinIO (Object Storage)

Request Flow

1. Proxied Request Flow

2. Dashboard Query Flow

Data Storage Strategy

What Goes Where?

Data Retention

High Availability Architecture

Security Architecture

Authentication Flow

Data Security

Scalability

Horizontal Scaling

Vertical Scaling

Monitoring

Health Checks

Metrics

Next Steps