Sentinel

essentially a single node, splunk-inspired log engine

A log aggregation platform with custom query language, real-time indexing, and sub-100ms search.

Try Sentinel →    _(mirror)

Performance

• Search Latency: <100ms
• Query Complexity: O(1) keyword and field lookups
• Propagation: <50ms from ingestion to websocket broadcast
• Persistence: 100% data recovery via segmented JSON storage
• Volume: Synthetic log generation via GitHub Actions for sustained load testing

Tech Stack

• Backend: Go, Gorilla WebSocket, Custom Inverted Index
• Frontend: Next.js, TypeScript, Tailwind CSS
• Infrastructure: Fly.io, Vercel, GitHub Actions

Architecture

• Ingestion: Receives JSON logs, generates unique IDs, and stores them in memory.
• Storage: Periodically flushes logs to disk in JSON format once a segment hits 10MB to ensure persistence.
• Indexing: Real-time indexing into a text-based inverted index and a time-bucketed index for fast lookups.
• Queries: Parses query strings to find log IDs in the indexes first, then retrieves the full log entries.
• Live Stream: Broadcasts incoming logs to all connected WebSocket clients with no delay.
• Concurrency: Manages simultaneous writes and searches using Go's thread-safe maps and mutexes.

How to Run

Local

1. Clone repository

   git clone https://github.com/rsashank/sentinel
   cd sentinel

2. Start Backend

   go run cmd/server/main.go

3. Start Generator (Optional)

   go run cmd/generator/main.go

4. Start Frontend

   cd frontend
   npm install
   npm run dev

Docker

docker build -t sentinel .
docker run -p 8080:8080 sentinel