Muhamed Ramees Cheriya Mukkolakkal

Muhamed Ramees Cheriya Mukkolakkal

This paper presents two independent contributions for Apache Pulsar practitioners. First, we validate 1,499,947 msg/s at 3.88 ms median publish latency on just three bare-metal Kubernetes nodes running Pulsar 4.0.8 with Java 21 and ZGC Generational garbage collection, and project a hardware-driven path to 15 million msg/s on 15 machines using five independent clusters with key-based partition routing. Hardware selection -- specifically dedicated NVMe journals achieving 0.02 ms fdatasync and 25 Gbps network interfaces -- is the primary determinant of throughput ceiling, not compute or software tuning. Second, we trace the complete latency optimization journey from 213 ms GC spikes and 13-18 ms median publish latency in production to 3.88 ms through Java Flight Recorder guided root cause analysis. Three independent root causes are identified and resolved: G1GC pauses eliminated by switching to ZGC Generational; journal fdatasync latency reduced from 5.1 ms to 0.02 ms through NVMe journal dedication; and a previously undocumented Linux kernel page cache writeback interaction inside BookKeeper's ForceWriteThread that degrades fdatasync from under 1 ms to 15-22 ms even across physically separate NVMe drives sharing the kernel block layer. This finding is undocumented in official Apache Pulsar and BookKeeper documentation and is relevant to all Pulsar operators experiencing unexplained P99.9 latency spikes. The combined optimizations achieve a 4.7x latency improvement at 50x higher throughput.

Muhamed Ramees Cheriya Mukkolakkal

Operating Elasticsearch clusters at scale demands continuous human expertise spanning the full lifecycle -- from initial deployment through performance tuning, monitoring, failure prediction, and incident recovery. We present the ES Guardian Agent, an autonomous AI SRE system that manages the complete Elasticsearch lifecycle without human intervention through eleven distinct phases: Evaluate, Optimize, Deploy, Calibrate, Stabilize, Alert, Predict, Heal, Learn, and Upgrade. A critical differentiator is its multi-source predictive failure engine, which continuously ingests and correlates metrics trends, application logs, and kernel-level telemetry -- including Linux dmesg streams, NVMe SMART data, NIC bond statistics, and thermal sensors -- to anticipate failures hours before they materialize. By cross-referencing current system signatures against a persistent incident memory of resolved failures, the AI engine stages corrective actions proactively. Through four successive agent architectures -- culminating in a 4,589-line system with five monitoring layers and an iterative AI action loop -- we demonstrate that an LLM equipped with tool-use access can function as a full-lifecycle autonomous SRE targeting six-nines (99.9999%) availability. In production evaluation, the Guardian Agent executed 300 autonomous investigation-and-repair cycles, recovered a cluster from an 18-hour cross-system outage, diagnosed hardware NIC failures across all host nodes, and maintained continuous operational visibility. We establish that data volume per shard -- not tuning -- is the primary determinant of query performance, with latency scaling at 0.26 ms per MB/shard.