Principal Engineer

We are seeking a Sr. Staff / Principal AI Systems Engineer to build, scale, and operate the AI Agent Platform that powers reasoning-driven assistants and autonomous agents across Freshworks — and to make that platform operate itself. You will own the systems architecture and engineering of a multi-tenant agent runtime that serves agentic workloads at high throughput and low latency, and you will pioneer an Agentic AIOps approach where autonomous agents monitor, diagnose, and remediate the platform in production.

This is a hands-on systems engineering role at the staff/principal level, with a strong platform-scale and agentic-operations center of gravity. You'll design how thousands of agents are orchestrated, served, observed, and kept healthy at enterprise scale, and you'll set the technical direction which the broader engineering organization builds against. The level (Sr. Staff vs. Principal) will be calibrated to your scope of technical ownership and organizational impact.

Key Responsibilities:

Scalable Agent Platform Systems

Architect and build the core AI Agent Platform — agent runtime, orchestration, tool/API invocation, state and memory management, and the retrieval/knowledge services agents reason over

Design for scale and efficiency: high-throughput multi-tenant serving, concurrency and queueing for agent workloads, model/inference routing, caching, and cost-aware execution

Build the control plane and systems primitives other teams use to define, deploy, version, and operate agents safely

Drive latency, throughput, and cost optimization across the agentic request path (planning → retrieval → tool calls → generation)

Agentic AIOps & Autonomous Operations

Architect agentic operations workflows where autonomous agents observe platform telemetry, reason about anomalies, perform root-cause analysis, and execute remediation — shifting operations from human-driven to agent-driven

Design multi-agent operational loops (detection, diagnosis, remediation) that collaborate, escalate, and hand off to on-call humans with clear rationale and audit trails

Build closed-loop self-healing for the platform: auto-detection and repair of failing agents, degraded tools/connectors, stale knowledge, failed ingestion, and retrieval/index drift

Define guardrails, confidence thresholds, and human-in-the-loop controls that make autonomous remediation safe at multi-tenant scale

Apply LLMs to operations directly — incident summarization, runbook generation, on-call copilots, and natural-language querying of platform telemetry

Observability for Agentic Systems

Instrument the platform end to end: distributed tracing across planning-retrieval-tool-generation loops, metrics, structured logging, and event correlation so multi-agent behavior is explainable and debuggable

Define golden signals for both system health and agent quality — task success rate, tool-call accuracy, grounding/hallucination rates, latency, cost-per-task, throughput — as first-class telemetry the operating agents act on

Establish SLOs/SLIs and error budgets for agent workflows, with alerting that feeds the agentic-ops layer

Reliability, SRE & Distributed Systems

Engineer the platform as a resilient, event-driven, cloud-native distributed system (Kubernetes, streaming pipelines, microservices) across regions and tenants

Drive SRE practices — capacity planning, graceful degradation, failover, chaos/resilience testing, blameless incident response — and progressively automate them through agents

Build for operability first: every component designed to be observed, diagnosed, and acted on autonomously

Knowledge & Retrieval (the domain agents operate over)

Guide the RAG/retrieval and knowledge services agents depend on, ensuring health, freshness, and quality are continuously monitored and remediated by the agentic-ops layer

Oversee enterprise content ingestion and sync (Confluence, SharePoint, Google Drive, Salesforce KB, ServiceNow KB) and multi-modal retrieval at platform scale

Technical Leadership

Set technical direction for the agent platform and agentic operations; mentor engineers and drive cross-team systems architecture decisions.

Partner with product and platform leadership on the long-term strategy for self-operating, enterprise-grade agentic AI

Required

10+ years building production software, with deep experience designing and operating large-scale distributed systems and platforms

Proven experience building scalable AI/agent platforms or high-throughput ML serving systems in production — orchestration, multi-tenancy, latency/cost optimization

Hands-on experience designing agentic or autonomous workflows — multi-agent reasoning, tool/API invocation, planning loops — applied to real production problems

Strong AIOps and SRE background: observability tooling (OpenTelemetry, Prometheus, Grafana, distributed tracing), SLOs/error budgets, anomaly detection, incident management, and closed-loop automation with human-in-the-loop safeguards

Hands-on experien…