See how candidates think and approach the work this role demands, before the phone screen. We'll build a video challenge from this posting, and you can edit or share it before it goes live.
A pharmaceutical company standing up a dedicated principal-level support function for AI products and agents suggests it is moving agentic AI from pilots into day-to-day operations that need production-grade reliability, much like it would run any critical clinical or manufacturing system.
At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.
About Tech@Lilly
At Lilly, everything we do starts with patients. We unite caring with discovery to make life better for people around the world. Headquartered in Indianapolis, Indiana, our global team of over 50,000 employees work with urgency and purpose to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. We bring our best to this work because people depend on it. If you're driven by purpose and determined to make a meaningful difference for patients, we invite you to bring your skill and your commitment to Lilly.
At Lilly, technology is not a support function. It is how a global medicine company operates, innovates, and delivers. Lilly in Hyderabad builds the capabilities that make this possible, cloud platforms, AI systems, and automation at enterprise scale, all in service of a purpose that makes this technology work genuinely distinctive, from advancing drug discovery to enabling connected clinical trials to keeping a global medicine company running at the standard patients deserve.
Principal Engineer – Operations, Reliability & AI Platforms (R3)
About the Role
As a Principal Engineer – Operations, Reliability & AI Platforms, you will provide technical leadership for enterprise-scale production systems, cloud platforms, and AI-powered applications. You will drive reliability, operational excellence, observability, automation, and resiliency across multiple products and platforms while serving as a trusted engineering leader for critical business services.
This role combines deep technical expertise, architectural leadership, operational ownership, and strategic thinking. You will partner with Engineering, Product, Platform, Security, and Operations teams to establish reliability standards, eliminate operational toil through automation, improve system resilience, and ensure mission-critical services operate securely and efficiently at scale.
What You'll Be Doing
Technical Leadership & Strategy
• Define and drive the operational engineering strategy for enterprise applications, cloud-native platforms, and AI-enabled systems.
• Establish standards for reliability, scalability, availability, performance, security, and operational excellence.
• Lead architecture reviews with a focus on supportability, observability, resiliency, disaster recovery, and long-term sustainability.
• Provide technical leadership during major incidents, service disruptions, and high-severity production events.
• Influence product and engineering roadmaps to proactively address technical debt, operational risks, and reliability improvements.
Production Operations & Reliability Engineering
• Own reliability outcomes through Service Level Objectives (SLOs), Service Level Indicators (SLIs), error budgets, and operational health metrics.
• Drive adoption of Site Reliability Engineering (SRE) and operational excellence practices across teams.
• Lead complex root cause investigations involving distributed systems, cloud platforms, integrations, databases, and AI services.
• Establish operational readiness reviews, production acceptance criteria, release governance, and supportability standards.
• Define and implement resiliency engineering practices including failover validation, disaster recovery testing, capacity planning, and business continuity programs.
• Drive continuous improvement through incident trend analysis, reliability metrics, and operational insights.
AI Platform & Agent Operations
• Provide technical leadership for AI-powered applications, copilots, virtual assistants, chatbots, and agent-based systems operating in production environments.
• Establish operational standards for Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), AI agents, vector databases, MCP ecosystems, and orchestration platforms.
• Lead investigations involving AI model failures, retrieval issues, prompt orchestration, tool-calling workflows, agent interactions, and integration failures.
• Define monitoring strategies for AI quality, token consumption, latency, throughput, reliability, and operational health.
• Drive AI governance, responsible AI practices, security, compliance, and operational risk management.
Automation & Operational Excellence
• Lead initiatives to eliminate manual operational effort through automation, intelligent workflows, self-healing capabilities, and autonomous remediation.
• Design and implement operational automation using scripting, Infrastructure as Code, CI/CD pipelines, observability platforms, and AIOps solutions.
• Establish enterprise runbooks, playbooks, operational standards, and knowledge management frameworks.
• Drive adoption of platform engineering principles to improve reliability, developer productivity, and operational efficiency.
Collaboration & Mentorship
• Partner with Engineering, Product, Platform, Security, Compliance, and Operations teams to improve service quality and customer outcomes.
• Mentor engineers and technical leaders in reliability engineering, distributed systems, cloud operations, and AI platform support.
• Communicate technical risks, operational health, and strategic recommendations to both technical and executive stakeholders.
• Lead cross-functional initiatives that improve reliability, resilience, supportability, and operational maturity across the organisation.
How You Will Succeed
• Demonstrate exceptional analytical, troubleshooting, and systems-thinking capabilities across complex distributed environments.
• Drive measurable improvements in service reliability, availability, performance, and operational efficiency.
• Build scalable observability, monitoring, tracing, and automation capabilities that proactively detect and resolve issues.
• Operate effectively across application, platform, infrastructure, cloud, and AI technology stacks.
• Influence engineering organisations through technical leadership, collaboration, and data-driven decision making.
• Balance business priorities, operational risk, engineering investment, and long-term sustainability.
• Establish measurable operational KPIs, SLOs, SLIs, and reliability programmes.
• Become the escalation point for the most complex technical and operational challenges.
What You Should Bring
Experience
• 10+ years of experience in Software Engineering, Site Reliability Engineering, DevOps, Platform Engineering, Cloud Engineering, Production Operations, or related disciplines.
• Proven experience operating and supporting large-scale enterprise production environments.
• Experience leading reliability, operational excellence, and platform transformation initiatives.
• Demonstrated success driving operational improvements across multiple products, systems, and teams.
• Experience operating within regulated industries such as Life Sciences, Healthcare, Financial Services, or similar environments.
Technical Expertise
• Deep expertise in distributed systems, microservices, APIs, event-driven architectures, and cloud-native platforms.
• Strong understanding of software architecture, performance engineering, resiliency, security, and operational excellence.
• Advanced troubleshooting capabilities across applications, integrations, databases, networking, infrastructure, and cloud services.
• Experience leading major incident management, post-incident reviews, root cause analysis, and corrective action programmes.
• Strong understanding of governance, compliance, risk management, and secure engineering practices.
Cloud & Platform Engineering Skills
• Expert knowledge of AWS, Azure, and/or GCP cloud platforms.
• Experience designing and supporting highly available, fault-tolerant, enterprise-scale cloud architectures.
• Strong expertise with Kubernetes, Docker, and container orchestration technologies.
• Experience with Infrastructure as Code technologies such as Terraform, CloudFormation, Bicep, or similar platforms.
• Deep understanding of networking, identity management, security, and cloud governance.
• Experience implementing platform engineering principles and Internal Developer Platforms (IDPs).
• Strong knowledge of high availability, failover, disaster recovery, backup, and resilience engineering concepts.
DevOps, SRE & Automation Skills
• Expert understanding of SRE principles, reliability engineering, error budgets, SLOs, and SLIs.
• Strong experience with GitHub Actions, Azure DevOps, Jenkins, GitLab CI/CD, and modern deployment automation platforms.
• Proven experience implementing self-healing systems, automated remediation, and operational automation solutions.
• Deep expertise with observability platforms including OpenTelemetry, Grafana, Datadog, Splunk, ELK, Prometheus, Dynatrace, or equivalent technologies.
• Experience implementing AIOps, intelligent alerting, event correlation, and operational analytics platforms.
• Strong automation and scripting capabilities using Python, PowerShell, Bash, JavaScript, or similar technologies.
AI & Emerging Technology Skills
• Strong understanding of Generative AI ecosystems and enterprise AI architecture patterns.
• Experience supporting:
• Familiarity with Azure OpenAI, AWS Bedrock, Google Vertex AI, OpenAI APIs, Anthropic Claude, Azure AI Services, or similar AI platforms.
• Understanding of MCP Server architecture, MCP Clients, Tool Calling, Enterprise AI integrations, and AI platform operations.
• Experience monitoring AI reliability, latency, throughput, token consumption, usage patterns, and operational health metrics.
• Knowledge of Responsible AI, AI governance, security, compliance, model evaluation, and operational risk management.
Nice to Have
• Experience leading enterprise-wide operational transformation initiatives.
• Experience building reliability engineering or operational excellence programmes.
• Experience supporting Kubernetes-based cloud-native architectures at scale.
• Familiarity with LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI, OpenAI Agents SDK, or similar orchestration frameworks.
• Experience supporting MCP-based enterprise integrations and AI ecosystems.
• Experience implementing AIOps capabilities, autonomous operations, and intelligent automation solutions.
• Cloud, Kubernetes, DevOps, SRE, or AI/Generative AI certifications.
• Experience supporting mission-critical, global, 24x7 production environments.
Key Competencies
• Principal Engineering Leadership
• Production Operations & Support
• Site Reliability Engineering (SRE)
• Operational Excellence
• Reliability Engineering
• Cloud Architecture & Operations
• Platform Engineering
• Incident & Problem Management
• Root Cause Analysis (RCA)
• Observability & Telemetry
• Automation & Self-Healing Systems
• AI Platform Operations
• MCP & AI Integration Support
• Generative AI & Agentic Systems
• Disaster Recovery & Resilience Engineering
• Security, Risk & Compliance
• Executive Stakeholder Management
• Cross-Functional Technical Leadership
• Continuous Improvement & Innovation
• Customer-Focused Problem Solving
Additional Information
Availability to work flexible work hours is/may be required. This team will support continuous operations across two shifts and therefore, this role will require non-standard work hours, and some work on weekends and holidays. Appropriate adjustments in benefits will be provided for employees working non-standard hours where applicable.
Candidate should be open to work on different shifts - 6AM to 2PM and 2PM to 11PM when required.
At Lilly, caring is not only what we do for patients. It is how we work. We believe the people who dedicate themselves to making medicines better deserve an environment that makes their lives better too, one where they are supported, respected, and given the space to do their best work. This is not just a policy. It is who we are.
Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.
#WeAreLillyAnswer a McCoy challenge. Companies hiring in engineering see your response and reach out when they want to talk.
About 15 minutes to prepare and 5 minutes to record. Unlimited retakes and a teleprompter help you answer. No AI training, and you can delete your response anytime.
This isn't an application to Eli Lilly.
lilly:R-113936See how we measure for definitions, or our corrections log for known issues. Found something wrong? Flag a correction.
wd115|lilly|LLYThis posting hasn't been probed by our closure verifier yet. Stream C runs on a rolling schedule against postings approaching the close-decision threshold.