OpenSearch Logging Platform
Migrated a legacy on-premises ELK stack to AWS using Amazon OpenSearch. Designed log ingestion with Filebeat, Logstash on ECS Fargate, and improved operational visibility and scalability.
Context
An enterprise needed to replace an aging, self-managed on-premises ELK stack with a cloud-native logging solution that could scale with their growing workload count and reduce maintenance overhead.
Problem / Constraints
The existing ELK stack was unstable under load, required significant manual maintenance, and lacked the access controls and scalability needed for a growing platform. Log ingestion was inconsistent across teams.
What I owned
I owned the OpenSearch deployment, log ingestion pipeline design (Filebeat and Logstash on ECS Fargate), and index lifecycle management. I also contributed to dashboard standardisation.
Approach
Deployed Amazon OpenSearch (including OpenSearch Serverless where applicable). Designed the log ingestion pipeline using Filebeat on application servers feeding into Logstash hosted on ECS with AWS Fargate. Implemented ISM policies for index lifecycle management and created standardised dashboard templates.
Architecture
Architecture diagram placeholder