Fully automated end-to-end data pipeline built on AWS that continuously ingests YouTube trending data via the YouTube Data API, cleans and transforms it through multiple stages, and stores it in structured format.
- Implemented Medallion Architecture with Bronze, Silver, and Gold layers.
- Used AWS Lambda for triggers, Glue for distributed ETL, and S3 for partition storage.
- Orchestrated entire pipeline with Step Functions, achieving 100% automation.
AWS S3
Lambda
AWS Glue
Step Functions