Aws Data Engineer

Roles and Responsibilities:

• API & Service Development: Design, build, and maintain high-performance, asynchronous microservices and RESTful APIs using FastAPI and Python.
• Data Pipeline Engineering: Architect, optimize, and scale distributed data pipelines using PySpark on AWS (EMR, Glue, or Databricks).
• AWS Cloud Integration: Deploy and manage cloud-native applications utilizing AWS services including Lambda, S3, ECS/EKS, API Gateway, DynamoDB, RDS, and SQS/SNS.
• Database Management: Write optimized queries, design schemas, and handle integrations across relational (PostgreSQL, MySQL) and NoSQL databases.
• Performance Tuning: Monitor and optimize Spark job performance, memory management, and API latency for high-throughput enterprise workloads.
• CI/CD & DevOps: Implement automated testing, continuous integration, and continuous deployment (CI/CD) pipelines using tools like GitHub Actions, GitLab CI, or Jenkins, backed by Infrastructure as Code (Terraform/CloudFormation).
• Code Quality & Testing: Write clean, modular, testable code adhering to PEP 8 standards with high unit and integration test coverage (pytest).

Required Qualifications & Skills

• Python: 3+ years of professional experience with advanced Python (asyncio, type hinting, object-oriented design).
• FastAPI: Hands-on experience developing, documenting (OpenAPI/Swagger), and securing RESTful APIs using FastAPI.
• PySpark: Proven experience developing distributed data processing applications and ETL jobs using PySpark/Apache Spark.
• AWS Cloud: Strong working experience with core AWS services (S3, EMR, Glue, Lambda, ECS/EKS, IAM, CloudWatch).
• Databases & Data Warehousing: Experience with SQL, relational databases (PostgreSQL/MySQL), and data storage formats (Parquet, ORC, Delta Lake).
• Containerization: Proficiency with Docker for containerizing microservices and local development.
• Agile & Version Control: Strong command of Git and experience working within Agile/Scrum delivery frameworks.