Aws Data Engineer

Job Summary:

We are seeking an experienced AWS Data Engineer with strong expertise in Databricks, PySpark, and cloud-based data engineering solutions. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, data lakes, and data warehouse solutions on AWS. The role requires hands-on experience in processing large-scale datasets, optimizing data workflows, and supporting analytics and business intelligence initiatives.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT data pipelines using PySpark and Databricks.
• Build and optimize data ingestion frameworks from various structured and unstructured data sources.
• Develop and manage data lakes and data warehouse solutions on AWS.
• Implement data transformation, cleansing, validation, and enrichment processes.
• Work with large datasets and optimize Spark jobs for performance and scalability.
• Integrate data from APIs, databases, streaming platforms, and third-party systems.
• Collaborate with Data Scientists, Analysts, and Business stakeholders to understand data requirements.
• Monitor, troubleshoot, and enhance data pipeline performance and reliability.
• Implement data quality, governance, security, and compliance best practices.
• Participate in code reviews, architecture discussions, and technical design sessions.

Required Skills:

• Strong experience in AWS Cloud Services (S3, EMR, Glue, Lambda, IAM, Redshift, Athena, CloudWatch).
• Hands-on experience with Databricks and Apache Spark.
• Strong programming skills in PySpark and Python.
• Experience in building ETL/ELT pipelines and data integration solutions.
• Strong knowledge of SQL and relational databases.
• Experience with Delta Lake, Spark SQL, and performance tuning.
• Understanding of data modeling concepts and data warehousing principles.
• Experience with version control systems such as Git.
• Knowledge of CI/CD practices and deployment automation.

Preferred Skills:

• Experience with Apache Kafka or other streaming technologies.
• Knowledge of Airflow, AWS Step Functions, or workflow orchestration tools.
• Experience with Snowflake or Redshift.
• Familiarity with Terraform or Infrastructure as Code (IaC).
• Exposure to machine learning data pipelines and MLOps.
• AWS Certification is a plus.

Educational Qualification:

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.

Key Competencies:

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work independently and in a collaborative environment.
• Strong focus on performance optimization and data quality.