Aws Data Engineer

Key Responsibilities

Develop, maintain, and optimise data pipelines and ETL processes leveraging AWS and Databricks.

Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver robust solutions.

Implement data quality, data governance, and security best practices in all workflows.

Monitor and troubleshoot job performance and failures, ensuring high availability and reliability of data processes.

Automate workflows using Databricks notebooks, jobs, and integration with AWS services such as S3, Redshift, Glue, and Lambda.

Document technical solutions, workflows, and best practices for future reference.

Required Skills and Qualifications

Bachelors degree in Computer Science, Information Technology, or related field.

Proven experience with Databricks and AWS data services (e.g., S3, Redshift, Glue, EMR).

Proficiency in Python, Scala, or SQL for data engineering tasks.

Hands-on experience with Spark for large-scale data processing.

Strong understanding of ETL concepts, data warehousing, and data modelling.

Familiarity with CI/CD pipelines and version control systems such as Git.

Excellent problem-solving skills and attention to detail.

Preferred Qualifications

AWS or Databricks certification.

Experience with real-time data processing and streaming (e.g., Kafka, Kinesis).

Knowledge of data security and compliance standards relevant to cloud data solutions.

Strong communication and collaboration skills.