Senior Azure Databricks & Snowflake Data Engineer

We are looking for a highly skilled Azure Databricks & Snowflake Data Engineer with strong experience in Spark, PySpark, Python/Scala, Azure Cloud, SQL, ETL/ELT pipelines, and Snowflake development . The ideal candidate should have hands-on experience in building and migrating enterprise data pipelines and should be comfortable working across the complete Software Development Life Cycle.

Exposure to Snowflake Cortex/CoCo, Databricks Genie, and AI-assisted coding tools such as GitHub Copilot is highly desirable. Candidates with strong core engineering skills and a willingness to learn emerging AI capabilities will also be considered.

Key Responsibilities

• Design, develop, optimize, and maintain scalable data engineering solutions using Azure Databricks, Apache Spark, PySpark, Python, and/or Scala.
• Develop and optimize complex ETL/ELT data pipelines for large-scale data processing.
• Perform migration of existing ETL/ELT pipelines and data workloads to Azure Databricks and Snowflake.
• Develop robust data transformation, ingestion, validation, and processing frameworks.
• Work extensively with Snowflake for data loading, transformation, optimization, and integration.
• Collaborate with data architects, developers, analysts, and business teams to understand requirements and deliver scalable solutions.
• Implement CI/CD pipelines and automated deployment processes using Git, GitHub, GitHub Workflows, and Azure DevOps.
• Follow software engineering best practices including version control, code reviews, unit testing, debugging, and documentation.
• Work with workflow/orchestration technologies such as Azure Data Factory (ADF), Azure Synapse, and Airflow.
• Participate in architecture discussions and contribute to improving data platform performance, reliability, and scalability.
• Utilize Terraform and Infrastructure-as-Code concepts where applicable.
• Explore and leverage AI-assisted development tools such as GitHub Copilot and Databricks Genie to improve engineering productivity.
• Gain exposure to and contribute to solutions using Snowflake Cortex/CoCo and other AI capabilities.
• Troubleshoot production issues and provide root-cause analysis and permanent fixes.
• Participate in all phases of the Software Development Life Cycle (SDLC).

Mandatory Technical Skills

• 5+ years of hands-on experience with Azure Databricks.
• 5+ years of experience with Apache Spark / PySpark.
• 5+ years of programming experience using Python and/or Scala.
• 5+ years of experience working on the Azure Cloud Platform.
• 3+ years of hands-on Snowflake development experience.
• Strong knowledge of SQL and database concepts with 5+ years of experience.
• Strong experience in ETL/ELT pipeline development and migration.
• Hands-on experience with CI/CD, Git, and GitHub Workflows.
• Strong software development and coding skills.
• 5+ years of complete SDLC experience.

Good-to-Have Skills

• Knowledge or hands-on experience with Snowflake Cortex / CoCo.
• Exposure to Databricks Genie.
• Experience using GitHub Copilot or other AI-assisted coding tools.
• Knowledge of Azure DevOps.
• Knowledge of Terraform / Infrastructure as Code.
• Experience with Azure Data Factory (ADF).
• Exposure to Azure Synapse Analytics.
• Exposure to Apache Airflow.
• Experience with Delta Lake and modern data lakehouse architecture.
• Understanding of data engineering best practices, performance tuning, and cloud-native architectures.

Core Competencies

• Strong programming and problem-solving skills.
• Excellent understanding of data structures, SQL, databases, and data processing concepts.
• Ability to develop clean, maintainable, reusable, and production-ready code.
• Strong analytical and debugging skills.
• Ability to work independently as well as collaboratively in a distributed team.
• Good communication and stakeholder-management skills.
• Strong willingness to learn emerging technologies, particularly Generative AI and AI-assisted development.

Education

Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related discipline .

Evaluation / Skill WeightageEvaluation AreaWeightageAzure Databricks / Spark / PySpark25%Python / Scala & Coding Skills15%Snowflake Development15%Azure Cloud & Data Engineering10%ETL/ELT & Pipeline Migration10%SQL / Database Expertise10%CI/CD, Git, GitHub & DevOps5%ADF / Synapse / Airflow / Terraform5%Snowflake Cortex/CoCo, Da…