AWS Databricks Developer

Key Responsibilities

• Design, develop, and maintain scalable data processing solutions using Databricks on AWS.
• Build and maintain ETL/ELT pipelines for batch and near real-time data ingestion.
• Develop data transformation and processing workflows using Apache Spark, PySpark, Spark SQL, Python, and Scala.
• Implement Delta Lake solutions, including ACID transactions, schema evolution, and performance optimization.
• Design and implement analytical and dimensional data models, including Star and Snowflake schemas.
• Work with AWS cloud storage services to manage data ingestion, storage, and processing.
• Optimize data pipelines for performance, scalability, reliability, and maintainability.
• Follow coding standards, documentation practices, and data engineering best practices.
• Collaborate with solution architects, BI teams, and data science teams to deliver end-to-end data solutions.
• Support data quality, troubleshooting, and pipeline performance improvements.

Required Skills and Qualifications

• 8+ years of relevant experience in data engineering and development.
• Hands-on development experience with AWS Databricks.
• Strong experience developing and maintaining ETL/ELT pipelines.
• Proficiency in Apache Spark, PySpark, Spark SQL, and Python.
• Working knowledge of Scala and Power BI.
• Hands-on experience with Delta Lake, including ACID transactions, schema evolution, and performance tuning.
• Experience building batch and near real-time data ingestion pipelines.
• Knowledge of analytical data modelling and dimensional modelling, including Star and Snowflake schemas.
• Experience with AWS cloud-based storage systems.
• Strong understanding of data engineering best practices, coding standards, and technical documentation.
• Ability to collaborate effectively with cross-functional technical teams.

Secondary Skills and Preferred Experience

• Exposure to AI/ML workloads on Databricks and feature engineering.
• Familiarity with MLflow or similar model tracking tools.
• Experience with Databricks CI/CD practices and Git-based version control.
• Understanding of data governance, security, and compliance.
• Familiarity with Agile/Scrum delivery methodologies.
• Experience in the insurance domain.

Pay: ₹1,400,000.00 - ₹1,800,000.00 per year

Application Question(s):

• By applying, you agree that your resume and personal details may be stored securely for up to 2 years for recruitment purposes.

Work Location: Hybrid remote in Bangalore City, Bengaluru, Karnataka