SENIOR DATA ENGINEER - Azure Databricks

Job Title: Senior Data Engineer   Years of Experience:  5-8 Years   

Job Summary:  We are seeking a highly skilled Senior Data Engineer with extensive experience in Azure Databricks, Azure Data Factory, and Palantir to join our dynamic team in Hyderabad. The ideal candidate will have a strong background in data engineering, with a proven track record of designing and implementing data solutions that drive business insights and decision-making. This role requires a deep understanding of data architecture, ETL processes, and cloud-based data platforms.   Responsibilities:    Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.  Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.  Implement data integration processes to ingest, transform, and store data from various sources into data lakes and warehouses.  Optimize data workflows and ensure data quality, reliability, and performance.  Utilize Palantir for data analysis and visualization, providing actionable insights to the business.  Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime.  Stay updated with the latest industry trends and technologies related to data engineering and cloud computing.  Mentor junior data engineers and contribute to team knowledge sharing.    

Mandatory Skills:    Proven experience with Azure Databricks (deal breaker skill).  Strong proficiency in Azure Data Factory for data integration and orchestration.  Experience with Palantir for data analysis and visualization.  Solid understanding of data modelling, ETL processes, and data warehousing concepts.  Proficient in programming languages such as Python, SQL, or Scala.    

Good understanding of Generative AI and Machine Learning concepts, including LLM, RAG, Vector Databases, Embeddings, Token Management, Feature Selection, Model Evaluation, and Neural Network fundamentals

Good to have : Ability to demonstrate practical knowledge of ML algorithms such as KNN, Linear Regression, CNN

Preferred Skills:    Experience with cloud platforms such as Azure, AWS, or Google Cloud.  Familiarity with big data technologies like Apache Spark, Hadoop, or Kafka.  Knowledge of data governance and data security best practices.  Experience with CI/CD pipelines and DevOps practices in data engineering.    

Qualifications:    Bachelor’s degree in Computer Science, Information Technology, or a related field.  5-8 years of experience in data engineering or a related role.  Strong analytical and problem-solving skills.  Excellent communication and collaboration abilities.   If you are passionate about data engineering and want to work in a fast-paced environment, we encourage you to apply for this exciting opportunity.