Data Engineer | PySpark | Databricks | Azure Data Factory
- Source: Naukri
- Infosys
- Hybrid
- Bengaluru
- Full-Time
- Mid level
Job Description
Responsibilities
- Design, develop, and maintain scalable data pipelines using PySpark, Spark, Databricks, Azure Data Factory (ADF), and AWS Glue.
- Build and optimize ETL/ELT workflows for processing large-scale structured and unstructured data.
- Develop and manage cloud-based data solutions on AWS, Azure, and GCP platforms.
- Work with Databricks, Delta Lake, ADLS, S3, BigQuery, Redshift, and Snowflake for data ingestion, transformation, and storage.
- Implement data quality, monitoring, and performance optimization best practices.
- Collaborate with data analysts, data scientists, and business stakeholders to understand requirements and deliver scalable solutions.
- Ensure data security, governance, and compliance standards are followed across cloud platforms.
- Optimize Spark jobs and SQL queries for improved performance and cost efficiency.
- Support real-time and batch data processing frameworks.
- Participate in code reviews, troubleshooting, and production support activities.
- Create technical documentation and maintain data engineering standards and best practices.
Skills
- pyspark
- azure-databricks
- databricks
- microsoft-azure
- azure-data-factory
- Spark
- amazon-web-services
- aws-glue
- google-cloud-platform
- delta-lake
- Snowflake
