AWS Databricks, Pyspark

Role- AWS Databricks, Pyspark

Desired Experience Range- 7-11 Years

Location- Chennai, Pan India

Roles & Responsibility -

• Strong hands-on experience with AWS Databricks, Pyspark, Delta Lake, Unity Catalog, Workflows/Jobs and Lakehouse architecture
• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Delta Lake, Auto Loader and Delta Live Tables (DLT).
• Design and implement metadata-driven data ingestion and transformation frameworks to enable scalable, reusable and configuration-based pipeline development across multiple data sources.
• Develop framework components for managing source configurations, ingestion rules, schema evolution, data quality validations, audit logging and pipeline execution monitoring.
• Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate data quality checks, validation frameworks, testing, and monitoring.
• Develop ingestion frameworks using Databricks Auto Loader, including checkpointing, schema evolution and handling structured/semi-structured data.
• Configure and manage Unity Catalog including catalogs, schemas, access controls, audit logging, data lineage and security policies.
• Optimize Spark workloads by tuning Spark jobs, cluster configurations, partitioning strategies, and Delta Lake storage layouts for performance and cost efficiency.
• Build and manage workflows using Databricks Jobs/Lakeflow Jobs for batch and streaming data processing.
• Implement CI/CD practices and integrate with Git-based DevOps processes.
• Integrate Databricks solutions with AWS services such as S3, IAM, Glue, Step Functions and other AWS data services.
• Enable analytics consumption through BI tools such as Power BI, Tableau or Looker using optimized connectivity patterns.
• Collaborate with data architects, analysts, data scientists and business stakeholders to deliver enterprise data platform solutions.
• Troubleshoot pipeline failures, performance issues and operational challenges while maintaining technical documentation.

Nice to Have-

• Experience with Databricks advanced capabilities such as MLflow, Feature Store, Vector Search and GenAI workloads.
• Knowledge of Delta Sharing and Databricks Marketplace.
• Experience migrating workloads from platforms such as Snowflake, Azure Databricks or traditional data warehouses.
• Exposure to enterprise-scale data platform implementations.