Software Development Engineer

Role & responsibilities

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines and ETL/ELT processes using Databricks and other data engineering tools to support analytics and BI use cases.
• Build and deploy cloud-native data and analytics solutions on AWS, leveraging services such as EKS, S3, and other relevant components for scalable and resilient architectures.
• Develop backend services and APIs using Python and Flask to enable data access, automation workflows, and integration with downstream applications and dashboards.
• Implement containerization and orchestration using Docker and Kubernetes (EKS preferred) to ensure reliable, portable, and scalable deployment of data and analytics workloads.
• Set up and maintain CI/CD pipelines using Jenkins/Jenkins Core to automate build, test, and deployment processes for data engineering and analytics solutions.
• Collaborate with data scientists, analysts, and BI developers to enable AI, machine learning, and advanced analytics use cases through well-structured, high-quality data.
• Optimize data models, queries, and pipelines for performance, cost efficiency, and reliability across large-scale datasets and complex analytics workloads.
• Implement data quality checks, monitoring, logging, and alerting to ensure data reliability, integrity, and timely availability for business stakeholders.
• Contribute to data automation initiatives by identifying opportunities to streamline manual processes and implementing robust automation solutions.
• Ensure adherence to best practices in cloud security, data governance, and compliance for all data engineering and analytics solutions.
• Support troubleshooting, root cause analysis, and performance tuning for data pipelines, cloud infrastructure, and analytics platforms.
• Mentor junior team members and contribute to knowledge sharing, coding standards, and best practices within the data engineering and analytics team.