Software Engineer Consultant I

Job Summary

Highly skilled and hands-on Senior Data Engineer to design, build, optimize, and support scalable, reliable, and high-performing data platforms and pipelines. This role is focused on developing enterprise data solutions that enable analytics, reporting, AI/ML, and data-driven decision making across the organization.

The ideal candidate possesses strong expertise in modern data engineering practices, Apache Spark, cloud-based data platforms, and lakehouse architectures. As a senior individual contributor, the candidate will provide technical leadership, mentor junior engineers, drive engineering best practices, and contribute directly to the design and implementation of enterprise-scale data solutions.

The role requires close collaboration with Analytics Engineers, Data Scientists, Product Managers, Platform Engineers, Security teams, and business stakeholders to deliver trusted, scalable, and analytics-ready data products.

Key Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark, cloud-native technologies, and modern data platforms.
• Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse enterprise source systems.
• Develop and implement data solutions leveraging Microsoft Fabric, OneLake, Lakehouse, Data Warehouse, and Semantic Model capabilities.
• Design data models, storage strategies, and data processing architectures that support enterprise analytics and reporting requirements.
• Partner with Analytics Engineers, BI teams, Data Scientists, and business stakeholders to deliver trusted, reusable, and analytics-ready datasets.
• Drive adoption of engineering best practices including code quality, peer reviews, automated testing, CI/CD, and documentation standards.
• Ensure data quality, governance, lineage, metadata management, security, privacy, and compliance requirements are incorporated into data solutions.
• Monitor and optimize performance, scalability, reliability, and cost efficiency of distributed data processing workloads.
• Develop reusable frameworks, libraries, templates, and engineering patterns that improve development productivity and consistency.
• Implement and support CI/CD pipelines, Infrastructure-as-Code principles, and deployment automation for data engineering solutions.
• Troubleshoot complex data integration, performance, and operational issues across platforms and pipelines.
• Collaborate with platform, infrastructure, architecture, and security teams to build resilient, scalable, and secure enterprise data solutions.
• Create and maintain technical documentation, architecture diagrams, operational runbooks, and implementation guides.
• Participate in technical design sessions, architecture reviews, and engineering discussions while providing recommendations and solutions.
• Mentor and guide junior and mid-level data engineers through technical coaching, code reviews, design reviews, and knowledge sharing.
• Evaluate and adopt emerging technologies, engineering practices, and AI-assisted development capabilities to improve delivery velocity and quality.
• Support production operations through monitoring, root cause analysis, incident resolution, and continuous improvement initiatives.
• Contribute to Agile delivery processes including sprint planning, estimation, backlog refinement, and execution of engineering commitments.
• Promote DataOps practices, automation, observability, and engineering excellence across development teams.
• Serve as a technical subject matter expert for modern data platforms, Spark processing, Microsoft Fabric, and lakehouse architectures.

Required Skills Experience

• Strong experience in Data Engineering and Big Data technologies.
• Deep expertise in Apache Spark (PySpark/Spark SQL).
• Experience with Microsoft Fabric, OneLake, Lakehouse Architecture, Data Warehouse, and Semantic Models.
• Strong SQL and data modeling skills.
• Experience building and optimizing ETL/ELT pipelines.
• Knowledge of streaming and batch data processing patterns.
• Experience with CI/CD, Git-based development, testing, and deployment automation.
• Understanding of Data Governance, Data Quality, Security, and Privacy controls.
• Experience working in cloud-based analytics and data platform environments.
• Strong problem-solving, troubleshooting, and performance tuning capabilities.
• Ability to mentor junior engineers and provide technical leadershi…