Senior Python PySpark Data Engineering Lead AI & Data Platforms

Date Posted: 08/31/2026

Hiring Organization: Rose International

Position Number: 506588

Industry: Financial Services

Job Title: Senior Python PySpark Data Engineering Lead AI & Data Platforms

Job Location: Rutherford, NJ, USA, 07070

Work Model: Hybrid

Work Model Details: 3 days onsite and 2 days remote

Employment Type: Temp to Hire

FT/PT: Full-Time

Estimated Duration (In months): 6

Min Hourly Rate($): 65.00

Max Hourly Rate($): 74.00

Must Have Skills/Attributes: ETL, PySpark, Python

Experience Desired: Experience in a Python and PySpark engineering (5 yrs); Experience with YAML, JSON, FAST API or Spring Boot, GitHub Copilot (5 yrs); Experience developing and automating microservice integrations (5 yrs)

Required Minimum Education: Bachelor’s Degree

**C2C is not available**

Job Description Required Experience, Knowledge & Skills:

• 8+ years of overall experience in large-scale application development with a recent mandatory platform for the secure and scalable deployment of AI agents into application contexts
• Minimum of 5+ years of proven experience in a Python and PySpark engineering lead role focused on building enterprise-grade, high-volume ELT/ETL processes using PySpark
• Hands-on experience with YAML, JSON, FAST API or Spring Boot, GitHub Copilot
• Proven experience developing and automating microservice integrations to support data-intensive applications
• Proficiency in at least one programming language commonly used for data analytics and engineering, such as Python or Scala
• Strong SQL skills and experience with various relational databases
• Deep understanding of data modeling, data warehousing concepts, Data Mesh architecture, and data federation
• Excellent communication, collaboration, and problem-solving skills

Responsibilities

• Design, develop, and maintain scalable, enterprise-grade AI agents, supporting ELT/ETL processes to handle large data volumes using Python, FAST API, Microservices, PySpark, Kafka, and the Databricks ecosystem.
• Develop, deploy, and automate microservice integrations to support data-intensive applications, ensuring scalability, resilience, and maintainability using cloud native infrastructure and OpenShift or Kubernetes architecture, including CI/CD pipelines.
• Ensure data quality, integrity, and security throughout the entire data lifecycle.
• Contribute to the continuous improvement of data engineering processes, standards, and best practices within the team.
• **Only those lawfully authorized to work in the designated country associated with the position will be considered.**
• **Please note that all Position start dates and duration are estimates and may be reduced or lengthened based upon a client’s business needs and requirements.**

Benefits For information and details on employment benefits offered with this position, please visit here. Should you have any questions/concerns, please contact our HR Department via our secure website.

California Pay Equity For information and details on pay equity laws in California, please visit the State of California Department of Industrial Relations’ website here.

Rose International is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, age, sex, sexual orientation, gender (expression or identity), national origin, arrest and conviction records, disability, veteran status or any other characteristic protected by law. Positions located in San Francisco and Los Angeles, California will be administered in accordance with their respective Fair Chance Ordinances. If you need assistance in completing this application, or during any phase of the application, interview, hiring, or employment process, whether due to a disability or otherwise, please contact our HR Department. Rose International has an official agreement (ID #132522), effective June 30, 2008, with the U.S. Department of Homeland Security, U.S. Citizenship and Immigration Services, Employment Verification Program (E-Verify). (Posting required by OCGA 13/10-91.).