Machine Learning Engineer I - Large Language Models - AI & Human Health Research

Description We are seeking a skilled LLM Engineer I to join our team in the SinAI Assurance Lab. The LLM Engineer will play a key role in designing, building, and deploying large language model (LLM) applications including retrieval-augmented generation (RAG) systems, agentic platforms, and clinical chatbots and will be responsible for designing, maintaining, and optimizing data infrastructure and model validation pipelines that ensure all AI systems, Generative and Non-Generative, deployed across the Mount Sinai Health System (MSHS) are rigorously validated for compliance, performance, and patient safety.

You will work closely with AI product teams, clinical and technical stakeholders, DevOps engineers, and the AI Governance Committee to engineer scalable data flows that support model validation, real-time monitoring, and Machine Learning Engineer I will be primarily responsible for contributing to the development and enhancement of machine learning applications and systems. They will work closely with other engineers and data scientists to design and implement scalable and efficient machine learning systems.

Responsibilities General Data Engineering

• Build and maintain robust ETL pipelines for structured and unstructured clinical data from EHR, imaging, and text sources.
• Design systems to automate data preparation, lineage tracking, and reproducibility for AI model inputs and outputs.
• Develop data infrastructure for benchmarking and stress-testing models in clinical simulation environments.
• Collaborate with DevOps and cloud teams to ensure deployment pipelines meet compliance and performance standards.
• Set up and monitor model tracking infrastructure for evaluation metrics and drift detection.
• Assist in the development of standards and procedures affecting data management, design and maintenance. Documents all standards and procedures.

AI Assurance & Governance

• Engineer and maintain pipelines that support pre-deployment model validation and post-deployment monitoring.
• Collaborate with Data Scientists and Clinical Product Owners to validate data integrity, reproducibility, and fairness in AI workflows.
• Ensure compliance with HIPAA, ethical guidelines, and institutional governance policies on sensitive health data use.
• Build dashboards and tools that provide observability across the ML lifecycle: data, models, outcomes.

LLM Engineering & Applied Generative AI

• Design, build, and deploy LLM-powered applications including clinical chatbots, copilots, and decision-support tools for end-users across MSHS.
• Develop retrieval-augmented generation (RAG) pipelines that integrate vector databases with clinical knowledge sources, EHR data, and institutional documents.
• Build agentic platforms and multi-agent workflows using frameworks such as LangChain, LlamaIndex, LangGraph, CrewAI, or equivalent, including tool use, function calling, and orchestration logic.
• Operationalize LLM deployment, including inference optimization, latency and cost tuning, model serving (e.g., Azure), and integration of safety guardrails.
• Implement prompt engineering, prompt versioning, and structured prompt-evaluation workflows across model providers and versions.
• Fine-tune and adapt foundation models (full fine-tuning, LoRA/PEFT, instruction tuning) to clinical and operational use cases where appropriate.
• Build LLM evaluation harnesses covering accuracy, hallucination, safety, bias, sycophancy, and clinical appropriateness, with red-teaming and stress-testing of deployed systems.

Stakeholder Engagement & Others

• Effectively communicate technical findings related to model and data integrity to governance teams, clinical stakeholders, and leadership.
• Maintain clear and well-organized documentation of data workflows, platform architecture, and validation processes.
• Help write internal reports on data infrastructure resilience, validation system status, and operational risk.
• Stay informed on industry best practices in data engineering and healthcare-focused machine learning.
• Possess an extremely flexible attitude. Willing to work with multiple types of technologies and languages with an open mind and without technology bia
• Continuous interest in updating skill sets and knowledge of trends in the Big Data Technology space.
• Work closely with cross-functional teams including data scientists, healthcare providers, and IT professionals to understand data requirements, de…