AI Architect - Databricks

Greymatter Innovationz helps you stay digitally relevant across domains, technologies, and skillsets, every day.

We are looking for:

AI Architect - Databricks

Location: New York City Onsite - 5d/ wk

Duration: Long Term

Experience Level: Senior / Principal (10+ years of total experience with 3+ years specifically on Databricks & AI Architectures)

Key Responsibilities

Architecture & Design: Design secure, scalable, and enterprise-grade AI, Generative AI (LLMs), and Machine Learning solutions leveraging the Databricks Lakehouse platform.

Business Translation: Collaborate closely with stakeholders to translate business use cases into robust technical architectures and actionable roadmaps.

Technical Guidance: Lead and mentor engineering, data science, and MLOps teams on best practices for model training, fine-tuning (e.g., using Hugging Face/Transformers on Databricks), and prompt engineering.

Data Integration & Governance: Oversee large-scale data pipelines, ETL/ELT processes, and ensure robust data governance, lineage, and security compliance utilizing Unity Catalog.

MLOps & Deployment: Architect and streamline automated CI/CD pipelines for ML models using MLflow, ensuring seamless model monitoring, tracking, and deployment to production.

Performance & Cost Optimization: Continuously optimize Databricks clusters, storage, and compute resources for cost efficiency and high throughput.

Ecosystem Integration: Integrate Databricks AI solutions with cloud ecosystems (AWS/Azure/GCP), enterprise security frameworks, and downstream applications.

Key Requirements & Qualifications

Experience: 10+ years in IT/Software Engineering with a minimum of 5 years focused on Data Engineering/Machine Learning, and at least 3+ years specifically in an Architecture or Lead role working with Databricks.

Databricks Expertise: Deep hands-on experience with Databricks Lakehouse Platform, Spark, Delta Lake, and Unity Catalog.

AI/ML & GenAI Mastery: Strong command over Machine Learning pipelines, Deep Learning frameworks (PyTorch, TensorFlow), and Large Language Models (LLMs) / RAG architectures.

MLOps Proficiency: Proven expertise in model lifecycle management using MLflow and deploying scalable ML workloads in production.

Programming Skills: Advanced proficiency in Python, PySpark, and SQL.

Cloud & Security: Solid understanding of cloud infrastructure (AWS/Azure/GCP), IAM, enterprise security, and network architecture.

We offer:

Motivating Work Environment.

Excellent Work Culture.

Help you to upgrade yourself to the next level.

And More!!!