Staff Software Engineer, AI Inference

Job Description Staff Software Engineer - AI Inference

Location: New York, NY (On-site, Remote)

About Syllo Syllo is a unified litigation platform enabling lawyers to use AI throughout the litigation lifecycle. We serve enterprise customers including major law firms and corporations.

About The Role We are building our own inference stack for open-source language models. You will define architecture, evaluate technologies, and build production systems for model serving, optimization, and monitoring.

Responsibilities

• Lead design and development of production inference platform
• Define technical roadmap for inference infrastructure and model serving
• Build scalable, cost-effective systems for LLM serving
• Evaluate and integrate inference technologies and runtimes
• Optimize latency, throughput, GPU utilization, memory, cost
• Develop deployment, routing, autoscaling, observability systems
• Partner with ML engineers to productionize models
• Establish benchmarking methodologies
• Make architectural decisions (build vs. leverage OSS/commercial)
• Mentor engineers and establish AI infrastructure best practices

Qualifications

• Significant experience with production AI inference systems
• Leadership in production LLM serving infrastructure
• Deep experience with inference runtimes (vLLM, SGLang, TensorRT-LLM, Triton, TGI, Dynamo, etc.)
• Strong background in distributed systems, backend infrastructure, or high-performance platform engineering
• Experience optimizing GPU inference performance (latency, throughput, batching, memory)
• Production GPU infrastructure experience
• Proficiency in Python and systems programming (Go, Rust, C++)
• Proven ability to lead technical architecture for complex initiatives
• Excellent communication and cross-team influence skills

Compensation Salary Range: $190,000 - $230,000 USD (plus health insurance and equity)

Remote: United States