About Us At Sully.ai http://Sully.ai, We're Building the Most Impactful Healthcare Company on Earth

We believe that access to a great doctor is a basic human right. Today, that's not a reality. Delays, misdiagnoses, administrative chaos, and burnout plague the system.

Our Mission One Human, One Doctor. We enable our customers to staff 30% of their workforce with AI by creating a shared agent architecture for scale and efficiency. All powered by our own patented, world-class models and deployed in real-world care.

Key Responsibilities

- Build and scale automated evaluation pipelines (LLM-as-judge + human review) with clinical-grade benchmarks.

Hard Requirements

• Proven experience designing agentic processes and LLM evaluation/benchmarking frameworks.
• Strong Python and ML background (PyTorch/TensorFlow, Hugging Face, LangChain/LlamaIndex).
• Demonstrated ability to design rigorous experiments and translate findings into production.
• Track record of published research or deep applied work in LLMs and agent evaluation.
• Strong communication and technical writing skills to articulate complex findings clearly.
• First-Month Focus
• Audit existing evaluation approaches for clinical and agentic tasks.
• Define initial benchmarks and build early automated pipelines.
• Partner with engineering to land first set of CI gates for accuracy, factuality, and safety.
• 90 Days
• Deliver a repeatable evaluation framework with automated pipelines in production.
• Demonstrate measurable improvements in robustness, hallucination reduction, or safety.
• Publish or present internal research findings that directly shape product reliability.
• If you've ever said, “I want to do work that actually matters”, this is it. Let's build something life-changing, together.

Key Results

• (FIRST 90 DAYS)
• Deliver a repeatable evaluation framework with automated pipelines in production.
• Demonstrate measurable improvements in robustness, hallucination reduction, or safety.
• Publish or present internal research findings that directly shape product reliability. Why Join Sully.ai http://Sully.ai? 🔥 Revolutionizing the antiquated $800B+ Healthcare market 🧠 60%+ Ex-founders who have built, scaled, exited. We hire A-players ⚡️ Speed matters: we operate with urgency, autonomy, and ownership 🧪 You'll work on real, first-of-their-kind problems at the edge of AI and medicine ❤️ Your work directly unlocks doctors to reclaim their time, and patients get better, faster care Sully.ai is an equal opportunity employer. In addition to EEO being the law, it is a policy that is fully consistent with our principles. All qualified applicants will receive consideration for employment without regard to status as a protected veteran or a qualified individual with a disability, or other protected status such as race, religion, color, national origin, sex, sexual orientation, gender identity, genetic information, pregnancy or age. Sully.ai prohibits any form of workplace harassment.