Applied AI Researcher, Benchmarking
Applied AI Researcher, Benchmarking
Applied AI Researcher, Benchmarking
Distyl
Distyl AI is seeking an Applied AI Researcher for their Benchmarking team to redefine software utilization in enterprise AI. The role involves designing evaluation frameworks and benchmarks to measure AI system performance, collaborating with industry leaders, and contributing to innovative AI-native technologies.
Qualification
- Experience designing and running evaluations, including benchmarks and test suites.
- Strong statistical and analytical skills to design reproducible experiments and extract meaningful insights from data.
- Experience in developing intelligent systems using models, focusing on techniques like ensembling and agentic collaboration.
- Proven track record of research results, including publications or contributions to the field of AI.
- Ability to creatively redefine software usage in enterprise settings.
Responsibility
- Define how progress is measured through evaluation frameworks capturing reasoning depth and interaction quality.
- Construct benchmarks reflecting real-world complexity for new architectures and techniques.
- Explore new paradigms for evaluating intelligent systems, including adversarial robustness testing and human-in-the-loop assessment.
- Investigate how metrics shape model behavior and establish methodologies for quantifying emergent capabilities.
- Drive internal research priorities and contribute to industry-wide standards.
