Shreya Shankar

Shreya Shankar is a computer science researcher at UC Berkeley specializing in machine learning systems, data management, and operational tooling for AI evaluation.

Key Research & Systems

1. Error Discovery & Active Learning: Pioneered active-learning workflows where an AI assistant observes real-time human trace annotations, updates a dynamic failure taxonomy, and proactively queries similar unlabeled records.
2. Criteria Drift in Evaluation: Demonstrated empirically that humans often cannot specify complete evaluation rubrics up front; criteria naturally emerge through the iterative act of labeling and observing model behaviors.
3. Model Cascades (BARGAIN): Created algorithms for optimal model cascading, routing high-confidence queries to cheap, small models while reserving frontier models for ambiguous edge cases to cut inference costs up to 86% without sacrificing quality.
4. Data Agent Benchmark (DAB): Designed benchmarks reflecting realistic, messy multi-database environments to measure planning and execution failures in analytical agents.

Related

• hamel husain
• error analysis and evals
• automated eval engineering
• closed loop agent improvement