Accelerating Algorithm Autoresearch
Bryce Adelstein Lelbach
Optimizing software has never been easier with the advent of agentic AI... right? It turns out it's not quite that simple!
Autoresearch is a powerful technique for optimization; an agent iteratively modifies, tests, and benchmarks changes to the codebase with the goal of maximizing the benchmark score.
While powerful, this process faces challenges, especially at scale:
- Cheating / reward hacking (just as in reinforcement learning).
- Overfitting to benchmark's inputs.
- Tradeoffs between input breadth and build/test/benchmark cost.
- Local maximums.
In this talk, we'll discuss some of these challenges and how to overcome them, based on examples from the open source GPUMODE kernel competitions and infrastructure. We'll explore how to take autoresearch results from "cool idea" to "shipped in production".
Bryce Adelstein Lelbach has spent over a decade developing programming languages, compilers, and libraries. He is passionate about parallel programming and strives to make it more accessible for everyone. Bryce is a Principal Engineer at NVIDIA, where he founded the CUDA Core Compute Libraries team and now leads the Vanguard Programming group that drives NVIDIA's roadmap for programming languages, compilers, and core libraries. He is a leader of the systems programming language community, having served as chair of the C++ Library Evolution and the US programming language standards committee. He has been an organizer and program chair for many conferences over the years. On the C++ committee, he has worked on concurrency primitives, parallel algorithms, senders, and multidimensional arrays. He previously worked at Lawrence Berkeley National Laboratory and Louisiana State University. He is one of the founding developers of the HPX parallel runtime system. Outside of work, Bryce is passionate about airplanes and watches. He lives in Midtown Manhattan with his girlfriend and dog.