Listen and Subscribe
Welcome to AI in the Real World! In this episode, Foundation Capital Partner Jaya Gupta sits down with Rohan Taori, a researcher on Anthropic's multimodal pre-training team.
Within the AI community, Rohan is best known for co-creating Alpaca, a project that demonstrated how fine-tuning Meta’s LLaMA model could achieve ChatGPT-level performance for under $600.
Rohan shares his journey from early work in computer vision at UC Berkeley to his Ph.D. at Stanford, where he explored methods for making AI more accessible.
He explains the technical breakthroughs behind Alpaca, including self-instruct, a method that uses a stronger language model (OpenAI’s text-davinci-003) to generate synthetic data that is then used to fine-tune a weaker model (Llama). This approach, which underpins Alpaca and its follow-up projects AlpacaFarm and AlpacaEval, illustrates how small-scale post-training can significantly enhance model performance.
The conversation also covers:
The promise and challenges of synthetic data for training AI models
What it will take to build foundation models that are 100x better
The future of multimodal AI and why it matters
Why better evals are critical to the next wave of AI advances
Chapters
00:00 Cold open
1:40 Rohan’s journey into AI research
04:50 Transitioning from vision research to LLMs
06:18 The story behind Alpaca
08:55 How Alpaca works
10:45 The AI community’s reception of Alpaca
12:26 The evolution of Alpaca related projects
14:22 The role of synthetic data
19:38 Challenges in multimodal AI
24:31 Future of foundation models
30:00 Importance of data in AI
34:48 Staying up to date with the latest AI research
36:12 Advice for founders