Let's Build a Nutrition Agent with Unsloth: RL Fine-Tuning a Small Model

John Lu

John Lu

Day 2 • Sun, Oct 18
14:00 - 14:45
Location
R1
Language
English
Category • Level
Machine Learning • Expert

Snap a photo of your lunch and have an app log it (“Lentil Curry, ~320 kcal, 34g protein”). The obvious build throws a giant multimodal model at the job, but that’s slow and costly to run all day.

We build this nutrition coach as a multimodal agent on Gemma 4, then take on the real problem: making it cheap. We right-size the model for the machine it will run on, post-train it with Unsloth (SFT, then GRPO reinforcement learning) on free Colab/Kaggle GPUs, and serve it with vLLM.

Along the way you’ll get a practical tour of the whole LLM lifecycle. You’ll see how to choose among the Gemma 4 family and how RL for LLMs evolved from PPO to DPO to GRPO, including why rule-based rewards can replace a learned reward model. You’ll also see what makes vLLM fast: PagedAttention, prefix caching, continuous batching and speculative decoding. You’ll leave knowing where SFT ends and RL begins: SFT sets the floor, RL raises the ceiling. All demos are runnable at home on free GPUs.

Description

We build up with runnable demos and a public repo attendees can use afterward, following the four stages of building an LLM (model, pre-training, post-training, inference) through one running example.

  1. The product and the big picture. We open with a demo: meal photo in, structured nutrition log out, with a vision-language model identifying the food and calling a nutrition lookup. Then we zoom out to the four stages of building an LLM, and why more capability now comes from inference-time compute as thinking models spend tokens reasoning. Training and inference have different compute needs, so their software stacks should be designed differently. That frames the talk: pick the model, post-train it with Unsloth, serve it with vLLM.
  2. Choosing the right model: Gemma 4. A tour of the family: five sizes, four architectures. The dense, on-device E2B and E4B use per-layer embeddings and dedicated audio and vision encoders. The laptop-class 12B is encoder-free, feeding inputs like audio straight into the LLM. The 26B A4B is a mixture-of-experts (26B parameters, 4B active) with a larger vision encoder for harder vision tasks. The 31B dense model is the most capable. The rule we apply: size the model to the machine you deploy on. Demo: running Gemma 4 locally with Unsloth on a free Colab/Kaggle GPU.
  3. Pre-training, briefly. Next-token prediction is self-supervised, so no labeling effort is needed, and the model implicitly learns language patterns, writing styles and world knowledge. The hard part is scale: parallelism across thousands of GPUs, automated crash recovery and petabyte-scale data pipelines. Even then, the result can be unpredictable and unaligned, which is where post-training comes in.
  4. Post-training with Unsloth: SFT, then RL. The heart of the talk. SFT teaches instruction following and tool use; RL then pushes toward the behaviors we want and away from the ones we don’t. We take a gentle tour of the math, followed by a demo: post-training a Gemma model with Unsloth on free GPUs.
    • PPO: advantage, clipped policy ratio, and a KL penalty against a frozen reference policy.
    • DPO: optimizing directly on preferences, with no explicit reward model.
    • GRPO, as used in DeepSeek-R1-Zero: sample a group of responses per prompt, score them with rule-based rewards (checkable answers, test cases, output format), and normalize rewards within the group so better-than-average responses get encouraged.
  5. Serving with vLLM. At scale, serving is less about raw compute than memory management, scheduling and hardware utilization. We unpack vLLM’s answers: PagedAttention (KV cache in fixed-size blocks), prefix caching (ideal for agentic and conversational workloads), continuous batching (token-level scheduling for irregular traffic) and speculative decoding, including the diffusion-style variant that drafts a block of tokens in a single pass. A unified backend means you don’t rewrite your application layer or serving stack. Demo: serving Gemma 4 with vLLM.
  6. Takeaways. One per stage: right-size the model for your deployment machine; pre-training is a compute-and-parallelism problem; SFT sets the floor and RL raises the ceiling; vLLM gives you high-throughput, memory-efficient inference.
John Lu
John Lu

John is a Senior Machine Learning Engineer from a Top Silicon Microchip Company, with hands-on, production experience training Large Language Models, Vision Language Models, and building RAG and tool-calling systems. He likes pulling apart the "magic" of modern AI and rebuilding it in clear, runnable code, and applying it to problems he lives with: optimizing his own diet and tracking nutrient-dense foods is a long-standing habit, which makes this project something he actually uses. He has spoken at PyCon APAC and COSCUP on applied ML and on-device AI, including TasteGram (image assessment with TensorFlow & Python) and Kimoji (an On-Device LLM mood-tracker system).

Related Speeches