Learn Japanese with Manga! Building a Multimodal Search Engine with Gemini Embedding 2

@x0mg

John Lu
Snap a photo of your lunch and have an app log it (“Lentil Curry, ~320 kcal, 34g protein”). The obvious build throws a giant multimodal model at the job, but that’s slow and costly to run all day.
We build this nutrition coach as a multimodal agent on Gemma 4, then take on the real problem: making it cheap. We right-size the model for the machine it will run on, post-train it with Unsloth (SFT, then GRPO reinforcement learning) on free Colab/Kaggle GPUs, and serve it with vLLM.
Along the way you’ll get a practical tour of the whole LLM lifecycle. You’ll see how to choose among the Gemma 4 family and how RL for LLMs evolved from PPO to DPO to GRPO, including why rule-based rewards can replace a learned reward model. You’ll also see what makes vLLM fast: PagedAttention, prefix caching, continuous batching and speculative decoding. You’ll leave knowing where SFT ends and RL begins: SFT sets the floor, RL raises the ceiling. All demos are runnable at home on free GPUs.
We build up with runnable demos and a public repo attendees can use afterward, following the four stages of building an LLM (model, pre-training, post-training, inference) through one running example.

John is a Senior Machine Learning Engineer from a Top Silicon Microchip Company, with hands-on, production experience training Large Language Models, Vision Language Models, and building RAG and tool-calling systems. He likes pulling apart the "magic" of modern AI and rebuilding it in clear, runnable code, and applying it to problems he lives with: optimizing his own diet and tracking nutrient-dense foods is a long-standing habit, which makes this project something he actually uses. He has spoken at PyCon APAC and COSCUP on applied ML and on-device AI, including TasteGram (image assessment with TensorFlow & Python) and Kimoji (an On-Device LLM mood-tracker system).