Small Models, Big Agents: Fine-Tuning and Serving SLMs on Kubernetes

Key Takeaways

• Match the model to the task: which agent steps suit small CPU models, and which genuinely need a larger LLM.
• Deploy efficient SLM inference on CPUs with llama.cpp.
• Build an agent whose routing, tool selection, argument extraction, and self-critique run on small CPU models, handing off to a larger model for open-ended reasoning when that's the right call.
• Train task-specific CPU specialists (classification, scoring, extraction, embedding) for each decision in the pipeline.
• Give agents safe, read-only tools so answers stay grounded in live cluster state.
• Right-size infrastructure with KEDA and Karpenter to balance cost and performance.
• Apply "right tool for the right task" thinking to production agentic AI.

Target Audience

• Platform Engineers
• Kubernetes Engineers
• MLOps Engineers
• AI/ML Engineers
• Cloud Architects
• DevOps Engineers
• Technical Leaders responsible for AI platforms

Requirements for Attendees

Technical Requirements

• Basic understanding of Kubernetes concepts.
• Familiarity with containers and cloud-native applications.
• General knowledge of Generative AI, LLMs, and AI agents.

Equipment Requirements

Laptop required.

Experience Level

Intermediate.

Duration

2–3 hours.