Asking the most powerful model to confirm "one margherita pizza, takeaway" is like calling a surgeon to apply a plaster. This subject is about sending every request to the right model, at the right price, without degrading quality.
Context
Every conversation with Ordy, our ordering agent for restaurants, and with Tenjah, our AI teacher for the baccalaureate, has a cost. Yet the most powerful model is not always needed. We need an infrastructure layer that chooses, caches, continuously measures quality and stays up when a provider goes down.
What you will do
- Design a learned router that picks, for each request, the cheapest model able to answer it (difficulty classifier, contextual bandit)
- Build a semantic cache with freshness control
- Evaluate response quality automatically (LLM-as-judge calibrated on human ratings)
- Detect quality drift over time
- Build a high-availability gateway: provider fallback, rate limiting, per-client cost tracking
- Expose the metrics in dashboards people can act on
Technical challenges
- Balancing cost, quality and latency without optimising one at the expense of the others
- Evaluating without ground truth: an automatic judge is only useful once calibrated
- Keeping a semantic cache from serving a stale or slightly wrong answer
- Keeping the gateway simple to operate and debug
Deliverables
- A working LLM gateway integrated into the B2H platform
- A router and cache evaluated on representative traffic
- A continuous evaluation and drift-detection setup
- Dashboards, technical documentation and a demo
- The final-year report
Success criteria
- Cost reduced by at least 40% at equal quality (target)
- p95 latency measured and kept under control
- Gateway availability measured, including during simulated provider outages
- Agreement between the LLM judge and human ratings measured
Tech stack
- LLM gateway
- Contextual bandits
- Redis
- Prometheus, Grafana
- Docker
Who we are looking for
- A final-year engineering or master's student in software engineering or MLOps
- A good grasp of distributed systems and the cloud
- A taste for measurement: you cannot cut a cost you do not measure
- Rigour around reliability and operations
- Fluent French and technical English
Practical details
- Duration: 4 to 6 months (February – June 2027, indicative)
- Supervised by the B2H engineering team, with a weekly check-in
- Integrated into the internal B2H platform (single identity, event bus, shared data)
- How to apply: CV + cover letter, at most 2 subjects per candidate
- Deadline: 15/12/2026