Motion-Y LogoMotion-Y
←All projects

Evaluation/Case StudyShipped/ 2025

Quiver Eval

LLM evaluation harness.

EvaluationLLMOpsPython
GitHubLive Demo
/quiver-eval
quiver eval / onlineEvaluation · LLMOps · Python

Problem and solution

01/Problem

Most teams can't tell whether their LLM feature got better or worse after a change. Evals are scattered across notebooks, dashboards, and vibes.

02/Solution

Quiver is a small CLI + dashboard that turns evals into CI gates. Define a rubric, write cases, run on every PR, review failures in one place.

03/How it works

The system, end to end.

  1. 01next →

    Author

    Write eval cases + rubrics.

  2. 02next →

    Run

    Execute locally or in CI.

  3. 03next →

    Compare

    Diff results against the last green.

  4. 04next →

    Gate

    Block merges on regression.

  5. 05loop

    Review

    Annotate failures for the next pass.

04/Architecture

Components and data flow.

How the moving parts fit together. Solid arrows are synchronous calls; dashed arrows are queued or batched.

  1. 01

    CLI

  2. 02

    Case runner

  3. 03

    Rubric scorer

  4. 04

    LLM judge

  5. 05

    Result store

  6. 06

    CI integration

  7. 07

    Web dashboard

↓ flow continues

05/Technology

The stack.

AI / ML
GPT-class judgeCustom rubric scorers
Frontend
ReactVite
Backend
PythonFastAPI
Database
PostgresDuckDB
Infrastructure
GitHub ActionsDocker
APIs
OpenAIAnthropic

06/Engineering decisions

What we picked, and why.

  1. 01

    Code-first cases

    Choice

    Cases as Python + YAML, not a web form

    Alternatives

    • Web-only authoring

    Why

    Version control and reviewability beat any UI for eval authoring.

07/Evaluation

What we measured.

  • 01

    2.4k

    Eval runs / week

  • 02

    <1d

    Median regression catch

  • 03

    2.1%

    False positive gate

08/Challenges

What almost broke us.

  • 01

    Judge drift

    What happened

    LLM judge scores drifted over time.

    What we did

    Pinned judge model + monthly calibration set.

09/Results

Caught 14 regressions pre-merge in the first quarter across 6 product teams.

10/Future improvements

  • ›Multi-judge consensus
  • ›Eval coverage reports

/explore the build

GitHubLive Demo

Have an interesting AI problem?

Let's build something.

If you're working on something that needs real AI engineering — not just a demo — let's talk.

Start a Conversation
© 2026
Motion-Y

Hello, I'm your Motion-Y Assistant. How can I help you today?