AI

Evaluations and K-LLM ensembles

Measure quality and combine several models.

Evaluations

Evals measure how well a model, Logic function or agent performs on a fixed set of cases.

• A suite holds test cases and how to score them.
• A run executes the suite and stores per-case results. Runs can be compared side by side, and variance across repeated runs is reported.
• Experiments compare configurations over the same suite.
• Results can be written to a dataset for further analysis.

Creating and editing suites requires an administrator; analysts and operators can run them.

K-LLM ensembles

An ensemble sends one prompt to K models in parallel (up to 8) and combines their answers. You may repeat the same model to measure how consistent it is with itself.

Synthesis modes:

Disagreement between models is recorded, not hidden.

In auto mode, Converge picks the models for a task from its history: win rate, agreement and cost per task tag. With too little history it falls back to the default models set by policy.

Each leg of an ensemble goes through the gateway like any other call, with the same data rules, rate limits and usage accounting. Logic functions can use ensembles through the ensemble block.