In DeepEval, you can collect test cases with input, response and criteria. Some metrics use a different model, so thresholds need to be checked against examples.
How to spot a silent regression
After editing the instruction, the assistant responds shorter, but began to skip an important exception. DeepEval helps to turn such a case into a repeatable test. Save the question, source material and expected condition. If you need a model-based evaluator for a metric, first check it out on multiple answers with known markup. The phrase “appreciate the quality” is too broad. For example, it is important to find out if an exception is mentioned when it applies to a particular input.
Two answers with a similar beginning
Make the correct answer, the answer with the lost condition and the answer with the statement that is not in the source. Run the selected check and compare its solution with a manual label. If the criterion misses an error, clarify it or add a separate rule. Include a few border examples after setting up, so as not to adjust the threshold for the first selection.
With the new version of the instructions run the same set. Keep the scores at the level of individual questions, otherwise one average will hide the changes.
When a model check is appropriate
If you can reliably extract a required field, check it in code. A model evaluator can assess meaning, but review a sample of its judgments. Record the evaluator version, settings, and run cost. If results fluctuate around the threshold, inspect the actual answers before making the test a release gate. Keep the metric definition with the test set.
Suitable tasks
- Track the deterioration of responses after the instruction is changed
- Check the suitability of the selected metric on marked examples
Strengths
- Document test cases and metric options
- Supports test runs from command line
Limitations
- An evaluator model can make mistakes or change its judgment between calls
- Compare criteria and thresholds with examples reviewed by people
Selection and conditions
- Free access
- Yes
- Trial period
- Not confirmed
- No sign-up required
- Not confirmed
- Russian interface
- Not confirmed
- Local installation
- Yes
- API
- Not confirmed
“Not confirmed” means there is not enough information. The feature may still be available; check the developer’s terms.
- Input formats
- Text, Data
- Result
- Data, Text
Terms
The open library requires no paid plan. Evaluator model calls and cloud platform features may incur separate charges. An evaluator model can make mistakes or change its judgment between calls.From task to first result
Practical example
- Task
- Track the deterioration of responses after the instruction is changed
- What to prepare
- Training questions, expected conditions, correct and erroneous answers.
Task brief or prompt
Evaluate the answers according to one approved criterion. Repeat the run after changing the instructions and show cases in which the score has changed.
Expected result
A list of regressions with estimates and verifiable examples.
How to check the result
- Check the scores with manual markings.
- Leave a few border cases out of the threshold setting.
An editorial teaching example of a task brief, not the result of hands-on testing of the service. Check that the required mode is available, that you have rights to the source material and that the intended use of the result is permitted.
Check details
Official sources
This description is based on the developer’s material. Features, prices and availability may change; check the primary source before use.
