The library helps to evaluate the search and generation bundle from the saved examples. The Faithfulness metric checks which statements follow from the fragments found.
Save the fragments that the model saw
An assistant confidently gives a deadline for returns that does not appear in the retrieved article. Save the question, the answer, and the passages actually sent to the model. The Ragas Faithfulness metric helps identify claims unsupported by that context. Evaluate retrieval separately if the relevant passage was missed or ranked below irrelevant results.
How to collect a training test
Take the fictional store rules and a few queries. In one example, the answer adds a non-existent term, in another correctly conveys the found rule, in the third you need an additional question. The manual mark is useful for checking the automatic evaluation. After changing the way you search, compare not only the overall figure, but also individual answers to the same questions. So it’s easier to see which setting has improved the context and which errors remain.
Where the assessment may be wrong
If a retrieved document contains an outdated rule, an answer can still be well grounded in it. Fact checking needs an up-to-date source. The evaluator model may also misunderstand negation or exceptions. Select examples like these for manual review and store the metric version with the result. Before using internal material, decide whether it can be shared with your evaluation provider and limit the dataset accordingly.
Suitable tasks
- Find answers with unconfirmed claims
- Compare two search options for the same questions
Strengths
- There are separate metrics for answering and searching
- The certificate describes in detail the expected inputs and calculation
Limitations
- Model-based scoring can be wrong on ambiguous examples
- An answer supported by the context can still rely on a false source
Selection and conditions
- Free access
- Yes
- Trial period
- Not confirmed
- No sign-up required
- Not confirmed
- Russian interface
- Not confirmed
- Local installation
- Yes
- API
- Not confirmed
“Not confirmed” means there is not enough information. The feature may still be available; check the developer’s terms.
- Input formats
- Text, Data
- Result
- Data, Text
Terms
The library is open source. Requests to an external evaluator model may incur charges at the provider's rates. Model-based scoring can be wrong on ambiguous examples.From task to first result
Practical example
- Task
- Find answers with unconfirmed claims
- What to prepare
- Training questions, expected conditions, correct and erroneous answers.
Task brief or prompt
Evaluate the answers according to one approved criterion. Repeat the run after changing the instructions and show cases in which the score has changed.
Expected result
A list of regressions with estimates and verifiable examples.
How to check the result
- Check the scores with manual markings.
- Leave a few border cases out of the threshold setting.
An editorial teaching example of a task brief, not the result of hands-on testing of the service. Check that the required mode is available, that you have rights to the source material and that the intended use of the result is permitted.
Check details
Official sources
This description is based on the developer’s material. Features, prices and availability may change; check the primary source before use.
