Promptfoo launches entry and verification suites for AI applications. It can be used to evaluate changes and verify agreed rules in your environment.
Check not only the wording of the refusal
An agent may say it changed nothing after a tool has already issued a command. In Promptfoo, write cases that check both the answer and available execution evidence. Start with your own test application and a stub for the external tool. Define allowed operations and keep the call log. That gives you evidence beyond the model’s wording.
Educational boundary of authority
The test assistant can read order status, while changes require human approval. Include a normal status question, an ambiguous request, and an explicit request to change the record. Inspect the response and the number of write-tool calls. When you change the model, rerun the cases on the same data. Verify that allowed reads still work after correcting unsafe behavior.
What Makes Testing Repeatable?
Store the inputs, expected results and app version together. When the result is evaluated by another model, review the disputed cases manually. For a structured signal, such as having a log entry, use an accurate check in the code. The set of tests covers the given situations. Add new examples of real role and tool changes, while maintaining the validation permissions and test environment boundaries. For each run, note what access the agent actually received. Repeat the script with a different role if the application serves multiple groups of users.
Suitable tasks
- Check if the test agent is performing only authorized actions
- Compare the results of two configurations on one set
Strengths
- Supports reproducible test sets and result checks
- There are official guidelines for integration and evaluation
Limitations
- Test only your own application or an environment you have explicit permission to test
- Checking the response text alone does not verify external actions or permissions
Selection and conditions
- Free access
- Yes
- Trial period
- Not confirmed
- No sign-up required
- Not confirmed
- Russian interface
- Not confirmed
- Local installation
- Yes
- API
- Not confirmed
“Not confirmed” means there is not enough information. The feature may still be available; check the developer’s terms.
- Input formats
- Text, Code
- Result
- Text, Data
Terms
The open-source library requires no subscription. Model calls, compute, and managed features are billed separately when applicable. Test only your own application or an environment you have explicit permission to test.From task to first result
Practical example
- Task
- Check if the test agent is performing only authorized actions
- What to prepare
- A test copy of the assistant and a fictional read and write policy.
Task brief or prompt
Run regular and border requests. Check that the valid reading is working, and the action without permission is not in the journal.
Expected result
Report with inputs, observed actions and comments to the rules.
How to check the result
- Check the tool log separately from the response text.
- Test only in an agreed environment.
An editorial teaching example of a task brief, not the result of hands-on testing of the service. Check that the required mode is available, that you have rights to the source material and that the intended use of the result is permitted.
Check details
Official sources
This description is based on the developer’s material. Features, prices and availability may change; check the primary source before use.
