Canonical: https://instain.ru/en/ai/promptfoo/ Language: en OpenAI AI Application Security Checker Tool # Promptfoo. Repeatable checks for applications with language models 4.40 / 5 [Directory score](https://instain.ru/en/ai/promptfoo/#editorial-rating) [Official website ](https://developers.openai.com/api/docs/guides/promptfoo-codex) Promptfoo launches entry and verification suites for AI applications. It can be used to evaluate changes and verify agreed rules in your environment. ![OpenAI Guide with example of Promptfoo report](https://instain.ru/assets/images/promptfoo-screen-a04439ab6fe4-1200.webp) OpenAI Guide with example of Promptfoo report. [Open the source ](https://developers.openai.com/api/docs/guides/promptfoo-codex) ## Check not only the wording of the refusal An agent may say it changed nothing after a tool has already issued a command. In Promptfoo, write cases that check both the answer and available execution evidence. Start with your own test application and a stub for the external tool. Define allowed operations and keep the call log. That gives you evidence beyond the model’s wording. ## Educational boundary of authority The test assistant can read order status, while changes require human approval. Include a normal status question, an ambiguous request, and an explicit request to change the record. Inspect the response and the number of write-tool calls. When you change the model, rerun the cases on the same data. Verify that allowed reads still work after correcting unsafe behavior. ## What Makes Testing Repeatable? Store the inputs, expected results and app version together. When the result is evaluated by another model, review the disputed cases manually. For a structured signal, such as having a log entry, use an accurate check in the code. The set of tests covers the given situations. Add new examples of real role and tool changes, while maintaining the validation permissions and test environment boundaries. For each run, note what access the agent actually received. Repeat the script with a different role if the application serves multiple groups of users. ## Suitable tasks - Check if the test agent is performing only authorized actions - Compare the results of two configurations on one set Features ## Strengths - Supports reproducible test sets and result checks - There are official guidelines for integration and evaluation Things to consider ## Limitations - Test only your own application or an environment you have explicit permission to test - Checking the response text alone does not verify external actions or permissions ## Selection and conditions Free access Yes Trial period Not confirmed No sign-up required Not confirmed Russian interface Not confirmed Local installation Yes API Not confirmed “Not confirmed” means there is not enough information. The feature may still be available; check the developer’s terms. Input formats Text, Code Result Text, Data ### Terms The open-source library requires no subscription. Model calls, compute, and managed features are billed separately when applicable. Test only your own application or an environment you have explicit permission to test. From task to first result ## Practical example Task Check if the test agent is performing only authorized actions What to prepare A test copy of the assistant and a fictional read and write policy. ### Task brief or prompt Run regular and border requests. Check that the valid reading is working, and the action without permission is not in the journal. ### Expected result Report with inputs, observed actions and comments to the rules. ### How to check the result - Check the tool log separately from the response text. - Test only in an agreed environment. An editorial teaching example of a task brief, not the result of hands-on testing of the service. Check that the required mode is available, that you have rights to the source material and that the intended use of the result is permitted. Check details ## Official sources - [The OpenAI Promptfoo Guide ](https://developers.openai.com/api/docs/guides/promptfoo-codex) - [Official Introduction to Promptfoo ](https://www.promptfoo.dev/docs/intro/) This description is based on the developer’s material. Features, prices and availability may change; check the primary source before use. About the tool Developer OpenAI Category [Code](https://instain.ru/en/category/code/) Access Run the open-source tool from your project. Connect the application under test and, if needed, a model to evaluate it. Payment The open-source library requires no subscription. Model calls, compute, and managed features are billed separately when applicable. API No separate API specified [Developer terms ](https://www.promptfoo.dev/docs/intro/) Directory score 4.40 / 5 Features 4.50 / 5 Workflows 4.10 / 5 Ecosystem 4.40 / 5 Documentation 4.60 / 5 Scores are an editorial guide to the directory. They are not user reviews or the results of personal testing. [See the methodology ](https://instain.ru/en/methodology/) Keep exploring ## For similar tasks [Full category ](https://instain.ru/en/category/code/) Code ### [Inspect AI](https://instain.ru/en/ai/inspect-ai/) Tasks with defined inputs, steps, and result checks 4.38 / 5 Framework for Language Model Assessment Similar tasks; compare terms, formats and limitations Code ### [NVIDIA NeMo Guardrails](https://instain.ru/en/ai/nvidia-nemo-guardrails/) Programmable rules for model inputs, outputs, and actions 4.30 / 5 rule library for applications with models Similar tasks; compare terms, formats and limitations