This node utilizes the provided knowledge base and the task description of the RAG system to automatically generate a diverse set of test questions that are designed to assess the performance of various components of the RAG system, such as the retriever, generator, and knowledge base quality. The questions target specific aspects of the RAG system, helping to identify potential weaknesses and areas for improvement.
The different types of questions include
- Simple questions : Simple questions generated from an excerpt of the knowledge base.
- Complex questions : More complex questions that use paraphrasing.
- Distracting questions : Questions containing distracting information to test the retrieval part of the RAG system.
- Situational questions : Questions that include context information to evaluate if the system can produce answers relevant to the context.
- Double questions : Questions consisting of two separate parts.
This node does not support Giskard's conversational questions.
For more details and examples refer to the Giskard documentation .
The output table will contain the following columns:
- Question : The generated question.
- Reference Answer : A reference answer to the question.
- Reference Context : The context used to create the reference answer.
- Metadata : Additional information specific to the type of question.
Note : If you use the Credentials Configuration node and do not select the "Save password in configuration (weakly encrypted)" option for passing the API key for the chat model or embeddings connector node(s), the Credentials Configuration node will need to be reconfigured upon reopening the workflow, as the credentials flow variable was not saved and will therefore not be available to downstream nodes.