Evaluate AI Agent Response Relevance using OpenAI and Cosine Similarity
n8n template #4425Summary
Use templateThis n8n template demonstrates how to calculate the evaluation metric "Relevance" which in this scenario, measures the relevance of the agent's response to the user's question. The scoring approach is adapted from the open-source evaluations project RAGAS and you can see the source here https://github.com/explodinggradients/ragas/blob/main/ragas/src/ragas/metrics/answerrelevance.py How it works This evaluation works best for Q&A agents. For our scoring, we analyse the agent's response and ask another AI to generate a question from it. This generated question is then compared to the original question using cosine similarity. A high score indicates relevance and the agent's successful ability
Hand off to your agent
Prompt
Help me set up the n8n workflow "Evaluate AI Agent Response Relevance using OpenAI and Cosine Similarity" (https://n8n.io/workflows/4425). It uses: HTTP Request, Code, AI Agent, Basic LLM Chain, OpenAI Chat Model, Structured Output Parser, Evaluation. Import the template JSON into my n8n instance, list every credential I need to create, and walk me through testing it.
Paste into Claude Code and it will do the rest.
Apps and nodes
Similar workflows
NameViews
- Evaluate AI Agent Response Correctness with OpenAI and RAGAS Methodology648
- Automate SEO blog content creation with GPT-4, Perplexity AI and WordPress6,949
- Automate SEO blog creation + social media with GPT-4, Perplexity and WordPress5,798
- Explore n8n Nodes in a Visual Reference Library5,197
- Automated HR Service System with WhatsApp, GPT-4 Classification & Google Workspace4,541
- Evaluate RAG Response Accuracy with OpenAI: Document Groundedness Metric638
- Transform quotes to viral videos with Gemini, GPT & ElevenLabs for social media214
- WordPress blog automation with Airtable interface, human review & AI research v2174