Evaluation Metric: Summarization

This n8n template demonstrates how to calculate the evaluation metric "Summarization" which in this scenario, measures the LLM's accuracy and faithfulness in producing summaries which are based on an incoming Youtube transcript. The scoring approach is adapted from https://cloud.google.com/vertex-ai/generative-ai/docs/models/metrics-templatespointwisesummarizationquality How it works This evaluation works best for an AI summarization workflows. For our scoring, we simple compare the generated response to the original transcript. A key factor is to look out information in the response which is not mentioned in the documents. A high score indicates LLM adherence and alignment whereas a low sco

Help me set up the n8n workflow "Evaluation Metric: Summarization" (https://n8n.io/workflows/4428). It uses: Google Drive, Basic LLM Chain, OpenAI Chat Model, Structured Output Parser, Google Gemini Chat Model, Evaluation. Import the template JSON into my n8n instance, list every credential I need to create, and walk me through testing it.