Orchestrator

Graph

Node Configuration

Run with Parameters

Pipeline JSON

{
  "edges": [
    {
      "to": "judge",
      "from": "__start__"
    },
    {
      "to": "__end__",
      "from": "judge"
    }
  ],
  "nodes": [
    {
      "name": "judge",
      "type": "llm",
      "config": {
        "llm": {
          "model": "gpt-4o",
          "temperature": 0
        },
        "output_schema": {
          "type": "object",
          "title": "SummarizationChecklistJudgeResult",
          "required": [
            "faithfulness",
            "coverage",
            "instruction_adherence",
            "clarity",
            "unsupported_claims",
            "missing_points",
            "error_types",
            "rationale",
            "verdict"
          ],
          "properties": {
            "clarity": {
              "enum": [
                "pass",
                "fail"
              ],
              "type": "string",
              "description": "SUMMARY coherent and usable?"
            },
            "verdict": {
              "enum": [
                "accept",
                "review",
                "reject"
              ],
              "type": "string",
              "description": "Suggested gate; prefer recomputing from aspects in code"
            },
            "coverage": {
              "enum": [
                "pass",
                "partial",
                "fail"
              ],
              "type": "string",
              "description": "PROMPT-required points present in SUMMARY?"
            },
            "rationale": {
              "type": "string",
              "description": "Brief evidence-backed justification citing SOURCE/PROMPT"
            },
            "error_types": {
              "type": "array",
              "items": {
                "enum": [
                  "omission",
                  "distortion",
                  "hallucination",
                  "verbosity",
                  "wrong_focus"
                ],
                "type": "string"
              },
              "description": "Closed error taxonomy; empty if none"
            },
            "faithfulness": {
              "enum": [
                "pass",
                "fail"
              ],
              "type": "string",
              "description": "Every material SUMMARY claim supported by SOURCE?"
            },
            "missing_points": {
              "type": "array",
              "items": {
                "type": "string"
              },
              "description": "Required points omitted from SUMMARY"
            },
            "unsupported_claims": {
              "type": "array",
              "items": {
                "type": "string"
              },
              "description": "SUMMARY claims not grounded in SOURCE"
            },
            "instruction_adherence": {
              "enum": [
                "pass",
                "fail"
              ],
              "type": "string",
              "description": "SUMMARY follows PROMPT constraints?"
            }
          }
        },
        "system_prompt": "You are an expert editor evaluating a summarization produced by another model.\n\nThe human message contains:\n- SOURCE: original document (or excerpt)\n- PROMPT: instructions given to the summarizer\n- SUMMARY: model output to evaluate\n\nJudge ONLY the SUMMARY against SOURCE and PROMPT. Do not rewrite the summary.\n\nEvaluate these aspects independently (do not collapse into a single 1-5 score):\n\n1. faithfulness \u2014 pass if every material claim in SUMMARY is supported by SOURCE; fail if any claim is unsupported, contradicted, or invented.\n2. coverage \u2014 pass if SUMMARY covers the points required by PROMPT; partial if important but not critical points are missing; fail if required substance is absent.\n3. instruction_adherence \u2014 pass if SUMMARY follows PROMPT constraints (format, length, tone, required fields); fail otherwise.\n4. clarity \u2014 pass if SUMMARY is coherent and usable as written; fail if it is garbled, truncated nonsense, or unusable.\n\nAlso list:\n- unsupported_claims: concrete SUMMARY claims not grounded in SOURCE (empty if none)\n- missing_points: required content from PROMPT/SOURCE that SUMMARY omitted (empty if none)\n- error_types: zero or more of: omission, distortion, hallucination, verbosity, wrong_focus\n\nverdict guidance (suggestion only; downstream may recompute):\n- reject if faithfulness is fail OR clarity is fail OR coverage is fail\n- review if any aspect is partial OR error_types is non-empty but not reject-worthy\n- accept otherwise"
      }
    }
  ],
  "description": "Checklist LLM-as-judge for ACE summarization. Seed messages with SOURCE, PROMPT, and SUMMARY. Returns per-aspect pass/fail labels; verdict is a model suggestion that callers may recompute from aspects."
}