Model eval gets caught in a loop if the endpoint returns HTTP errors.

[valerie-autumn-skye]

I ran metacoder without having OpenAI configued, and the evaluation process gets stuck in a failure loop.
This should be detected so that the process can exit cleanly.

(metacoder) PS C:\Users\CTParker\PycharmProjects\metacoder> uv run metacoder eval .\tests\input\literature_mcp_eval_config.yaml
🔬 Running evaluations from: tests\input\literature_mcp_eval_config.yaml
📊 Loaded dataset: pubmed tools evals
   Models: claude-sonnet
   Coders: goose, dummy (all available)
   Cases: 1
   Total evaluations: 2

🚀 Starting evaluations...
Progress: 1/1 - goose/claude-sonnet/PMID_28027860_Full_Text with servers: mcp-simple-pubmed
Running goose with claude-sonnet on case 'PMID_28027860_Full_Text'
📁 Preparing workdir: eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text
🔒 Obtaining lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder      
🔧 Writing config object: .config/goose/config.yaml type=yaml
🔓 Releasing lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder      
🔒 Obtaining lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder      
🦆 Running command: goose run -t What is the first sentence of section 2 in PMID: 28027860?
🦆 Command took 25.10282039642334 seconds
🔓 Releasing lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder      
Evaluating with CorrectnessMetric
✨ You're running DeepEval's latest Correctness [GEval] Metric! (using gpt-4.1, strict=False, async_mode=True)...
Evaluating 1 test case(s) in parallel ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0:00:00
    🎯 Evaluating test case #0        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0:00:00HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Evaluating 1 test case(s) in parallel ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0:00:01
    🎯 Evaluating test case #0        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0:00:01HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Evaluating 1 test case(s) in parallel ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0:00:02
    🎯 Evaluating test case #0        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0:00:02HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
OpenAI Error: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.ope

Aborted!                                                                                                                                                                                                   
Evaluating 1 test case(s) in parallel ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0:00:05
    🎯 Evaluating test case #0        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0:00:05

[github-actions]

Issue Summary

This issue reports a critical problem where the metacoder evaluation framework gets stuck in an infinite loop when encountering HTTP errors from the OpenAI API, specifically HTTP 429 "Too Many Requests" errors. The evaluation process fails to handle these errors gracefully and should exit cleanly instead of retrying indefinitely.

Analysis

The problem occurs in the evaluation pipeline when using DeepEval's CorrectnessMetric with GPT-4 for evaluation. From the logs:

1. The goose coder successfully completes its task (25 seconds)
2. DeepEval attempts to evaluate the output using OpenAI's API
3. Receives HTTP 429 errors (quota exceeded)
4. Gets caught in a retry loop without proper error handling
5. Eventually shows "Aborted!" but continues the infinite loop

Root Cause

The issue stems from insufficient error handling in the evaluation framework when external APIs (OpenAI) return non-recoverable HTTP errors. The system should:

1. Detect quota/rate limit errors (HTTP 429) and other non-recoverable errors
2. Implement exponential backoff with maximum retry limits
3. Gracefully exit when API quotas are exceeded rather than infinite retries
4. Provide clear error messages about configuration issues (missing/invalid API keys)

Related Issues

This is part of a cluster of evaluation robustness issues:

• Issue Divide by zero error in metacoder eval summary. #15: Divide by zero error when no valid coder combinations are available
• Issue Coders should not rely on default encoding from operating system. #14: Unicode encoding errors on Windows breaking evaluations
• Issue --coder claude option returned non-zero exit status 1 #8 (closed): Non-zero exit status handling for claude coder

These suggest the evaluation framework needs broader error handling improvements.

Recommended Solutions

Immediate Fix

1. Add retry limits and exponential backoff to DeepEval API calls
2. Catch and handle specific HTTP error codes (429, 401, 403, etc.)
3. Exit gracefully with informative error messages

Broader Improvements

1. Pre-flight checks: Validate API keys and quotas before starting evaluations
2. Configuration validation: Ensure all required services are properly configured
3. Graceful degradation: Allow evaluations to continue with other metrics if one fails
4. Better logging: More detailed error reporting for debugging

Technical Implementation

The fix likely needs to be implemented in:

• src/metacoder/evals/runner.py - Add error handling around evaluation calls
• DeepEval integration - Wrap API calls with proper retry logic and error detection
• Configuration validation - Check API connectivity before starting evaluations

This appears to be a high-priority bug that significantly impacts the usability of the evaluation framework, especially for users with limited API quotas or misconfigured credentials.

[valerie-autumn-skye]

Here is my assessment:

• This is only happening with the Correctness metric.
• GEval uses OpenAI by default (but appears to have an option to specify a different model).
• When calls to OpenAI fail (e.g., 429 Too Many Requests from exceeding quota) the metric throws a connection error.
• DeepEval is being invoked asynchronously and has some sort of internal fail/retry logic.

So this is more of a bug with DeepEval, but might be able to be mitigated by invoking it with different parameters. Not yet sure if it can be run synchronously or force it to stop on fail.

Here is how evaluate is being invoked via runner.py:

            # Evaluate
            logger.info(f"Evaluating with {metric_name}")
            eval_results = evaluate([test_case], [metric])

and this is how runner.py invokes GEval:

def get_default_metrics() -> Dict[str, BaseMetric]:
    """Get default metrics. Creates instances lazily to avoid network calls during import."""
    return {
        "CorrectnessMetric": GEval(
            name="Correctness",
            criteria="Determine whether the actual output is factually correct based on the expected output.",
            # NOTE: you can only provide either criteria or evaluation_steps, and not both
            evaluation_steps=[
                "Check whether the facts in 'actual output' contradicts any facts in 'expected output'",
                "You should also heavily penalize omission of detail",
                "Vague language, or contradicting OPINIONS, are OK",
            ],
            threshold=0.8,
            evaluation_params=[
                LLMTestCaseParams.INPUT,
                LLMTestCaseParams.ACTUAL_OUTPUT,
                LLMTestCaseParams.EXPECTED_OUTPUT,
            ],
        ),
        "DummyMetric": DummyMetric(threshold=0.5),
    }

The method signature for GEval (from g_eval.py) is:

Need to look into what options are available for the model parameter.

class GEval(BaseMetric):
    def __init__(
        self,
        name: str,
        evaluation_params: List[LLMTestCaseParams],
        criteria: Optional[str] = None,
        evaluation_steps: Optional[List[str]] = None,
        rubric: Optional[List[Rubric]] = None,
        model: Optional[Union[str, DeepEvalBaseLLM]] = None,
        threshold: float = 0.5,
        top_logprobs: int = 20,
        async_mode: bool = True,
        strict_mode: bool = False,
        verbose_mode: bool = False,
        _include_g_eval_suffix: bool = True,
    ):