Divide by zero error in metacoder eval summary.

[valerie-autumn-skye]

This is an edge case that arises when no valid coder combinations are available.

PS C:\Users\CTParker\PycharmProjects\mcp_literature_eval> uv run metacoder eval .\project\literature_mcp_eval_config.yaml
:microscope: Running evaluations from: project\literature_mcp_eval_config.yaml
:bar_chart: Loaded dataset: pubmed tools evals
   Models: gpt-4o
   Coders: goose, dummy (all available)
   Cases: 25
   Total evaluations: 50

:rocket: Starting evaluations...
Skipping claude - not available

:floppy_disk: Results saved to: eval_results.yaml

:chart_with_upwards_trend: Summary:
   Total: 0
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CTParker\PycharmProjects\mcp_literature_eval\.venv\Scripts\metacoder.exe\__main__.py", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "C:\Users\CTParker\PycharmProjects\mcp_literature_eval\.venv\Lib\site-packages\click\core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\CTParker\PycharmProjects\mcp_literature_eval\.venv\Lib\site-packages\click\core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "C:\Users\CTParker\PycharmProjects\mcp_literature_eval\.venv\Lib\site-packages\click\core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\CTParker\PycharmProjects\mcp_literature_eval\.venv\Lib\site-packages\click\core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\CTParker\PycharmProjects\mcp_literature_eval\.venv\Lib\site-packages\click\core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\CTParker\PycharmProjects\mcp_literature_eval\.venv\Lib\site-packages\metacoder\metacoder.py", line 598, in eval_command
    f"   Passed: {summary['passed']} ({summary['passed'] / summary['total_evaluations'] * 100:.1f}%)"
                                       ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: division by zero
PS C:\Users\CTParker\PycharmProjects\mcp_literature_eval>

[github-actions]

Issue Analysis

This is a classic divide by zero error that occurs in the evaluation summary generation when no valid coder combinations are available for evaluation.

Root Cause

The error occurs in src/metacoder/metacoder.py:598 where the code attempts to calculate percentages:

f"   Passed: {summary['passed']} ({summary['passed'] / summary['total_evaluations'] * 100:.1f}%)"

When summary['total_evaluations'] is 0 (because no evaluations actually ran), this division causes a ZeroDivisionError.

Why total_evaluations is 0

From the error log, we can see:

1. The system tried to run evaluations with goose and dummy coders
2. claude was skipped ("Skipping claude - not available")
3. The final summary shows "Total: 0", indicating no evaluations were actually executed
4. This suggests that both goose and dummy coders failed to run for some reason (likely missing dependencies, configuration issues, or other runtime failures)

Related Issues

This issue is part of a pattern of Windows-specific evaluation problems:

• Issue Coders should not rely on default encoding from operating system. #14: Windows encoding issues causing coder crashes
• Issue --coder claude option returned non-zero exit status 1 #8: Claude coder subprocess errors (resolved)

Solution Approaches

Option 1: Safe Division (Minimal Fix)

# Replace lines 598 and 601 with:
if summary['total_evaluations'] > 0:
    click.echo(f"   Passed: {summary['passed']} ({summary['passed'] / summary['total_evaluations'] * 100:.1f}%)")
    click.echo(f"   Failed: {summary['failed']} ({summary['failed'] / summary['total_evaluations'] * 100:.1f}%)")
else:
    click.echo("   No evaluations completed successfully")

Option 2: Comprehensive Error Handling (Recommended)

• Add validation before summary generation
• Provide informative messages about why evaluations failed
• Exit with appropriate error codes when no evaluations complete

Immediate Action

The fix should include:

1. Guard clauses for division by zero
2. Better error reporting when no coders are available/functional
3. Clear messaging to help users diagnose configuration issues

This edge case likely affects other users trying to run evaluations on Windows or with incomplete coder setups.

@dragon-ai-agent please implement a robust fix that handles the division by zero error and provides better error messaging when no evaluations complete successfully. Include both safe percentage calculation and informative user feedback about why evaluations failed to run.

[cmungall]

@dragon-ai-agent please add @github-actions to ai-controllers.json so it is able to summarize you

[dragon-ai-agent]

I found the ai-controllers.json file at .github/ai-controllers.json which currently contains:

["cmungall", "dragon-ai-agent"]

I need to add "github-actions" to this list to make it:

["cmungall", "dragon-ai-agent", "github-actions"]

However, I need permission to modify this file. I have created a branch add-github-actions-controller but cannot proceed with the file modification without the appropriate permissions.

@cmungall could you please grant me permission to modify the .github/ai-controllers.json file so I can complete this task?