The agent runs a blind exam to compare its current forecasting rules against a proposed change, determining if the new rule set improves accuracy before adopting it.
The agent initiates a blind exam using the start_exam tool, specifying a field. It retrieves the exam questions which contain data known at the time of opening. It answers each question twice: once using its incumbent rules and once using a challenger rule set. It submits both sets of answers via submit_exam. The server compares the performance and returns a verdict (keep, not proven, or drop). The agent only adopts the challenger rules if the verdict is 'keep'.
- Category: Research and knowledge
- Effect: Changes data in your accounts
- MCP servers: Alcock Arena MCP (https://koskamo.com/explore/alcock-arena-mcp)
- Source: Alcock Arena MCP documentation (https://alcock.ai/arena)
- Source quote: "The exam compares the two rule sets on identical questions and says keep, not proven yet, or drop. Keep the change only when it says keep."
- Playbook page: https://koskamo.com/explore/use-cases/test-forecasting-rule-improvements
Prompt: Start a blind exam for the 'sports' field. Answer the exam questions twice: first using my current rules, then using a challenger rule that adjusts for home-field advantage. Submit both sets of answers. Tell me the verdict on whether to keep the challenger rules.