Compare model responses

Compare is an AI Playground feature for exploring how two models respond to the same prompt. EidoStack sends the request to both selected models and displays their streamed and completed answers side by side. You can use it for a quick comparison or a controlled test; for model-only comparisons, keep the instructions and context the same.

Plan limits: Free accounts can compare models in each of up to five chats, using models from one configured provider. Pro accounts can compare models across their configured providers without chat or message limits.

Set up a fair comparison

Before enabling comparison, unlock the vault and save the required provider key or keys. Free accounts compare two models from their one configured provider; Pro accounts can compare models from different configured providers as long as each selected model has usable provider access.

1. Open or create the chat that contains the task you want to test.
2. Select Compare in the composer toolbar.
3. Select Model A and Model B using the two model controls.

4. Set the instruction for each side with its Sys Prompt control. Use the same instruction for an apples-to-apples model test, or intentionally vary the instructions when instruction design is the variable under test.

5. Review the shared Context Memory setting, then enter and send the prompt.

The context strategy applies to the chat, so both sides use the same selected context policy. This is usually the right baseline for a model choice. If you need to test two different context policies, run separate comparisons rather than treating them as a single model-only result.

Select None to send only the current prompt. It also omits configured system prompts, conversation history, and retrieved memory.

Read the result side by side

EidoStack starts one stream for each model. The responses appear in parallel as they arrive, and completed paired answers stay together in the conversation. Explore differences in their explanations, formatting, completeness, instruction following, and handling of uncertainty or refusal. For a structured evaluation, compare each response against the same rubric.

One side can fail while the other succeeds. Read the displayed error and use Try again only after checking the affected provider key, provider quota, and model access. Retrying is a new request, so do not assume it is a no-cost operation.

Check token and cost consequences

Open the usage button in the prompt toolbar after the responses complete. The chat summary combines the messages in the chat and can show a breakdown by model, including input tokens, output tokens, total tokens, and estimated cost. Comparison mode naturally sends two model requests, so the total will usually be higher than a single-model run.

For an analysis over multiple comparisons, open Usage Metrics from the workspace header and use its Per Model and Per Chat views. Usage, tokens, and costs explains the available date filters and reports.

Keep comparison evidence useful

Run a small, representative prompt set instead of choosing a winner from one response. Name or pin the chat that contains the result, and keep edge cases in the same evaluation folder so later reviewers can see the prompts, instructions, model choices, and cost context together.

Comparison mode makes differences visible; it does not automatically score models or declare a winner. Use it to explore possible approaches, and follow Evaluate a model when you need explicit criteria for a model-selection decision.