# Calibrate

> An interactive quiz app that tests human reasoning against real AI benchmark datasets. Answers are validated server-side and never exposed in the browser before submission.

Live at: https://h3manth.com/fun/calibrate/

## What it does

Calibrate lets you quiz yourself on five peer-reviewed AI benchmark datasets used to evaluate large language models. After each answer, you see how GPT-4, Claude 3, Llama 3 70B, and GPT-3.5 score on the same benchmark — enabling direct human-vs-AI comparison.

## Links

- [Calibrate App](https://h3manth.com/fun/calibrate/): The live quiz application
- [GitHub](https://github.com/hemanth/Calibrate): Source code
- [CommonsenseQA Dataset](https://huggingface.co/datasets/tau/commonsense_qa): Everyday world knowledge benchmark
- [GSM8K Dataset](https://huggingface.co/datasets/openai/gsm8k): Grade school math reasoning benchmark
- [ARC Challenge Dataset](https://huggingface.co/datasets/allenai/ai2_arc): Hard science questions by AllenAI
- [TruthfulQA Dataset](https://huggingface.co/datasets/truthful_qa): Misconceptions and truthfulness benchmark
- [HellaSwag Dataset](https://huggingface.co/datasets/Rowan/hellaswag): Physical common sense completion benchmark

## Features

- Quiz yourself on 5 peer-reviewed AI benchmark datasets
- Multiple-choice (CommonsenseQA, ARC, TruthfulQA, HellaSwag) and open-ended (GSM8K) question types
- Server-side answer validation — correct answers never appear in browser DevTools before submission
- Real-time AI model performance comparisons shown after each answer
- Questions fetched live from Hugging Face Datasets Server API with random offset each round
- Custom datasets: any public Hugging Face dataset or paste your own JSON Q&A pairs
- Keyboard shortcuts: A/B/C/D to select choices, Enter to advance
- Fully accessible — WCAG AA compliant, screen reader friendly
- Mobile responsive, no framework, no build step

## API

```
# Get questions (answers NOT included in response)
GET /fun/calibrate/api/questions/:dataset?count=10
→ { questions: [{ id, question, choices, category, type }], baselines: [...] }

# Validate an answer (server-side)
POST /fun/calibrate/api/check
{ questionId: "uuid", answer: "B" }
→ { correct: true, correctAnswer: "B", explanation: "..." }

# Custom Hugging Face dataset
GET /fun/calibrate/api/custom-hf?dataset=rajpurkar/squad&split=train&count=10

# Custom JSON Q&A
POST /fun/calibrate/api/custom-json
{ questions: [{ question, choices, answer, explanation }] }
```

## Datasets

- **CommonsenseQA** — 1,221 multiple-choice questions testing everyday world knowledge
- **GSM8K** — 8,500 grade school math problems requiring multi-step reasoning
- **ARC Challenge** — 1,172 hard science questions designed to defeat pattern-matching
- **TruthfulQA** — 817 questions designed to surface human misconceptions
- **HellaSwag** — 10,042 physical common sense sentence completion tasks

## AI baselines shown in-app

GPT-4, Claude 3, Llama 3 70B, GPT-3.5 — accuracy scores per benchmark displayed after each answer.

## Tech stack

- Node.js + Express (server-side answer store with 2h TTL per session)
- Vanilla HTML/CSS/JavaScript — no framework, no build step
- Hugging Face Datasets Server API for live question loading
- OKLCH color system with WCAG AA contrast compliance
- Google Fonts: Sora (display) + JetBrains Mono (code/data)

## Author

Hemanth HM — https://h3manth.com