mmlu

A large-scale 7B pretraining language model developed by BaiChuan-Inc.

A series of large language models developed by Baichuan Intelligent Technology

A 13B large language model developed by Baichuan Intelligent Technology

A Contamination-free Multi-task Language Understanding Benchmark [Official, ACL 2025]

Dual-engine (llama.cpp + vLLM) LLM benchmarking pipeline for GGUF & safetensors on NVIDIA GPUs — speed, quality, live dashboard, publishable cards.

[NeurIPS 2023 Spotlight] In-Context Impersonation Reveals Large Language Models' Strengths and Biases

Production-grade LLM Evaluation & Benchmarking Framework - GPT-4, Claude, Gemini, Mistral. Accuracy, latency, cost, hallucination, reasoning metrics.

[NeurIPS 2025] AGI-Elo: How Far Are We From Mastering A Task?

A pip-installable benchmark runner for LLMs and agents. Five minutes to your first eval.

A Massive Multitask Benchmark for Urdu Language Understanding

Pre-registered non-inferiority study: where is a quantized 1.7B model on a laptop CPU good enough to replace a hosted 70B? On this benchmark, nowhere, and the local path was also slower. Margin fixed before data collection.

Benchmark suite for open-source language models on the edge. Evaluates inference efficiency, MMLU accuracy, and LLM-rated teaching effectiveness.

Enterprise-grade LLM evaluation framework | Multi-model benchmarking, honest dashboards, system profiling | Academic metrics: MMLU, TruthfulQA, HellaSwag | Zero fake data | PyPI: llm-benchmark-toolkit | Blog: https://dev.to/nahuelgiudizi/building-an-honest-llm-evaluation-framework-from-fake-metrics-to-real-benchmarks-2b90

Code and data accompanying the article "The impact of quantising a small open source LLM". This repository explores how quantisation affects performance, VRAM usage, and inference speed in Qwen3 1.7B.

CLI tool to evaluate LLM factuality on MMLU benchmark.

ChoiceBench is a lightweight framework for MCQ evaluation-method research on LLMs, with built-in support for answer-order bias analysis and mitigation methods.

An easy-to-use and standardised framework for evaluating Large Language Models (LLMs) on the Massive Multitask Language Understanding (MMLU) dataset. Currently supported: Hugging Face transformer models and Bedrock models.

Proper scoring rules, reduces LLM overconfidence in multiple-choice QA.

Two-stage CPT + SFT domain adaptation pipeline (QLoRA/Unsloth) with an evaluation harness measuring catastrophic forgetting via MMLU/GSM8K retention vs. domain-accuracy gains. FastAPI serving + CI/CD.

Full-stack app comparing large language models on published benchmarks (MMLU, HumanEval, GSM8K) and pricing: relational data model, REST API with filtering and sorting, responsive React dashboard.