#llm-evaluation (30 Repositories)
Ranked open-source repositories tagged with #llm-evaluation, scored by pull request acceptance likelihood and maintainer engagement velocity.
57.1%
61.1h
30 repositories tagged #llm-evaluation
Marker-Inc-Korea/AutoRAG
AutoRAG: Now your agent can find anything in your computer. It gets smarter if you are using it frequently.
comet-ml/opik
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
lmnr-ai/lmnr
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
truera/trulens
Evaluation and Tracking for LLM Experiments and AI Agents
future-agi/future-agi
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
PhiloLabs/agentic-vbench
AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?
EvolvingLMMs-Lab/lmms-eval
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
juanjuandog/FinSight-AI
AI equity research agent with resilient workflows, evidence-grounded RAG, versioned reports, and automated quality evaluation.
Continuum-AI-Corp/OrcaReplay
OrcaReplay — Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.
Giskard-AI/giskard-oss
🐢 Open-Source Evaluation & Testing library for LLM Agents
EverMind-AI/SkillCorpus
Open-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corpora—with retrieval and evaluation tooling included.
JasonColapietro/suede-creator-skills
Open-source AI skills for SEO, AI search visibility, conversion copy, marketing strategy, and business operations. Reusable workflows for Claude Code and Codex, plus code review, app delivery, and creator tools.
microsoft/prompty
Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.
huggingface/aisheets
Build, enrich, and transform datasets using AI models with no code
alphadl/AdaRubrics
AdaRubric: Adaptive Dynamic Rubric Evaluator for Agent Trajectories
msoedov/agentic_security
Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪
ktwu01/benchmark-radar
Track 20,710+ AI benchmark, eval, dataset, and data-quality records from 39 public sources, with linked evidence and daily updates.
Jwuthri/Tracely
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
scaleapi/agentenv-framework
Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.
confident-ai/deepeval
The LLM Evaluation Framework
Jwuthri/Tracely-ai
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
ValueByte-AI/Awesome-LLM-in-Social-Science
Awesome papers involving LLMs in Social Science.
OpenBMB/UltraEval-Audio
Your faithful, impartial partner for audio evaluation — know yourself, know your rivals. 真实评测，知己知彼。A unified benchmark framework for ASR/TTS/Audio Codec/audio LLM evaluation
weavebench/WeaveBench
[EMNLP 2026] WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
darkrishabh/agent-skills-eval
A test runner for agentskills.io-style AI agent skills
superwesleyhys-ux/factcircuit
FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
ukanwat/overtime
Give a coding agent a brief, not a chat, and it works on its own across sessions. Includes an example run: an open-world city built in a real game engine with no human help. In early development.
ukanwat/selfstarter
Give a coding agent a brief, not a chat, and it works on its own across sessions. Includes an example run: an open-world city built in a real game engine with no human help. In early development.
overmind-core/overmind
The platform for continuously improving AI agents.
suncityldp/zx-bench
A reproducible local LLM benchmarking platform with versioned datasets, deterministic scoring, and auditable reports.