#evaluation (30 Repositories)
Ranked open-source repositories tagged with #evaluation, scored by pull request acceptance likelihood and maintainer engagement velocity.
67.7%
66.0h
30 repositories tagged #evaluation
Marker-Inc-Korea/AutoRAG
AutoRAG: Now your agent can find anything in your computer. It gets smarter if you are using it frequently.
AgentX-ai/AgentX-Trace-Eval
The trust layer for your AI agent on any platform. Trace and evaluate your agent. Full Observability. Run Model portability analysis. Autotune your agent.
langchain-ai/langsmith-sdk
LangSmith Client SDK Implementations
mlflow/mlflow
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
modelscope/evalscope
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
NVIDIA-NeMo/Gym
Evaluate and improve models and agents using environments
langfuse/skills
Agent Skills for Langfuse, the open source LLM engineering platform for tracing, prompt management, and evaluation
INGInious/INGInious
INGInious is a secure and automated exercises assessment platform using your own tests, also providing a pluggable interface with your existing LMS.
langwatch/langwatch
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
EvolvingLMMs-Lab/lmms-eval
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
NVIDIA/SkillEvaluator
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
CodeSoul-co/Hypha
Harness-oriented agent system framework for production-grade LLM agent applications
promptfoo/promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
TIGER-AI-Lab/ClawBench
Open-source benchmark for browser AI agents on daily tasks.
Cloud-CV/EvalAI
:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
ncalc/ncalc
NCalc is a fast and lightweight expression evaluator library for .NET, designed for flexibility and high performance. It supports a wide range of mathematical and logical operations.
IBM/unitxt
🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking
strands-agents/evals
A comprehensive evaluation framework for AI agents and LLM applications.
trpc-group/trpc-agent-go
A Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.
FeiZhuNiU-INFJA/LIFT
An Evaluation Framework for Self-Evolving-Agent (Loaded Impact on Holdout Final Task)
coze-dev/coze-loop
Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.
ktwu01/benchmark-radar
Track 20,710+ AI benchmark, eval, dataset, and data-quality records from 39 public sources, with linked evidence and daily updates.
open-compass/opencompass
OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
Jwuthri/Tracely
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
Jwuthri/Tracely-ai
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
huggingface/lighteval
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
comet-ml/opik-openclaw
🦞 Official plugin for OpenClaw that exports agent traces to Opik. See and monitor agent behaviour, cost, tokens, errors and more.
alipay/ant-application-security-testing-benchmark
xAST评价体系，让安全工具不再“黑盒”. The xAST evaluation benchmark makes security tools no longer a "black box".
NOAA-OWP/inundation-mapping
Flood inundation mapping and evaluation software configured to work with U.S. National Water Model.
OpenBMB/UltraEval-Audio
Your faithful, impartial partner for audio evaluation — know yourself, know your rivals. 真实评测，知己知彼。A unified benchmark framework for ASR/TTS/Audio Codec/audio LLM evaluation