#multimodal (30 Repositories)

ysr666/dsh-vision-router

Eyes for text-only DeepSeek Harness agents: built-in free vision chain (no key) + pixel-level vision tools (Q&A, grounding, crop, pixel diff, colors, OCR, SVG trace, cutout, screenshots). One-command install, no Python, image turns work like ordinary tool-calling turns.

verl-project/verl-omni

Multimodal RL training framework for diffusion & omni models

vortex-data/vortex

An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.

antflydb/antfly

vllm-project/vllm-omni

A framework for efficient model inference with omni-modality models

sgl-project/sglang-omni

SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.

Code-Amadeus/Amadeus

Real-time multimodal desktop agent evolving toward a persistent AI OS interface (0.16 α).

pixeltable/pixeltable

The backend agents build with - Multimodal database, orchestration, and serving in one file

RunanywhereAI/runanywhere-sdks

Production ready toolkit to run AI locally

NVIDIA-NeMo/DataDesigner

🎨 NeMo Data Designer: Generate high-quality synthetic data from scratch or from seed data.

clawdotnet/openclaw.net

Self-hosted Enterprise and Personal AI + agent runtime in .NET (NativeAOT-friendly)

screenpipe/screenpipe

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

modelscope/ms-swift

Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).

EvolvingLMMs-Lab/lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

Eventual-Inc/Daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

InternLM/xtuner

A Next-Generation Training Engine Built for Ultra-Large MoE Models

however-yir/knowledgeops-agent

Production-oriented Spring AI platform prototype for RAG, tool calling, async ingestion, JWT/RBAC, and observability.

redai-studio/Relax

An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

rerun-io/rerun

Visualize, query, and stream to train on multimodal robotics data.

vinhnx/VT.ai

VT.ai - multimodal AI chat app with dynamic conversation routing

genkit-ai/genkit

Open-source framework for building agentic apps in JavaScript, Go, Dart, and Python, built and used in production by Google

TanStack/ai

🤖 Type-safe, provider-agnostic TypeScript AI SDK for streaming chat, tool calling, agents, and multimodal apps across OpenAI, Anthropic, Gemini, React, Vue, Svelte, and Solid.

Blaizzy/mlx-audio

A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.

datachain-ai/datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

Mintplex-Labs/anything-llm

Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience

volcengine/ark-cli

The fastest way to put Volcengine Ark in your terminal and your AI agent — go from prompt to generated media, multimodal answer, or deployed endpoint in a single command, no API glue code.

HUANGCHIHHUNGLeo/claude-real-video

Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.

OpenMOSS/MOVA

A foundation model that generates synchronized video and audio in a single model

tong-io/tongflow

TongFlow — Multimodal GenAI Studio

cerul-ai/cerul

Open-source video processing core and CLI. Search and annotate local videos with your own model endpoints.