fossasia/voxbento

Summary As VoxBento evolves, we are increasingly relying on running intensive Machine Learning models locally. While we currently run Speech-to-Text (faster-whisper) and Text-to-Text (ctranslate2), our future roadmap may include local Text-to-Speech (TTS), local LLMs, or audio/video analysis models. Currently, we run these models directly within the main web server process. This approach does not scale. It requires immense manual engineering to prevent models from crashing the server, leaves hardware underutilized, and makes adding new models incredibly difficult. This Epic tracks the creation of a Generalized ML Serving Architecture using Ray serve By migrating to Ray Serve, we establish a robust, scalable foundation for all future local ML models. We will decouple ML inference from the web server, enabling massive throughput, autoscaling, and out-of-the-box memory management for any model we deploy. The Problems : To understand why a generalized ML architecture is required, the eng

Summary portal/translations/worker.py _translate_and_broadcast is called once per target language per segment. Each invocation independently: Opens a new DB session for vocabulary lookup (line 152) Creates a new httpx.AsyncClient for the LLM call (line 218) Opens another DB session to save the translation (line 177) With 5 target languages, one transcription segment triggers 15 DB session opens and 5 httpx client creates. Evidence # Called N times via asyncio.gather (line 120) async def _translate_and_broadcast(self, ...): # Session 1: vocabulary lookup async with get_session() as vocab_session: # line 152 vocab_entries = await resolve_vocabulary_entries(...) # Session 2 (inside _call_llm): new httpx client async with httpx.AsyncClient(timeout=timeout) as client: # line 218 ... # Session 3: save translation async with get_session() as local_session: # line 177 ... Impact Connection pool pressure on both the database

Summary portal/websockets/manager.py is a 344-line god module that mixes two responsibilities: Connection management (3 classes: ConnectionManager, ListenerConnectionManager, TTSConnectionManager) — lines 25-116 Business logic handlers (8 functions: _handle_join, _handle_leave, _handle_chat, _handle_set_active, _handle_update_state, _handle_set_broadcast_unlocked, _handle_initiate_handoff, _handle_accept_handoff, _handle_cancel_handoff) — lines 137-343 The handlers do DB queries, auth checks, and business logic — this is not connection management. Current Module Layout portal/websockets/ __init__.py (empty) handlers.py (157 lines — route definitions only, imports handlers from manager.py) manager.py (344 lines — connection managers + all handler functions) Proposed Refactor portal/websockets/ __init__.py routes.py (rename handlers.py — WebSocket route definitions) manager.py (keep only connection manager classes + broadcast_t

Summary Nearly every template file uses inline style= attributes instead of CSS classes. This makes the styles impossible to override, theme, or maintain consistently. Key Offenders File Lines What is inline-styled templates/base.html 13-23 Footer (8+ properties), toast container (5 properties) templates/admin/base.html 33-45 Modal margins, flex layout templates/account.html 12, 50 Brand image height, badge background override templates/403.html 10-17 5 separate inline styles for layout templates/listener-event.html 157-187 Flex layouts, display, gaps, borders templates/login.html 14 Image height/margin templates/register.html 14 Image height/margin Impact Inline styles cannot be overridden by CSS classes without !important Breaks theming and dark mode support Ad-hoc styling instead of intentional design system Maintenance burden: every page has its own one-off styles Proposed Fix For each template: Identify inline styles Create or reuse a named CSS class in th

Summary home.html and listener_join.html load the Tailwind CSS CDN in production: <script src="https://cdn.tailwindcss.com"></script> This violates the project convention (AGENTS.md: "No framework. Frontend is plain ES modules") and introduces several problems: 330+ KB JIT compiler loaded and executed in the browser on every page load Flash of unstyled content (FOUC) while Tailwind compiles External CDN dependency — if cdn.tailwindcss.com goes down, pages are unstyled Tailwind docs explicitly say: "Do not use the CDN in production" Files Affected templates/home.html (764 lines, heavy Tailwind usage) templates/listener_join.html (35 lines, light Tailwind usage) Proposed Fix listener_join.html (quick win): Convert ~10 Tailwind utility classes to vanilla CSS classes Add them to an appropriate stylesheet home.html (larger effort): Extract the used Tailwind classes into a compiled/static CSS file Or rewrite the page using the project vanilla CSS approach Move inline JS to static/js