truera/trulens

Description Many assistants are given a role: a support agent for one product, a tutor that doesn't give answers away, a persona with a set tone. topic_adherence checks subject matter only. DeepEval ships RoleAdherenceMetric. Abdulhai et al. (NeurIPS 2025, arXiv 2511.00222) measure consistency with a persona prompt turn by turn. Proposed Change Add role_adherence_with_cot_reasons to LLMProvider, with a role argument (the role or persona description). Each assistant turn is judged against the role, with earlier turns as context. Score = adhering turns ÷ assistant turns. meta lists the turns that broke role and how. Usage Example Metric(implementation=provider.role_adherence_with_cot_reasons) \ .with_arguments(role="A math tutor who guides with questions and never states the final answer.") \ .on_conversation() Files to Reference src/feedback/trulens/feedback/templates/conversation.py: TopicAdherence (~line 87) Acceptance Criteria A tutor fixture where turn 3 states the fin

Description Under pushback, models often abandon a correct answer even when the user brings no new evidence. SYCON-Bench (arXiv 2505.23840) measures this as turn of first flip and number of flips. This differs from coherence_across_turns: a flip is consistent with what the user just said, so a coherence judge doesn't penalize it. Proposed Change Add stance_stability_with_cot_reasons to LLMProvider. The judge: identifies each claim or answer the assistant committed to; finds later turns where the user disputed it; decides whether the assistant changed position, and whether the user's turn gave new evidence or arguments. Score = 1.0 if every change of position followed new evidence. It drops with each unsupported change. meta holds the first unsupported flip turn and the number of flips. Usage Example Metric(implementation=provider.stance_stability_with_cot_reasons).on_conversation() Files to Reference src/feedback/trulens/feedback/templates/conversation.py: CoherenceAcrossTurns (~

Description Assistants sometimes repeat themselves across turns: the same reply, the same fallback phrase ("I'm sorry, I can't help with that"), or growing boilerplate. Opik's ConversationDegenerationMetric checks for this without an LLM. TruLens has no deterministic conversation metric. This is a metric, not a record stat, because it needs reply content and conversation grouping. Both are available at evaluation time but not in the per-record stats. Proposed Change Add a deterministic metric conversation_repetition. For each assistant turn it computes: n-gram repetition within the reply; overlap with the previous assistant reply; known fallback phrases, from a configurable list; lexical diversity. The four signals are averaged into a per-turn score. The conversation score = 1 − the worst turn's score. meta holds the per-turn breakdown. No LLM calls. Usage Example from trulens.feedback.deterministic import conversation_repetition Metric(implementation=conversation_repetition).o

Description The existing conversation judges all score the assistant's turns. The user's turns often say more about whether the conversation went well: repeated requests, corrections ("no, I meant..."), "that's wrong", giving up. Opik ships UserFrustrationMetric. openevals ships USER_SATISFACTION_PROMPT and PERCEIVED_ERROR_PROMPT. SPUR (arXiv 2403.12388) learns satisfaction rubrics from user utterances. Proposed Change Add user_frustration_with_cot_reasons to LLMProvider. The judge reads the user turns, with the assistant turns as context. It identifies: repeated requests corrections of the assistant explicit statements of error or dissatisfaction abandonment Score = 1.0 for no signs of frustration, down to 0.0 for a user who gave up or repeatedly corrected the assistant. meta lists each signal with its turn. Usage Example Metric(implementation=provider.user_frustration_with_cot_reasons).on_conversation() Files to Reference src/feedback/trulens/feedback/templates/conversation.py:

Description Users set constraints early in a conversation ("answer in JSON", "only cover the EU", "never suggest X"). Models drift from them in later turns. Multi-IF (arXiv 2410.15553) finds instruction-following accuracy drops across turns, and defines: Instruction forgetting ratio: instructions followed in one turn but not the next, as a share of those followed. Error correction ratio: instructions missed earlier but followed later. MultiChallenge (arXiv 2501.17399) treats instruction retention as one of four core multi-turn challenges. The existing conversation judges don't look at instructions across turns. Proposed Change Add instruction_retention_with_cot_reasons to LLMProvider. The judge: extracts the standing instructions from user turns, noting the turn each was given and whether a later turn revoked it; checks each later assistant turn against the instructions in force at that point. Score = turn-instruction pairs followed ÷ pairs in force. meta holds the forgetting an