A scalable system provides automated conversation review that can identify potential miscommunications. The system may provide suggested actions to fix errors in intelligent virtual assistant (IVA) understanding, may prioritize areas of language model repair, and may automate the review of conversations. By the use of an automated system for conversation review, problematic interactions can be surfaced without exposing the entire set of conversation logs to human reviewers, thereby minimizing privacy invasion. A scalable system processes conversations and autonomously marks the interactions where the IVA is misunderstanding the user.