AI Dialogue Quality Analyst

About The Role

Conversational AI is one of the fastest-moving areas in tech --- and the quality of every chatbot interaction depends on people like you. We're looking for sharp, detail-oriented analysts to evaluate AI chatbot conversations and help shape how the next generation of conversational products thinks, responds, and communicates.

Your feedback will directly influence how AI systems handle real-world dialogue --- from simple questions to complex, multi-turn conversations.

• Organization: Alignerr
• Type: Hourly Contract
• Location: Remote
• Commitment: 10--40 hours/week

What You'll Do

• Evaluate AI chatbot conversations across a wide range of topics and use cases
• Assess dialogue quality for context retention, coherence, natural flow, and helpfulness
• Identify where and how conversations break down across extended multi-turn interactions
• Design test scenarios that push conversational AI to its limits
• Document failure patterns, edge cases, and quality issues using structured feedback frameworks
• Rate chatbot responses for accuracy, tone, safety, and overall helpfulness
• Provide clear, actionable recommendations for improving dialogue system performance
• Collaborate asynchronously with research and product teams to refine evaluation standards

Who You Are

• Experience in conversational AI evaluation, chatbot QA, dialogue system testing, or UX research for conversational products
• Strong understanding of how context and coherence can degrade across extended conversations
• Analytical mindset with a sharp eye for subtle quality issues in natural language
• Excellent written communication --- you can clearly explain what went wrong and why
• Detail-oriented and systematic in how you approach testing and evaluation
• Self-motivated and comfortable managing your own schedule independently
• Familiar with conversation design principles and common chatbot failure modes

Nice to Have

• Background in NLP, large language models, or machine learning concepts
• Experience in linguistics, cognitive science, or human-computer interaction
• Familiarity with evaluation frameworks such as human preference ratings or custom rubrics
• Experience writing test plans or QA documentation for conversational products
• Prior work with prompt engineering or red-teaming AI systems

Why Join Us

• Work on cutting-edge AI projects with top research labs and industry leaders
• Fully remote and flexible --- work on your own schedule, anywhere
• Freelance perks: autonomy, variety, and asynchronous global collaboration
• Build deep expertise in one of the fastest-growing areas of AI
• Meaningful work that directly shapes how AI communicates with people
• Potential for ongoing work and contract extension