AI Chatbot Quality Analyst

About The Role

Conversational AI is only as good as the people who evaluate it. We're looking for sharp, analytical thinkers to join Alignerr as AI Chatbot Quality Analysts --- reviewing real AI dialogue, identifying where conversations break down, and helping build smarter, more natural AI systems.

Your insights will directly influence how the next generation of conversational AI products think, respond, and hold context across complex, multi-turn interactions.

• Organization: Alignerr
• Type: Hourly Contract
• Location: Remote
• Commitment: 10--40 hours/week

What You'll Do

• Conduct structured evaluations of AI chatbot conversations across a wide range of topics and use cases
• Assess dialogue quality for context retention, coherence, natural flow, and state management
• Identify where and how conversations degrade across extended multi-turn interactions
• Design and execute test scenarios that stress-test conversational AI capabilities
• Document failure patterns, edge cases, and quality issues using structured feedback frameworks
• Rate chatbot responses for accuracy, helpfulness, safety, and tone consistency
• Provide actionable recommendations that improve dialogue system performance
• Collaborate asynchronously with research and product teams to refine evaluation criteria

Who You Are

• 2+ years of experience in conversational AI evaluation, chatbot QA, dialogue system testing, or UX research for conversational products
• Strong understanding of how context, state, and coherence degrade across extended conversations
• Analytical mindset with a sharp eye for subtle quality issues in natural language interactions
• Excellent written communication skills --- you can clearly articulate what went wrong and why
• Detail-oriented and systematic in your approach to testing
• Comfortable working independently and managing your own schedule
• Familiar with conversation design principles and common chatbot failure modes

Nice to Have

• Experience with NLP, LLMs, or machine learning concepts
• Background in linguistics, cognitive science, or human-computer interaction
• Familiarity with evaluation frameworks like BLEU scores, human preference ratings, or custom rubrics
• Experience writing test plans or QA documentation for conversational products
• Prior work with prompt engineering or red-teaming AI systems

Why Join Us

• Work on cutting-edge AI projects shaping the future of conversational technology
• Fully remote and flexible --- set your own schedule and work asynchronously
• Collaborate with top AI research labs and industry leaders
• Autonomy and independence in how you approach your work
• Build deep, marketable expertise in one of the fastest-growing areas of AI
• Potential for ongoing work and contract extension