Job description
About The Role
Conversational AI is only as good as the people who test it. We're looking for sharp, detail-oriented Chatbot Quality Analysts to evaluate AI-powered dialogue systems and help shape the next generation of conversational products.
You'll dig into real AI conversations, spot where things go wrong, and deliver structured feedback that directly improves how these systems think, respond, and hold context across extended interactions. This is meaningful, intellectually engaging work --- and you can do it entirely on your own schedule.
- Organization: Alignerr
- Type: Hourly Contract
- Location: Remote
- Commitment: 10--40 hours/week
What You'll Do
- Evaluate AI chatbot conversations across a wide range of topics and use cases
- Assess dialogue quality for context retention, coherence, natural flow, and state management
- Identify where and how conversations degrade over extended multi-turn interactions
- Design and run test scenarios that stress-test conversational AI capabilities
- Document failure patterns, edge cases, and quality issues using structured feedback frameworks
- Rate chatbot responses for accuracy, helpfulness, safety, and tone consistency
- Provide clear, actionable recommendations to improve dialogue system performance
- Collaborate asynchronously with research and product teams to refine evaluation criteria
Who You Are
- 2+ years of experience in conversational AI evaluation, chatbot QA, dialogue system testing, or UX research for conversational products
- Strong understanding of how context, state, and coherence degrade in extended conversations
- Analytical mindset with a sharp eye for subtle quality issues in natural language
- Excellent written communication skills in English --- you can clearly explain what went wrong and why
- Detail-oriented, systematic, and comfortable working independently
- Familiar with conversation design principles and common chatbot failure modes
Nice to Have
- Experience with NLP, LLMs, or machine learning concepts
- Background in linguistics, cognitive science, or human-computer interaction
- Familiarity with evaluation frameworks like BLEU scores, human preference ratings, or custom rubrics
- Experience writing QA documentation or test plans for conversational products
- Prior work with prompt engineering or red-teaming AI systems
- Proficiency in Portuguese
Why Join Us
- Work on cutting-edge AI projects with top research labs and industry leaders
- Fully remote and async --- set your own hours and work from anywhere
- Genuine autonomy in how you approach and structure your work
- Build deep, marketable expertise in one of the fastest-growing areas of AI
- Potential for ongoing work and contract extension