About The Company
YO IT Consulting is a forward-thinking technology firm dedicated to advancing artificial intelligence and machine learning solutions. With a focus on innovation and excellence, the company collaborates with leading AI teams worldwide to develop, evaluate, and refine cutting-edge conversational AI systems. YO IT Consulting prides itself on fostering a dynamic and inclusive work environment that encourages continuous learning and professional growth. Our mission is to harness the power of AI to solve complex problems and deliver impactful solutions across various industries, ensuring our clients stay ahead in the digital age.
About The Role
We are seeking a highly skilled and detail-oriented AI Evaluation Specialist to join our team as an independent contractor. This remote, full-time or part-time contract position offers an exciting opportunity to contribute to the development and enhancement of general-purpose conversational AI systems, specifically focusing on their reasoning about code, programming solutions, and technical explanations. The successful candidate will play a critical role in assessing AI responses, ensuring accuracy, clarity, and reliability in software engineering contexts. This role requires a deep understanding of programming, AI evaluation methodologies, and a passion for improving AI capabilities through rigorous analysis and feedback.
Qualifications
- Bachelor's, Master's, or PhD degree in Computer Science or a closely related field.
- Significant professional experience in software engineering or related technical roles.
- Expertise in at least one programming language such as Python, Java, C++, JavaScript, Go, or Rust.
- Proven ability to independently solve complex coding challenges on platforms like HackerRank or LeetCode at medium and hard levels.
- Experience contributing to well-known open-source projects, including merging pull requests and participating in code reviews.
- Hands-on experience working with large language models (LLMs), understanding their strengths and limitations in coding contexts.
- Strong attention to detail, with the ability to evaluate complex technical reasoning, identify subtle bugs, and logical flaws.
- Excellent communication skills in English, with the ability to articulate technical feedback clearly and effectively.
Responsibilities
- Evaluate responses generated by large language models to coding and software engineering queries, focusing on accuracy, reasoning, clarity, and completeness.
- Conduct fact-checking of AI responses using trusted public sources and authoritative references to ensure factual correctness.
- Perform accuracy testing by executing code snippets, validating outputs, and using appropriate tools to verify correctness.
- Annotate AI-generated responses by identifying strengths, weaknesses, and areas for improvement, including factual or conceptual inaccuracies.
- Assess the quality, readability, and algorithmic soundness of code, as well as the clarity of technical explanations provided by AI systems.
- Ensure responses adhere to expected conversational behavior and system guidelines, maintaining consistency across evaluations.
- Follow established evaluation standards, taxonomies, and benchmarks to maintain consistency and objectivity in assessments.
- Provide detailed feedback that contributes to the refinement of AI models, enhancing their robustness, correctness, and user trustworthiness.
Benefits
As an independent contractor with YO IT Consulting, you will enjoy the flexibility of remote work, allowing you to manage your schedule either full-time or part-time. You will have the opportunity to work on innovative projects that push the boundaries of AI technology, contributing to meaningful advancements in conversational AI systems. The role offers competitive compensation aligned with your expertise and experience. You will also gain exposure to cutting-edge AI evaluation methodologies and collaborate with a talented global team dedicated to excellence in AI development. This position provides an excellent platform for professional growth, skill enhancement, and establishing yourself as a key contributor in the AI community.
Equal Opportunity
YO IT Consulting is committed to creating an inclusive environment that values diversity and equal opportunity. We do not discriminate based on race, ethnicity, gender, age, sexual orientation, disability, or any other protected characteristic. We believe that diverse perspectives foster innovation and excellence, and we are dedicated to providing a supportive and respectful workplace for all contractors and employees.