About the role
Job Title: Data Science & Analytic Methodology SME
Role Type: Contractor Location: Remote
Job Summary
We are seeking experienced Data Science & Analytic Methodology Subject Matter Experts to contribute to a high-impact project focused on advanced analytical pipelines and large language models (LLMs). In this role, you will apply your domain expertise to evaluate AI-generated outputs, assess analytical workflows, and help improve the accuracy, reliability, and performance of next-generation AI systems. Prior experience in AI is not required, although relevant technical experience is highly valued.
Key Responsibilities
1. Map large-scale analytical and pipeline integration scenarios to established taxonomies across key evaluation areas. 2. Develop detailed analytical cases reflecting real-world deployment patterns, including automated triage, data enrichment, analysis at scale, and common operational failure modes. 3. Evaluate and grade responses based on source handling, reasoning quality, understanding of limitations, and practical utility. 4. Identify and flag instances of model over-refusal that may disrupt legitimate analytical workflows or pipeline continuity. 5. Document complex edge cases, emerging methodological challenges, and opportunities for improvement, providing actionable recommendations. 6. Deliver clear, insightful written and verbal feedback to support iterative development and high-quality project outcomes.
Preferred Qualifications
1. Demonstrated experience integrating large language models (LLMs), including government-specific GPT variants, into complex data or analytical pipelines. 2. Proven experience conducting analysis at scale, with a strong understanding of automation failure modes and relevant evaluation metrics. 3. Experience in teaching, mentoring, or providing technical leadership in data science or related analytical disciplines. 4. Strong technical writing skills, with the ability to document methodologies clearly and communicate complex concepts in English. 5. Familiarity with designing evaluation frameworks, benchmarks, or assessment systems. 6. Experience in defense, intelligence, or government analytical environments is highly valued. 7. Ability to work effectively in distributed, international project environments with minimal clearance-related constraints.
Ideal Candidate Profile
The ideal candidate combines strong data science expertise with practical experience in analytical methodology, automation, and LLM integration. They can assess complex workflows, identify limitations and failure patterns, evaluate model performance, and provide actionable feedback to improve the reliability and effectiveness of AI-driven analytical systems.
Originally posted on Himalayas