About the role
Description
We’re looking for a musically literate specialist to help evaluate the quality of AI models working with music notation and audio. You’ll contribute to QA and data labeling across OMR, Audio2Score, and Autograde, helping improve our models and build high-quality golden datasets. Strong musical understanding, attention to detail, and the ability to evaluate musical outputs in a structured way are essential.
This role is offered on a contract basis, with the number of working hours to be discussed and agreed upon.
Key responsibilities
* Expert-score every OMR and A2S release each sprint (biweekly QA: 5-20 scores for OMR, 5-10 for A2S) * Run deeper quarterly QA cycles with AB-tests (100-300 scores for OMR, ~100 for A2S) * Run quarterly competitor benchmarking against N competitors on the golden dataset (first cycle planned October 2026) * Provide fast sanity checks on model candidates between releases (5-10 scores/week for OMR when updates are available; 1-2 scores, 2-3 times/month for A2S) * Spot-check outputs against MSCZ ground truth and flag dataset/benchmark inconsistencies * Build and label the Autograde golden test set: record specific mistake types and annotate each recording against the score (what should have been played, what error occurred) — needed before the Oct 31, 2026 MatchMySound cutover * Run head-to-head comparison of Autograde vs MatchMySound on the same golden set * Extend the golden dataset by transcribing audio where possible * Extend the same labeling/QA process to adjacent projects (AutoSync, Audio2Chords) as they scale
Requirements
* Strong musical literacy: able to read and evaluate music notation, scores, and audio performances * Experience judging OMR (optical music recognition) and audio-to-score transcription output for accuracy * Familiarity with structured/rubric-based QA scoring methodology * Comfortable evaluating a range of score types (multilingual lyrics, orchestral vs. ensemble vs. solo, tabs, chord notation) * Attention to detail for spot-checking datasets against ground truth * Comfortable giving structured written feedback usable by engineering/product teams
Originally posted on Himalayas