Read-only mirror of the MIREX Wiki, snapshot of 29 October 2026.
Open this page on the original wiki to edit it or to see newer changes.
2026:Audio Instrument Recognition Results
From MIREX Wiki
Contents
Submissions[edit]
| Submission | Title | Authors | |
|---|---|---|---|
| Baseline: OpenMIC-RF | OpenMIC-2018: An Open Dataset for Multiple Instrument Recognition | Link | |
| SSR | Full-Excerpt Instrument Recognition with Four LoRA-Adapted Audio Encoders | Shuangrun Shao |
Evaluation[edit]
Systems output a confidence score for each of the 20 OpenMIC-2018 instrument labels for every excerpt. Scores ≥ 0.5 are counted as positive predictions; missing (file, label) pairs count as 0. The primary metric is clip-level macro-F1 over the evaluated instrument classes; micro-F1 and mean average precision (mAP) are also reported. Higher is better for all metrics.
Test Sets[edit]
- OpenMIC-2018 test (public reference): the official
split01_testpartition of OpenMIC-2018 [1], 5,085 ten-second excerpts. Following the dataset's partial-label design, each class is evaluated only on excerpts where that class is annotated (Y_mask= 1), with ground truthY_true≥ 0.5. All 20 classes are evaluated. Audio was converted from the distributed Ogg files to 44.1 kHz 16-bit WAV (235 excerpts with other sample rates were resampled). - Hidden set: 316 ten-second excerpts from 115 multitrack recordings of MedleyDB 1.0 [2] (at most 3 non-adjacent excerpts per recording), mixture audio as 44.1 kHz 16-bit WAV. Labels were derived automatically from the official MedleyDB stem activation annotations: an instrument is present if a stem mapped to it is active (activation confidence ≥ 0.5) for at least 1 s of the excerpt and absent if never active; stems mapped ambiguously to a class (e.g. viola for violin, drum kits for cymbals, brass sections for trumpet/trombone/saxophone), and activity shorter than 1 s, make that label excluded from scoring. Labels were not manually verified. Every excerpt has on average 3.43 present instruments. Only the 12 classes with at least 20 positive excerpts are evaluated; organ and ukulele do not occur.
Results[edit]
Systems are ranked by macro-F1 on the hidden set. The OpenMIC-2018 results are reported separately as a public reference. Bold indicates the best value in each column (per class: in each row); the baseline is shown in italics.
Hidden set (MedleyDB)[edit]
| System | Macro-F1 | Micro-F1 | mAP | Predicted labels / clip |
|---|---|---|---|---|
| Baseline: OpenMIC-RF | 0.497 | 0.607 | 0.632 | 3.16 |
| SSR | 0.488 | 0.451 | 0.688 | 11.18 |
Ground-truth present labels per clip: 3.43.
| Instrument | Positives | Baseline: OpenMIC-RF | SSR |
|---|---|---|---|
| accordion† | 15 | 0.000 | 0.074 |
| banjo† | 6 | 0.087 | 0.077 |
| bass | 177 | 0.333 | 0.818 |
| cello | 28 | 0.627 | 0.258 |
| clarinet | 25 | 0.000 | 0.243 |
| cymbals† | 16 | 0.429 | 0.351 |
| drums | 155 | 0.774 | 0.789 |
| flute | 21 | 0.286 | 0.186 |
| guitar | 142 | 0.821 | 0.734 |
| mallet_percussion | 23 | 0.000 | 0.219 |
| mandolin† | 19 | 0.278 | 0.129 |
| organ† | 0 | — | — |
| piano | 139 | 0.600 | 0.687 |
| saxophone† | 15 | 0.682 | 0.189 |
| synthesizer | 73 | 0.420 | 0.529 |
| trombone† | 7 | 0.000 | 0.028 |
| trumpet | 22 | 0.604 | 0.224 |
| ukulele† | 0 | — | — |
| violin | 41 | 0.667 | 0.309 |
| voice | 160 | 0.834 | 0.856 |
† Fewer than 20 positive excerpts; reported for information, not included in the averages.
OpenMIC-2018 test (public reference)[edit]
| System | Macro-F1 | Micro-F1 | mAP |
|---|---|---|---|
| Baseline: OpenMIC-RF | 0.707 | 0.787 | 0.794 |
| SSR | 0.801 | 0.841 | 0.854 |
| Instrument | Positives | Baseline: OpenMIC-RF | SSR |
|---|---|---|---|
| accordion | 115 | 0.448 | 0.676 |
| banjo | 140 | 0.585 | 0.707 |
| bass | 134 | 0.620 | 0.721 |
| cello | 226 | 0.774 | 0.819 |
| clarinet | 137 | 0.146 | 0.471 |
| cymbals | 297 | 0.956 | 0.954 |
| drums | 278 | 0.932 | 0.932 |
| flute | 175 | 0.459 | 0.642 |
| guitar | 286 | 0.983 | 0.973 |
| mallet_percussion | 211 | 0.729 | 0.821 |
| mandolin | 193 | 0.595 | 0.681 |
| organ | 121 | 0.391 | 0.566 |
| piano | 285 | 0.957 | 0.963 |
| saxophone | 305 | 0.826 | 0.886 |
| synthesizer | 268 | 0.967 | 0.965 |
| trombone | 228 | 0.632 | 0.783 |
| trumpet | 318 | 0.688 | 0.844 |
| ukulele | 182 | 0.617 | 0.734 |
| violin | 394 | 0.883 | 0.915 |
| voice | 224 | 0.943 | 0.964 |
Notes[edit]
- No system was trained on MedleyDB. SSR was trained on the OpenMIC-2018 training partition (the public test partition was excluded) together with seven external datasets (see its PDF); the baseline was trained on the OpenMIC-2018 training partition only.
- Baseline: random forests trained exactly as in the official OpenMIC-2018 notebook on the distributed VGGish features. At test time, VGGish features are computed from the WAV input with peak normalisation as in the dataset's feature-extraction script; its OpenMIC-2018 test macro-F1 (0.707) matches that obtained with the distributed test features (0.705).
References[edit]
- E. J. Humphrey, S. Durand, and B. McFee, "OpenMIC-2018: An open dataset for multiple instrument recognition," in Proc. ISMIR, 2018, pp. 438–444.
- R. M. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. P. Bello, "MedleyDB: A multitrack dataset for annotation-intensive MIR research," in Proc. ISMIR, 2014, pp. 155–160.