2013:Audio Cover Song Identification Results
Contents
Introduction[edit]
Audio Cover Song (ACS) Identification was evaluated against two datasets: Mixed Collection and Sapp's Mazurka Collection.
Mixed Collection Information[edit]
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.
Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.
Sapp's Mazurka Collection Information[edit]
In addition to our original ACS dataset, we used the Mazurka.org dataset put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.
General Legend[edit]
| Sub code | Submission name | Abstract | Contributors |
|---|---|---|---|
| YCW1 | MCY_COVER | Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang | |
| CYWW1 | CYC_COVER | Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang
|
Results[edit]
Mixed Collection[edit]
Summary Results[edit]
| CYWW1 | YCW1 | |
|---|---|---|
| Total number of covers identified in top 10 | 1714.00 | 925.00 |
| Mean number of covers identified in top 10 (average performance) | 1.71 | 0 |
| Mean (arithmetic) of Avg. Precisions | 0.52 | 0.28 |
| Mean rank of first correctly identified cover | 11.27 | 22.29 |
Number of Correct Covers at Rank X Returned in Top Ten[edit]
| Rank | CYWW1 | YCW1 |
|---|---|---|
| 1 | 229 | 187 |
| 2 | 222 | 161 |
| 3 | 209 | 141 |
| 4 | 204 | 124 |
| 5 | 188 | 103 |
| 6 | 168 | 70 |
| 7 | 149 | 52 |
| 8 | 151 | 44 |
| 9 | 116 | 22 |
| 10 | 78 | 21 |
| Total | 1714 | 925 |
Average Performance per Query Group[edit]
| Group | YCW1 | CYWW1 |
|---|---|---|
| 1 | 0.13 | 1 |
| 2 | 0.05 | 0.01 |
| 3 | 0.22 | 0.51 |
| 4 | 0.01 | 0.92 |
| 5 | 0.52 | 0.84 |
| 6 | 0.30 | 1 |
| 7 | 0.34 | 0.12 |
| 8 | 0.02 | 0.02 |
| 9 | 0.01 | 0.17 |
| 10 | 0.43 | 0.65 |
| 11 | 0.02 | 0.02 |
| 12 | 0.61 | 0.89 |
| 13 | 0.50 | 1 |
| 14 | 0.01 | 0.04 |
| 15 | 0.70 | 0.93 |
| 16 | 0.74 | 0.77 |
| 17 | 0.15 | 0.31 |
| 18 | 0.75 | 0.82 |
| 19 | 0.48 | 0.66 |
| 20 | 0.43 | 0.64 |
| 21 | 0.04 | 0.12 |
| 22 | 0.12 | 0.15 |
| 23 | 0.01 | 0.13 |
| 24 | 0.53 | 0.88 |
| 25 | 0.06 | 0.68 |
| 26 | 0.12 | 0.63 |
| 27 | 0.67 | 0.90 |
| 28 | 0.01 | 0.06 |
| 29 | 0.59 | 1 |
| 30 | 0.80 | 0.91 |
Sapp's Mazuraka Collection[edit]
Summary results[edit]
| CYWW1 | YCW1 | |
|---|---|---|
| Total number of covers identified in top 10 | 5129.00 | 1913.00 |
| Mean number of covers identified in top 10 (average performance) | 9.51 | 3.54 |
| Mean (arithmetic) of Avg. Precisions | 0.94 | 0.37 |
| Mean rank of first correctly identified cover | 1.86 | 5.65 |
Number of Correct Covers at Rank X Returned in Top Ten[edit]
| Rank | CYWW1 | YCW1 |
|---|---|---|
| 1 | 520 | 415 |
| 2 | 521 | 339 |
| 3 | 519 | 273 |
| 4 | 521 | 221 |
| 5 | 520 | 167 |
| 6 | 523 | 151 |
| 7 | 517 | 125 |
| 8 | 509 | 88 |
| 9 | 506 | 71 |
| 10 | 473 | 63 |
| Total | 5129 | 1913 |
Average Performance per Query Group[edit]
| Group | YCW1 | CYWW1 |
|---|---|---|
| 1 | 0.13 | 1 |
| 2 | 0.05 | 0.01 |
| 3 | 0.22 | 0.51 |
| 4 | 0.01 | 0.92 |
| 5 | 0.52 | 0.84 |
| 6 | 0.30 | 1 |
| 7 | 0.34 | 0.12 |
| 8 | 0.02 | 0.02 |
| 9 | 0.01 | 0.17 |
| 10 | 0.43 | 0.65 |
| 11 | 0.02 | 0.02 |
| 12 | 0.61 | 0.89 |
| 13 | 0.50 | 1 |
| 14 | 0.01 | 0.04 |
| 15 | 0.70 | 0.93 |
| 16 | 0.74 | 0.77 |
| 17 | 0.15 | 0.31 |
| 18 | 0.75 | 0.82 |
| 19 | 0.48 | 0.66 |
| 20 | 0.43 | 0.64 |
| 21 | 0.04 | 0.12 |
| 22 | 0.12 | 0.15 |
| 23 | 0.01 | 0.13 |
| 24 | 0.53 | 0.88 |
| 25 | 0.06 | 0.68 |
| 26 | 0.12 | 0.63 |
| 27 | 0.67 | 0.90 |
| 28 | 0.01 | 0.06 |
| 29 | 0.59 | 1 |
| 30 | 0.80 | 0.91 |
Individual Results Files[edit]
Mixed Collection[edit]
Average Precision by Query
YCW1 : Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang
CYWW1 : Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang
Rank Lists
YCW1 : Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang
CYWW1 : Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang
Sapp's Mazurka Collection[edit]
Average Precision by Query
YCW1 : Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang
CYWW1 : Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang
Rank Lists
YCW1 : Ming-Chi Yen, Chuan-Yau Chan, Hsin-Min Wang
CYWW1 : Chuan-Yau Chan, Ming-Chi Yen, Ju-Chiang Wang, Hsin-Min Wang