2018:Audio Cover Song Identification Results
Contents
Introduction[edit]
Audio Cover Song (ACS) Identification was evaluated against two datasets: Mixed Collection and Sapp's Mazurka Collection.
Mixed Collection Information[edit]
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.
Using each of these cover song files in turn as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.
Sapp's Mazurka Collection Information[edit]
In addition to our original ACS dataset, we used the Mazurka.org dataset put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.
General Legend[edit]
| Sub code | Submission name | Abstract | Contributors |
|---|---|---|---|
| CL1 | CL1 | Ning Chen, Mingyu Li | |
| LCLL1 | CoverNet | juheon lee, Sungkyun Chang, Donmoon Lee, Kyogu Lee |
Results[edit]
Mixed Collection[edit]
Summary Results[edit]
| CL1 | LCLL1 | |
|---|---|---|
| Total number of covers identified in top 10 | 2060.00 | 2104.00 |
| Mean number of covers identified in top 10 (average performance) | 6.24 | 6.38 |
| Mean (arithmetic) of Avg. Precisions | 0.66 | 0.66 |
| Mean rank of first correctly identified cover | 6.89 | 20.10 |
Number of Correct Covers at Rank X Returned in Top Ten[edit]
| Rank | CL1 | LCLL1 |
|---|---|---|
| 1 | 247 | 268 |
| 2 | 249 | 259 |
| 3 | 238 | 255 |
| 4 | 227 | 252 |
| 5 | 223 | 243 |
| 6 | 216 | 223 |
| 7 | 193 | 207 |
| 8 | 188 | 175 |
| 9 | 174 | 144 |
| 10 | 105 | 78 |
| Total | 2060 | 2104 |
Average Performance per Query Group[edit]
| Group | CL1 | LCLL1 |
|---|---|---|
| 1 | 0.82 | 0.82 |
| 2 | 0.13 | 0.24 |
| 3 | 0.69 | 0.65 |
| 4 | 0.85 | 1.00 |
| 5 | 0.78 | 0.43 |
| 6 | 0.97 | 0.97 |
| 7 | 0.71 | 0.70 |
| 8 | 0.49 | 0.62 |
| 9 | 0.18 | 0.05 |
| 10 | 0.37 | 0.52 |
| 11 | 0.28 | 0.47 |
| 12 | 0.87 | 0.81 |
| 13 | 0.87 | 1 |
| 14 | 0.29 | 0.20 |
| 15 | 0.97 | 0.97 |
| 16 | 0.74 | 0.60 |
| 17 | 0.40 | 0.31 |
| 18 | 0.97 | 0.94 |
| 19 | 0.71 | 0.78 |
| 20 | 0.67 | 0.70 |
| 21 | 0.52 | 0.63 |
| 22 | 0.73 | 0.79 |
| 23 | 0.59 | 0.65 |
| 24 | 0.85 | 0.83 |
| 25 | 0.37 | 0.16 |
| 26 | 0.40 | 0.41 |
| 27 | 0.82 | 0.82 |
| 28 | 0.78 | 0.78 |
| 29 | 1 | 1.00 |
| 30 | 0.90 | 0.98 |
Sapp's Mazuraka Collection[edit]
Summary results[edit]
| CL1 | LCLL1 | |
|---|---|---|
| Total number of covers identified in top 10 | 5179.00 | 5072.00 |
| Mean number of covers identified in top 10 (average performance) | 9.61 | 9.41 |
| Mean (arithmetic) of Avg. Precisions | 0.96 | 0.94 |
| Mean rank of first correctly identified cover | 1.36 | 1.55 |
Number of Correct Covers at Rank X Returned in Top Ten[edit]
| Rank | CL1 | LCLL1 |
|---|---|---|
| 1 | 519 | 522 |
| 2 | 517 | 525 |
| 3 | 524 | 525 |
| 4 | 523 | 511 |
| 5 | 524 | 510 |
| 6 | 517 | 515 |
| 7 | 524 | 508 |
| 8 | 516 | 495 |
| 9 | 510 | 501 |
| 10 | 505 | 460 |
| Total | 5179 | 5072 |
Average Performance per Query Group[edit]
| Group | CL1 | LCLL1 |
|---|---|---|
| 1 | 1 | 1 |
| 2 | 1 | 1 |
| 3 | 1 | 1 |
| 4 | 1 | 0.73 |
| 5 | 1 | 1 |
| 6 | 1 | 1 |
| 7 | 1 | 1 |
| 8 | 1 | 0.99 |
| 9 | 1 | 0.93 |
| 10 | 1 | 1 |
| 11 | 1 | 0.98 |
| 12 | 1 | 1 |
| 13 | 1 | 1 |
| 14 | 1 | 1 |
| 15 | 1 | 1 |
| 16 | 1 | 1 |
| 17 | 1 | 1 |
| 18 | 1 | 1 |
| 19 | 1 | 1 |
| 20 | 1 | 1 |
| 21 | 1 | 1 |
| 22 | 1 | 1 |
| 23 | 1 | 1 |
| 24 | 1 | 1 |
| 25 | 1 | 0.98 |
| 26 | 0.59 | 0.66 |
| 27 | 0.81 | 0.74 |
| 28 | 0.47 | 0.45 |
| 29 | 0.35 | 0.35 |
| 30 | 1 | 1 |
| 31 | 1 | 0.97 |
| 32 | 1 | 1 |
| 33 | 1 | 0.97 |
| 34 | 1 | 1 |
| 35 | 1 | 0.98 |
| 36 | 1 | 0.91 |
| 37 | 1 | 1 |
| 38 | 0.94 | 0.99 |
| 39 | 1 | 0.93 |
| 40 | 1 | 1 |
| 41 | 1 | 1 |
| 42 | 0.88 | 0.90 |
| 43 | 1 | 1 |
| 44 | 1 | 1 |
| 45 | 1 | 0.94 |
| 46 | 1 | 1 |
| 47 | 1 | 1 |
| 48 | 1 | 1 |
| 49 | 0.83 | 0.77 |
Individual Results Files[edit]
Mixed Collection[edit]
Average Precision by Query
CL1 : Ning Chen, Mingyu Li
LCLL1 : Juheon Lee, Sungkyun Chang, Donmoon Lee, Kyogu Lee
Rank Lists
CL1 : Ning Chen, Mingyu Li
LCLL1 : Juheon Lee, Sungkyun Chang, Donmoon Lee, Kyogu Lee
Mazurkas[edit]
Average Precision by Query
CL1 : Ning Chen, Mingyu Li
LCLL1 : Juheon Lee, Sungkyun Chang, Donmoon Lee, Kyogu Lee
Rank Lists
CL1 : Ning Chen, Mingyu Li
LCLL1 : Juheon Lee, Sungkyun Chang, Donmoon Lee, Kyogu Lee