2009:Audio Cover Song Identification Results
Contents
Introduction[edit]
This year, we ran Audio Cover Song (ACS) Identfication with two datasets: Mixed Collection and Sapp's Mazurka Collection.
Mixed Collection Information[edit]
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files (16bit, monophonic, 22.05khz, wav). The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations.
Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.
Sapp's Mazurka Collection Information[edit]
In addition to our original ACS dataset, we used the Mazurka.org dataset put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems returned a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.
General Legend[edit]
Team ID[edit]
RE = Suman Ravuri, Daniel P. W. Ellis
SZA = Joan Serra, Massimiliano Zanin, Ralph G. Andrzejak
TA = Teppo E. Ahonen
Results[edit]
Mixed Collection[edit]
Summary Results[edit]
| RE | SZA | TA | |
|---|---|---|---|
| Total number of covers identified in top 10 | 2046.00 | 2426.00 | 646.00 |
| Mean number of covers identified in top 10 (average performance) | 6.20 | 7.35 | 1.96 |
| Mean (arithmetic) of Avg. Precisions | 0.66 | 0.75 | 0.20 |
| Mean rank of first correctly identified cover | 2.28 | 6.15 | 29.90 |
Number of Correct Covers at Rank X Returned in Top Ten[edit]
| Rank | RE | SZA | TA |
|---|---|---|---|
| 1 | 285 | 284 | 118 |
| 2 | 271 | 285 | 108 |
| 3 | 245 | 282 | 85 |
| 4 | 246 | 279 | 80 |
| 5 | 228 | 265 | 58 |
| 6 | 208 | 252 | 61 |
| 7 | 196 | 230 | 46 |
| 8 | 160 | 227 | 33 |
| 9 | 139 | 186 | 32 |
| 10 | 68 | 136 | 25 |
| Total | 2046 | 2426 | 646 |
Average Performance per Query Group[edit]
| Group | RE | SZA | TA |
|---|---|---|---|
| 1 | 0.56 | 0.97 | 0.02 |
| 2 | 0.90 | 0.75 | 0.04 |
| 3 | 0.29 | 0.05 | 0.06 |
| 4 | 0.57 | 1 | 0.04 |
| 5 | 0.03 | 0.01 | 0.02 |
| 6 | 0.77 | 0.76 | 0.11 |
| 7 | 0.89 | 1 | 0.15 |
| 8 | 0.91 | 1 | 0.61 |
| 9 | 0.94 | 0.96 | 0.32 |
| 10 | 0.75 | 0.99 | 0.14 |
| 11 | 0.22 | 0.72 | 0.08 |
| 12 | 0.65 | 0.51 | 0.29 |
| 13 | 1 | 1 | 0.90 |
| 14 | 0.45 | 0.93 | 0.02 |
| 15 | 0.92 | 0.90 | 0.07 |
| 16 | 0.42 | 0.21 | 0.01 |
| 17 | 0.88 | 0.90 | 0.49 |
| 18 | 0.84 | 0.90 | 0.05 |
| 19 | 0.95 | 1 | 0.79 |
| 20 | 0.46 | 0.06 | 0.01 |
| 21 | 0.57 | 0.21 | 0.03 |
| 22 | 0.85 | 1 | 0.04 |
| 23 | 0.94 | 0.99 | 0.07 |
| 24 | 0.86 | 0.98 | 0.58 |
| 25 | 0.91 | 0.91 | 0.49 |
| 26 | 0.43 | 0.91 | 0.05 |
| 27 | 0.23 | 0.81 | 0.04 |
| 28 | 0.82 | 0.80 | 0.26 |
| 29 | 0.79 | 0.89 | 0.02 |
| 30 | 0.84 | 0.96 | 0.02 |
Friedman's Test for Significant Differences[edit]
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| SZA | RE | -0.44 | 0.17 | 0.77 | FALSE |
| SZA | TA | 0.88 | 1.48 | 2.09 | TRUE |
| RE | TA | 0.71 | 1.32 | 1.92 | TRUE |
Run Times[edit]
TBA
Sapp's Mazuraka Collection[edit]
Summary results[edit]
| RE | SZA | TA | |
|---|---|---|---|
| Total number of covers identified in top 10 | 4757.00 | 5165.00 | 2843.00 |
| Mean number of covers identified in top 10 (average performance) | 8.83 | 9.58 | 5.27 |
| Mean (arithmetic) of Avg. Precisions | 0.91 | 0.96 | 0.56 |
| Mean rank of first correctly identified cover | 1.68 | 1.61 | 5.49 |
Number of Correct Covers at Rank X Returned in Top Ten[edit]
| Rank | RE | SZA | TA |
|---|---|---|---|
| 1 | 522 | 527 | 451 |
| 2 | 522 | 522 | 413 |
| 3 | 522 | 525 | 368 |
| 4 | 523 | 516 | 320 |
| 5 | 518 | 522 | 316 |
| 6 | 513 | 522 | 271 |
| 7 | 495 | 518 | 235 |
| 8 | 452 | 506 | 195 |
| 9 | 415 | 511 | 160 |
| 10 | 285 | 496 | 114 |
| Total | 4767 | 5165 | 2843 |
Average Performance per Query Group[edit]
| Group | RE | SZA | TA |
|---|---|---|---|
| 1 | 1.00 | 1 | 0.61 |
| 2 | 1 | 1 | 0.94 |
| 3 | 1 | 1 | 0.72 |
| 4 | 0.98 | 0.98 | 0.92 |
| 5 | 1 | 1 | 0.02 |
| 6 | 0.89 | 1 | 0.47 |
| 7 | 1 | 1 | 0.97 |
| 8 | 1 | 1 | 0.45 |
| 9 | 1 | 1 | 0.96 |
| 10 | 0.92 | 1 | 0.68 |
| 11 | 0.97 | 1 | 0.90 |
| 12 | 0.85 | 1 | 0.32 |
| 13 | 1 | 1 | 0.31 |
| 14 | 0.99 | 1 | 0.26 |
| 15 | 0.97 | 1 | 0.80 |
| 16 | 0.99 | 1 | 0.34 |
| 17 | 0.99 | 1 | 0.05 |
| 18 | 0.98 | 1 | 0.81 |
| 19 | 1 | 0.94 | 0.86 |
| 20 | 1 | 1 | 0.98 |
| 21 | 1 | 1 | 0.49 |
| 22 | 0.90 | 1 | 0.97 |
| 23 | 1 | 1 | 0.90 |
| 24 | 0.75 | 1 | 0.56 |
| 25 | 0.99 | 1 | 0.41 |
| 26 | 0.71 | 0.73 | 0.08 |
| 27 | 0.90 | 0.84 | 0.60 |
| 28 | 0.58 | 0.45 | 0.45 |
| 29 | 0.45 | 0.49 | 0.32 |
| 30 | 0.93 | 1 | 0.30 |
| 31 | 0.91 | 1 | 0.30 |
| 32 | 0.94 | 1 | 0.87 |
| 33 | 1 | 1 | 0.74 |
| 34 | 0.94 | 1 | 0.28 |
| 35 | 0.92 | 1 | 0.40 |
| 36 | 0.90 | 1 | 0.75 |
| 37 | 0.91 | 1 | 0.27 |
| 38 | 0.75 | 1 | 0.05 |
| 39 | 0.99 | 1 | 0.87 |
| 40 | 1 | 1 | 0.04 |
| 41 | 0.89 | 1 | 0.94 |
| 42 | 0.91 | 0.99 | 0.47 |
| 43 | 0.83 | 1 | 0.30 |
| 44 | 0.90 | 1 | 0.18 |
| 45 | 0.75 | 1 | 0.03 |
| 46 | 1 | 1 | 0.41 |
| 47 | 1 | 1 | 0.91 |
| 48 | 0.98 | 1 | 0.72 |
| 49 | 0.89 | 0.90 | 0.46 |
Friedman's Test for Significant Differences[edit]
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| SZA | RE | 0.1764 | 0.6327 | 1.0889 | TRUE |
| SZA | TA | 1.2988 | 1.7551 | 2.2114 | TRUE |
| RE | TA | 0.6662 | 1.1224 | 1.5787 | TRUE |
Run Times[edit]
TBA
Individual Results Files[edit]
Mixed Collection[edit]
Average Precision by Query
RE : S. Ravuri, D.P.W Ellis
SZA : J. Serra et al.
TA : T. Ahonen
Average Precision by Cover Group
RE : S. Ravuri, D.P.W Ellis
SZA : J. Serra et al.
TA : T. Ahonen
Rank Lists
RE : S. Ravuri, D.P.W Ellis
SZA : J. Serra et al.
TA : T. Ahonen
Sapp's Mazurka Collection[edit]
Average Precision by Query
RE : S. Ravuri
SZA : J. Serra et al.
TA : T. Ahonen
Average Precision by Cover Group
RE : S. Ravuri
SZA : J. Serra et al.
TA : T. Ahonen
Rank Lists
RE : S. Ravuri
SZA : J. Serra et al.
TA : T. Ahonen