Read-only mirror of the MIREX Wiki, snapshot of 29 October 2026. Open this page on the original wiki to edit it or to see newer changes.

2006:Symbolic Melodic Similarity Results

From MIREX Wiki

Introduction[edit]

These are the results for the 2006 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the 2006:Symbolic Melodic Similarity page.

Each system was given a query and returned the 10 most melodically similar songs from a given collection where the collections were RISM (monophonic; 10,000), Karoke (polyphonic; 1,000), Mixed (polyphonic; 15,741). Then, for each query, the returned results from all participants were grouped and were evaluated by human graders, each query being evaluated by 3 different graders with two scores (using the Evalutron 6000 system). Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 10).

Evalutron 6000 Summary Data[edit]

Number of evaluators = 20
Number of evaluations per query/candidate pair = 3
Number of queries per grader = 15
Ave. size of the candidate lists = 15
Ave. number of query/candidate pairs evaluated per grader: 225
 Number of queries (across all subtasks = 17

General Legend[edit]

Team ID[edit]

Prefix R = RISM collection, K = Karaoke collection, M = Polyphonic collection

FH = Pascal Ferraro and Pierre Hanna
NM = Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen
RT = Rainer Typke, Frans Wiering and Remco C. Veltkamp
KF = Klaus Frieler
AU = Alexandra Uitdenbogerd

Broad Categories[edit]

NS = Not Similar
SS = Somewhat Similar
VS = Very Similar

Table Headings[edit]

ADR = Average Dynamic Recall
NRGB = Normalize Recall at Group Boundaries
AP = Average Precision (non-interpolated)
PND = Precision at N Documents

Calculating Summary Measures[edit]

Fine(1) = Sum of fine-grained human similarity decisions (0-10).
PSum(1) = Sum of human broad similarity decisions: NS=0, SS=1, VS=2.
WCsum(1) = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar).
SDsum(1) = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar).
Greater0(1) = NS=0, SS=1, VS=1 (binary relevance judgement).
Greater1(1) = NS=0, SS=0, VS=1 (binary relevance judgement using only Very Similar).

(1)Normalized to the range 0 to 1.

Overall Summary Results[edit]

Visualizations[edit]

Rainer Typke has created a series of 2006:Symbolic Melodic Similarity Graphs that help us visualize the results.
Rainer Typke has also created a set of detailed representations of the results that is definitely with exploring at [http://rainer.typke.org/mirex06.0.html].

Task I: RISM Overall Summary[edit]

RFH RRT RKF2 RAU RKF3 RNM2 RNM1 RKF1
ADR 0.707 0.715 0.670 0.577 0.555 0.541 0.268 0.000
NRGB 0.622 0.626 0.568 0.515 0.466 0.484 0.277 0.000
AP 0.623 0.607 0.594 0.461 0.391 0.393 0.167 0.000
PND 0.526 0.547 0.504 0.441 0.426 0.403 0.250 0.000
Fine 0.490 0.488 0.478 0.433 0.418 0.292 0.208 0.145
Psum 0.594 0.592 0.567 0.522 0.500 0.364 0.258 0.172
WCsum 0.544 0.544 0.519 0.457 0.450 0.344 0.250 0.119
SDsum 0.519 0.521 0.494 0.425 0.425 0.335 0.246 0.092
Greater0 0.744 0.733 0.711 0.717 0.650 0.422 0.283 0.333
Greater1 0.444 0.450 0.422 0.328 0.333 0.306 0.233 0.011

download these results as csv

Task I: RISM Runtime Data[edit]

Team ID Machine Run-time(seconds)
AU indexing beer 4 33
AU query beer 4 31
FH query beer 4 807
KF indexing black 210
KF1 query black 2880
KF2 query black 2220
KF3 optip black 3960
NM1 query beer 6 68
NM2 query beer 6 188
RT query beer 4 59

download these results as csv

Task IIa: Karaoke Overall Summary[edit]

KRT KAU KFH KNM2 KNM1
ADR 0.819 0.378 0.150 0.000 0.000
NRGB 0.764 0.333 0.150 0.000 0.000
AP 0.875 0.363 0.100 0.000 0.000
PND 0.833 0.333 0.100 0.000 0.000
Fine 0.267 0.207 0.153 0.112 0.105
Psum 0.327 0.260 0.200 0.147 0.120
WCsum 0.289 0.211 0.151 0.098 0.082
SDsum 0.270 0.187 0.127 0.073 0.063
Greater0 0.440 0.407 0.347 0.293 0.233
Greater1 0.213 0.113 0.053 0.007 0.000

download these results as csv

Task IIa: Karaoke Runtime Data[edit]

Team ID Machine Run-time(seconds)
AU indexing beer 4 397
AU query beer 4 5
FH query beer 4 1338
NM1 query beer 6 386
NM2 query beer 6 1875
RT query beer 4 32

download these results as csv

Task IIb: Mixed Polyphonic Overall Summary[edit]

MRT MAU MFH MNM1 MNH2
ADR 0.784 0.587 0.218 0.070 0.000
NRGB 0.806 0.516 0.174 0.040 0.000
AP 0.903 0.517 0.170 0.032 0.004
PND 0.903 0.549 0.187 0.044 0.000
Fine 0.663 0.489 0.283 0.128 0.114
Psum 0.772 0.558 0.342 0.153 0.153
WCsum 0.752 0.535 0.294 0.115 0.113
SDsum 0.742 0.524 0.271 0.096 0.093
Greater0 0.833 0.628 0.483 0.267 0.272
Greater1 0.711 0.489 0.200 0.039 0.033

download these results as csv

Task IIb: Mixed Polyphonic Runtime Data[edit]

Team ID Machine Run-time(seconds)
AU indexing beer 4 2785
AU query beer 4 51
FH query beer 4 14440
NM1 both beer 6 3271
NM2 both beer 6 16314
RT query beer 4 108

download these results as csv

Task I: RISM Collection Summary Results[edit]

There is an error with this data set...please stand by.

ADR:
queryID RFH RRT RKF2 RAU RKF3 RNM2 RNM1 RKF1
qr000001 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
qr000002 0.768 0.802 0.518 0.144 0.298 0.572 0.103 0.000
qr000003 0.786 0.840 0.707 0.740 0.642 0.000 0.555 0.000
qr000004 0.930 0.795 0.866 0.707 0.609 0.817 0.222 0.000
qr000005 0.870 0.870 0.960 0.960 0.870 0.870 0.613 0.000
qr000006 0.887 0.984 0.967 0.912 0.912 0.984 0.113 0.000
Ave. ADR score: 0.707 0.715 0.670 0.577 0.555 0.541 0.268 0.000
NRGB:
queryID RFH RRT RKF2 RAU RKF3 RNM2 RNM1 RKF1
qr000001 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
qr000002 0.630 0.630 0.296 0.296 0.333 0.370 0.185 0.000
qr000003 0.625 0.688 0.563 0.625 0.500 0.000 0.375 0.000
qr000004 0.917 0.615 0.771 0.531 0.406 0.708 0.260 0.000
qr000005 0.840 0.840 0.920 0.920 0.840 0.840 0.560 0.000
qr000006 0.719 0.984 0.859 0.719 0.719 0.984 0.281 0.000
Ave. NRGB score: 0.622 0.626 0.568 0.515 0.466 0.484 0.277 0.000
AP:
queryID RFH RRT RKF2 RAU RKF3 RNM2 RNM1 RKF1
qr000001 0.500 0.500 0.500 0.333 0.000 0.000 0.000 0.000
qr000002 0.604 0.514 0.222 0.059 0.137 0.383 0.050 0.000
qr000003 0.500 0.604 0.375 0.446 0.384 0.000 0.250 0.000
qr000004 0.750 0.548 0.683 0.375 0.375 0.500 0.186 0.000
qr000005 0.600 0.600 0.800 0.800 0.600 0.600 0.419 0.000
qr000006 0.785 0.875 0.986 0.750 0.847 0.875 0.098 0.000
Ave. AP score: 0.623 0.607 0.594 0.461 0.391 0.393 0.167 0.000
PND:
queryID RFH RRT RKF2 RAU RKF3 RNM2 RNM1 RKF1
qr000001 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
qr000002 0.556 0.556 0.222 0.222 0.333 0.444 0.222 0.000
qr000003 0.500 0.625 0.375 0.500 0.500 0.000 0.250 0.000
qr000004 0.750 0.625 0.750 0.375 0.375 0.500 0.375 0.000
qr000005 0.600 0.600 0.800 0.800 0.600 0.600 0.400 0.000
qr000006 0.750 0.875 0.875 0.750 0.750 0.875 0.250 0.000
Ave. PND score: 0.526 0.547 0.504 0.441 0.426 0.403 0.250 0.000
Fine:
queryID RFH RRT RKF2 RAU RKF3 RNM2 RNM1 RKF1
qr000001 0.322 0.265 0.359 0.309 0.230 0.042 0.007 0.154
qr000002 0.566 0.569 0.407 0.423 0.452 0.358 0.153 0.214
qr000003 0.401 0.489 0.352 0.396 0.355 0.017 0.191 0.076
qr000004 0.581 0.544 0.554 0.357 0.419 0.390 0.341 0.111
qr000005 0.387 0.386 0.475 0.483 0.379 0.276 0.278 0.140
qr000006 0.681 0.675 0.720 0.632 0.672 0.671 0.278 0.178
Ave. Fine Score: 0.490 0.488 0.478 0.433 0.418 0.292 0.208 0.145
Psum:
queryID RFH RRT RKF2 RAU RKF3 RNM2 RNM1 RKF1
qr000001 0.417 0.317 0.433 0.383 0.267 0.067 0.033 0.200
qr000002 0.717 0.700 0.483 0.533 0.533 0.467 0.200 0.250
qr000003 0.500 0.617 0.417 0.483 0.450 0.050 0.250 0.100
qr000004 0.667 0.650 0.633 0.417 0.517 0.450 0.400 0.100
qr000005 0.483 0.450 0.550 0.567 0.450 0.350 0.350 0.183
qr000006 0.783 0.817 0.883 0.750 0.783 0.800 0.317 0.200
Ave. Psum Score: 0.594 0.592 0.567 0.522 0.500 0.364 0.258 0.172
WCsum:
queryID RFH RRT RKF2 RAU RKF3 RNM2 RNM1 RKF1
qr000001 0.322 0.244 0.344 0.289 0.189 0.044 0.022 0.133
qr000002 0.689 0.667 0.422 0.456 0.467 0.444 0.200 0.178
qr000003 0.444 0.556 0.378 0.433 0.422 0.033 0.233 0.067
qr000004 0.633 0.600 0.600 0.378 0.456 0.433 0.400 0.067
qr000005 0.433 0.411 0.500 0.489 0.400 0.333 0.333 0.122
qr000006 0.744 0.789 0.867 0.700 0.767 0.778 0.311 0.144
Ave. WCsum Score: 0.544 0.544 0.519 0.457 0.450 0.344 0.250 0.119
SDsum:
queryID RRT RFH RKF2 RAU RKF3 RNM2 RNM1 RKF1
qr000001 0.208 0.275 0.300 0.242 0.150 0.033 0.017 0.100
qr000002 0.650 0.675 0.392 0.417 0.433 0.433 0.200 0.142
qr000003 0.525 0.417 0.358 0.408 0.408 0.025 0.225 0.050
qr000004 0.575 0.617 0.583 0.358 0.425 0.425 0.400 0.050
qr000005 0.392 0.408 0.475 0.450 0.375 0.325 0.325 0.092
qr000006 0.775 0.725 0.858 0.675 0.758 0.767 0.308 0.117
Ave. SDsum Score: 0.521 0.519 0.494 0.425 0.425 0.335 0.246 0.092
Greater0:
queryID RFH RRT RAU RKF2 RKF3 RNM2 RNM1 RKF1
qr000001 0.700 0.533 0.667 0.700 0.500 0.133 0.067 0.400
qr000002 0.800 0.800 0.767 0.667 0.733 0.533 0.200 0.467
qr000003 0.667 0.800 0.633 0.533 0.533 0.100 0.300 0.200
qr000004 0.767 0.800 0.533 0.733 0.700 0.500 0.400 0.200
qr000005 0.633 0.567 0.800 0.700 0.600 0.400 0.400 0.367
qr000006 0.900 0.900 0.900 0.933 0.833 0.867 0.333 0.367
Ave. greater0 Score: 0.744 0.733 0.717 0.711 0.650 0.422 0.283 0.333
Greater1:
queryID RRT RFH RKF2 RKF3 RAU RNM2 RNM1 RKF1
qr000001 0.100 0.133 0.167 0.033 0.100 0.000 0.000 0.000
qr000002 0.600 0.633 0.300 0.233 0.300 0.400 0.200 0.033
qr000003 0.433 0.333 0.300 0.367 0.333 0.000 0.200 0.000
qr000004 0.500 0.567 0.533 0.333 0.300 0.400 0.400 0.000
qr000005 0.333 0.333 0.400 0.300 0.333 0.300 0.300 0.000
qr000006 0.733 0.667 0.833 0.733 0.600 0.733 0.300 0.033
Ave. greater1 Score: 0.450 0.444 0.422 0.333 0.328 0.306 0.233 0.011

download these results as csv

Task IIa: Karaoke Collection Summary Results[edit]

ADR:
queryID KRT KAU KFH KNM2 KNM1
qk000001 1.000 0.889 0.000 0.000 0.000
qk000002 0.542 0.000 0.000 0.000 0.000
qk000003 0.556 0.000 0.000 0.000 0.000
qk000004 1.000 0.000 0.000 0.000 0.000
qk000005 1.000 1.000 0.750 0.000 0.000
Ave. ADR score: 0.820 0.378 0.150 0.000 0.000
NRGB:
queryID KRT KAU KFH KNM2 KNM1
qk000001 1.000 0.667 0.000 0.000 0.000
qk000002 0.375 0.000 0.000 0.000 0.000
qk000003 0.444 0.000 0.000 0.000 0.000
qk000004 1.000 0.000 0.000 0.000 0.000
qk000005 1.000 1.000 0.750 0.000 0.000
Ave. NRGB score: 0.764 0.333 0.150 0.000 0.000
AP:
queryID KRT KAU KFH KNM2 KNM1
qk000001 1.000 0.667 0.000 0.000 0.000
qk000002 0.707 0.000 0.000 0.000 0.000
qk000003 0.667 0.067 0.000 0.000 0.000
qk000004 1.000 0.083 0.000 0.000 0.000
qk000005 1.000 1.000 0.500 0.000 0.000
Ave. AP score: 0.875 0.363 0.100 0.000 0.000
PND:
queryID KRT KAU KFH KNM2 KNM1
qk000001 1.000 0.667 0.000 0.000 0.000
qk000002 0.500 0.000 0.000 0.000 0.000
qk000003 0.667 0.000 0.000 0.000 0.000
qk000004 1.000 0.000 0.000 0.000 0.000
qk000005 1.000 1.000 0.500 0.000 0.000
Ave. PND score: 0.833 0.333 0.100 0.000 0.000
Fine
queryID KRT KAU KFH KNM2 KNM1
qk000001 0.337 0.268 0.108 0.079 0.116
qk000002 0.297 0.094 0.106 0.048 0.109
qk000003 0.222 0.189 0.169 0.177 0.073
qk000004 0.267 0.199 0.152 0.088 0.078
qk000005 0.214 0.283 0.230 0.168 0.149
Ave. Fine Score: 0.267 0.207 0.153 0.112 0.105
Psum:
queryID KRT KAU KFH KNM2 KNM1
qk000001 0.417 0.317 0.150 0.100 0.133
qk000002 0.350 0.117 0.117 0.067 0.133
qk000003 0.300 0.250 0.233 0.233 0.050
qk000004 0.317 0.267 0.217 0.117 0.117
qk000005 0.250 0.350 0.283 0.217 0.167
Ave. Psum Score: 0.327 0.260 0.200 0.147 0.120
WCsum:
queryID KRT KAU KFH KNM2 KNM1
qk000001 0.378 0.278 0.100 0.067 0.089
qk000002 0.322 0.078 0.078 0.044 0.089
qk000003 0.244 0.189 0.189 0.156 0.033
qk000004 0.278 0.211 0.167 0.078 0.078
qk000005 0.222 0.300 0.222 0.144 0.122
Ave. WCsum Score: 0.289 0.211 0.151 0.098 0.082
SDsum:
queryID KRT KAU KFH KNM2 KNM1
qk000001 0.358 0.258 0.075 0.050 0.067
qk000002 0.308 0.058 0.058 0.033 0.067
qk000003 0.217 0.158 0.167 0.117 0.025
qk000004 0.258 0.183 0.142 0.058 0.058
qk000005 0.208 0.275 0.192 0.108 0.100
Ave. SDsum Score: 0.270 0.187 0.127 0.073 0.063
Greater0:
queryID KRT KAU KFH KNM2 KNM1
qk000001 0.533 0.433 0.300 0.200 0.267
qk000002 0.433 0.233 0.233 0.133 0.267
qk000003 0.467 0.433 0.367 0.467 0.100
qk000004 0.433 0.433 0.367 0.233 0.233
qk000005 0.333 0.500 0.467 0.433 0.300
Ave. greater0 Score: 0.440 0.407 0.347 0.293 0.233
Greater1:
queryID KRT KAU KFH KNM1 KNM2
qk000001 0.300 0.200 0.000 0.000 0.000
qk000002 0.267 0.000 0.000 0.000 0.000
qk000003 0.133 0.067 0.100 0.000 0.000
qk000004 0.200 0.100 0.067 0.000 0.000
qk000005 0.167 0.200 0.100 0.033 0.000
Ave. greater1 Score: 0.213 0.113 0.053 0.007 0.000

download these results as csv

Task IIb: Mixed Polyphonic Collection Summary Results[edit]

ADR:
queryID MRT MAU MFH MNM1 MNM2
qm000001 0.569 0.158 0.000 0.000 0.000
qm000002 0.592 0.322 0.000 0.408 0.000
qm000003 0.793 0.663 0.000 0.000 0.000
qm000004 0.893 0.602 0.012 0.000 0.000
qm000005 1.000 1.000 1.000 0.000 0.000
qm000006 0.857 0.776 0.293 0.010 0.000
Ave. ADR score: 0.784 0.587 0.218 0.070 0.000
NRGB:
queryID MRT MAU MFH MNM1 MNM2
qm000001 0.583 0.139 0.000 0.000 0.000
qm000002 0.772 0.517 0.000 0.228 0.000
qm000003 0.857 0.510 0.000 0.000 0.000
qm000004 0.836 0.383 0.012 0.000 0.000
qm000005 0.909 0.909 0.909 0.000 0.000
qm000006 0.879 0.637 0.121 0.010 0.000
Ave. NRGB score: 0.806 0.516 0.174 0.040 0.000
AP:
queryID MRT MAU MFH MNM1 MNM2
qm000001 1.000 0.167 0.000 0.000 0.000
qm000002 0.833 0.403 0.000 0.167 0.021
qm000003 1.000 0.524 0.000 0.000 0.000
qm000004 0.778 0.397 0.012 0.000 0.000
qm000005 0.909 0.909 0.909 0.000 0.000
qm000006 0.900 0.700 0.100 0.025 0.000
Ave. AP score: 0.903 0.517 0.170 0.032 0.004
PND:
queryID MRT MAU MFH MNM1 MNM2
qm000001 1.000 0.167 0.000 0.000 0.000
qm000002 0.833 0.500 0.000 0.167 0.000
qm000003 1.000 0.571 0.000 0.000 0.000
qm000004 0.778 0.444 0.111 0.000 0.000
qm000005 0.909 0.909 0.909 0.000 0.000
qm000006 0.900 0.700 0.100 0.100 0.000
Ave. AP score: 0.903 0.549 0.187 0.044 0.000
Fine:
queryID MRT MAU MFH MNM1 MNM2
qm000001 0.502 0.162 0.130 0.097 0.101
qm000002 0.479 0.306 0.108 0.203 0.088
qm000003 0.670 0.438 0.087 0.105 0.073
qm000004 0.621 0.430 0.226 0.097 0.235
qm000005 0.866 0.880 0.870 0.147 0.078
qm000006 0.838 0.719 0.279 0.122 0.110
Ave. Fine Score: 0.663 0.489 0.283 0.128 0.114
Psum:
queryID MRT MAU MFH MNM1 MNM2
qm000001 0.600 0.217 0.167 0.083 0.150
qm000002 0.583 0.367 0.133 0.233 0.150
qm000003 0.783 0.483 0.133 0.117 0.083
qm000004 0.717 0.450 0.267 0.117 0.250
qm000005 1.000 1.000 1.000 0.167 0.100
qm000006 0.950 0.833 0.350 0.200 0.183
Ave. Psum Score: 0.772 0.558 0.342 0.153 0.153
WCsum:
queryID MRT MAU MFH MNM1 MNM2
qm000001 0.567 0.167 0.122 0.056 0.100
qm000002 0.533 0.344 0.089 0.189 0.122
qm000003 0.767 0.456 0.089 0.078 0.056
qm000004 0.700 0.422 0.189 0.078 0.200
qm000005 1.000 1.000 1.000 0.122 0.067
qm000006 0.944 0.822 0.278 0.167 0.133
Ave. WCsum Score: 0.752 0.535 0.294 0.115 0.113
SDsum:
queryID MRT MAU MFH MNM1 MNM2
qm000001 0.550 0.142 0.100 0.042 0.075
qm000002 0.508 0.333 0.067 0.167 0.108
qm000003 0.758 0.442 0.067 0.058 0.042
qm000004 0.692 0.408 0.150 0.058 0.175
qm000005 1.000 1.000 1.000 0.100 0.050
qm000006 0.942 0.817 0.242 0.150 0.108
Ave. SDsum Score: 0.742 0.524 0.271 0.096 0.093
Greater0:
queryID MRT MAU MFH MNM2 MNM1
qm000001 0.700 0.367 0.300 0.300 0.167
qm000002 0.733 0.433 0.267 0.233 0.367
qm000003 0.833 0.567 0.267 0.167 0.233
qm000004 0.767 0.533 0.500 0.400 0.233
qm000005 1.000 1.000 1.000 0.200 0.300
qm000006 0.967 0.867 0.567 0.333 0.300
Ave. greater0 Score: 0.833 0.628 0.483 0.272 0.267
Greater1:
queryID MRT MAU MFH MNM1 MNM2
qm000001 0.500 0.067 0.033 0.000 0.000
qm000002 0.433 0.300 0.000 0.100 0.067
qm000003 0.733 0.400 0.000 0.000 0.000
qm000004 0.667 0.367 0.033 0.000 0.100
qm000005 1.000 1.000 1.000 0.033 0.000
qm000006 0.933 0.800 0.133 0.100 0.033
Ave. greater1 Score: 0.711 0.489 0.200 0.039 0.033

download these results as csv

Raw Scores[edit]

The raw data derived from the Evalutron 6000 human evaluations are located on the 2006:Symbolic Melodic Similarity Raw Data page.