Read-only mirror of the MIREX Wiki, snapshot of 29 October 2026. Open this page on the original wiki to edit it or to see newer changes.

2009:Query-by-Tapping Results

From MIREX Wiki

Introduction[edit]

These are the results for the 2008 running of the Query-by-tappingn task. For background information about this task set please refer to the 2009:Query_by_Tapping page.

Task Descriptions[edit]

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

  • Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
  • Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.


Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.

General Legend[edit]

Team ID[edit]

CSJ = Chun-Ta Chen and Jyh-Shing Roger Jang
HAFR = Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine
HL = Shu-Jen Show Hsiao,Tyne Liang

Task 1 Results[edit]

Task 1a, Jang's Dataset Results[edit]

Task 1a Overall Results[edit]
CSJ HAFR HL
Simple Count 0.87 0.88 0.89
MRR 0.78 0.77 0.79
Total count 890 890 890

download these results as csv

Task 1a Friedman's Test for Significant Differences[edit]

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID TeamID Lowerbound Mean Upperbound Significance
HL CSJ -0.1863 -0.0184 0.1496 FALSE
HL HAFR -0.1716 -0.0037 0.1643 FALSE
CSJ HAFR -0.1533 0.0147 0.1827 FALSE

download these results as csv

2009 sqbttask1asimple friedman mean ranks.png

MRR Method:

TeamID TeamID Lowerbound Mean Upperbound Significance
HL CSJ -0.2337 -0.0074 0.2190 FALSE
HL HAFR -0.0646 0.1618 0.3881 FALSE
CSJ HAFR -0.0572 0.1691 0.3955 FALSE

download these results as csv

2009 sqbt1atask2mrr friedman mean ranks.png

Task 1a Summary Results by Query Group[edit]

Simple Hit/Miss Counting

CSJ HAFR HL
1 1 0.92 0.92
2 1 1 1
3 1 1 1
4 1 0 1
5 1 1 1
6 1 1 0.50
7 1 1 1
8 1 1 1
9 1 1 1
10 1 1 1
11 0.50 1 0.50
12 1 1 1
13 1 1 1
14 1 1 1
15 1 1 1
16 1 1 0.95
17 0.87 0.93 1
18 0.77 0.85 0.69
19 0.38 0.50 1
20 1 1 1
21 1 1 1
22 1 0.50 0.50
23 0 0 0
24 1 1 1
25 1 1 1
26 0 0 1
27 1 0.94 1
28 1 1 1
29 1 1 1
30 0.58 0.75 0.67
31 1 1 1
32 1 1 0.67
33 0.78 0.75 0.94
34 0.92 0.92 0.92
35 1 1 1
36 1 1 1
37 0 1 1
38 1 1 1
39 1 1 1
40 1 1 1
41 0.83 0.75 0.92
42 1 1 1
43 0.80 1 1
44 0.75 0.75 0.50
45 1 1 0.50
46 1 1 1
47 1 1 1
48 1 1 1
49 1 1 1
50 1 1 1
51 0.60 0.80 0.60
52 1 1 1
53 0.80 0.60 0.60
54 1 1 1
55 1 0.98 1
56 1 1 1
57 1 0.80 1
58 1 1 1
59 0.20 0.60 0.20
60 0.43 1 0.71
61 1 1 1
62 1 1 1
63 1 1 1
64 0.50 0.50 0.50
65 1 1 1
66 1 1 0.50
67 1 1 1
68 0.67 0.78 0.61
69 0.82 0.82 0.94
70 1 1 0.75
71 1 0 1
72 0.83 0.92 0.75
73 0.93 0.86 0.93
74 1 1 1
75 0.83 1 0.83
76 1 1 1
77 0 0 1
78 1 0 0
79 1 0.50 0.75
80 1 1 1
81 1 1 1
82 1 1 1
83 1 1 1
84 1 1 1
85 0.86 0.86 0.86
86 1 1 1
87 0.58 0.75 0.75
88 1 1 1
89 1 0.67 1
90 0.50 1 1
91 1 1 1
92 0.25 0.25 0.50
93 1 1 1
94 0.50 0.67 0.67
95 1 1 1
96 1 1 1
97 1 1 1
98 0.50 0.50 0.50
99 0.88 0.88 0.81
100 1 1 1
101 0.83 1 0.83
102 0.88 1 1
103 1 1 1
104 1 1 1
105 1 1 0.96
106 0.73 0.68 0.50
107 0.94 0.94 1
108 0.69 0.56 0.88
109 0.94 0.81 0.94
110 1 1 1
111 1 1 0.78
112 1 1 1
113 1 1 1
114 1 1 1
115 0.80 0.60 0.60
116 0.75 0.75 0.75
117 1 1 1
118 1 1 1
119 0.67 0.67 0.67
120 0.67 0.67 1
121 0.33 0.67 0.33
122 0.33 0.67 0.67
123 0.67 0.33 0.33
124 0.33 0.33 0.33
125 1 0.33 0.67
126 0.67 0.67 1
127 1 1 1
128 0.67 0.67 0.67
129 0.50 0.50 0
130 1 0.50 1
131 1 1 1
132 1 1 1
133 1 1 1
134 1 1 1
135 1 1 1
136 1 1 1

download these results as csv

MRR Method

CSJ HAFR HL
1 0.94 0.88 0.88
2 1 1 1
3 1 1 1
4 1 0 1
5 1 1 1
6 0.78 0.88 0.50
7 1 1 1
8 0.75 0.75 0.75
9 1 0.25 1
10 1 1 1
11 0.50 0.60 0.50
12 1 1 1
13 1 1 0.50
14 1 0.94 0.94
15 1 1 1
16 0.93 0.82 0.35
17 0.76 0.93 1
18 0.71 0.64 0.59
19 0.28 0.30 0.50
20 1 1 1
21 0.56 1 1
22 1 0.50 0.50
23 0 0 0
24 1 0.50 1
25 0.90 0.90 0.89
26 0 0 0.14
27 0.97 0.76 0.88
28 0.83 1 1
29 1 1 1
30 0.52 0.75 0.53
31 1 0.33 1
32 1 1 0.67
33 0.50 0.72 0.91
34 0.92 0.92 0.92
35 1 1 1
36 0.88 0.78 0.88
37 0 1 0.25
38 1 1 1
39 1 1 1
40 0.96 0.96 1
41 0.63 0.58 0.60
42 1 1 1
43 0.80 0.88 0.84
44 0.75 0.63 0.50
45 0.60 0.42 0.17
46 1 0.71 0.92
47 1 0.57 1
48 0.41 1 0.78
49 1 1 0.75
50 0.73 0.44 0.78
51 0.33 0.63 0.27
52 1 1 1
53 0.67 0.60 0.60
54 1 0.50 1
55 0.98 0.96 0.98
56 1 1 1
57 0.90 0.80 1
58 1 1 0.89
59 0.03 0.43 0.20
60 0.43 0.59 0.52
61 1 1 1
62 1 0.96 0.94
63 0.33 0.11 1
64 0.38 0.28 0.38
65 1 1 1
66 1 1 0.50
67 1 1 0.50
68 0.56 0.55 0.56
69 0.77 0.67 0.77
70 1 0.78 0.50
71 1 0 0.50
72 0.69 0.82 0.68
73 0.87 0.82 0.89
74 1 0.80 1
75 0.71 0.85 0.83
76 1 1 1
77 0 0 0.12
78 1 0 0
79 0.65 0.38 0.75
80 0.73 1 1
81 1 1 1
82 1 1 1
83 1 0.88 1
84 1 0.50 1
85 0.63 0.16 0.86
86 1 1 1
87 0.33 0.69 0.45
88 1 1 1
89 0.83 0.42 1
90 0.25 1 1
91 1 1 1
92 0.25 0.13 0.13
93 1 1 1
94 0.42 0.29 0.36
95 0.91 0.94 1
96 1 1 1
97 1 1 0.88
98 0.50 0.50 0.50
99 0.88 0.79 0.70
100 1 1 1
101 0.83 0.57 0.83
102 0.88 1 1
103 1 1 1
104 0.98 1 1
105 0.94 0.94 0.96
106 0.60 0.59 0.48
107 0.73 0.68 0.82
108 0.48 0.46 0.69
109 0.90 0.65 0.89
110 1 0.97 1
111 0.89 0.82 0.78
112 0.78 0.94 0.94
113 1 1 0.90
114 0.80 0.80 0.90
115 0.48 0.31 0.25
116 0.53 0.54 0.63
117 1 1 1
118 0.47 1 1
119 0.38 0.67 0.67
120 0.67 0.67 1
121 0.33 0.40 0.17
122 0.08 0.50 0.39
123 0.38 0.33 0.33
124 0.33 0.33 0.33
125 0.70 0.17 0.67
126 0.23 0.67 0.67
127 1 1 1
128 0.67 0.67 0.67
129 0.50 0.25 0
130 0.58 0.50 1
131 0.67 1 1
132 0.75 1 0.67
133 0.17 0.14 0.20
134 1 1 1
135 1 1 1
136 0.50 1 1

download these results as csv

Task 1a Summary Results by Query[edit]

Simple Hit/Miss Counting [1]

MRR Method [2]


Task 1b, Hsiao's Dataset Results[edit]

Task 1b Overall Results[edit]
CSJ HAFR HL
Simple Count 0.82 0.88 0.90
MRR 0.66 0.70 0.69
Total count 410.00 410.00 410.00

download these results as csv

Task 1b Friedman's Test for Significant Differences[edit]

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID TeamID Lowerbound Mean Upperbound Significance
HL HAFR -0.1073 0.0352 0.1776 FALSE
HL CSJ 0.0451 0.1875 0.3299 TRUE
HAFR CSJ 0.0099 0.1523 0.2948 TRUE

download these results as csv

2009 sqbttask1bsimple friedman mean ranks.png

MRR Method:

TeamID TeamID Lowerbound Mean Upperbound Significance
HAFR HL -0.1002 0.1328 0.3659 FALSE
HAFR CSJ -0.0260 0.2070 0.4401 FALSE
HL CSJ -0.1588 0.0742 0.3073 FALSE

download these results as csv

2009 sqbttask1bmrr friedman mean ranks.png

Task 1b Summary Results by Query Group[edit]

Simple Hit/Miss Counting

CSJ HAFR HL
1 1 1 1
2 NaN NaN NaN
3 1 1 1
4 1 1 1
5 1 1 1
6 1 1 1
7 1 1 1
8 1 1 1
9 1 1 1
10 0.67 0.67 0.67
11 1 1 1
12 0.75 1 0.50
13 1 1 1
14 1 1 1
15 NaN NaN NaN
16 1 1 1
17 1 1 1
18 1 1 1
19 1 1 1
20 1 1 1
21 0.67 1 1
22 1 1 1
23 1 0.83 0.83
24 0.67 0.67 0.67
25 0 1 1
26 0 0 0
27 1 1 1
28 NaN NaN NaN
29 1 1 1
30 1 1 1
31 NaN NaN NaN
32 1 1 1
33 1 1 1
34 1 1 1
35 NaN NaN NaN
36 0 0 0
37 0.50 0 0.50
38 1 0.86 1
39 1 1 1
40 0 0.50 0.50
41 1 1 1
42 0.67 1 1
43 1 1 1
44 0 1 0.50
45 1 1 1
46 1 1 1
47 1 1 1
48 1 1 1
49 1 0.67 1
50 1 1 1
51 1 1 1
52 1 1 1
53 0.89 0.89 0.89
54 0.33 1 1
55 0.50 1 1
56 1 0.50 1
57 1 1 1
58 NaN NaN NaN
59 1 1 1
60 1 1 1
61 0.75 0.75 1
62 0 0.50 0.50
63 1 1 1
64 NaN NaN NaN
65 1 1 1
66 NaN NaN NaN
67 1 1 1
68 NaN NaN NaN
69 0.40 1 1
70 0.20 1 1
71 NaN NaN NaN
72 1 1 1
73 1 1 0.50
74 NaN NaN NaN
75 1 1 1
76 0.40 1 0.40
77 NaN NaN NaN
78 1 1 1
79 0.50 1 1
80 1 1 0
81 0 1 1
82 0 1 1
83 1 1 1
84 1 1 1
85 0 0 1
86 1 1 1
87 1 1 1
88 0.80 0.60 0.80
89 0.80 1 0.20
90 1 1 1
91 1 1 1
92 1 1 0.80
93 0.50 1 1
94 1 0.75 1
95 0.67 0.67 0.67
96 1 1 1
97 0 0 0
98 1 1 1
99 1 1 1
100 1 1 1
101 1 1 1
102 1 1 1
103 NaN NaN NaN
104 1 1 1
105 1 1 1
106 0 0.33 0
107 NaN NaN NaN
108 1 1 1
109 1 1 1
110 0 1 1
111 1 1 1
112 0.67 1 0.67
113 1 1 1
114 0.50 0.50 0.50
115 NaN NaN NaN
116 0 0 1
117 1 1 1
118 1 1 1
119 0 0 0
120 0.67 0.67 0.67
121 1 0.67 1
122 1 1 1
123 1 1 1
124 1 1 1
125 1 0 1
126 1 1 1
127 1 1 1
128 1 1 1
129 1 1 1
130 0 0 0
131 0 0 1
132 1 1 1
133 0.33 1 0.67
134 0.50 0.50 0.50
135 1 1 1
136 1 1 1
137 1 1 1
138 1 1 1
139 0.60 1 1
140 0 0 1
141 1 1 1
142 1 1 1
143 1 1 1

download these results as csv

MRR Method

CSJ HAFR HL
1 0.56 1 1
2 NaN NaN NaN
3 0.88 0.83 0.83
4 0.50 0.75 0.50
5 1 1 1
6 1 0.50 1
7 1 1 1
8 1 1 1
9 1 1 1
10 0.44 0.50 0.67
11 1 1 1
12 0.75 0.81 0.50
13 1 1 1
14 1 0.42 1
15 NaN NaN NaN
16 1 1 1
17 1 1 1
18 1 1 1
19 1 1 1
20 0.83 1 0.61
21 0.45 1 1
22 1 1 0.88
23 0.88 0.47 0.64
24 0.67 0.67 0.67
25 0 0.75 0.15
26 0 0 0
27 0.50 0.67 0.44
28 NaN NaN NaN
29 1 1 0.75
30 1 1 0.75
31 NaN NaN NaN
32 1 0.50 1
33 0.33 0.56 0.50
34 0.78 0.61 0.58
35 NaN NaN NaN
36 0 0 0
37 0.13 0 0.25
38 0.89 0.26 0.90
39 0.18 1 1
40 0 0.07 0.08
41 1 1 0.50
42 0.50 0.83 0.75
43 0.75 0.75 0.88
44 0 0.42 0.10
45 0.33 0.56 0.22
46 0.78 1 1
47 1 1 0.75
48 1 1 1
49 0.53 0.14 0.28
50 1 1 1
51 0.10 0.13 0.17
52 1 1 1
53 0.89 0.83 0.89
54 0.11 0.37 0.32
55 0.06 1 0.75
56 0.42 0.25 0.67
57 0.80 0.90 0.60
58 NaN NaN NaN
59 1 0.50 0.50
60 1 0.63 1
61 0.50 0.75 0.54
62 0 0.50 0.50
63 1 0.67 0.50
64 NaN NaN NaN
65 0.69 0.49 1
66 NaN NaN NaN
67 0.58 0.20 0.67
68 NaN NaN NaN
69 0.05 0.85 1
70 0.20 0.41 0.90
71 NaN NaN NaN
72 1 1 0.60
73 1 1 0.50
74 NaN NaN NaN
75 1 1 1
76 0.20 0.85 0.30
77 NaN NaN NaN
78 0.67 0.83 0.67
79 0.13 0.58 0.88
80 0.13 0.33 0
81 0 0.35 0.29
82 0 0.50 0.25
83 1 1 1
84 1 0.63 1
85 0 0 0.25
86 1 1 1
87 1 1 1
88 0.67 0.43 0.47
89 0.38 0.69 0.02
90 0.83 0.78 0.53
91 0.88 1 0.88
92 1 1 0.80
93 0.50 1 1
94 0.33 0.16 0.29
95 0.67 0.67 0.67
96 1 1 1
97 0 0 0
98 1 0.75 1
99 1 1 1
100 1 1 1
101 1 1 0.80
102 1 1 1
103 NaN NaN NaN
104 0.33 1 1
105 0.35 0.33 0.40
106 0 0.11 0
107 NaN NaN NaN
108 1 1 1
109 0.33 0.33 0.33
110 0 0.10 0.17
111 0.50 0.50 0.13
112 0.50 0.78 0.20
113 1 1 1
114 0.50 0.17 0.50
115 NaN NaN NaN
116 0 0 0.60
117 0.75 0.75 0.75
118 0.83 0.67 0.33
119 0 0 0
120 0.21 0.50 0.20
121 0.56 0.67 0.71
122 0.65 0.88 0.43
123 0.79 0.78 1
124 0.70 0.70 0.70
125 0.19 0 0.23
126 1 1 1
127 0.50 0.88 0.50
128 0.56 0.78 0.50
129 1 1 1
130 0 0 0
131 0 0 0.29
132 1 0.75 0.63
133 0.03 0.48 0.15
134 0.50 0.06 0.07
135 1 1 0.78
136 0.56 1 1
137 1 1 1
138 0.23 0.38 0.33
139 0.35 0.44 0.80
140 0 0 0.50
141 0.56 0.69 0.42
142 0.50 0.50 0.39
143 1 1 1

download these results as csv

Task 1b Summary Results by Query[edit]

Simple Hit/Miss Counting [3]

MRR Method [4]

Task 2 Results[edit]

Task 2 Overall Results[edit]
CSJ HAFR HL
Simple Count 0.86 0.79 0.84
MRR 0.77 0.66 0.73
Total count 890 890 880

download these results as csv

Task 2 Friedman's Test for Significant Differences[edit]

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID TeamID Lowerbound Mean Upperbound Significance
CSJ HL -0.1404 0.0515 0.2434 FALSE
CSJ HAFR 0.0103 0.2022 0.3941 TRUE
HL HAFR -0.0412 0.1507 0.3426 FALSE

download these results as csv

2009 sqbttask2simple friedman mean ranks.png

MRR Method:

TeamID TeamID Lowerbound Mean Upperbound Significance
CSJ HL -0.1350 0.1066 0.3482 FALSE
CSJ HAFR 0.1702 0.4118 0.6533 TRUE
HL HAFR 0.0636 0.3051 0.5467 TRUE

download these results as csv

2009 sqbttask2mrr friedman mean ranks.png

Task 2 Summary Results by Query Group[edit]

Simple Hit/Miss Counting

CSJ HAFR HL
1 1 0.69 0.77
2 1 1 1
3 1 1 1
4 1 0 1
5 1 1 1
6 1 1 0.75
7 1 1 1
8 1 1 1
9 1 1 1
10 1 1 1
11 0.50 1 1
12 1 1 1
13 1 1 1
14 1 1 1
15 1 1 1
16 1 0.90 0.85
17 0.80 0.93 1
18 0.77 0.62 0.69
19 0.38 0.75 1
20 1 1 1
21 1 1 1
22 1 1 0.50
23 0 0 0
24 1 1 1
25 1 1 1
26 0 0 1
27 1 0.71 0.88
28 1 0.67 1
29 1 1 1
30 0.67 0.67 0.75
31 1 0 1
32 1 1 1
33 0.71 0.59 0.82
34 0.83 0.92 0.92
35 1 1 1
36 1 1 1
37 0 1 1
38 1 1 1
39 1 1 1
40 1 0.86 0.93
41 0.75 0.83 0.92
42 1 1 1
43 0.80 0.90 0.90
44 0.75 0.75 0.50
45 1 1 0.50
46 1 0.83 1
47 1 1 1
48 1 1 1
49 1 1 0.50
50 1 1 1
51 0.60 0.80 0.80
52 1 1 1
53 0.80 0.20 0.60
54 1 1 1
55 1 0.84 0.98
56 1 1 1
57 1 0.80 0.80
58 1 1 1
59 0.20 0.40 0.20
60 0.43 0.57 0.57
61 1 0.82 0.91
62 0.92 0.75 0.83
63 1 0 1
64 0.50 0.50 0.50
65 1 1 1
66 1 1 0.50
67 1 1 1
68 0.56 0.56 0.61
69 0.82 0.64 0.91
70 1 0.75 0.75
71 1 0 1
72 0.83 0.92 0.75
73 0.86 0.71 0.93
74 1 1 1
75 0.83 0.83 0.83
76 1 1 1
77 0 0 1
78 1 0 0
79 1 0.75 0.75
80 1 1 1
81 1 1 1
82 1 1 1
83 1 1 1
84 1 1 1
85 0.86 0.43 0.71
86 1 1 1
87 0.58 0.67 0.67
88 1 1 1
89 1 1 1
90 0.50 1 1
91 1 1 1
92 0.25 0.25 0.25
93 1 1 1
94 0.50 0.83 0.67
95 1 0.88 1
96 1 1 1
97 1 1 1
98 0.50 0.50 0.50
99 0.81 0.88 0.75
100 1 1 1
101 0.83 1 0.83
102 0.88 1 1
103 1 1 1
104 0.96 0.89 0.89
105 1 0.81 0.89
106 0.64 0.73 0.45
107 0.94 0.72 0.89
108 0.69 0.44 0.81
109 0.88 0.81 0.75
110 0.93 0.93 0.87
111 1 1 1
112 1 1 0.88
113 0.86 0.71 0.86
114 1 0.60 0.60
115 0.80 0.60 0.40
116 0.50 0.75 0.75
117 1 1 1
118 1 1 1
119 0.67 0.67 1
120 1 0.67 1
121 1 1 1
122 0.33 0.67 1
123 0.67 0.67 1
124 0.33 0.33 0.50
125 0.67 1 1
126 1 1 1
127 1 1 0.67
128 0.67 0.67 0.67
129 0.50 0.50 0
130 1 0.50 1
131 1 1 1
132 1 1 1
133 1 0 0
134 1 1 1
135 1 1 1
136 1 0 0

download these results as csv

MRR Method

CSJ HAFR HL
1 0.94 0.69 0.77
2 1 1 1
3 1 1 1
4 1 0 1
5 1 0.67 1
6 0.78 0.83 0.75
7 1 1 1
8 0.75 1 0.50
9 1 0.10 1
10 1 1 1
11 0.50 0.67 0.63
12 1 1 1
13 1 1 0.50
14 1 0.89 0.94
15 1 1 1
16 0.91 0.70 0.32
17 0.74 0.79 0.81
18 0.71 0.34 0.59
19 0.28 0.33 0.44
20 1 1 1
21 0.55 1 1
22 1 0.55 0.50
23 0 0 0
24 1 0.50 1
25 0.90 0.93 0.89
26 0 0 0.14
27 0.94 0.63 0.68
28 0.83 0.37 1
29 1 1 1
30 0.60 0.53 0.61
31 1 0 1
32 1 1 0.72
33 0.43 0.56 0.76
34 0.83 0.92 0.92
35 1 1 1
36 0.88 0.58 0.88
37 0 1 0.25
38 1 1 1
39 1 1 1
40 0.96 0.82 0.93
41 0.52 0.47 0.60
42 1 1 1
43 0.73 0.78 0.83
44 0.75 0.75 0.50
45 0.58 0.33 0.17
46 1 0.63 0.92
47 1 1 1
48 0.42 0.43 0.78
49 1 1 0.25
50 0.73 0.51 0.78
51 0.32 0.70 0.47
52 1 1 0.63
53 0.67 0.20 0.60
54 1 0.33 1
55 0.98 0.75 0.98
56 1 1 1
57 1 0.80 0.80
58 1 0.91 0.89
59 0.03 0.09 0.20
60 0.43 0.40 0.50
61 1 0.82 0.91
62 0.92 0.69 0.79
63 0.33 0 1
64 0.38 0.50 0.38
65 1 1 1
66 1 1 0.50
67 1 0.25 0.50
68 0.50 0.44 0.56
69 0.76 0.49 0.75
70 1 0.54 0.50
71 1 0 0.50
72 0.69 0.80 0.61
73 0.86 0.61 0.93
74 1 0.90 1
75 0.71 0.83 0.75
76 1 1 1
77 0 0 0.12
78 1 0 0
79 0.65 0.38 0.75
80 0.73 1 1
81 1 1 1
82 1 1 1
83 1 0.88 1
84 1 0.50 1
85 0.66 0.22 0.62
86 1 1 1
87 0.33 0.50 0.37
88 1 0.75 1
89 0.83 0.58 0.83
90 0.25 0.56 1
91 1 0.87 1
92 0.25 0.06 0.08
93 1 1 1
94 0.42 0.49 0.36
95 0.91 0.70 1
96 1 0.50 1
97 1 0.79 0.88
98 0.50 0.50 0.50
99 0.81 0.79 0.69
100 1 1 1
101 0.83 0.52 0.83
102 0.88 1 1
103 1 0.89 1
104 0.90 0.78 0.85
105 0.97 0.81 0.89
106 0.58 0.58 0.40
107 0.78 0.67 0.66
108 0.47 0.33 0.57
109 0.83 0.62 0.65
110 0.88 0.90 0.87
111 1 0.81 1
112 0.88 1 0.88
113 0.76 0.71 0.64
114 0.80 0.44 0.37
115 0.30 0.35 0.22
116 0.50 0.58 0.35
117 1 0.75 1
118 0.47 1 1
119 0.38 0.50 1
120 0.83 0.67 1
121 0.83 0.83 0.75
122 0.08 0.25 0.25
123 0.40 0.44 0.57
124 0.33 0.17 0.50
125 0.67 0.28 0.75
126 0.48 1 0.75
127 1 1 0.67
128 0.67 0.67 0.67
129 0.50 0.05 0
130 1 0.50 1
131 1 1 1
132 0.75 0.60 1
133 0.14 0 0
134 1 1 1
135 1 1 1
136 0.20 0 0

download these results as csv

Task 2 Summary Results by Query[edit]

Simple Hit/Miss Counting [5]

MRR Method [6]

Runtime Results[edit]

Participant Task Runtime (min) Machine
CSJ 1a ~30 BEER 2
CSJ 1b 7 BEER 2
CSJ 2 ~35 BEER 2
HAFR 1a 15 BEER 2
HAFR 1b 5 BEER 2
HAFR 2 35 BEER 2
HL 1a ~95 FAST
HL 1b ~40 FAST
HL 2 ~95 FAST

download these results as csv