Tackling Winograd Schemas
by Formalizing Relevance Theory in Knowledge Graphs

Conference: International Conference on Principles of Knowledge Representation and Reasoning 2014

Author: Peter Schüller, Marmara University, İstanbul, Turkey

This work has been supported by Marmara University, Scientific Research Research Projects Committee (BAPKO Project No. FEN-D-120514-0177, 2014).

Contents

• Publication
• Software
• How to execute benchmarks
• Visualizations of Knowledge Graphs
• Benchmark Results

Publication

The paper can be downloaded here.

Software

Knowledge graphs of the four Winograd Schemas in our tests are encoded in the following files.

• schoolbag.graph
• drain.graph
• drawing.graph
• pillar.graph

Background knowledge graphs and graph-based reasoning as described in the submission is implemented in the following files.

• graphprocessor.4.2.1.gringo (compatible with clingo 4.2.1)
• graphprocessor.4.5.4.gringo (compatible with clingo 4.5.4)
• mixer.gringo
• costfunctionhelpers.gringo
• min_diameter.gringo
• background.graphs
• background.ontology

How to Execute Benchmarks

The benchmarks were performed using clingo-4.2.1. Note that previous clingo versions do not work due to a bug that was fixed in version 4.2.1.

Experiments were performed using the following commandline, where <input> is one of the input graph files, <schema> is either 1 or 2 (selecting the first or second variant of the schema).

The relevance fitness function is defined by <frel> and composed from the following components where C_* are constants as described in the submission.

• C_com_in,N : nodestats(N,orig,Coll,0)
• C_bg+C_com_bg*Coll+C_dis_bg*Dist,N : nodestats(N,new,Coll,Dist)
• C_om,G,A : activated_unused(G,A)
• C_rad*N : diameter(N)

For example to configure the third line of Table 1 [3 (0,1,-2) -10 -5] you would create the following configuration file example-config.gringo: (percent signs are comments)
  schema1.
  mix.
  #maximize {
    % C_com_in = 3
    N : nodestats(N,orig,Coll,0) ;
    % C_bg = 0, C_com_bg = 1, C_dis_bg = -2
    1*Coll-2*Dist,N : nodestats(N,new,Coll,Dist) ;
    % C_om = -10
    0,G,A : activated_unused(G,A) ;
    % C_rad = -5
    -5*N : diameter(N)
}.

The program is then executed as follows, where you need to use the graphprocessor.gringo compatible with your clingo version and replace schoolbag.graph with the graph you want to use.
  $ clingo-4.2.1-bin --configuration=trendy schoolbag.graph example-config.gringo \
    graphprocessor.4.2.1.gringo mixer.gringo costfunctionhelpers.gringo min_diameter.gringo \
    background.graphs background.ontology --project 0

The output contains answer sets and graphviz-compatible output for visualization. At the moment there is no option to write the graphviz file to a separate file, you need to extract it with a script or manually from the output to render it with neato.

Visualization of Knowledge Graphs of Winograd Schemas in the KR 2014 Paper

Knowledge Graph of (5) plus activated Background Knowledge Graphs

Knowledge Graph of (6) plus activated Background Knowledge Graphs

Knowledge Graph of (7) plus activated Background Knowledge Graphs

Knowledge Graph of (8) plus activated Background Knowledge Graphs

Knowledge Graph of (9) plus activated Background Knowledge Graphs

Knowledge Graph of (10) plus activated Background Knowledge Graphs

Knowledge Graph of (11) plus activated Background Knowledge Graphs

Knowledge Graph of (12) plus activated Background Knowledge Graphs

Benchmark Results: Accumulated Over Configurations

(Sorted by correctness percentage, descending, and then by number of result graphs, ascending.)

C_com_in	(C_bg,C_com_bg,C_dis_bg)	C_om	C_rad	correct	results	correct	time to optimum		time to enumerate
				#	#	%	sec	# timeouts	sec	# timeouts
3	(0,1,-2)	0	-5	10	10	100.0	731	0	587	0
3	(0,1,-2)	0	-10	10	10	100.0	645	0	421	0
3	(0,1,-2)	-10	-5	10	10	100.0	598	0	520	0
3	(0,1,-2)	-10	-10	10	10	100.0	708	0	468	0
3	(0,0,-1)	0	-5	10	10	100.0	490	0	464	0
3	(0,0,-1)	-10	-5	10	10	100.0	687	0	533	0
1	(0,0,-1)	-10	-5	10	10	100.0	705	0	610	0
3	(0,0,-1)	-10	-10	10	10	100.0	585	0	375	0
2	(-1,0,0)	0	-5	10	10	100.0	269	0	297	0
3	(-1,0,0)	0	-5	10	10	100.0	364	0	237	0
3	(0,1,-1)	0	-5	10	10	100.0	815	0	950	0
3	(0,0,-1)	-30	-5	11	11	100.0	599	0	706	0
2	(0,1,-2)	-30	-10	11	11	100.0	387	0	341	0
2	(-1,0,0)	-10	-5	11	11	100.0	226	0	277	0
3	(0,1,-2)	-30	-5	11	11	100.0	520	0	461	0
3	(0,1,-2)	-20	-10	11	11	100.0	658	0	346	0
1	(0,0,-1)	-30	-5	11	11	100.0	671	0	526	0
3	(-1,0,0)	-10	-10	11	11	100.0	229	0	226	0
3	(-1,0,0)	-10	-5	11	11	100.0	270	0	331	0
2	(-1,0,0)	-30	-5	11	11	100.0	197	0	189	0
3	(0,0,-1)	-30	-10	11	11	100.0	364	0	311	0
3	(0,1,-1)	-30	-5	11	11	100.0	782	0	619	0
1	(-1,0,0)	-10	-5	11	11	100.0	219	0	199	0
3	(0,1,-2)	-30	-10	11	11	100.0	352	0	368	0
2	(0,0,-1)	-30	-5	11	11	100.0	612	0	571	0
3	(0,1,-1)	-20	-5	11	11	100.0	786	0	768	0
2	(0,0,-1)	-20	-5	11	11	100.0	639	0	851	0
2	(0,1,-2)	-30	-5	11	11	100.0	605	0	496	0
3	(-1,0,0)	-20	-10	11	11	100.0	202	0	185	0
3	(0,0,-1)	-20	-5	11	11	100.0	631	0	502	0
3	(0,0,-1)	-20	-10	11	11	100.0	465	0	456	0
1	(-1,0,0)	-30	-5	11	11	100.0	163	0	150	0
2	(-1,0,0)	-20	-5	11	11	100.0	236	0	165	0
3	(0,1,-2)	-20	-5	11	11	100.0	465	0	573	0
2	(0,1,-2)	-20	-10	11	11	100.0	426	0	277	0
2	(0,1,-2)	-20	-5	11	11	100.0	476	0	474	0
1	(0,0,-1)	-20	-5	11	11	100.0	873	0	489	0
1	(-1,0,0)	-20	-5	11	11	100.0	138	0	139	0
3	(-1,0,0)	-30	-5	11	11	100.0	198	0	187	0
3	(-1,0,0)	-30	-10	11	11	100.0	188	0	165	0
3	(-1,0,0)	-20	-5	11	11	100.0	254	0	259	0
1	(0,0,-1)	0	0	59	59	100.0	421	0	393	0
1	(0,1,-2)	0	0	67	67	100.0	426	0	520	0
2	(0,1,-2)	0	0	89	89	100.0	206	0	366	0
1	(0,1,-2)	-10	0	89	89	100.0	242	0	231	0
2	(0,1,-2)	-10	0	89	89	100.0	200	0	247	0
0	(0,0,-1)	-10	0	89	89	100.0	50	0	36	0
3	(-1,0,0)	0	0	89	89	100.0	104	0	102	0
3	(0,1,-2)	-10	0	89	89	100.0	193	0	260	0
1	(0,0,-1)	-10	0	89	89	100.0	390	0	470	0
3	(0,0,-1)	0	0	89	89	100.0	196	0	133	0
2	(-1,0,0)	0	0	89	89	100.0	85	0	123	0
3	(0,1,-2)	0	0	89	89	100.0	226	0	255	0
2	(0,0,-1)	-10	0	89	89	100.0	311	0	231	0
2	(0,0,-1)	0	0	91	91	100.0	339	0	227	0
1	(-1,0,0)	0	0	91	91	100.0	132	0	180	0
1	(0,1,-1)	-30	0	122	122	100.0	281	0	362	0
2	(0,1,-1)	-20	0	122	122	100.0	242	0	395	0
2	(-1,0,0)	-30	0	122	122	100.0	99	0	76	0
3	(0,1,-1)	-30	0	122	122	100.0	187	0	423	0
1	(-1,0,0)	-10	0	122	122	100.0	101	0	100	0
1	(0,1,-2)	-20	0	122	122	100.0	143	0	292	0
3	(0,0,-1)	-30	0	122	122	100.0	150	0	233	0
0	(-1,0,0)	-30	0	122	122	100.0	37	0	38	0
2	(0,0,-1)	-30	0	122	122	100.0	321	0	176	0
3	(0,1,-2)	-30	0	122	122	100.0	152	0	218	0
3	(-1,0,0)	-10	0	122	122	100.0	115	0	102	0
1	(0,1,-1)	-10	0	122	122	100.0	271	0	774	0
3	(0,1,-1)	-20	0	122	122	100.0	252	0	376	0
2	(-1,0,0)	-10	0	122	122	100.0	92	0	121	0
0	(-1,0,0)	-20	0	122	122	100.0	40	0	33	0
2	(0,1,-1)	-10	0	122	122	100.0	298	0	496	0
3	(0,0,-1)	-20	0	122	122	100.0	247	0	122	0
0	(0,0,-1)	-30	0	122	122	100.0	47	0	35	0
2	(0,1,-1)	-30	0	122	122	100.0	234	0	488	0
2	(0,1,-2)	-20	0	122	122	100.0	188	0	197	0
1	(-1,0,0)	-30	0	122	122	100.0	87	0	98	0
1	(0,0,-1)	-20	0	122	122	100.0	233	0	288	0
2	(0,1,-2)	-30	0	121	122	99.2	221	0	177	0
3	(0,0,-1)	-10	0	121	122	99.2	204	0	181	0
3	(0,1,-1)	-10	0	121	122	99.2	252	0	412	0
2	(0,0,0)	-10	0	158	160	98.8	71	0	41	0
0	(0,0,-1)	-20	0	120	122	98.4	46	0	33	0
0	(-1,0,0)	-10	0	115	122	94.3	34	0	35	0
3	(0,1,-1)	0	0	112	122	91.8	370	0	566	0
2	(0,0,-1)	-20	0	112	122	91.8	294	0	217	0
3	(0,1,-1)	-30	-10	11	12	91.7	613	0	675	0
2	(0,0,-1)	-30	-10	11	12	91.7	440	0	278	0
2	(-1,0,0)	-10	-10	11	12	91.7	228	0	177	0
2	(0,0,-1)	-20	-10	11	12	91.7	589	0	372	0
3	(-1,0,0)	-20	-15	11	12	91.7	192	0	175	0
3	(-1,0,0)	-30	-15	11	12	91.7	169	0	146	0
1	(0,1,-1)	-20	0	111	122	91.0	370	0	397	0
3	(0,1,-2)	-20	-15	10	11	90.9	472	0	380	0
1	(0,0,-1)	-30	-15	10	11	90.9	446	0	179	0
1	(0,0,-1)	-30	-10	10	11	90.9	500	0	294	0
2	(0,1,-1)	-20	-15	10	11	90.9	349	0	388	0
1	(-1,0,0)	-20	-15	10	11	90.9	144	0	99	0
1	(-1,0,0)	-20	-10	10	11	90.9	99	0	129	0
2	(-1,0,0)	-30	-15	10	11	90.9	144	0	120	0
2	(-1,0,0)	-30	-10	10	11	90.9	198	0	135	0
3	(0,0,-1)	-30	-15	10	11	90.9	499	0	184	0
2	(0,1,-2)	-10	-10	10	11	90.9	704	0	346	0
3	(0,1,-1)	-30	-15	10	11	90.9	564	0	424	0
1	(-1,0,0)	-30	-10	10	11	90.9	161	0	101	0
2	(0,1,-1)	-10	-15	10	11	90.9	656	0	341	0
2	(0,1,-1)	-10	-10	10	11	90.9	955	0	706	0
3	(0,1,-2)	-30	-15	10	11	90.9	451	0	227	0
2	(0,1,-1)	-20	-10	10	11	90.9	515	0	348	0
3	(-1,0,0)	-10	-15	10	11	90.9	241	0	184	0
2	(0,0,-1)	-30	-15	10	11	90.9	343	0	396	0
3	(0,1,-1)	-20	-15	10	11	90.9	790	0	393	0
2	(0,0,-1)	-20	-15	10	11	90.9	439	0	431	0
3	(0,0,-1)	0	-10	10	11	90.9	407	0	292	0
2	(0,1,-2)	-30	-15	10	11	90.9	407	0	209	0
3	(0,0,-1)	-20	-15	10	11	90.9	488	0	232	0
2	(-1,0,0)	-20	-10	10	11	90.9	190	0	181	0
2	(-1,0,0)	-20	-15	10	11	90.9	138	0	114	0
2	(0,1,-1)	-30	-15	10	11	90.9	450	0	351	0
2	(0,1,-1)	-30	-10	10	11	90.9	721	0	582	0
1	(-1,0,0)	-30	-15	10	11	90.9	83	0	86	0
3	(0,1,-1)	-10	-15	10	11	90.9	458	0	423	0
1	(0,0,-1)	-20	-10	10	11	90.9	362	0	226	0
1	(0,0,-1)	-20	-15	10	11	90.9	428	0	207	0
1	(-1,0,0)	-10	-10	9	10	90.0	156	0	152	0
3	(-1,0,0)	0	-10	9	10	90.0	280	0	216	0
2	(-1,0,0)	-10	-15	9	10	90.0	161	0	120	0
2	(0,1,-2)	-20	-15	9	10	90.0	484	0	316	0
3	(0,1,-2)	-20	0	108	122	88.5	186	0	222	0
1	(0,1,-2)	-30	0	107	122	87.7	151	0	188	0
3	(-1,0,0)	-30	0	107	122	87.7	89	0	91	0
3	(-1,0,0)	-20	0	106	122	86.9	103	0	86	0
2	(-1,0,0)	-20	0	106	122	86.9	96	0	97	0
1	(-1,0,0)	-20	0	106	122	86.9	86	0	80	0
2	(0,0,-1)	0	-10	10	12	83.3	394	0	412	0
1	(-1,0,0)	0	-5	14	17	82.4	194	0	309	0
3	(-1,0,0)	0	-15	9	11	81.8	258	0	153	0
3	(0,1,-1)	0	-10	9	11	81.8	554	0	702	0
2	(0,0,-1)	-10	-10	9	11	81.8	397	0	450	0
2	(0,0,-1)	0	-15	8	10	80.0	388	0	201	0
2	(0,1,-2)	0	-15	8	10	80.0	412	0	520	0
2	(0,1,-2)	-10	-15	8	10	80.0	432	0	349	0
1	(-1,0,0)	-10	-15	8	10	80.0	125	0	103	0
3	(0,1,-1)	0	-15	8	10	80.0	643	0	791	0
3	(0,1,-2)	-10	-15	8	10	80.0	331	0	270	0
3	(0,0,-1)	-10	-15	8	10	80.0	349	0	375	0
1	(0,0,-1)	-10	-15	8	10	80.0	397	0	211	0
1	(0,0,-1)	-10	-10	8	10	80.0	519	0	223	0
2	(0,1,-1)	0	-10	8	10	80.0	760	0	778	0
2	(0,1,-1)	0	-15	8	10	80.0	519	0	661	0
3	(0,0,-1)	0	-15	8	10	80.0	393	0	198	0
2	(-1,0,0)	0	-10	8	10	80.0	309	0	174	0
2	(-1,0,0)	0	-15	8	10	80.0	258	0	190	0
3	(0,1,-2)	0	-15	8	10	80.0	524	0	499	0
2	(0,0,-1)	-10	-15	8	10	80.0	552	0	287	0
1	(0,1,-2)	-20	-5	16	20	80.0	603	0	501	0
1	(0,0,-1)	-30	0	97	122	79.5	262	0	313	0
0	(0,1,-2)	-30	0	98	124	79.0	56	0	39	0
3	(0,0,0)	-30	-5	59	75	78.7	160	0	138	0
1	(0,0,-1)	0	-5	10	13	76.9	515	0	460	0
0	(0,0,-1)	-20	-15	16	21	76.2	68	0	32	0
2	(0,0,0)	0	-5	56	74	75.7	109	0	113	0
1	(0,1,-1)	-30	-10	15	20	75.0	440	0	465	0
1	(0,1,-1)	-30	-15	15	20	75.0	374	0	230	0
1	(0,1,-2)	-20	-10	15	20	75.0	306	0	307	0
0	(-1,0,0)	-30	-10	15	20	75.0	46	0	32	0
0	(-1,0,0)	-30	-15	15	20	75.0	40	0	30	0
0	(-1,0,0)	-30	-5	15	20	75.0	52	0	34	0
1	(0,1,-1)	-20	-15	15	20	75.0	409	0	331	0
1	(0,1,-1)	-20	-10	15	20	75.0	671	0	573	0
0	(0,0,-1)	-20	-10	15	20	75.0	58	0	35	0
0	(0,0,-1)	-20	-5	15	20	75.0	73	0	31	0
1	(0,1,-1)	-10	-15	15	20	75.0	588	0	402	0
0	(-1,0,0)	-20	-5	15	20	75.0	46	0	30	0
0	(-1,0,0)	-20	-15	15	20	75.0	42	0	29	0
0	(-1,0,0)	-20	-10	15	20	75.0	44	0	30	0
0	(-1,0,0)	-10	-5	15	20	75.0	50	0	33	0
0	(0,0,-1)	-30	-5	15	20	75.0	61	0	29	0
0	(0,0,-1)	-30	-15	15	20	75.0	59	0	29	0
0	(0,0,-1)	-30	-10	15	20	75.0	63	0	32	0
1	(0,1,-2)	-30	-15	15	20	75.0	308	0	228	0
1	(0,1,-2)	-30	-10	15	20	75.0	594	0	366	0
3	(0,0,0)	-10	0	119	160	74.4	83	0	47	0
3	(0,0,0)	-20	-5	55	75	73.3	201	0	166	0
1	(0,1,-2)	-20	-15	13	18	72.2	347	0	308	0
0	(0,1,-2)	-20	0	89	124	71.8	48	0	45	0
0	(0,0,0)	-30	-10	126	178	70.8	43	0	28	0
2	(0,0,0)	-30	-5	53	75	70.7	145	0	153	0
2	(0,0,0)	0	0	110	160	68.8	68	0	51	0
3	(0,0,0)	-20	0	110	160	68.8	72	0	47	0
1	(0,0,0)	0	0	110	160	68.8	65	0	48	0
3	(0,0,0)	0	0	110	160	68.8	65	0	49	0
0	(0,1,-2)	-30	-5	6	9	66.7	79	0	48	0
0	(0,1,-2)	-20	-5	6	9	66.7	68	0	64	0
1	(-1,0,0)	0	-10	12	18	66.7	198	0	207	0
2	(0,0,0)	-20	-5	49	75	65.3	170	0	120	0
0	(0,1,-2)	-10	-5	7	11	63.6	61	0	58	0
0	(0,0,0)	-10	-15	98	154	63.6	43	0	28	0
1	(0,1,-1)	0	-15	12	19	63.2	548	0	574	0
1	(0,1,-1)	0	-10	12	19	63.2	660	0	652	0
1	(0,1,-2)	-10	-15	12	19	63.2	417	0	407	0
1	(0,1,-2)	-10	-10	12	19	63.2	460	0	388	0
0	(0,0,-1)	-10	-15	12	19	63.2	59	0	35	0
0	(0,0,-1)	-10	-10	12	19	63.2	61	0	33	0
0	(0,0,-1)	-10	-5	12	19	63.2	59	0	39	0
0	(-1,0,0)	-10	-10	12	19	63.2	44	0	35	0
0	(-1,0,0)	-10	-15	12	19	63.2	46	0	32	0
2	(0,0,0)	-10	-5	47	75	62.7	180	0	142	0
0	(0,1,-2)	-30	-15	5	8	62.5	56	0	50	0
0	(0,1,-2)	-30	-10	5	8	62.5	75	0	54	0
0	(0,1,-2)	-20	-10	5	8	62.5	73	0	52	0
0	(0,1,-2)	0	0	60	97	61.9	46	0	48	0
1	(0,0,-1)	0	-10	8	13	61.5	383	0	289	0
1	(0,0,0)	-10	0	98	160	61.3	62	0	44	0
3	(0,0,0)	-30	0	98	160	61.3	62	0	47	0
2	(0,0,0)	-20	0	98	160	61.3	67	0	41	0
1	(-1,0,0)	0	-15	11	18	61.1	120	0	114	0
0	(0,1,-2)	-20	-15	6	10	60.0	65	0	59	0
3	(0,0,0)	0	-5	45	75	60.0	131	0	132	0
0	(-1,0,0)	0	0	52	89	58.4	37	0	30	0
0	(0,0,-1)	0	0	52	89	58.4	38	0	32	0
0	(0,2,-1)	-20	0	116	200	58.0	232	0	555	0
3	(0,0,0)	-10	-5	43	75	57.3	233	0	175	0
0	(0,1,-1)	-10	-15	8	14	57.1	67	0	66	0
1	(0,1,-2)	0	-10	17	30	56.7	484	0	332	0
0	(0,0,-1)	0	-5	5	9	55.6	47	0	33	0
0	(0,1,-2)	0	-5	5	9	55.6	63	0	73	0
0	(0,1,-2)	-10	-15	5	9	55.6	72	0	51	0
0	(0,1,-2)	-10	-10	5	9	55.6	61	0	58	0
0	(0,1,-2)	-10	0	59	107	55.1	50	0	39	0
1	(0,0,-1)	0	-15	7	13	53.8	487	0	220	0
2	(0,0,0)	-30	0	86	160	53.8	77	0	47	0
0	(0,2,-1)	-30	0	107	200	53.5	336	0	242	0
0	(-1,0,0)	0	-15	8	15	53.3	43	0	34	0
0	(-1,0,0)	0	-10	8	15	53.3	47	0	33	0
0	(0,1,-1)	-30	-5	8	15	53.3	62	0	57	0
0	(0,0,-1)	0	-10	8	15	53.3	48	0	33	0
0	(0,0,-1)	0	-15	8	15	53.3	48	0	36	0
0	(0,1,-1)	-20	-5	8	15	53.3	88	0	71	0
0	(0,1,-1)	-10	-5	8	15	53.3	81	0	63	0
1	(0,0,0)	-30	0	85	160	53.1	63	0	51	0
0	(-1,0,0)	0	-5	9	17	52.9	39	0	29	0
0	(0,1,-2)	0	-10	10	19	52.6	80	0	65	0
0	(0,1,-2)	0	-15	10	19	52.6	80	0	70	0
0	(0,2,-1)	0	-15	14	27	51.9	847	0	498	0
0	(0,2,-1)	-10	0	102	200	51.0	288	0	351	0
0	(0,1,-1)	-30	-10	6	12	50.0	78	0	60	0
0	(0,1,-1)	-30	-15	6	12	50.0	81	0	53	0
0	(0,1,-1)	-20	-15	6	12	50.0	61	0	62	0
0	(0,1,-1)	-20	-10	6	12	50.0	67	0	61	0
0	(0,1,-1)	-10	-10	6	12	50.0	77	0	68	0
0	(0,2,-1)	-10	-15	13	26	50.0	380	0	396	0
0	(0,2,-1)	-10	-10	13	26	50.0	397	0	535	0
0	(0,2,-1)	-20	-10	13	26	50.0	569	0	322	0
0	(0,2,-1)	-20	-15	13	26	50.0	378	0	343	0
0	(0,2,-1)	0	-10	13	26	50.0	518	0	515	0
0	(0,2,-1)	-30	-15	13	26	50.0	369	0	343	0
0	(0,2,-1)	-30	-10	13	26	50.0	569	0	269	0
0	(0,2,-1)	0	0	100	200	50.0	393	0	634	0
0	(0,2,-1)	-20	-5	15	31	48.4	660	0	671	0
2	(0,0,0)	-10	-10	27	57	47.4	121	0	133	0
2	(0,0,0)	-10	-15	27	57	47.4	136	0	100	0
1	(0,0,0)	-10	-15	27	57	47.4	100	0	100	0
1	(0,0,0)	-10	-10	27	57	47.4	138	0	110	0
1	(0,0,0)	-10	-5	27	57	47.4	151	0	171	0
2	(0,0,0)	-30	-15	27	57	47.4	165	0	95	0
3	(0,0,0)	-10	-15	27	57	47.4	168	0	132	0
3	(0,0,0)	-10	-10	27	57	47.4	181	0	148	0
1	(0,0,0)	-30	-10	27	57	47.4	97	0	86	0
3	(0,0,0)	-20	-10	27	57	47.4	168	0	161	0
3	(0,0,0)	-20	-15	27	57	47.4	183	0	126	0
1	(0,0,0)	-20	-10	27	57	47.4	114	0	75	0
1	(0,0,0)	-20	-15	27	57	47.4	90	0	74	0
1	(0,0,0)	-20	-5	27	57	47.4	104	0	94	0
2	(0,0,0)	-30	-10	27	57	47.4	151	0	101	0
3	(0,0,0)	-30	-15	27	57	47.4	173	0	121	0
3	(0,0,0)	-30	-10	27	57	47.4	154	0	111	0
1	(0,0,0)	-30	-5	27	57	47.4	108	0	95	0
1	(0,0,0)	-30	-15	27	57	47.4	71	0	62	0
2	(0,0,0)	-20	-15	27	57	47.4	105	0	87	0
2	(0,0,0)	-20	-10	27	57	47.4	156	0	134	0
1	(0,1,-2)	0	-15	15	32	46.9	420	0	337	0
3	(0,0,0)	0	-10	26	56	46.4	185	0	118	0
0	(0,0,0)	-10	-10	82	178	46.1	35	0	29	0
0	(0,0,0)	0	0	92	200	46.0	-	0	20	0
0	(0,1,-1)	0	0	73	160	45.6	59	0	67	0
2	(0,0,0)	0	-15	25	56	44.6	111	0	94	0
2	(0,0,0)	0	-10	25	56	44.6	122	0	125	0
1	(0,0,0)	0	-5	25	56	44.6	110	0	114	0
1	(0,0,0)	0	-10	25	56	44.6	104	0	87	0
1	(0,0,0)	0	-15	25	56	44.6	88	0	78	0
3	(0,0,0)	0	-15	25	56	44.6	111	0	107	0
0	(0,0,0)	-30	0	88	200	44.0	34	0	30	0
0	(0,0,0)	-10	0	88	200	44.0	35	0	32	0
0	(0,0,0)	-20	0	88	200	44.0	29	0	28	0
0	(0,0,0)	-20	-15	77	178	43.3	41	0	30	0
0	(0,0,0)	-20	-5	77	178	43.3	42	0	29	0
0	(0,0,0)	-10	-5	74	178	41.6	35	0	26	0
0	(0,0,0)	-30	-15	74	178	41.6	40	0	28	0
0	(0,0,0)	-20	-10	74	178	41.6	36	0	30	0
0	(0,1,-1)	-30	0	62	160	38.8	51	0	51	0
0	(0,1,-1)	-20	0	60	160	37.5	54	0	60	0
0	(0,1,-1)	0	-5	4	11	36.4	73	0	68	0
0	(0,1,-1)	0	-15	4	11	36.4	88	0	77	0
0	(0,1,-1)	0	-10	4	11	36.4	78	0	76	0
1	(0,0,0)	-20	0	58	160	36.3	63	0	40	0
0	(0,0,0)	0	-15	50	150	33.3	41	0	32	0
0	(0,0,0)	0	-5	50	150	33.3	37	0	30	0
0	(0,0,0)	0	-10	50	150	33.3	36	0	30	0
0	(0,1,-1)	-10	0	46	160	28.8	45	0	55	0
0	(0,0,0)	-30	-5	24	178	13.5	37	0	28	0
Parameter Settings with at least one Timeout
2	(0,1,-2)	-10	-5	8	8	100.0	844	0	704	1
2	(0,0,-1)	-10	-5	10	10	100.0	961	1	817	0
3	(0,1,-1)	-10	-5	11	11	100.0	808	0	961	1
1	(0,1,-1)	0	0	65	65	100.0	511	0	873	1
2	(0,2,-1)	0	0	75	75	100.0	995	2	973	2
2	(0,2,-1)	-30	0	78	78	100.0	975	2	957	2
2	(0,1,-1)	0	0	87	87	100.0	356	0	768	1
2	(0,2,-1)	-20	0	99	99	100.0	971	2	670	1
3	(0,2,-1)	-30	0	99	100	99.0	964	2	608	1
3	(0,2,-1)	0	0	74	75	98.7	994	2	994	2
3	(0,2,-1)	-10	0	73	74	98.6	962	2	979	2
3	(0,2,-1)	-20	-10	38	40	95.0	1106	2	867	1
2	(0,1,-1)	-30	-5	16	17	94.1	749	1	569	0
3	(0,1,-1)	-20	-10	11	12	91.7	765	1	661	0
3	(0,1,-1)	-10	-10	11	12	91.7	499	0	701	1
2	(0,1,-2)	0	-10	10	11	90.9	625	0	699	1
3	(0,2,-1)	0	-15	10	11	90.9	1106	2	1116	2
3	(0,2,-1)	0	-5	29	32	90.6	1154	2	882	1
3	(0,2,-1)	-20	0	67	75	89.3	993	2	967	2
2	(0,2,-1)	-10	0	108	123	87.8	971	2	788	1
2	(0,2,-1)	-30	-10	7	8	87.5	1118	2	1060	2
3	(0,2,-1)	-10	-15	28	32	87.5	1064	2	707	1
3	(0,2,-1)	-20	-15	6	7	85.7	1082	2	1018	2
2	(0,2,-1)	-20	-10	12	14	85.7	1058	2	1093	2
1	(0,1,-1)	-10	-5	32	38	84.2	927	1	454	0
1	(0,1,-2)	-30	-5	16	20	80.0	670	1	369	0
2	(0,2,-1)	-10	-5	26	33	78.8	1182	2	1017	1
2	(0,2,-1)	0	-10	7	9	77.8	1163	2	1086	2
3	(0,2,-1)	-30	-15	7	9	77.8	1078	2	1033	2
2	(0,1,-2)	0	-5	21	27	77.8	755	1	625	0
1	(0,1,-1)	-30	-5	15	20	75.0	905	1	856	0
1	(0,1,-1)	-20	-5	15	20	75.0	872	1	608	0
1	(0,1,-1)	-10	-10	15	20	75.0	992	1	478	0
3	(0,2,-1)	-30	-5	11	15	73.3	1077	2	1039	2
2	(0,0,-1)	0	-5	24	33	72.7	901	1	465	0
2	(0,1,-1)	-10	-5	29	40	72.5	991	2	521	0
1	(0,2,-1)	-30	-5	27	38	71.1	1135	2	789	1
2	(0,2,-1)	-20	-15	31	45	68.9	1052	2	943	1
1	(0,1,-2)	0	-5	19	28	67.9	738	1	650	0
2	(0,2,-1)	0	-15	21	33	63.6	1137	2	871	1
1	(0,1,-2)	-10	-5	21	33	63.6	720	1	399	0
1	(0,1,-1)	0	-5	12	19	63.2	986	1	821	1
1	(0,2,-1)	-10	-5	8	13	61.5	1139	2	1049	2
2	(0,2,-1)	-10	-10	34	58	58.6	1168	2	1040	0
2	(0,2,-1)	0	-5	18	33	54.5	1214	2	1023	1
2	(0,1,-1)	0	-5	17	32	53.1	987	2	730	1
1	(0,2,-1)	-20	0	61	116	52.6	975	2	946	2
0	(0,2,-1)	0	-5	13	25	52.0	831	0	794	1
0	(0,2,-1)	-30	-5	13	25	52.0	755	0	668	1
1	(0,2,-1)	-30	0	68	135	50.4	991	2	793	1
2	(0,2,-1)	-10	-15	16	33	48.5	1084	2	868	1
0	(0,2,-1)	-10	-5	15	31	48.4	814	1	656	0
3	(0,2,-1)	-20	-5	27	57	47.4	1120	2	788	0
3	(0,2,-1)	-10	-5	23	50	46.0	1083	2	1052	1
1	(0,2,-1)	-10	0	62	135	45.9	964	2	795	1
3	(0,2,-1)	-10	-10	20	44	45.5	1099	2	1048	1
1	(0,2,-1)	-20	-5	25	58	43.1	1144	2	867	1
1	(0,2,-1)	-20	-15	15	35	42.9	1043	2	937	1
2	(0,1,-1)	-20	-5	14	34	41.2	1010	1	620	1
1	(0,2,-1)	0	0	45	111	40.5	978	2	995	2
1	(0,2,-1)	-20	-10	14	35	40.0	1066	2	947	1
2	(0,2,-1)	-30	-15	22	58	37.9	1014	2	768	0
1	(0,2,-1)	0	-15	13	35	37.1	1047	2	812	1
1	(0,2,-1)	-10	-15	13	35	37.1	1033	2	938	1
1	(0,2,-1)	-30	-15	16	46	34.8	1017	2	593	1
3	(0,2,-1)	0	-10	11	32	34.4	1139	2	1178	1
1	(0,2,-1)	0	-5	13	38	34.2	1140	2	803	1
1	(0,2,-1)	0	-10	20	60	33.3	1109	2	643	0
1	(0,2,-1)	-30	-10	11	35	31.4	1033	2	947	1
2	(0,2,-1)	-30	-5	10	33	30.3	1104	2	840	1
2	(0,2,-1)	-20	-5	10	33	30.3	1115	2	794	1
3	(0,2,-1)	-30	-10	10	36	27.8	1117	2	774	1
1	(0,2,-1)	-10	-10	14	60	23.3	1040	2	435	0