The OWL reasoner evaluation (ORE) 2015 competition report. (English) Zbl 1425.68402
Summary: The OWL Reasoner Evaluation competition is an annual competition (with an associated workshop) that pits OWL 2 compliant reasoners against each other on various standard reasoning tasks over naturally occurring problems. The 2015 competition was the third of its sort and had 14 reasoners competing in six tracks comprising three tasks (consistency, classification, and realisation) over two profiles (OWL 2 DL and EL). In this paper, we discuss the design, execution and results of the 2015 competition with particular attention to lessons learned for benchmarking, comparative experiments, and future competitions.
MSC:
| 68T30 | Knowledge representation |
| 68T27 | Logic in artificial intelligence |
| 68T35 | Theory of languages and software systems (knowledge-based systems, expert systems, etc.) for artificial intelligence |
| 68T37 | Reasoning under uncertainty in the context of artificial intelligence |
Software:
HermiT; FaCT++; Chainsaw; RacerPro; TPTP; Racer; ELepHant; BioPortal; Konclude; ELK; GitHub; JFact; TrOWL; OpenStreetMap; PelletReferences:
| [1] | Armas Romero, A., Cuenca Grau, B., Horrocks, I.: MORe: modular combination of OWL reasoners for ontology classification. In: The Semantic Web—ISWC 2012—11th International Semantic Web Conference, Boston, MA, USA, November 11-15, 2012, Proceedings, Part I, pp. 1-16 (2012). doi:10.1007/978-3-642-35176-1_1 |
| [2] | Baader, F., Calvanese, D., McGuinness, D., Nardi, D., Patel-Schneider, P. (eds.): The Description Logic Handbook: Theory, Implementation, and Applications, 2nd edn. Cambridge University Press, Cambridge (2007) · Zbl 1132.68055 |
| [3] | Bail, S., Glimm, B., Gonçalves, R.S., Jiménez-Ruiz, E., Kazakov, Y., Matentzoglu, N., Parsia, B. (eds.): Informal Proceedings of the 2nd International Workshop on OWL Reasoner Evaluation (ORE-2013), Ulm, Germany, July 22, 2013, CEUR Workshop Proceedings, vol. 1015. CEUR-WS.org. http://ceur-ws.org/Vol-1015 (2013) · Zbl 1185.68636 |
| [4] | Bail, S., Glimm, B., Jiménez-Ruiz, E., Matentzoglu, N., Parsia, B., Steigmiller, A. (eds.): Informal Proceedings of the 3rd International Workshop on OWL Reasoner Evaluation (ORE 2014) Co-located with the Vienna Summer of Logic (VSL 2014), Vienna, Austria, July 13, 2014, CEUR Workshop Proceedings, vol. 1207. CEUR-WS.org. http://ceur-ws.org/Vol-1207 (2014) · Zbl 1314.68280 |
| [5] | Beisswanger, E., Schulz, S., Stenzhorn, H., Hahn, U.: Biotop: an upper domain ontology for the life sciencesa description of its current structure, contents and interfaces to OBO ontologies. Appl. Ontol. 3(4), 205-212 (2008). doi:10.3233/AO-2008-0057 · doi:10.3233/AO-2008-0057 |
| [6] | Connolly, D., van Harmelen, F., Horrocks, I., McGuinness, D.L., Patel-Schneider, P.F., Stein, L.: DAML + OIL (March 2001) reference description. W3C Note. https://www.w3.org/TR/daml+oil-reference (2001) |
| [7] | Dumontier, M., Glimm, B., Gonçalves, R.S., Horridge, M., Jiménez-Ruiz, E., Matentzoglu, N., Parsia, B., Stamou, G.B., Stoilos, G. (eds.): Informal Proceedings of the 4th International Workshop on OWL Reasoner Evaluation (ORE-2015) Co-located with the 28th International Workshop on Description Logics (DL 2015), Athens, Greece, June 6, 2015, CEUR Workshop Proceedings, vol. 1387. CEUR-WS.org. http://ceur-ws.org/Vol-1387 (2015) |
| [8] | Eiter, T., Schneider, P., Simkus, M., Xiao, G.: Using OpenStreetMap data to create benchmarks for description logic reasoners. In: Bail et al. [4], pp. 51-57. http://ceur-ws.org/Vol-1207/paper_10.pdf |
| [9] | Glimm, B., Horrocks, I., Motik, B., Stoilos, G., Wang, Z.: HermiT: an OWL 2 reasoner. J. Autom. Reason. 53(3), 245-269 (2014). doi:10.1007/s10817-014-9305-1 · Zbl 1314.68280 · doi:10.1007/s10817-014-9305-1 |
| [10] | Gonçalves, R.S., Matentzoglu, N., Parsia, B., Sattler, U.: The empirical robustness of description logic classification. In: Proceedings of the ISWC 2013 Posters and Demonstrations Track, Sydney, Australia, October 23, 2013, pp. 277-280. http://ceur-ws.org/Vol-1035/iswc2013_poster_25.pdf (2013) |
| [11] | Haarslev, V., Hidde, K., Möller, R., Wessel, M.: The RacerPro knowledge representation and reasoning system. Semant. Web 3(3), 267-277 (2012). doi:10.3233/SW-2011-0032 · doi:10.3233/SW-2011-0032 |
| [12] | Herre, H., Hoehndorf, R., Kelso, J., Loebe, F., Schulz, S.: OBML—ontologies in biomedicine and life sciences. J. Biomed. Semant. 2(S-4), I1. http://www.jbiomedsem.com/content/2/S4/I1 (2011) |
| [13] | Hitzler, P., Krötzsch, M., Parsia, B., Patel-Schneider, P.F., Rudolph, S.: OWL 2 Web Ontology Language: Primer, 2nd edn. W3C Recommendation. https://www.w3.org/TR/owl-primer/ (2012) · Zbl 1358.68276 |
| [14] | Horridge, M., Bechhofer, S.: The OWL API: a Java API for OWL ontologies. Semant. Web 2(1), 11-21 (2011). doi:10.3233/SW-2011-0025 · doi:10.3233/SW-2011-0025 |
| [15] | Horrocks, I., Yatskevich, M., Jiménez-Ruiz, E. (eds.): Proceedings of the 1st International Workshop on OWL Reasoner Evaluation (ORE-2012), Manchester, UK, July 1st, 2012, CEUR Workshop Proceedings, vol. 858. CEUR-WS.org. http://ceur-ws.org/Vol-858 (2012) |
| [16] | Kazakov, Y., Krötzsch, M., Simancik, F.: The incredible ELK—from polynomial procedures to efficient reasoning with EL ontologies. J. Autom. Reason. 53(1), 1-61 (2014). doi:10.1007/s10817-013-9296-3 · Zbl 1331.68216 · doi:10.1007/s10817-013-9296-3 |
| [17] | Keet, C.M., Lawrynowicz, A., d’Amato, C., Kalousis, A., Nguyen, P., Palma, R., Stevens, R., Hilario, M.: The data mining optimization ontology. J. Web Semant. 32, 43-53 (2015). doi:10.1016/j.websem.2015.01.001 · doi:10.1016/j.websem.2015.01.001 |
| [18] | Krötzsch, M., Simančík, F., Horrocks, I.: Description logics. IEEE Intell. Syst. 29(1), 12-19 (2014). doi:10.1109/MIS.2013.123 · Zbl 1358.68276 · doi:10.1109/MIS.2013.123 |
| [19] | Lee, M., Matentzoglu, N., Parsia, B., Sattler, U.: A multi-reasoner, justification-based approach to reasoner correctness. In: International Semantic Web Conference (2015) |
| [20] | Liaw, S., Taggart, J., Yu, H., de Lusignan, S., Kuziemsky, C.E., Hayen, A.: Integrating electronic health record information to support integrated care: practical application of ontologies to improve the accuracy of diabetes disease registers. J. Biomed. Inform. 52, 364-372. http://www.sciencedirect.com/science/article/pii/S1532046414001798 (2014) · Zbl 1019.68101 |
| [21] | Matentzoglu, N., Bail, S., Parsia, B.: A snapshot of the OWL web. In: The Semantic Web—ISWC 2013—12th International Semantic Web Conference, Sydney, NSW, Australia, October 21-25, 2013, Proceedings, Part I, pp. 331-346 (2013). doi:10.1007/978-3-642-41335-3_21 |
| [22] | Matentzoglu, N., Leo, J., Hudhra, V., Sattler, U., Parsia, B.: A survey of current, stand-alone OWL reasoners. In: Dumontier, M., Glimm, B., Gonçalves, R., Horridge, M., Jiménez-Ruiz, E., Matentzoglu, N., Parsia, B., Stamou, G., Stoilos, G. (eds.) Informal Proceedings of the 4th International Workshop on OWL Reasoner Evaluation, vol. 1387. CEUR-WS (2015) · Zbl 1314.68280 |
| [23] | Matentzoglu, N., Parsia, B.: The OWL full/DL gap in the field. In: Proceedings of the 11th International Workshop on OWL: Experiences and Directions (OWLED 2014) Co-located with 13th International Semantic Web Conference on (ISWC 2014), Riva del Garda, Italy, October 17-18, 2014, pp. 49-60 (2014) · Zbl 1358.68276 |
| [24] | Matentzoglu, N., Parsia, B.: ORE 2015 reasoner competition dataset (2015). doi:10.5281/zenodo.18578 · Zbl 1425.68402 |
| [25] | Mendez, J.: jcel: A modular rule-based reasoner. In: Proceedings of the 1st International Workshop on OWL Reasoner Evaluation (ORE-2012), Manchester, UK, July 1st, 2012. http://ceur-ws.org/Vol-858/ore2012_paper12.pdf (2012) |
| [26] | Motik, B., Cuenca Grau, B., Horrocks, I., Wu, Z., Fokoue, A., Lutz, C.: OWL 2 Web Ontology Language: Profiles, 2nd edn. W3C Recommendation. https://www.w3.org/TR/owl2-profiles/ (2012) |
| [27] | Motik, B., Nenov, Y., Piro, R., Horrocks, I., Olteanu, D.: Parallel materialisation of Datalog programs in centralised, main-memory RDF systems. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July 27-31, 2014, Québec City, Québec, Canada, pp. 129-137. http://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8505 (2014) |
| [28] | Motik, B., Patel-Schneider, P.F., Cuenca Grau, B.: OWL 2 Web Ontology Language: Direct semantics, 2nd edn. W3C Recommendation. https://www.w3.org/TR/owl2-direct-semantics/ (2012) |
| [29] | Motik, B., Patel-Schneider, P.F., Parsia, B., Bock, C., Fokoue, A., Haase, P., Hoekstra, R., Horrocks, I., Ruttenberg, A., Sattler, U., Smith, M.: OWL 2 Web Ontology Language: Structural specification and functional-style syntax, 2nd edn. W3C Recommendation. https://www.w3.org/TR/owl2-syntax/ (2012) |
| [30] | Noy, N.F., Shah, N.H., Whetzel, P.L., Dai, B., Dorf, M., Griffith, N., Jonquet, C., Rubin, D.L., Storey, M.A.D., Chute, C.G., Musen, M.A.: BioPortal: ontologies and integrated data resources at the click of a mouse. Nucleic Acids Res. 37(Web-Server-Issue), 170-173 (2009). doi:10.1093/nar/gkp440 · doi:10.1093/nar/gkp440 |
| [31] | Osumi-Sutherland, D., Marygold, S.J., Millburn, G.H., McQuilton, P., Ponting, L., Stefancsik, R., Falls, K., Brown, N.H., Gkoutos, G.V.: The drosophila phenotype ontology. J. Biomed. Semant. 4, 30 (2013). doi:10.1186/2041-1480-4-30 · doi:10.1186/2041-1480-4-30 |
| [32] | Palmisano, I.: JFact repository. https://github.com/owlcs/jfact (2015) |
| [33] | Parsia, B., Matentzoglu, N., Gonçalves, R.S., Glimm, B., Steigmiller, A.: The OWL reasoner evaluation (ORE) 2015 competition report. In: Liebig, T., Fokoue, A. (eds.) Proceedings of the 11th International Workshop on Scalable Semantic Web Knowledge Base Systems co-located with 14th International Semantic Web Conference (ISWC 2015), CEUR Workshop Proceedings, vol. 1457, pp. 2-15. CEUR-WS.org (2015) · Zbl 1425.68402 |
| [34] | Pelletier, F., Sutcliffe, G., Suttner, C.: The development of CASC. AI Commun. 15(2-3), 79-90 (2002) · Zbl 1019.68101 |
| [35] | Ren, Y., Pan, J.Z., Zhao, Y.: Soundness preserving approximation for TBox reasoning. In: Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010. http://www.aaai.org/ocs/index.php/AAAI/AAAI10/paper/view/1955 (2010) |
| [36] | Samwald, M.: Genomic CDS: an example of a complex ontology for pharmacogenetics and clinical decision support. In: Bail et al. [3], pp. 128-133. http://ceur-ws.org/Vol-1015/paper_12.pdf |
| [37] | Sattler, U., Matentzoglu, N.: List of Reasoners (owl.cs) http://owl.cs.manchester.ac.uk/tools/list-of-reasoners/. http://owl.cs.manchester.ac.uk/tools/list-of-reasoners/ (2014) |
| [38] | Schulz, S., Martínez-Costa, C.: How ontologies can improve semantic interoperability in health care. In: Riaño, D., Lenz, R., Miksch, S., Peleg, M., Reichert, M., ten Teije, A. (eds.) Process Support and Knowledge Representation in Health Care - AIME 2013 Joint Workshop, KR4HC 2013/ProHealth 2013, Murcia, Spain, June 1, 2013, Revised Selected Papers, Lecture Notes in Computer Science, vol. 8268, pp. 1-10. Springer (2013). doi:10.1007/978-3-319-03916-9_1 |
| [39] | Sertkaya, B.: The ELepHant reasoner system description. In: Informal Proceedings of the 2nd International Workshop on OWL Reasoner Evaluation (ORE-2013), Ulm, Germany, July 22, 2013, pp. 87-93. http://ceur-ws.org/Vol-1015/paper_15.pdf (2013) |
| [40] | Sirin, E., Parsia, B., Cuenca Grau, B., Kalyanpur, A., Katz, Y.: Pellet: a practical OWL-DL reasoner. J. Web Semant. 5(2), 51-53 (2007). doi:10.1016/j.websem.2007.03.004 · doi:10.1016/j.websem.2007.03.004 |
| [41] | Steigmiller, A., Liebig, T., Glimm, B.: Konclude: system description. J. Web Semant. 27, 78-85 (2014). doi:10.1016/j.websem.2014.06.003 · Zbl 1423.68486 · doi:10.1016/j.websem.2014.06.003 |
| [42] | Stevens, R., Matentzoglu, N., Sattler, U., Stevens, M.: A family history knowledge base in OWL 2. In: Bail et al. [4], pp. 71-76. http://ceur-ws.org/Vol-1207/paper_11.pdf |
| [43] | Sutcliffe, G.: The TPTP problem library and associated infrastructure: the FOF and CNF Parts, v3.5.0. J. Autom. Reason. 43(4), 337-362 (2009) · Zbl 1185.68636 · doi:10.1007/s10817-009-9143-8 |
| [44] | Sutcliffe, G.: The CADE ATP system competition—CASC. AI Mag. 37(2), 99-101 (2016) · Zbl 0916.68141 · doi:10.1609/aimag.v37i2.2620 |
| [45] | Thomas, E., Pan, J.Z., Ren, Y.: TrOWL: Tractable OWL 2 reasoning infrastructure. In: The Semantic Web: Research and Applications, 7th Extended Semantic Web Conference, ESWC 2010, Heraklion, Crete, Greece, May 30-June 3, 2010, Proceedings, Part II, pp. 431-435 (2010). doi:10.1007/978-3-642-13489-0_38 |
| [46] | Tsarkov, D., Horrocks, I.: FaCT++ description logic reasoner: system description. In: Automated Reasoning, Third International Joint Conference, IJCAR 2006, Seattle, WA, USA, August 17-20, 2006, Proceedings, pp. 292-297 (2006). doi:10.1007/11814771_26 |
| [47] | Tsarkov, D., Palmisano, I.: Chainsaw: a metareasoner for large ontologies. In: Proceedings of the 1st International Workshop on OWL Reasoner Evaluation (ORE-2012), Manchester, UK, July 1st, 2012. http://ceur-ws.org/Vol-858/ore2012_paper2.pdf (2012) |
| [48] | Wolstencroft, K., Brass, A., Horrocks, I., Lord, P.W., Sattler, U., Turi, D., Stevens, R.: A little semantic web goes a long way in biology. In: Gil, Y., Motta, E., Benjamins, V.R., Musen, M.A. (eds.) The Semantic Web—ISWC 2005, 4th International Semantic Web Conference, ISWC 2005, Galway, Ireland, November 6-10, 2005, Proceedings, Lecture Notes in Computer Science, vol. 3729, pp. 786-800. Springer (2005). doi:10.1007/11574620_56 |
| [49] | Wu, J., Hudek, A., Toman, D., Weddell, G.: Absorption for ABoxes. J. Autom. Reason. 1-29 (2014). doi:10.1007/s10817-014-9304-2 · Zbl 1314.68300 |
| [50] | Zhou, Y., Nenov, Y., Grau, B.C., Horrocks, I.: Pay-as-you-go OWL query answering using a triple store. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July 27-31, 2014, Québec City, Québec, Canada, pp. 1142-1148 (2014) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
