Abstract
Information Retrieval (IR) systems are developed to fulfill several needs of users. As a plethora of IR techniques has been developed, therefore the evaluation of these techniques has been of paramount importance. The evaluation of these techniques require test collections, which are composed of a collection of documents, queries and relevance judgement between query–document pairs. Recognizing the importance of the evaluation of IR techniques, Text REtrieval Conference (TREC) has been regularly organized for the last three decades with an aim to develop and continuously improve these techniques. However, most of the resource development effort has been directed to English and other Western languages, whereas resource development for Urdu has received little attention, which has thwarted the development of IR techniques. Furthermore, the available benchmarks have several limitations which include smaller size, unavailability of the benchmark, inadequate candidate documents for evaluation, and merely binary relevance judgement. To that end, this study has focused on constructing the largest-ever and semantic IR benchmark for the Urdu language that strictly complies with the procedures proposed by TREC. That is, firstly, a large collection of 2,887,169 Urdu documents is scrapped and converted into the standard format proposed by TREC. Secondly, 105 queries are generated, which includes detailed descriptions of 35 queries and three variants of each query in the TREC format. Thirdly, a pooling-based approach is employed using queries, as well as their variants, to identify a candidate pool of 13,392 documents for human judgement. Finally, two experts performed 13,392 query–document comparisons and ranked them on a scale of four types of relevance: highly relevant, fairly relevant, marginally relevant and irrelevant, which is a significant enhancement from the existing benchmarks. The benchmark can be used for the evaluation of existing IR techniques, as well as for future techniques.
•Large corpus of over 2,887,169 Urdu documents in the TREC defined SGML format.•A collection of 35 Urdu queries from 14 domains for assessment.•Human benchmark of candidate relevant documents using a pooling-based.•Non-binary relevance judgement at four-levels.