Figure 1. Figure 1: The project workflow
The poster presents interim results of an ongoing project that pursues the modeling of the social history of the premodern Islamicate world from the rich historical and biographical texts, which are characteristic of the Arabic written tradition. Leveraging advancements in language modeling and deep learning, the project utilizes a vast corpus of digitized texts to identify and connect relevant information, ultimately creating a novel knowledge network through which practically any historical issue—represented in those texts—can be investigated. Figure 1 illustrates the project workflow:
Corpus. The starting point of the project is a subcorpus of about 330 Arabic biographical and historical texts, amounting to 112 million tokens (an equivalent of about 135 million words in English). Although written in Arabic, these sources come from all the major regions of the Islamic world, spanning from Spain (al-Andalus) and North Africa (al-Maġrib), to Egypt (Miṣr) and Syria (al-Šām), to Iraq (al-ʿIrāq) and Iranian provinces (Fārs, Ḫurasān, etc.), to Central Asia (Mā-warāʾ-l-nahr) and India (Sind/Hind). These sources are particularly valuable for the period before the 15th century and most of them come from the period 1000–1600 CE. Still growing and expanding, this corpus contains about 600 thousand biographical records and perhaps almost as many descriptions of different historical events. Each of these texts is carefully split into self-contained text units: complete mini-texts of sorts that describe discrete historical entities, most commonly, historical events and biographies of individuals. These MIUs are treated as nodes in the new network of knowledge that the project is creating.
Annotation Pipeline, Active Learning Cycle. To extract the relevant data from these sources, a custom annotation pipeline was developed to extract information like persons, places, dates, and relationships. This pipeline is used for both training and production phases, employing fine-tuned models to annotate texts for data extraction. In order to ensure high-quality data extraction, we are relying on an active learning cycle where the automatically annotated data is continuously reviewed and fed back to models to improve their accuracy.
Data Enrichment, Linking, and Modeling. Annotated entities undergo further classification and metadata enrichment. Through computational comparison, the project identifies texts that discuss the same entities (for example, biographies of the same individuals) or entities of the same kind (for example, events of the same type). Furthermore, different types of extracted data are used to re-structure historical data into new objects. Thus, rich onomastic data is used to build extensive machine-readable profiles for each individual (family relationships, religious affiliations, professional specializations, etc.). Dates are used to re-align historical and biographical data chronologically. Places are used to re-align data geographically as well as to model geographical networks. Persons mentioned in shared contexts are used to construct complex networks of personal connections. Finally, by using different similarity-based approaches and the active learning cycle for the revision of results, we build connections between categorically similar text units; these connections are then used to re-aggregate our historical data into the new network of knowledge.
Research Framework. A web-based research framework is being developed to make our networked data more easily accessible. This framework will enable researchers to engage with the data through querying, filtering, and visualization options. The framework will support both distant and close reading modes, providing visual summaries of data and facilitating navigation across the vast network of historical records. The framework is being designed also to allow researchers to create new connections between the nodes of text units, thus forging and curating their own networks to be tailored to their specific research questions.
To sum up, the project breaks new ground in Islamicate studies by combining quantitative and qualitative historical methods with a broad variety of computational methods. The combination of deep learning and historical studies allows us to create a comprehensive research tool for exploring the social history of the Islamic world. The scalability of the approach allows for the continuous incorporation of new texts, while the active learning cycle ensures that our fine-tuned transformer models for classical Arabic NLP tasks are always at their highest possible accuracy. For the first time, the vast Arabic historical and biographical tradition is being brought together into a unified network of knowledge, which will advance the study of the record of Islamic history to a qualitatively new level.