Developing an Ontology for Intricate Visual Content Annotation: Experience from the Panorama of the Battle of Murten

1. Context

Realized by Louis Braun (1836-1916) in 1893, the Panorama of the Battle of Murten ¹ stands as a Swiss national treasure. This artwork commemorates the Swiss victory in 1476 that reinforced Swiss autonomy, shaped the national border, and elevated the prestige of Swiss mercenaries (Fuhrer / Geiger 2002; Schmidt 2002) .

The panorama also stands as a global visual heritage, particularly for the history of art and panorama. It is a rare surviving example of a painted panorama that depicts a medieval battle scene ² rather than a contemporary one (Grau 2003) . Moreover, it is the only surviving panorama by Louis Braun (Alexandre et al. 2012) , making it essential for the preservation of the artist’s creation. Recognizing its cultural significance, and to make the global visual heritage accessible, the Foundation for the Panorama of the Battle of Murten partners with Laboratory for Experimental Museology (eM+), EPFL to deliver a digital twin using advanced imaging and content augmentation techniques.

2. Intricate Visual Content as Big Cultural Data

The panorama, which measures 10 x 100 meters, was digitized in October 2023 at 1,000 dpi. The resulting digital twin is a 1,600 Gigapixels and 9.6 Terabytes (16-bit RGB uncompressed) single image. The content encapsulated within the panorama is remarkably rich, encompassing a myriad of elements such as geolocations, heraldic representations, historical characters, events, armaments, and costumes.

Intricate visual content represents an alternative big data challenge arising from a monolithic digital representation, deviating from the mainstream definition of “big data” ³ which emphasizes the sheer volume of documents in a corpus. Intricate visual content is characterized by a high density of compositions, motifs, decorations, or figurative narratives, as seen in cabinet of curiosities (Kunstkamer) paintings. Such content is often massive in extent, either physically—exemplified by painted panoramas and mural paintings—or digitally, such as in Chinese court paintings requiring gigapixel imaging to capture intricate details.

3. Our Objective and Method

To capture the extensive network of iconographical, historical, and cultural connections of intricate visual content, we propose a deep semantic annotation approach for data curation down to Point of Interest (POI) level. Our objective is to introduce a semantic annotation platform for, but not limited to, intricate visual content. Harnessing Linked Open Data (LOD) technologies, POI-based annotation can be summarized into three levels,

Table 1. Three levels POI-based Annotation

Scholarly interpretation is a nuanced process which encompasses multiple levels of conceptualization, aggregation of observations and evidence, and formation of interpretative statements using domain-specific terminology. Building a competent system for scholarly annotation demands the system to attain the deep semantic annotation level mentioned above. This involves two stages. Firstly, to characterize the visual scholarly domain and formalize an ontology to support fine-grained scholarly annotation. Secondly, choosing appropriate technologies for developing the annotation platform, supported by user study to validate the system's efficacy.

Supported by the Knowledge Graph Management Platform (KGMP), the primary objective of the semantic annotation platform is to function as a comprehensive knowledge base. This knowledge base will facilitate the accumulation of scholarly interpretations along with their corresponding evidence.

The proposed annotation ontology comprises the following components:

• Interpretation components: We will work with domain experts from various disciplines to identify alignment paths for specializing ontologies, if any are drawn.

• Influence, Provenance and Reference: Provides a semantic representation of the digital scholarly annotation in the humanities.

• Annotation Selector: This component links to the digital representation or citation of the annotated object or the sources/theories supporting the annotation.

• Upper Ontology: Provides alignments of key classes and properties to upper ontology.

• Narratology: Combines conceptual objects, events, and actors to provide ontological grounding for plots and characters. It serves as the knowledge model to disentangle historical events from the plots depicted in the visual content.

Figure 1. High-level ontology network view of the proposed annotation ontology

This paper will report the current progress of the annotation ontology, Our formalization process employs a diverse range of methods. Domain characterization relies on grounded evidence to identify the key notion to be modelled and addressed in the ontology. The reuse of existing ontologies fosters interoperability, building upon prior critical efforts in formalization (Azzi et al. 2019; Carriero et al. 2021). The formalization process also references a collection of ontology design methods, such as SAMOD (Peroni 2016), ontology network (Gómez-Pérez / Suárez-Figueroa 2009), ontology design pattern (ODP) (Gangemi / Presutti 2009) and modular design (Shimizu et al. 2020).

Figure 2. Selected grounded evidence for bottom-up ontology modeling

4. Conclusion and Outlook

The development of the annotation ontology is an iterative process, involving deployment, evaluation on an annotation platform, and subsequent redevelopment. It is crucial not to perceive these stages in isolation. One important consideration is annotation ergonomics (Burghardt, 2014). Fine-grained ontology might not be optimal during annotation, in contrast to machine-based knowledge graph extraction.

Deep data curation into POI will open new avenues for art historical research. Iconographical databases, like Arkyves (Brandhorst / Posthumus 2014) and the Warburg Institute Iconographic Database (Warburg Institute 2010) can yield even more fine-grained search results. By leveraging the relative positions of annotated motifs, whether through human or machine, within intricate visual content, we can compute the relative importance of these motifs against a chosen centroid motif using a weighted graph, potentially offering a new perspective for distant viewing.