A digital 4D information system at world scale: Research challenges, approaches and preliminary results

1. Introduction

Imagine you’re exploring the historic center of a city with its impressive town houses, churches and monuments. What if you could just use your mobile device to find out about the historic buildings around you, with detailed visual information about how they were built and the story behind them, making history come alive before your eyes? Photographs and plans are an essential source for historical research (Burke, 2003, Paul, 2006, Pérez-Gómez and Pelletier, 1997). Numerous digital image archives, containing vast numbers of photographs, have been set up in the context of digitization projects. Within the Jena4D research group we are investigating and developing methods and technologies for transferring historical media and their contextual information into a 4D – 3D spatial and temporal scaled - model to support research and education on urban history. Content is made accessible as a location-dependent virtual reality 4D browser application for mobile devices and a 4D desktop browser application. Within former articles we highlighted the prospected research agenda (Münster et al., 2020) as well as technological venues (Muenster et al., 2021, Münster et al., 2021) but also conceptual challenges to automatically reconstruct the past (Münster et al., 2022, Münster et al., 2024). The main purpose of this contribution is to (1) highlight current conceptual and technological challenges, (2) examine a state of the art and (3) our approach towards data collection, 4D modelling and visualization at world scale, as well as present (4) results from demo cases in Dresden, Jena and Amsterdam and at world-scale.

2. Methodical approach

Our pipeline comprises three steps: (1) to collect data as images, models and information from multiple sources and via automated and community-based approaches (c.f. Fig 1), to (2) automatically form a model of a historical architecture and cityscapes via a 4D reconstruction from historical sources. The model is made accessible in two ways of (c) visualization; via a 4D browser and a location-dependent augmented reality (AR) representation.

2.1. Data collection

Data was collected in various ways: (a) Data retrieval comprises approaches to gather images, links to information resources and 3D models from different sources. So far ca. 20.000 images and 4.000 3D models has been retrieved. (b) To aggregate historical images of Jena, a citizen contest was held between 25th October 2022 and 4th December 2022. In total, over 4000 historical photos and rephotographs were submitted of which a selection is used to evaluate the 3D reconstruction methods. (c) As another option for gathering 3D content we implemented a low-end 3D digitization pipeline to document heritage with images taken with a smart phone – this is combined with a server-based 3D reconstruction. Educational courses comprise (d) two issues of student hackathons to virtually recreate historic cityscapes and (e) three courses designed by students of art history and history teaching, which were offered to primary and secondary school students in the context of school and extracurricular working groups and a project week in a youth centre and are now made available to other teachers on a public platform. In total 95 pupils participated in these educational courses.

Fig 1. Data collection and retrieval approaches used by the Jena4D group

2.2. 4D Modelling pipeline

Our main approach (c.f. Fig 2) includes (a) to georeferenced historical plans and to extract building footprints – this step is currently done semi-automatically. As a parallel pipeline task the spatialization of historical photographs takes place. The first step is (b) to collect and identify spatiotemporally corresponding images via textual image metadata (e.g. placenames) (Wu et al., 2021, Hessel et al., 2019). Then to (c) detect similar views via overlapping segments in historical and contemporary photographs and (g) a relative position is calculated via a feature-based orientation/positioning pipeline, leading in combination with contemporary, oriented data to absolute positions. This step is currently available as prototype. A final step creates the 3D building by extruding the footprints identified in step a to building walls, calculate a roof shape and map the photographs according to the calculated position and orientation from step d onto the façade – this is already in production use. These steps make it possible to create large-scale models with basic geometric features.

Fig 2. Workflow for footprint extraction, photo orientation and parametric generation of textured 3D building models

2.3. Visualization

Since 2016 our group has developed a modular 4D web-visualization software framework for 4D city and 4D content browsing to test and validate design hypotheses in virtual, augmented, and 2.5D visualization on mobile and desktop devices (Münster, 2018). As user acceptance of native applications is decreasing (Bender, 2020), especially for specific and short-term use, as relevant for most cityscape scenarios (Reips, 2002) this is a browser-based web application.

2.3.1. 4D City application for mobile devices

The 4D City application for mobile devices enables time-variant virtual 3D impressions of historic cities to a multi-device visual interface (Fig. 3) – capable to being accessed via desktop, mobile devices, via AR and VR glasses. The 4Dcity application does focus on the application layer and do feed in-time from other data providers without a own database except for caching. It enables virtual city tours and past-plays as e.g. mixed-reality escape or discovery games and provide access to knowledge assets from open source platforms such as Wikipedia or tourist information platforms as Trivago. Unique features of that application include a dynamic shader which is capable to create the 4D environments on the fly.

Fig 3. Browser-based mobile application showing textured 3D models of historical buildings and points of interest.

2.3.2. 4D Browser

The 4D Browser is used for the information presentation and as a research tool (Bearman and Geber, 2008). Linking digital images and their actual location allow for example to directly present resources and therefore proves a valuable support tool for historic research (Fig. 4). Users of the virtual archives can benefit extensively from effective searching functions and tools which work not only content- and theme-based but also location-based. A timeline provides information on the development of a city via filtering photos of different building conditions.

Fig. 4 . Graphical user interface of the 4D Browser application

3. Summary

The pipeline and applications are developed by my group since 2016 via various projects at national and European scale. At the current state the applications are already available to provide a contemporary viewer functionality at world scale with location-based Wikipedia articles included as POIs. So far, nine PhD and PostDoc endeavours in computing, humanities, geosciences, design and education are linked to the pipeline and applications and their usage.

Even after eight years of work, various research challenges and issues remain unresolved. For user studies, the next research steps is to better align visualizations to usage scenarios and user needs. We still know very little about how architectural historians visually judge sources or how laypersons cope with 4D reconstructions. Also, visual parameters and designs for 3D/4D visualizations of past architecture are rarely empirically validated. Consequently, a future step may be the comprehensive investigation of design recommendations for 3D/4D representations to serve architectural history studies. Concerning 4D modeling, open research tasks are to reduce the number and quality of historical images necessary to enable the use of dense matching and the creation of historical (generalized) 3D models. Another approach currently under development in a connecting project is large-scale automated workflow for image orientation including contemporary data and 3D models.

Finally, the distinction into digital disciplines and humanities fall short in the case of large scale and multi-facetted projects as the one described here. Consequently, a main task is to aligning between epistemic cultures and their paradigms, terminologies and quality criteria.