Jonathan Dentler, Chercheur postdoctoral

Content warning: This post contains images that may be upsetting or distressing and display violence and the aftermath of violence. It is shown for educational and scholarly purposes and we ask that the reader engage with the material ethically and responsibly. 

On May 9^th, EyCon team members and institutional partners from the Musée Quai Branly-Jacques Chirac, the Service Historique de la Défense (SHD), and the Établissement de Communication et de Production Audiovisuelle de la Défense (ECPAD) met at the ECPAD installation at the Fort d’Ivry to discuss the team’s exciting research into the possibilities opened up by multimodal visual AI in historical research in photography and visual culture. Daniel Foliard began by introducing the workshop’s objectives. Such tools can help group together similar digital images, tracing the circulation of images across different publications for example. They can also assist with functions such as detecting objects as well as stylistic or formal analysis. Finally, such tools can produce new metadata in a semi-automated manner by, for instance, suggesting probable attributions for variables such as a photograph’s date of production, medium or process, photographer, or place depicted. Visual similarity tools become much more powerful when combined with existing textual metadata as an additional vector in a multi-modal fashion, and this is precisely what EyCon project computer science intern Mohamed Salim Aissi has been working on recently. In this workshop, we wanted to demonstrate a proof-of-concept test based on a selection of photos from the EyCon database centered on the atrocities carried out by the Italian Army in Tripoli in the fall of 1911.

Figure 1: Pierre Schill, workshop presentation. “The Paradox of the Italo-Turkish War: an abundant production and wide distribution of images, and yet an ‘under-exposed’ conflict.” 

In order to demonstrate some of the possibilities and limitations of these tools, we invited Pierre Schill to share his research on the photographic coverage of the Italo-Turkish war. M. Schill is a researcher who has published on the photos from the war in Libya, and who is currently collaborating with the EyCon team on a forthcoming article on the topic. You can find his book on the topic here, which contains additional context and analysis for these images. The idea was to show how photo historians analyze and interpret historical photos in part by tracing the circulation of images from their production through their reproductions in various formats and publications. This can help establish context by making arguments for probable attributions, as well as show how photographs’ meanings are inflected by editorial choices such as cropping or captioning. (Figure 1) M. Schill began by suggesting that a paradox of the war in Tripoli is that, while it was highly photographed by newspaper correspondents, it is little remembered in Europe today. In part, he suggested, this is because of the difficulty in identifying camera operators as well as the diversity of sites and modes of conservation of the photographs. To get a better idea of the conflict and its visual records, it is necessary to draw links between the various archives holding the record in order to make attributions and establish context.

Figure 2: Pierre Schill, workshop presentation. “A scene with two prisoners and three soldiers: two photographs in circulation in the press.”

(Figure 2) Based on his painstaking archival work, M. Schill showed us a number of examples of how he compared photographs in different collections in order to establish that they depicted the same event from different angles, or that the same photographer or persons appear in different scenes. (Figure 3) He also demonstrated that photos have been misattributed; visual cues such as the angle of the sun in different images of the same scene show that it was photographed multiple times, likely by different photographers. 

Figure 3: Pierre Schill, workshop presentation. “Play of shadows and identity: beyond the analogy of the scene (public hanging, Dec. 6, 1911).”

The key question for this workshop was whether it might be possible to produce visual similarity tools that are capable of capturing this level of nuance, helping to perform tasks such as suggesting probable attributions for photographs. For example, using a multi-modal approach, a visual similarity tool could quickly find two instances of the same image in different publications, different formats, and/or different archives. If the image is attributed to a photographer in one instance but not in the other, the text-processing part of the process could perceive this attribution, and then suggest that attribution for the other instance of the image.

However, there are a number of dangers here. First, without a sufficiently sophisticated tool, this might simply automate misattributions. In the case of the false attribution of the hanging photograph to Gaston Chérau that Pierre Schill shared with us, would a multi-modal visual similarity approaches be sufficiently nuanced to recognize that the changed angle of the sun indicated that significant time had elapsed between exposures, and that this might be strong evidence for different photographers? If the database with which the algorithm was working included the image from Domenica del Corriere in which the image was credited to “Dott Comboni” (likely an Italian military photographer who took the image later in the day), perhaps it could deliver the correct suggestion. This raises two important points. First, it is important to work with very large amounts of data. Second, and even more crucially, it is essentially that the researcher be aware of the limitations of the database with which the machine is working, and that semi-automation of this function in no way means that the suggestions it generates are infallible. A second-order problem also results from the coexistence of these two factors: while working with large data sets is important—indeed, it is central to the appeal of algorithmic approaches—an ever-larger data set also risks obscuring the tool’s limits from the researcher, lending the results a misleading aura of authority.

Following M. Schill’s presentation, Mohamed Salim Aissi presented his new work on a multimodal visual similarity tool, which will be built into EyCon’s public-facing database. We first constructed a limited image database to test the tools. This set included images of loose photographs documenting the Tripoli atrocities from the Forbin fonds at the SHD, as well as instances of those images reproduced in publications such as The Daily Mirror and Excelsior. In order to make sure that the tools could pick out similarities among a much larger set of non-similar photos, we also included the roughly 60,000 images from the fonds Valois, images produced by the French military during the Great War.

(Figures 4-5) M. Aissi explained how visual similarity algorithms and textual similarity algorithms function, translating the technical subtleties into terms that were comprehensible for the non-specialist audience.

Figure 4: Mohamed Salim Aissi. Graphic explanation of layers produced by a visual similarity tool. 

Figure 5: Mohammed Salim Aissi. Graphic explanation of how textual metadata is vectorized to create probabilistic predictions for word order. This is how the textual aspect of the multimodal tool operates. 

Figure 6: Mohamed Salim Aissi. Demonstration of multimodal search function for retrieving similar images from a large corpus.

(Figure 6) He then demonstrated the maquette he created, which can be used to search solely on the basis of visual similarity, solely on the basis of text, or with a multimodal approach. For example, a user will be able to query the database using an image, and ask for the ten most similar images in the database. Alternatively, the user could enter in a query using textual terms, and see what photographs are proposed. Finally, the user can search using an image file, and add additional vectors to the search on the basis of text that should be associated with the image.

Figure 7: Mohammed Salim Aissi. Multimodal vectorization of metadata and resulting clustering of terms and images. 

(Figure 7) Using several examples, M. Aissi demonstrated that this combined approach delivered a much higher degree of accuracy in finding different versions of an image across different publications for which cropping or other formal features of the reproduction varied.

Finally, Marina Giardinetti discussed some of the issues of metadata treatment raised by visual similarity tools. Noting that the structure of the metadata in the database was essential for such tools to function well, she explained the EyCon team’s methodology for conserving metadata and its sources in ways that allow its production to be traceable and for artificial intelligence tools to function in the best way possible. EyCon integrates the source of all metadata attached to the image so that its origin is documented and conserved. The EyCon website will show the metadata, allowing the user to get an idea of some of the variables that the multimodal algorithm used in making its determinations. If an algorithm suggests metadata, this would be noted in the display.

In the closing discussion, the participants considered the potential pitfalls and strengths of such tools. In comparison to many of the AI-powered tools that have come to the forefront of public discussion in the past year or two—notably ChatGPT—a strength of this system is that it clearly shows whether metadata was produced by a human or a machine, and presents at least some ways for the user to get an idea of how the algorithm has arrived at its conclusions. One key associated issue is metadata standards—EyCon is promoting discussion on this issue in France, and encouraging the use of the International Image Interoperability Framework (IIIF) standard. IIF standardizes the delivery of image files from servers to web displays where they can be used and interacted with. It eases the delivery of richer functionality for image or audio-visual files beyond simple viewing, and, crucially, it attaches metadata to the digital object, which makes it useful for preserving context when it comes to archival images.

Finally, we discussed the question of design and user experience. From our perspective, the display and design we propose for the public-facing database has several benefits. First, because the user has to do some work to define the search parameters, it makes the process slightly clunky. Instead of making the experience as seamless and smooth as possible, we wanted to show the system’s artificiality, finding a balance between ergonomics and resistance. Ideally, this should help the user realize that the suggestions made by the AI are not infallible and they do not exclude other possibilities that might have been missed due to the limitations of the inputs or of the algorithm. We also discussed the potential uses for researchers: while these tools may help us reanimate images and their circulations, they are ultimately most helpful in accelerating certain time-consuming operations performed by the researcher, allowing them to focus on other tasks requiring higher order analysis and synthesis. The key challenge for such tools going into the future may be finding ways to translate the impenetrable depths of the technological “black box” into terms that the average user can understand, mediating its function such that the user has insight into the variables at play, or the possibilities that might have been missed. If this is not accomplished, there is a very real danger that AI could degenerate from a research tool into a hindrance, obscuring aspects of the analogue archive behind a veneer of computational objectivity.

We would like to thank Véronique Pontillon Valedon, chargée des actions scientifique at the ECPAD, for welcoming the workshop participants at the facility at the Fort d’Ivry.

Jonathan Dentler, Chercheur postdoctoral

On the 6^th of March, the EyCon Project (funded by the Labex “Les Passés dans le Présent” and the AHRC) organized a workshop on the potential benefits and problems of using multi-modal artificial intelligence (AI) approaches to work with archives of sensitive digitized historical photographs. The workshop was attended by archivists and museum professionals from the Établissement de la Communication et de la Production Audiovisuelle de la Défense (ECPAD), the Service Historique de la Défense (SHD), and La Contemporaine museum and archive, and was held at La Contemporaine. The workshop’s aim was to collectively explore the possibilities for computational and machine-learning approaches to making potentially sensitive colonial archives more navigable and discoverable. The EyCon team presented a theoretical pipeline or workflow with which heritage institutions could carry out the computer-assisted analysis of images, enriching their metadata with a number of tools. In particular, we aimed to think through ways that these tools could be used to identify and contextualize sensitive images. After the presentation, the floor was opened for a dialogue between the research team and the archival partners about the contemporary working practices around metadata and sensitive images, and the possibilities and challenges of these technologies.

            After introductions, EyCon postdoctoral research fellow Dr. Jonathan Dentler introduced the project and the large questions that have been recently introduced in the public sphere and in scholarship about the ethics and politics of disseminating sensitive images online, particularly those depicting colonial violence. He suggested that a key proposition of the Eycon project is that, “critical and multimodal artificial intelligence offers a means of balancing the ethical treatment of material with the necessity of improving access to colonial archives for researchers and the public.” Using examples from the holdings of the ECPAD, he also introduced the issue of metadata, and how archives handle what terms to attach to photographs in order to contextualize them and make them searchable. AI could potentially solve the problem of scale in the task of adding metadata to large photography collections. Furthermore, by automatically detecting visual similarity, it could potentially help trace image circulation both transnationally and trans-imperially among linked archives, helping to de-silo national and imperial histories.

Another computational tool that could be used to enrich metadata are various forms of object detection and “safe search” functionalities that automatically detect violence and other forms of sensitive material.

Figure 1: Test of the Google Vision API “safe search” function on a photograph of the corpse of a german soldier from the fonds Valois.

(Figure 1) Unfortunately, as Jonathan Dentler discussed, such tools have been shaped by the legal and commercial needs of companies, and are not technically, scientifically, or ethically suited to historical material. (Figure 2) Indeed, out of 199 images extracted from the fonds Valois photo albums created by the French Service Photographique de l’Armée (SPA) during World War One and held by La Contemporaine that contain the word “corpse” in their textual metadata, only 12% were recognized by Google Vision API’s “safe search’ function as “possibly” or “probably” containing violence.

Figure 2: Results of a Google Vision API “Safe Search” with a corpus of 199 images containing the word “cadavre” or “corpse” in their metadata in the Valois albums.

            EyCon’s newly hired computer vision engineer Mohamed Salim Aissi, who recently earned a masters degree in computer science from Sorbonne Université, then introduced the technical details on visual similarity operations, explaining the architecture of auto-encoding that permits the extraction of information from images in order to determine similarity.

Figure 3: Visual Similarity method results.

(Figure 3) Using this technique, he was able to automatically find similar objects in the Valois albums, including, for example, shots of dugouts and trenches, injured faces, diplomatic meetings pictured from slightly different angles, and statues.

Figure 4: Images of statues made by the SPA and grouped in the same zone of the pixplot.

(Figure 4) These images were represented in clusters in a pixplot. Jonathan Dentler pointed out, however, that without fine-tuning of this approach, it could create false positives, for example grouping a photograph of burned bodies inside a destroyed tank with simple landscape shots on the basis of the tonal values of the pixels distributed across the image. The EyCon project believes that one way out of this problem is to treat textual metadata as a vector in the process of detecting similarity across images, so that the computation includes terms like “corpse” in the metadata as a hint that there may be violence in the picture, in addition to detecting objects or similarity. Such a process will likely be the subject of a second workshop in May.

            At this point, Jonathan Dentler also discussed the problem of implicit violence in photographs, using the example of the album La Prise de Samory, which is conserved by the SHD. An image that is apparently inoffensive can often have violent implications.

Figure 5: « Samory dans les rue de Saint Louis ». Album La Prise de Samory. Service Historique de la Défense (Vincennes).

(Figure 5) Extracted from its context, this image seems to show a simple parade or procession down the street of a town. In fact, this example is taken from an album created in 1899 by a French officer just after the conclusion of the war between the French army and Samory Touré’s Wassoulou Empire, which, at its apogee during the 1880s, extended over parts of present-day Guinea, Mali, and the Ivory Coast. In 1898 the French forces captured Touré and exiled him to Gabon.This photo shows the procession of Touré in Saint Louis after his capture. It is therefore a document of a form of public humiliation and a display of power, and is potentially sensitive in a way that simple object detection could not account for.

             Project leader Dr. Julien Schuh then presented on issues about how to make a pipeline for artificial intelligence work designed for scientific and university purposes. He discussed the need for what experts call “explainable AI,” which is a notion that the technology should not be “black boxed” and opaque from the perspective of its users. This involves, for example, making training datasets publicly available so that they can be critiqued. A good pipeline would also involve input from experts and source communities, perhaps through platforms such as Zooniverse. Such input could then be verified by archive professionals. From a meta-level perspective of the organization of archival materials, Dr. Schuh discussed the ways in which artificial intelligence could build new connections at scale that would allow researchers to draw new connections that were hidden by layers of archivation handed down by colonial institutions.

             The Eycon Project’s research engineer, Soumik Mallick, discussed the way in which various kinds of artificial intelligence tools such as pose detection or object detection can be deployed on datasets that have been extracted using the tool Layout Parser. He also explored reasons behind failures in pose recognition on historical photos. Data architect Marina Giardinetti discussed how the project uses Layout Parser in order to extract images from complex album and periodical page designs. Only when the images are extracted can “clean” datasets be created that would be useful in testing or producing tools informed by artificial intelligence. She explained how this methodology is build into the overall structure of EyCon’s database, which will survive after the end of the project, and demonstrated a “beta” version of the public-facing database.

            After a short coffee break, the workshop reconvened in an open discussion format to gather the responses of the project’s archival partners in attendance. Representing the ECPAD were Véronique Pontillon (chargée des actions scientifiques) and Eléonore Plantard (chargée des collections entrées par voie extraordinaire). They explained that the choice was made at ECPAD to put warning screens up on an image-by-image basis for potentially sensitive images due to the fact that they were being put on line, and that this involved a considerable amount of work. One employee within the service is responsible for validating these sensitivity notices. On the back-end of the database, this person can check a box which implements a warning on the selected contents to hide potentially disturbing images. There is great difficulty in making this determination for certain limit cases—images that do not necessarily appear violent at first sight, such as burned corpses that are not recognizable as such, or the moments just before an execution rather than during the act itself. While there is some direction from the heritage code and the principle of one’s right to their image and respect of human dignity, there is not a specific framework for deciding whether an image is sensitive or not. Instead, a trained expert takes all final decisions on this question, taking into account the suggestions of those who processed the documents.

            The database gives researchers the chance to alert archivists to consult the record to add additional information, including about sensitivity. These archivists can then add information on sensitive images in text form to the documentary records. Ideally, these notices are included for each image in a given file, but nevertheless occasionally there is only one notice for a whole batch of images. Another solution that was devised was putting up warning screens over sensitive images. Eléonore Plantard cautioned that it is important to realize that this platform is more of a publication tool than a tool for putting everything at the disposition of the researcher. There are files in the archives that contain pictures, without corresponding images put up on the site. These website pages simply alert researchers of the presence of a set of documents held in physical form only. The images put up on line are necessarily images on which ECPAD archivists have worked in order to enrich the metadata, and which have been considered from the point of view of potential sensitivity, but also from the point of view of copyright law.

            Julien Gueslin, head of the museum department at La Contemporaine, explained that the photographic prints in the collection are put online using the national tool Calames in EAD-XML with a number of headings attached, including persons, places, and subjects, which enables research within the corpus concerned. He felt sure that EyCon’s work has great potential to enrich these descriptions. As with other private archives in the public domain, these images are under open license, so the archive user can access them directly without mediation. This means that the archivist needs to consider what kind of information to furnish on the webpage. For other photographic archives that are private and under copyright, consultation can instead only take place on the premises, which in the case of sensitive images provides a form of mediation. Before the accession into the public archive from a private source, these documents can be submitted to rules that concern archival use and potentially to the need for special authorizations or exemptions from public use. Typically, however, the donor singles out these exceptions not on the basis of sensitive content but rather out of concerns for familial or private life. 

            Concerning the Valois Albums in particular, Julien Gueslin pointed out that it is interesting to conceive of these photos produced by the SPA as sensitive images today, given that during the war they were made available publicly for journalists or diplomats for propaganda purposes. They were controlled in terms of censorship and propaganda, but nonetheless widely disseminated without seeming to cause a public outcry at the time or in later years. He pointed out that this demonstrates how definitions of sensitivity can change and evolve over time. While there may always be new criticisms of the collection according to new definitions of sensitivity, most of the users of the archive are historians, who are more cognizant of the bias inherent in all such archival records. Overall, Julien Gueslin questioned whether it was the role of archivists to decide what is sensitive and prevent access to historical data. The fact of being an archive—a space dedicated to research—means that this institution is somewhat set aside from concerns about sensitivity in other spaces. In an archive one can contextualize documents in a way not possible in the digital media space, which provides some cover from pressure to censor or suppress data.

More than blocking what is sensitive, he believes that users most want to enrich data further. For example, Julien Gueslin suggested that it would be interesting to automatically enrich the location data attached photos, or to give these automatically-located place-names in different languages, particularly in areas that have many different linguistic groups or where the sovereignty and dominant language has changed, such as in Eastern Europe. EyCon’s work in enriching data, and being able to make connections and links between archival collections, is thus the most interesting part of the project from his perspective.  

            Finally, Benjamin Doizelet, head of the iconographic archives division at the SHD, explained that the SHD is an archival service and thus the patrimony code applies to it. This code requires that the archive be open for public access, and they provide this access. At the same time, there is no Internet portal for the photographs held by the SHD. However, there is a communication and follow-up process with everyone who accesses the photographs. Because it is specifically a military archive, it is assumed that all of the files are potentially sensitive, though less because of death and destruction (which is common) than because of potentially sensitive ways in which the army represents itself in the documents. Certain archival documents might offend the dignity of known or recognizable persons and thus must not be made publicly available. Over time, his predecessors had marked certain boxes with a red sticker to signify potentially sensitive content. However, this has always been subject to revision over time and is without any official legal status, indicating rather a subjective and ethical point of view.

            Ultimately, the workshop was a valuable opportunity to exchange with the project’s archival partners. The team showed both the potential and the problems of these tools, which are sure to reshape archival work. At the same time, our archival partners were able to educate us about the current practical, legal, and scientific conditions that currently govern they ways in which they make photographic heritage available to researchers and the public.