Jonathan Dentler, Chercheur postdoctoral
On the 6th of March, the EyCon Project (funded by the Labex “Les Passés dans le Présent” and the AHRC) organized a workshop on the potential benefits and problems of using multi-modal artificial intelligence (AI) approaches to work with archives of sensitive digitized historical photographs. The workshop was attended by archivists and museum professionals from the Établissement de la Communication et de la Production Audiovisuelle de la Défense (ECPAD), the Service Historique de la Défense (SHD), and La Contemporaine museum and archive, and was held at La Contemporaine. The workshop’s aim was to collectively explore the possibilities for computational and machine-learning approaches to making potentially sensitive colonial archives more navigable and discoverable. The EyCon team presented a theoretical pipeline or workflow with which heritage institutions could carry out the computer-assisted analysis of images, enriching their metadata with a number of tools. In particular, we aimed to think through ways that these tools could be used to identify and contextualize sensitive images. After the presentation, the floor was opened for a dialogue between the research team and the archival partners about the contemporary working practices around metadata and sensitive images, and the possibilities and challenges of these technologies.
After introductions, EyCon postdoctoral research fellow Dr. Jonathan Dentler introduced the project and the large questions that have been recently introduced in the public sphere and in scholarship about the ethics and politics of disseminating sensitive images online, particularly those depicting colonial violence. He suggested that a key proposition of the Eycon project is that, “critical and multimodal artificial intelligence offers a means of balancing the ethical treatment of material with the necessity of improving access to colonial archives for researchers and the public.” Using examples from the holdings of the ECPAD, he also introduced the issue of metadata, and how archives handle what terms to attach to photographs in order to contextualize them and make them searchable. AI could potentially solve the problem of scale in the task of adding metadata to large photography collections. Furthermore, by automatically detecting visual similarity, it could potentially help trace image circulation both transnationally and trans-imperially among linked archives, helping to de-silo national and imperial histories.
Another computational tool that could be used to enrich metadata are various forms of object detection and “safe search” functionalities that automatically detect violence and other forms of sensitive material.
Figure 1: Test of the Google Vision API “safe search” function on a photograph of the corpse of a german soldier from the fonds Valois.
(Figure 1) Unfortunately, as Jonathan Dentler discussed, such tools have been shaped by the legal and commercial needs of companies, and are not technically, scientifically, or ethically suited to historical material. (Figure 2) Indeed, out of 199 images extracted from the fonds Valois photo albums created by the French Service Photographique de l’Armée (SPA) during World War One and held by La Contemporaine that contain the word “corpse” in their textual metadata, only 12% were recognized by Google Vision API’s “safe search’ function as “possibly” or “probably” containing violence.
Figure 2: Results of a Google Vision API “Safe Search” with a corpus of 199 images containing the word “cadavre” or “corpse” in their metadata in the Valois albums.
EyCon’s newly hired computer vision engineer Mohamed Salim Aissi, who recently earned a masters degree in computer science from Sorbonne Université, then introduced the technical details on visual similarity operations, explaining the architecture of auto-encoding that permits the extraction of information from images in order to determine similarity.
Figure 3: Visual Similarity method results.
(Figure 3) Using this technique, he was able to automatically find similar objects in the Valois albums, including, for example, shots of dugouts and trenches, injured faces, diplomatic meetings pictured from slightly different angles, and statues.
Figure 4: Images of statues made by the SPA and grouped in the same zone of the pixplot.
(Figure 4) These images were represented in clusters in a pixplot. Jonathan Dentler pointed out, however, that without fine-tuning of this approach, it could create false positives, for example grouping a photograph of burned bodies inside a destroyed tank with simple landscape shots on the basis of the tonal values of the pixels distributed across the image. The EyCon project believes that one way out of this problem is to treat textual metadata as a vector in the process of detecting similarity across images, so that the computation includes terms like “corpse” in the metadata as a hint that there may be violence in the picture, in addition to detecting objects or similarity. Such a process will likely be the subject of a second workshop in May.
At this point, Jonathan Dentler also discussed the problem of implicit violence in photographs, using the example of the album La Prise de Samory, which is conserved by the SHD. An image that is apparently inoffensive can often have violent implications.
Figure 5: « Samory dans les rue de Saint Louis ». Album La Prise de Samory. Service Historique de la Défense (Vincennes).
(Figure 5) Extracted from its context, this image seems to show a simple parade or procession down the street of a town. In fact, this example is taken from an album created in 1899 by a French officer just after the conclusion of the war between the French army and Samory Touré’s Wassoulou Empire, which, at its apogee during the 1880s, extended over parts of present-day Guinea, Mali, and the Ivory Coast. In 1898 the French forces captured Touré and exiled him to Gabon.This photo shows the procession of Touré in Saint Louis after his capture. It is therefore a document of a form of public humiliation and a display of power, and is potentially sensitive in a way that simple object detection could not account for.
Project leader Dr. Julien Schuh then presented on issues about how to make a pipeline for artificial intelligence work designed for scientific and university purposes. He discussed the need for what experts call “explainable AI,” which is a notion that the technology should not be “black boxed” and opaque from the perspective of its users. This involves, for example, making training datasets publicly available so that they can be critiqued. A good pipeline would also involve input from experts and source communities, perhaps through platforms such as Zooniverse. Such input could then be verified by archive professionals. From a meta-level perspective of the organization of archival materials, Dr. Schuh discussed the ways in which artificial intelligence could build new connections at scale that would allow researchers to draw new connections that were hidden by layers of archivation handed down by colonial institutions.
The Eycon Project’s research engineer, Soumik Mallick, discussed the way in which various kinds of artificial intelligence tools such as pose detection or object detection can be deployed on datasets that have been extracted using the tool Layout Parser. He also explored reasons behind failures in pose recognition on historical photos. Data architect Marina Giardinetti discussed how the project uses Layout Parser in order to extract images from complex album and periodical page designs. Only when the images are extracted can “clean” datasets be created that would be useful in testing or producing tools informed by artificial intelligence. She explained how this methodology is build into the overall structure of EyCon’s database, which will survive after the end of the project, and demonstrated a “beta” version of the public-facing database.
After a short coffee break, the workshop reconvened in an open discussion format to gather the responses of the project’s archival partners in attendance. Representing the ECPAD were Véronique Pontillon (chargée des actions scientifiques) and Eléonore Plantard (chargée des collections entrées par voie extraordinaire). They explained that the choice was made at ECPAD to put warning screens up on an image-by-image basis for potentially sensitive images due to the fact that they were being put on line, and that this involved a considerable amount of work. One employee within the service is responsible for validating these sensitivity notices. On the back-end of the database, this person can check a box which implements a warning on the selected contents to hide potentially disturbing images. There is great difficulty in making this determination for certain limit cases—images that do not necessarily appear violent at first sight, such as burned corpses that are not recognizable as such, or the moments just before an execution rather than during the act itself. While there is some direction from the heritage code and the principle of one’s right to their image and respect of human dignity, there is not a specific framework for deciding whether an image is sensitive or not. Instead, a trained expert takes all final decisions on this question, taking into account the suggestions of those who processed the documents.
The database gives researchers the chance to alert archivists to consult the record to add additional information, including about sensitivity. These archivists can then add information on sensitive images in text form to the documentary records. Ideally, these notices are included for each image in a given file, but nevertheless occasionally there is only one notice for a whole batch of images. Another solution that was devised was putting up warning screens over sensitive images. Eléonore Plantard cautioned that it is important to realize that this platform is more of a publication tool than a tool for putting everything at the disposition of the researcher. There are files in the archives that contain pictures, without corresponding images put up on the site. These website pages simply alert researchers of the presence of a set of documents held in physical form only. The images put up on line are necessarily images on which ECPAD archivists have worked in order to enrich the metadata, and which have been considered from the point of view of potential sensitivity, but also from the point of view of copyright law.
Julien Gueslin, head of the museum department at La Contemporaine, explained that the photographic prints in the collection are put online using the national tool Calames in EAD-XML with a number of headings attached, including persons, places, and subjects, which enables research within the corpus concerned. He felt sure that EyCon’s work has great potential to enrich these descriptions. As with other private archives in the public domain, these images are under open license, so the archive user can access them directly without mediation. This means that the archivist needs to consider what kind of information to furnish on the webpage. For other photographic archives that are private and under copyright, consultation can instead only take place on the premises, which in the case of sensitive images provides a form of mediation. Before the accession into the public archive from a private source, these documents can be submitted to rules that concern archival use and potentially to the need for special authorizations or exemptions from public use. Typically, however, the donor singles out these exceptions not on the basis of sensitive content but rather out of concerns for familial or private life.
Concerning the Valois Albums in particular, Julien Gueslin pointed out that it is interesting to conceive of these photos produced by the SPA as sensitive images today, given that during the war they were made available publicly for journalists or diplomats for propaganda purposes. They were controlled in terms of censorship and propaganda, but nonetheless widely disseminated without seeming to cause a public outcry at the time or in later years. He pointed out that this demonstrates how definitions of sensitivity can change and evolve over time. While there may always be new criticisms of the collection according to new definitions of sensitivity, most of the users of the archive are historians, who are more cognizant of the bias inherent in all such archival records. Overall, Julien Gueslin questioned whether it was the role of archivists to decide what is sensitive and prevent access to historical data. The fact of being an archive—a space dedicated to research—means that this institution is somewhat set aside from concerns about sensitivity in other spaces. In an archive one can contextualize documents in a way not possible in the digital media space, which provides some cover from pressure to censor or suppress data.
More than blocking what is sensitive, he believes that users most want to enrich data further. For example, Julien Gueslin suggested that it would be interesting to automatically enrich the location data attached photos, or to give these automatically-located place-names in different languages, particularly in areas that have many different linguistic groups or where the sovereignty and dominant language has changed, such as in Eastern Europe. EyCon’s work in enriching data, and being able to make connections and links between archival collections, is thus the most interesting part of the project from his perspective.
Finally, Benjamin Doizelet, head of the iconographic archives division at the SHD, explained that the SHD is an archival service and thus the patrimony code applies to it. This code requires that the archive be open for public access, and they provide this access. At the same time, there is no Internet portal for the photographs held by the SHD. However, there is a communication and follow-up process with everyone who accesses the photographs. Because it is specifically a military archive, it is assumed that all of the files are potentially sensitive, though less because of death and destruction (which is common) than because of potentially sensitive ways in which the army represents itself in the documents. Certain archival documents might offend the dignity of known or recognizable persons and thus must not be made publicly available. Over time, his predecessors had marked certain boxes with a red sticker to signify potentially sensitive content. However, this has always been subject to revision over time and is without any official legal status, indicating rather a subjective and ethical point of view.
Ultimately, the workshop was a valuable opportunity to exchange with the project’s archival partners. The team showed both the potential and the problems of these tools, which are sure to reshape archival work. At the same time, our archival partners were able to educate us about the current practical, legal, and scientific conditions that currently govern they ways in which they make photographic heritage available to researchers and the public.