Ressources numériques en sciences humaines et sociales OpenEdition Nos plateformes OpenEdition Books OpenEdition Journals Hypothèses Calenda Bibliothèques OpenEdition Freemium Suivez-nous

Workshop Report: Multimodal Visual Similarity Algorithms and Digitized Photo Archives

Jonathan Dentler, Chercheur postdoctoral

Content warning: This post contains images that may be upsetting or distressing and display violence and the aftermath of violence. It is shown for educational and scholarly purposes and we ask that the reader engage with the material ethically and responsibly. 

On May 9th, EyCon team members and institutional partners from the Musée Quai Branly-Jacques Chirac, the Service Historique de la Défense (SHD), and the Établissement de Communication et de Production Audiovisuelle de la Défense (ECPAD) met at the ECPAD installation at the Fort d’Ivry to discuss the team’s exciting research into the possibilities opened up by multimodal visual AI in historical research in photography and visual culture. Daniel Foliard began by introducing the workshop’s objectives. Such tools can help group together similar digital images, tracing the circulation of images across different publications for example. They can also assist with functions such as detecting objects as well as stylistic or formal analysis. Finally, such tools can produce new metadata in a semi-automated manner by, for instance, suggesting probable attributions for variables such as a photograph’s date of production, medium or process, photographer, or place depicted. Visual similarity tools become much more powerful when combined with existing textual metadata as an additional vector in a multi-modal fashion, and this is precisely what EyCon project computer science intern Mohamed Salim Aissi has been working on recently. In this workshop, we wanted to demonstrate a proof-of-concept test based on a selection of photos from the EyCon database centered on the atrocities carried out by the Italian Army in Tripoli in the fall of 1911.

Figure 1: Pierre Schill, workshop presentation. “The Paradox of the Italo-Turkish War: an abundant production and wide distribution of images, and yet an ‘under-exposed’ conflict.” 

In order to demonstrate some of the possibilities and limitations of these tools, we invited Pierre Schill to share his research on the photographic coverage of the Italo-Turkish war. M. Schill is a researcher who has published on the photos from the war in Libya, and who is currently collaborating with the EyCon team on a forthcoming article on the topic. You can find his book on the topic here, which contains additional context and analysis for these images. The idea was to show how photo historians analyze and interpret historical photos in part by tracing the circulation of images from their production through their reproductions in various formats and publications. This can help establish context by making arguments for probable attributions, as well as show how photographs’ meanings are inflected by editorial choices such as cropping or captioning. (Figure 1) M. Schill began by suggesting that a paradox of the war in Tripoli is that, while it was highly photographed by newspaper correspondents, it is little remembered in Europe today. In part, he suggested, this is because of the difficulty in identifying camera operators as well as the diversity of sites and modes of conservation of the photographs. To get a better idea of the conflict and its visual records, it is necessary to draw links between the various archives holding the record in order to make attributions and establish context.

Figure 2: Pierre Schill, workshop presentation. “A scene with two prisoners and three soldiers: two photographs in circulation in the press.”

(Figure 2) Based on his painstaking archival work, M. Schill showed us a number of examples of how he compared photographs in different collections in order to establish that they depicted the same event from different angles, or that the same photographer or persons appear in different scenes. (Figure 3) He also demonstrated that photos have been misattributed; visual cues such as the angle of the sun in different images of the same scene show that it was photographed multiple times, likely by different photographers. 

Figure 3: Pierre Schill, workshop presentation. “Play of shadows and identity: beyond the analogy of the scene (public hanging, Dec. 6, 1911).”

The key question for this workshop was whether it might be possible to produce visual similarity tools that are capable of capturing this level of nuance, helping to perform tasks such as suggesting probable attributions for photographs. For example, using a multi-modal approach, a visual similarity tool could quickly find two instances of the same image in different publications, different formats, and/or different archives. If the image is attributed to a photographer in one instance but not in the other, the text-processing part of the process could perceive this attribution, and then suggest that attribution for the other instance of the image.

However, there are a number of dangers here. First, without a sufficiently sophisticated tool, this might simply automate misattributions. In the case of the false attribution of the hanging photograph to Gaston Chérau that Pierre Schill shared with us, would a multi-modal visual similarity approaches be sufficiently nuanced to recognize that the changed angle of the sun indicated that significant time had elapsed between exposures, and that this might be strong evidence for different photographers? If the database with which the algorithm was working included the image from Domenica del Corriere in which the image was credited to “Dott Comboni” (likely an Italian military photographer who took the image later in the day), perhaps it could deliver the correct suggestion. This raises two important points. First, it is important to work with very large amounts of data. Second, and even more crucially, it is essentially that the researcher be aware of the limitations of the database with which the machine is working, and that semi-automation of this function in no way means that the suggestions it generates are infallible. A second-order problem also results from the coexistence of these two factors: while working with large data sets is important—indeed, it is central to the appeal of algorithmic approaches—an ever-larger data set also risks obscuring the tool’s limits from the researcher, lending the results a misleading aura of authority.

Following M. Schill’s presentation, Mohamed Salim Aissi presented his new work on a multimodal visual similarity tool, which will be built into EyCon’s public-facing database. We first constructed a limited image database to test the tools. This set included images of loose photographs documenting the Tripoli atrocities from the Forbin fonds at the SHD, as well as instances of those images reproduced in publications such as The Daily Mirror and Excelsior. In order to make sure that the tools could pick out similarities among a much larger set of non-similar photos, we also included the roughly 60,000 images from the fonds Valois, images produced by the French military during the Great War.

(Figures 4-5) M. Aissi explained how visual similarity algorithms and textual similarity algorithms function, translating the technical subtleties into terms that were comprehensible for the non-specialist audience.

Figure 4: Mohamed Salim Aissi. Graphic explanation of layers produced by a visual similarity tool. 

Figure 5: Mohammed Salim Aissi. Graphic explanation of how textual metadata is vectorized to create probabilistic predictions for word order. This is how the textual aspect of the multimodal tool operates. 

Figure 6: Mohamed Salim Aissi. Demonstration of multimodal search function for retrieving similar images from a large corpus.

(Figure 6) He then demonstrated the maquette he created, which can be used to search solely on the basis of visual similarity, solely on the basis of text, or with a multimodal approach. For example, a user will be able to query the database using an image, and ask for the ten most similar images in the database. Alternatively, the user could enter in a query using textual terms, and see what photographs are proposed. Finally, the user can search using an image file, and add additional vectors to the search on the basis of text that should be associated with the image.

Figure 7: Mohammed Salim Aissi. Multimodal vectorization of metadata and resulting clustering of terms and images. 

(Figure 7) Using several examples, M. Aissi demonstrated that this combined approach delivered a much higher degree of accuracy in finding different versions of an image across different publications for which cropping or other formal features of the reproduction varied.

Finally, Marina Giardinetti discussed some of the issues of metadata treatment raised by visual similarity tools. Noting that the structure of the metadata in the database was essential for such tools to function well, she explained the EyCon team’s methodology for conserving metadata and its sources in ways that allow its production to be traceable and for artificial intelligence tools to function in the best way possible. EyCon integrates the source of all metadata attached to the image so that its origin is documented and conserved. The EyCon website will show the metadata, allowing the user to get an idea of some of the variables that the multimodal algorithm used in making its determinations. If an algorithm suggests metadata, this would be noted in the display.

In the closing discussion, the participants considered the potential pitfalls and strengths of such tools. In comparison to many of the AI-powered tools that have come to the forefront of public discussion in the past year or two—notably ChatGPT—a strength of this system is that it clearly shows whether metadata was produced by a human or a machine, and presents at least some ways for the user to get an idea of how the algorithm has arrived at its conclusions. One key associated issue is metadata standards—EyCon is promoting discussion on this issue in France, and encouraging the use of the International Image Interoperability Framework (IIIF) standard. IIF standardizes the delivery of image files from servers to web displays where they can be used and interacted with. It eases the delivery of richer functionality for image or audio-visual files beyond simple viewing, and, crucially, it attaches metadata to the digital object, which makes it useful for preserving context when it comes to archival images.

Finally, we discussed the question of design and user experience. From our perspective, the display and design we propose for the public-facing database has several benefits. First, because the user has to do some work to define the search parameters, it makes the process slightly clunky. Instead of making the experience as seamless and smooth as possible, we wanted to show the system’s artificiality, finding a balance between ergonomics and resistance. Ideally, this should help the user realize that the suggestions made by the AI are not infallible and they do not exclude other possibilities that might have been missed due to the limitations of the inputs or of the algorithm. We also discussed the potential uses for researchers: while these tools may help us reanimate images and their circulations, they are ultimately most helpful in accelerating certain time-consuming operations performed by the researcher, allowing them to focus on other tasks requiring higher order analysis and synthesis. The key challenge for such tools going into the future may be finding ways to translate the impenetrable depths of the technological “black box” into terms that the average user can understand, mediating its function such that the user has insight into the variables at play, or the possibilities that might have been missed. If this is not accomplished, there is a very real danger that AI could degenerate from a research tool into a hindrance, obscuring aspects of the analogue archive behind a veneer of computational objectivity.

We would like to thank Véronique Pontillon Valedon, chargée des actions scientifique at the ECPAD, for welcoming the workshop participants at the facility at the Fort d’Ivry.

Devika Mehra (Loughborough University) writes about her work on the EyCon project

I joined the EyCon (Visual AI and Early Conflict Photography) project as a postdoctoral research associate in January 2023. The project’s emphasis on using Artificial Intelligence to shed light on colonial history and the ethics of making sensitive archival records accessible to the public sparked my interest. Up until that point, my study focused on expanding access to children’s literature archives although with limited use of AI. I am curious about how larger audiences might interact with the history of children’s books and archive collections, which are generally studied by children’s literature scholars and archivists who have prior knowledge of these holdings and the required permissions to access them. I was keen to learn how AI could improve accessibility to diverse archival collections.

The fact that EyCon also addresses the ethics of AI concerning cultural heritage and the accessibility of sensitive archival collections was helpful. As someone who was exploring the issue of diversity in international children’s archives (print and visual), I had the chance to work with an interdisciplinary team of colleagues that gave me an insight into different facets of AI that can interpret not easily accessible history.

EyCon brings together archivists, Digital Humanists, Computer Scientists, and other stakeholders to explore how to make archives more accessible. These accessibility issues have arisen as collections become data in the process of digitising more materials. As they can include sensitive information or be so large that they are practically hard to discover and utilise, these digitised files might be difficult to be accessed easily by the public.

EyCon intends to connect, analyse, and comment on these sources, drawing on AI to improve the discoverability and usability of neglected and dispersed information on colonial armed conflicts up to 1918. EyCon’s main goal is to evaluate how beneficial computation is for visualising, navigating, and analysing massive visual corpora. EyCon intends to harness computation, question it, and evaluate new visualisation techniques when they are used in historical research. In doing so, it responds to the moral and intellectual concerns brought up by the use of AI techniques to analyse contentious historical events.

When I first arrived in January, I helped with the copyediting of the EyCon website. I learned more about the team, the project, and the way the website was created as a result of it. Soon after, I began work on helping organise the second workshop’s planning for the EyCon project. Here, we have gathered our regular working groups to discuss the prospects for AI and archives and to devise fresh initiatives.

“Using Visual AI Applied to Digital Archives,” the second EyCon workshop, has been organised in collaboration with the Imperial War Museums (IWM) Institute, the IWM’s centre for research and knowledge hub. The workshop will take place in person from June 8 through June 9, 2023, at the IWM in London. It will offer a forum for discussing the enormous difficulties that academics, archivists, and other professionals encounter when dealing with a sizable volume of visual data, especially sensitive material. A combination of presentations on Day 1 and interactive sessions on Day 2 will make up the workshop, which will enable interaction with French and UK cultural organisations affected by the difficulties of sensitive digitised archives.

I was given the chance to work on the AEOLIAN project at about the same time. The final AEOLIAN workshop provided me with more information about the project and the potential of the AI and GLAM sectors. Exploring the effect this project had on academic and professional practitioners was made possible by working on the AEOLIAN website and posting the films. These videos have been crucial for raising awareness of the project’s work and serving as a helpful frame of reference for describing some of the most critical problems facing AI today. The AEOLIAN network’s interdisciplinary character has produced articles that are applicable to a wide range of industries and show the potential of AI and some of its possible drawbacks in ways that go well beyond archives and cultural heritage.

I continued working on two special issues, one for the AEOLIAN network and the other for the EyCon project, in addition to the workshops. I helped with the AEOLIAN special issue, ‘Applying Innovative Technologies to Digitised and Born-Digital Archives’ for JOCCH (Journal on Computing and Cultural Heritage), which involved contacting authors and journal editors, inviting reviewers, and reviewing. It made me aware of how much work is being done in the areas of AI, digital archives, and cultural heritage organisations. So, keep an eye out for this special issue for the opportunity to explore the application of innovative technologies, including AI to digital and born-digital archives. DHQ (Digital Humanities Quarterly) will publish the EyCon special issue focusing on the theme of ’Using Visual AI Applied to Digital Archives’, which is currently being written.

Being a part of this interdisciplinary network over the past few months has been great. I am eager to see what this quickly developing field will bring us next. The discourse around archives and artificial intelligence that the AURA, AEOLIAN, EyCon, and LUSTRE projects have begun will pave the way for future innovations and collaborations. We look forward to growing the network in the future, and I am curious to find out more about how technology and history might combine to make archives more accessible.

The circulation of the image of Clicko “the wild dancing bushman” between French, British and American archives.

By Aude Dumont

Thanks to the Eycon project’s partnership with the Service historique de la Défense (SHD) in Vincennes, we have been able to digitise the fond Victor Forbin, an archive that contains about 55 000 photographs from all over the world dating from approximately 1880 to 1920. Forbin collected images from Cameroon, Palestine, Turkey, and South Africa, among other locations. Although it is not easy to trace the history of the circulation of these photographs, many of them were published in the British press, and often have stamps on the verso from the Daily Mirror or the Illustrations Bureau. Such stamps make it possible to trace their circulation at least partially. The collection at the SHD was acquired from Forbin’s home in Clamart, in the Paris suburbs, and the fond’s boxes are classified by country.

      Here we will focus on file 2K247 on South Africa, and on one photograph in particular that you can see below.

Photography held at the Service historique de la Défense, in Vincennes. SHD GR 2K247 11: “Fonds Rumpf, Archives photographiques de Victor Forbin”

On the right picture: The pencil caption at the top state, “Capt Courténay d’Esterre with the wild Dancing Bushman which he brought from S. Africa.” We then have Victor Forbin’s signature below, a measurement written in blue, unidentified numbers at the bottom left. There is also a stamp of the British Press illustration office.

There seems to be no information regarding the photographer’s identity. Similarly, Captain Courténay d’Esterre does not appear to be well known in other archives. He was certainly an English missionary in charge of documenting indigenous South African traditions and bringing people back to England to be exhibited in human zoos or universal exhibitions. This is the case with this man on the right, whose identity is unknown and is simply called the “wild dancing bushman”. We know only that he was brought back to England. Regarding the photograph’s date, it was certainly made around the beginning of the twentieth century, when English missions became increasingly numerous after the colonisation of this territory following the victory in the Boer wars. This can also be determined by the photograph’s varnish aspect, which is beginning to crack slightly, but also by a light yellowish or brown shade. The technique used was probably an old baryta silver print, also known as gelatino-chloro-silver bromide paper print, a process widely used from 1880 until 1930.

      One thing that is immediately obvious is the white outline that has been added over the image. The addition of this outline can be seen, as embossed on the reverse side of the image. Closer examination reveals that hints of white colour have been added to the two individuals to increase the figures’ contrast. It is likely that this addition of white paint was intended to make the two protagonists more visible once the photograph was published in smaller print. The stamp of the Illustrations Bureau on the back confirms this hypothesis of publication in the British press. The image is then divided into two parts. There is the background, rather dull and without any real importance. While the foreground looks like a staged scene in which the English missionary wants to show off his finding, with a man at his side who, to a British audience, would have appeared much smaller than average. This staging of difference, emphasised by the comparison between the missionary and the dancer, is part of a widespread phenomenon at the time of Europeans measuring physical differences between human populations, and turning these differences into spectacles through the press, or universal exhibitions. He is also shown shirtless with an animal skin around his waist. Underneath their feet is a wild beast skin, highlighted by white outlining, as if to give a more exotic feel to the captain’s mission.

Unfortunately, much information that would help to trace the photograph’s history and the story of its circulation in the press is missing. It is not known, for example, who added the white outline. Was it the photographer himself, or was it the newsroom artists of a British newspaper in which this photograph was published? Are there several copies of this photograph, and perhaps an original without the white line? To write this photograph’s history in a more complete form, to understand whether the man on the right was forced to pose for this photo or not, and to know what happened to him, it is necessary to try to connect this archive to other archives, to have access to other points of view on this moment.

From the name of Captain Courténay D’Esterre, it was only possible to find one newspaper clipping. This is the Daily Mirror of January 16, 1913 – available below and in the British Newspaper Archive website – in which we can find exactly the same photograph without the white outline, as well as three other photographs of this “wild dancing bushman.” In the other three figures, as well as in the caption, this man is presented almost as an animal, or at least as a subhuman who only dances and sleeps. He and his “tribe” are presented sometimes as hunters, dancers, and he is sometimes a person “in his wild state.” His height, which is the main object of curiosity here, is mentioned as “4ft. 1in.” The different angles from which the person is shot and the accompanying caption demonstrate how the press popularized scientific racism. It should also be noted that the real name of the captain is Courteney D’Esterre and not Courténay D’Esterre as written by Victor Forbin on the back of our archive.

“Wild bushman who sleeps in the day and dances at night” Daily Mirror, January 16, 1913

      After attempting to trace the history of this photograph from the time it was taken in South Africa to its publication in the Daily Mirror in England, a new challenge emerges. How to trace the history of the photographed subject so as to give him a name, to understand the history of his life, and to ensure he does not remain a static picture or an unknown “wild dancing bushman”? Firstly, we now know that the SHD photo dates from around 1913, as it was published in that year, and that this South African man was brought back to England around the same time.

“Clico Wild Dancing, South African Bushman,” unknown photographer, United States at Ringling Bros. and Barnum & Bailey Circus, circa 1928

Based on the vague term “wild dancing bushman”, it is possible to find a man named Clicko, sometimes spelled Clico, who also went by the name Franz Taaibosch. This man turns out to be the same as the one in our photograph. This can be seen from his facial features and the lump on his right arm – see the picture above. He was first “brought” back to Europe, as written in the Daily Mirror where he performed dances in Paris, London, but also in Spain and Germany between 1912 and 1913. Then he moved to Havana in 1916 to join the Cuban circus, Santos Artigas Gran Circo before arriving in New York in 1917 at Brooklyn’s Coney Island amusement park.[1] He joined the Ringling Bros. and Barnum & Bailey Circus Company formed in 1919 from the fusion of Ringling Bros with Barnum & Bailey. This company was familiar at that time with “freak shows” throughout the United States, i.e. the presentation of animals and people with abnormal physiognomy to amaze the American audience. The photo above mentions the name of this circus in which he took part. His name is often accompanied by the short description “wild dancing bushman” because his role in this circus was essentially to perform a traditional Korana dance, the ethnic group to which he belonged. It is important to mention that he was Korana, a tribe of the Khoekhoe ethnic group, and not a Bushman, which refers to another ethnic group, the San. The only academic book to treat this little-known history was written by Neil Parsons and published in 2010.[2] Thanks to this more detailed study, it is possible to question the veracity of Clicko’s description in the Daily Mirror. Indeed, Parsons wrote that he was probably born around 1870, which means that he would have been about 43 years old in 1913 and not around 60 to 70 years old as written in the newspaper clipping.

The intention to make him appear older than he really was in this newspaper was deliberate in order to attract public curiosity about him, and it was not just a mistake.

“He was billed as the “first specimen of the most primitive race ever brought before the public”, that he could dance for eight hours continuously and that he was more than one hundred years old. To attract paying audiences the unusual, the exotic, “a unique specimen of human nature”, had to be emphasized.”[3]

However, at no time did Neil Parsons mention Captain Courteney d’Esterre, and it is not possible to find any other trace of the four photographs published in the Daily Mirror, except for the one on the top right where he is in a tree which became a publicity postcard distributed in England in 1913 as seen in Bernth Lindfors’ book.[4] The card is composed of this picture and a writing in white paint saying “Yours faithfully, The Wild Dancing Bushman.”

      It was also possible to find evidence that Captain Courteney d’Esterre may not have been the first European to meet him before he left for Europe. In a small booklet available at the Milner Library of Illinois State University,[5] an Irishman named Captain Maurice Patrick (Paddy) Hepston, also known as Morris Epstein in Ireland, claimed to have found a Bushman who was Franz Taaibosch, in the Kalahari Desert while hunting ostriches. Bushmen were then considered an extinct race by Europeans who searched for them in vain. This man lived on a farm near Kimberley and he was a gold miner, farmer and diamond digger. He refers to himself in the third person and boasts about his discovery that Dr Livingstone spent years trying to make in the same desert with no result. He speaks of Franz as an animal, or as a “specimen” sometimes “docile” and compares him to the ostriches he was hunting.

It was the same Paddy Hepston whom Franz followed to Ireland and Havana and who pushed him to perform in circuses. Hepston wanted to escape from the Aborigines Protection Society in London, which was pursuing him because of his mistreatment towards Franz.

Hepston then did everything possible to keep track of him and continued to exploit him in circuses. One of the pieces of evidence is found in the United States, in a photograph taken by the famous photographer Charles Eisenmann, who produced cabinet card pictures or views of human oddities from the Barnum and Bailey circus. On the verso of one of his 1930 portraits of Franz, held at Syracuse University in the State of New York, it is written “Clicko the bushman actually captured from head hunters by Capt. Hepston 1930 with Ringling Circus.”[6]

      Nevertheless, this story of a discovery in the Kalahari Desert seems romanticised, and only served to add to the mystery of the Bushmen people, who were hard to be found. It is more likely that Hepston met Franz in Kimberley when he was already in the company of a certain Christiaan Willem Roberts, who kept sheep in Middleburg, and who employed Franz. They travelled to Kimbereley where Paddy Hepston later hired him in a local circus. Captain Courteney d’Esterre may also have met him in Kimberley. Neil Parsons himself admits that this year of Franz’s life, which corresponds to the year of our photograph and its publication in the Daily Mirror, remains unclear.

“The biggest gap in the known biography of Franz Taaibosh is the eleven months between July 1912, when Paddy Hepston was given official permission at Kimberley to take on “Bushman W.D.” as his servant, and June 1913, when the Wild Dancing Bushman made his first appearance in a London theatre.”[7]

      Since there is little historical evidence of Franz Taaibosch entering a local circus in Kimberley, it is difficult to know whether he was forced to join Paddy Hepston, or whether he did so voluntarily, to earn money. This requires a deeper understanding of the social and economic context of the Khoekhoe and San (Bushmen) ethnic groups at that time to determine if there was a proletarianization, and some form of indirect coercion of their labour.[8] Similarly, Captain Courteney d’Esterre claimed to have “brought” him to England, as if Franz had simply followed him without having the choice. We do not know if this statement is true. But we also know that he chose to stay with the Cook family in the United States. Tracing the history of his life choices or possible coercion is therefore difficult since there is no direct testimony from him. Furthermore, Neil Parsons’ book relies heavily on the testimony of Evelyn Cook, who herself may have had a biased view of Franz’s life, between choice and coercion.

      If we have finally been able to put a name on this face, then rehabilitating his memory and the story of his life cannot be done only through the prism of European or American sources, as with the Daily Mirror’s cut or Neil Parsons’ book. It is important to know his story from the point of view of his descendants – if he has any – or at least from the point of view of people of the same or a close ethnicity, who may have a different view of the racial discrimination he suffered in his time. They can tell us more about his life before he left South Africa, about everything that Franz himself could not convey to us and that we only know through European protagonists who told his story for him. It is important that his story is not only narrated here by Europeans and Americans, even though it took place on three continents. Only a connective and polyphonic history allows for a comprehensive view of Clicko’s life.

      In fact, among the few works on his life, a fairly detailed podcast has caught our attention. It was produced by Shaldon Ferris, a Khoisan and Indigenous Rights Radio Coordinator. The speaker talking about Franz/Clicko is Letitia Timas Peterson, an advocate for Indigenous African descent rights in South Africa. The podcast was published one year ago on the Cultural Survival website.[9] This organisation wants to give a voice to cultures and ethnicities oppressed by various states around the world, including South Africa. The podcast was produced on the occasion of the International Day For People Of African Descent, a celebration created by the United Nations in 2021. The Cultural Survival organization was created in 1972 by David Henry Peter Maybury-Lewis a British anthropologist, ethnologist, and an activist for indigenous peoples’ human rights.

Here is an interview conducted in February 2023 with Letitia Timas Peterson and Shaldon Ferris about Clicko/Franz Taaibosch.

Aude Dumont: How did you get so much information about Franz Taaibosch, whom you were able to present in your podcast, when so little historical work has been done on him?

Letitia T. Peterson and Shaldon Ferris: He is one of our ancestors, not direct ancestor but one from our ethnicity. Most of the information we have about Franz Taaibosch comes from the information Neil Parsons collected. But there are also oral testimonies circulating in his ethnic group, as indirect descendants are still with us so that this story will not be forgotten. I don’t have their contact. But these accounts tell of his life as a house servant in Kimberley before he left South Africa, and then of the possible mistreatments he received in the United States.

These oral testimonies are certainly based on Parsons’ book and other archives as it was certainly difficult, if not impossible, for Franz to have a direct contact with his family. Indeed, he died in the United States without ever having returned to South Africa. His grave is in Cedar Park Cemetery in Hudson, New York, with that of the Sullivan-Cook family, whom he stayed living with. It was this family, and in particular Evelyn Cook, who was able to give so much information about Franz to Neil Parsons. Not returning to South Africa was his choice, he was not forced and considered the Sullivan-Cook as his family. We also know that he was fluent in Afrikaans and perhaps his relatives were too, although this is difficult to know. However, he was a prisoner of the circus he worked for, where it was easier for the show to pass him off as an “ignorant savage” which may have prevented him from communicating. He also knew some English for example, but never spoke it.

A.D.: Do you know how to write exactly the tribal name of Franz?

L.T.P: The tribal name of Franz was !Gaub. He was from the clan called !Gaibe or Taaibosch !Korana, Korana or !Ora people.[10]

A.D.: Do you know something about the descendants of Franz Taaibosch? 

L.T.P and S.F.: There are many with the same surname, one will have to investigate to find out if he had children, and then we have to figure out who was directly linked to him. There is Taaibos, and Taaibosch, which has a Dutch spin on it.

For Neil Parsons, “There is at least one Taibos or Taaibosch family still living today in the town of Middleburg, close to Haydon on the northern edge of the Sneeubergen.”

A.D.: Did he talk about his life in America and the stereotypes and racism he possibly faced?

L.T.P and S.F.: We don’t really know sorry. Perhaps there is more information about this in the book of Neil Parsons.

A.D.: Why were you interested in the story of Franz Taaibosch?

L.T.P: I was first interested in the story of Sarah Baartman in 2002. You know this is the woman who was taken to England to be exhibited in human zoos because of her unusual appearance. She was called by Europeans the “Hottentot Venus”[11]. She was then exhibited for years in the National Museum of Natural History in Paris after her death. Cuvier had dissected her and then reconstructed her statue and skeleton for exhibition. If I was working on this subject in the early 2000s, it was because we had requested, with the support of the South African state, the restitution of her skeleton so that she could be buried according to our customs. After an initial refusal by the French state, we were finally able to repatriate her body in 2002. I was there for the burial. It was while making research on her that I heard about Franz Taaibosch, so I was able to make a podcast about him to tell all the things I discovered.

A.D: I could find you through the podcast you made about Franz Taaibosch on the Cultural Survival website. Why did you make a podcast about him on the Cultural Survival website? Can you tell us about the purpose of Cultural Survival?

L.T.P and S.F.: Even in the context of commemorations such as International Day for People of African descent (which is a new commemoration by the way), it is very seldom that people will talk about Indigenous peoples. Franz’s story is an international one, and one that is seldom heard. It is important to let the world know the history of Indigenous South Africans. Cultural Survival advocates for Indigenous Peoples’ rights and supports Indigenous communities’ self-determination, cultures and political resilience since 1972. Here is a link to the homepage of our website for all the people who are interested: cs.org.

A.D: What do you think of the archive from Vincennes I sent you? What was your reaction? Are you surprised to know that it is in France?

S.F.: Yes indeed. I think archives as well as artefacts should be returned one day. Why is it in Europe when it is our relatives? Europeans in the past have studied all that they can, for the sake of their own narrowmindedness, so they can return it now.

This research is taking place in the context of the claim of artefacts by various African states to European museums, be they statues, bodies such as that of Sarah Baartman, or cult objects. Besides, the digitisation of these archives that we have carried out with the Eycon project allows easier access for anyone in the world, not just a European audience. But in the case of our photograph, it is complicated to argue that it should be returned to South Africa or not, as this would imply that the material object itself belongs to South Africans. But, as we said earlier, the photo was probably taken by an English missionary. The issue here should then be considered differently. It is not the possession of the material object that is in question. Instead, the question is who owns the content of the picture, i.e. the representation of the person who was photographed, in this case Franz Taaibosch. It is difficult to guess whether he was forced to pose for this photo and the others that were published in the Daily Mirror. But if so, can we consider that his image is stolen?

A.D.: Do you know Captain Courtenay D’Esterre? Have you ever seen this photograph that I found in Vincennes?

L.T.P: No, never, I’m sorry.

A.D: I feel that there is a lack of historical works on Franz Taaibosch and maybe on other people from this ethnicity in general. What can you tell me about this? Do you feel the same way in South Africa? Do you think more historians should work on such subjects?

S.F: Yes, we have not learnt anything about Franz, or Sarah Baartman or Dawid Stuurman in school. The ones who write history, writes their own versions. Now, it should be us who research and tell these stories.

      This interview may be published later by Shaldon Ferris on the Cultural Survival website as a follow-up to the podcast on Franz Taaibosch.

References:

Neil Parsons, Clicko: The Wild Dancing Bushman, University of Chicago Press, December 2010

Neil Parsons: Clicko: The Wild Dancing Bushman. Review by: Sheridan Griswold. Botswana Notes and Records, Vol. 42 (2010), pp. 192-194

Lindfors, Bernth: Africans on Stage: Studies in Ethnological Show Business. Bloomington Cape Town: Indiana University Press, 1999

Capt. M. P. Hepston: The wild dancing bushman, his history and story of his discovery in the Kalahari Desert, South Africa. Dreamland Circus Sideshow (Coney Island, N. Y.), 1917/1918. Special Collections at Milner Library, Illinois State University (Circus and Allied Arts Collection), https://digital.library.illinoisstate.edu/digital/collection/CircusSelects/id/3

Becker, Ronald G.: “Ronald G. Becker collection of Charles Eisenmann photographs”, 1836-1960. Box 14: 1123-115. Syracuse University Libraries, Special Collections Research Center, https://library.syracuse.edu/digital/guides/b/becker_eisenmann.htm

Wilmsen, Edwin: Herders, Warriors, and Traders, Pastoralism in Africa. Chapter “Pastoro-Foragers to “Bushmen”: Transformations in Kalahari Relations of Property, Production and Labor”. Routledge, 1991

Good, Kenneth: Bushmen and Diamonds (Un)Civil Society in Botswana. Uppsala, Nordiska Afrikainstitutet, 2003

Shaldon Ferris and Letitia Timas Peterson: Cultural Survival podcast about Franz Taaibosch, https://rights.culturalsurvival.org/international-day-people-african-descent


[1] Neil Parsons: Clicko: The Wild Dancing Bushman. Review by: Sheridan Griswold. Botswana Notes and Records, Vol. 42 (2010), pp. 192-194

[2] Neil Parsons, Clicko: The Wild Dancing Bushman, University of Chicago Press, December 2010. About the age of Franz Taaibosch, see also the review of Sheridan Griswold.

[3] Clicko: The Wild Dancing Bushman by Neil Parsons. Review by: Sheridan Griswold. Botswana Notes and Records, Vol. 42 (2010), pp. 192-194

[4] Lindfors, Bernth: Africans on Stage: Studies in Ethnological Show Business. Bloomington Cape Town: Indiana University Press, 1999. Chapter 8: ““Clicko”: Franz Taaibosch, South African Bushman Enterntainer in England, France, Cuba, and the United States, 1908-1940” p. 203

[5] Capt. M. P. Hepston: The wild dancing bushman, his history and story of his discovery in the Kalahari Desert, South Africa. Dreamland Circus Sideshow (Coney Island, N. Y.), 1917/1918. Special Collections at Milner Library, Illinois State University (Circus and Allied Arts Collection), https://digital.library.illinoisstate.edu/digital/collection/CircusSelects/id/3

[6] Becker, Ronald G.: “Ronald G. Becker collection of Charles Eisenmann photographs”, 1836-1960. Box 14: 1123-115. Syracuse University Libraries, Special Collections Research Center, https://library.syracuse.edu/digital/guides/b/becker_eisenmann.htm

[7] Parsons, Neil: chapter 4: “Disappearance to Australasia or the Far East”

[8] On the changing labour market in the Kalahari, including the emergence of gold and diamond mines, see: Wilmsen, Edwin: Herders, Warriors, and Traders, Pastoralism in Africa. Chapter “Pastoro-Foragers to “Bushmen”: Transformations in Kalahari Relations of Property, Production and Labor”. Routledge, 1991

Good, Kenneth: Bushmen and Diamonds (Un)Civil Society in Botswana. Uppsala, Nordiska Afrikainstitutet, 2003.

[9] https://rights.culturalsurvival.org/international-day-people-african-descent

[10] The “!” symbol refers to the click sound. The nickname Clico also comes from the clicking sound of his native language.

[11] « Hottentot » was used by early Dutch settlers to talk about Khoekhoe people, and probably meant stutterers.

Workshop Report: Working with Potentially Sensitive Digital Photographic Archives with the Help of Multimodal Deep-learning

Jonathan Dentler, Chercheur postdoctoral

On the 6th of March, the EyCon Project (funded by the Labex “Les Passés dans le Présent” and the AHRC) organized a workshop on the potential benefits and problems of using multi-modal artificial intelligence (AI) approaches to work with archives of sensitive digitized historical photographs. The workshop was attended by archivists and museum professionals from the Établissement de la Communication et de la Production Audiovisuelle de la Défense (ECPAD), the Service Historique de la Défense (SHD), and La Contemporaine museum and archive, and was held at La Contemporaine. The workshop’s aim was to collectively explore the possibilities for computational and machine-learning approaches to making potentially sensitive colonial archives more navigable and discoverable. The EyCon team presented a theoretical pipeline or workflow with which heritage institutions could carry out the computer-assisted analysis of images, enriching their metadata with a number of tools. In particular, we aimed to think through ways that these tools could be used to identify and contextualize sensitive images. After the presentation, the floor was opened for a dialogue between the research team and the archival partners about the contemporary working practices around metadata and sensitive images, and the possibilities and challenges of these technologies.

            After introductions, EyCon postdoctoral research fellow Dr. Jonathan Dentler introduced the project and the large questions that have been recently introduced in the public sphere and in scholarship about the ethics and politics of disseminating sensitive images online, particularly those depicting colonial violence. He suggested that a key proposition of the Eycon project is that, “critical and multimodal artificial intelligence offers a means of balancing the ethical treatment of material with the necessity of improving access to colonial archives for researchers and the public.” Using examples from the holdings of the ECPAD, he also introduced the issue of metadata, and how archives handle what terms to attach to photographs in order to contextualize them and make them searchable. AI could potentially solve the problem of scale in the task of adding metadata to large photography collections. Furthermore, by automatically detecting visual similarity, it could potentially help trace image circulation both transnationally and trans-imperially among linked archives, helping to de-silo national and imperial histories.

Another computational tool that could be used to enrich metadata are various forms of object detection and “safe search” functionalities that automatically detect violence and other forms of sensitive material.

Figure 1: Test of the Google Vision API “safe search” function on a photograph of the corpse of a german soldier from the fonds Valois.

(Figure 1) Unfortunately, as Jonathan Dentler discussed, such tools have been shaped by the legal and commercial needs of companies, and are not technically, scientifically, or ethically suited to historical material. (Figure 2) Indeed, out of 199 images extracted from the fonds Valois photo albums created by the French Service Photographique de l’Armée (SPA) during World War One and held by La Contemporaine that contain the word “corpse” in their textual metadata, only 12% were recognized by Google Vision API’s “safe search’ function as “possibly” or “probably” containing violence.

Figure 2: Results of a Google Vision API “Safe Search” with a corpus of 199 images containing the word “cadavre” or “corpse” in their metadata in the Valois albums.

            EyCon’s newly hired computer vision engineer Mohamed Salim Aissi, who recently earned a masters degree in computer science from Sorbonne Université, then introduced the technical details on visual similarity operations, explaining the architecture of auto-encoding that permits the extraction of information from images in order to determine similarity.

Figure 3: Visual Similarity method results.

(Figure 3) Using this technique, he was able to automatically find similar objects in the Valois albums, including, for example, shots of dugouts and trenches, injured faces, diplomatic meetings pictured from slightly different angles, and statues.

Figure 4: Images of statues made by the SPA and grouped in the same zone of the pixplot.

(Figure 4) These images were represented in clusters in a pixplot. Jonathan Dentler pointed out, however, that without fine-tuning of this approach, it could create false positives, for example grouping a photograph of burned bodies inside a destroyed tank with simple landscape shots on the basis of the tonal values of the pixels distributed across the image. The EyCon project believes that one way out of this problem is to treat textual metadata as a vector in the process of detecting similarity across images, so that the computation includes terms like “corpse” in the metadata as a hint that there may be violence in the picture, in addition to detecting objects or similarity. Such a process will likely be the subject of a second workshop in May.

            At this point, Jonathan Dentler also discussed the problem of implicit violence in photographs, using the example of the album La Prise de Samory, which is conserved by the SHD. An image that is apparently inoffensive can often have violent implications.

Figure 5: « Samory dans les rue de Saint Louis ». Album La Prise de Samory. Service Historique de la Défense (Vincennes).

(Figure 5) Extracted from its context, this image seems to show a simple parade or procession down the street of a town. In fact, this example is taken from an album created in 1899 by a French officer just after the conclusion of the war between the French army and Samory Touré’s Wassoulou Empire, which, at its apogee during the 1880s, extended over parts of present-day Guinea, Mali, and the Ivory Coast. In 1898 the French forces captured Touré and exiled him to Gabon.This photo shows the procession of Touré in Saint Louis after his capture. It is therefore a document of a form of public humiliation and a display of power, and is potentially sensitive in a way that simple object detection could not account for.

             Project leader Dr. Julien Schuh then presented on issues about how to make a pipeline for artificial intelligence work designed for scientific and university purposes. He discussed the need for what experts call “explainable AI,” which is a notion that the technology should not be “black boxed” and opaque from the perspective of its users. This involves, for example, making training datasets publicly available so that they can be critiqued. A good pipeline would also involve input from experts and source communities, perhaps through platforms such as Zooniverse. Such input could then be verified by archive professionals. From a meta-level perspective of the organization of archival materials, Dr. Schuh discussed the ways in which artificial intelligence could build new connections at scale that would allow researchers to draw new connections that were hidden by layers of archivation handed down by colonial institutions.

             The Eycon Project’s research engineer, Soumik Mallick, discussed the way in which various kinds of artificial intelligence tools such as pose detection or object detection can be deployed on datasets that have been extracted using the tool Layout Parser. He also explored reasons behind failures in pose recognition on historical photos. Data architect Marina Giardinetti discussed how the project uses Layout Parser in order to extract images from complex album and periodical page designs. Only when the images are extracted can “clean” datasets be created that would be useful in testing or producing tools informed by artificial intelligence. She explained how this methodology is build into the overall structure of EyCon’s database, which will survive after the end of the project, and demonstrated a “beta” version of the public-facing database.

            After a short coffee break, the workshop reconvened in an open discussion format to gather the responses of the project’s archival partners in attendance. Representing the ECPAD were Véronique Pontillon (chargée des actions scientifiques) and Eléonore Plantard (chargée des collections entrées par voie extraordinaire). They explained that the choice was made at ECPAD to put warning screens up on an image-by-image basis for potentially sensitive images due to the fact that they were being put on line, and that this involved a considerable amount of work. One employee within the service is responsible for validating these sensitivity notices. On the back-end of the database, this person can check a box which implements a warning on the selected contents to hide potentially disturbing images. There is great difficulty in making this determination for certain limit cases—images that do not necessarily appear violent at first sight, such as burned corpses that are not recognizable as such, or the moments just before an execution rather than during the act itself. While there is some direction from the heritage code and the principle of one’s right to their image and respect of human dignity, there is not a specific framework for deciding whether an image is sensitive or not. Instead, a trained expert takes all final decisions on this question, taking into account the suggestions of those who processed the documents.

            The database gives researchers the chance to alert archivists to consult the record to add additional information, including about sensitivity. These archivists can then add information on sensitive images in text form to the documentary records. Ideally, these notices are included for each image in a given file, but nevertheless occasionally there is only one notice for a whole batch of images. Another solution that was devised was putting up warning screens over sensitive images. Eléonore Plantard cautioned that it is important to realize that this platform is more of a publication tool than a tool for putting everything at the disposition of the researcher. There are files in the archives that contain pictures, without corresponding images put up on the site. These website pages simply alert researchers of the presence of a set of documents held in physical form only. The images put up on line are necessarily images on which ECPAD archivists have worked in order to enrich the metadata, and which have been considered from the point of view of potential sensitivity, but also from the point of view of copyright law.

            Julien Gueslin, head of the museum department at La Contemporaine, explained that the photographic prints in the collection are put online using the national tool Calames in EAD-XML with a number of headings attached, including persons, places, and subjects, which enables research within the corpus concerned. He felt sure that EyCon’s work has great potential to enrich these descriptions. As with other private archives in the public domain, these images are under open license, so the archive user can access them directly without mediation. This means that the archivist needs to consider what kind of information to furnish on the webpage. For other photographic archives that are private and under copyright, consultation can instead only take place on the premises, which in the case of sensitive images provides a form of mediation. Before the accession into the public archive from a private source, these documents can be submitted to rules that concern archival use and potentially to the need for special authorizations or exemptions from public use. Typically, however, the donor singles out these exceptions not on the basis of sensitive content but rather out of concerns for familial or private life. 

            Concerning the Valois Albums in particular, Julien Gueslin pointed out that it is interesting to conceive of these photos produced by the SPA as sensitive images today, given that during the war they were made available publicly for journalists or diplomats for propaganda purposes. They were controlled in terms of censorship and propaganda, but nonetheless widely disseminated without seeming to cause a public outcry at the time or in later years. He pointed out that this demonstrates how definitions of sensitivity can change and evolve over time. While there may always be new criticisms of the collection according to new definitions of sensitivity, most of the users of the archive are historians, who are more cognizant of the bias inherent in all such archival records. Overall, Julien Gueslin questioned whether it was the role of archivists to decide what is sensitive and prevent access to historical data. The fact of being an archive—a space dedicated to research—means that this institution is somewhat set aside from concerns about sensitivity in other spaces. In an archive one can contextualize documents in a way not possible in the digital media space, which provides some cover from pressure to censor or suppress data.

More than blocking what is sensitive, he believes that users most want to enrich data further. For example, Julien Gueslin suggested that it would be interesting to automatically enrich the location data attached photos, or to give these automatically-located place-names in different languages, particularly in areas that have many different linguistic groups or where the sovereignty and dominant language has changed, such as in Eastern Europe. EyCon’s work in enriching data, and being able to make connections and links between archival collections, is thus the most interesting part of the project from his perspective.  

            Finally, Benjamin Doizelet, head of the iconographic archives division at the SHD, explained that the SHD is an archival service and thus the patrimony code applies to it. This code requires that the archive be open for public access, and they provide this access. At the same time, there is no Internet portal for the photographs held by the SHD. However, there is a communication and follow-up process with everyone who accesses the photographs. Because it is specifically a military archive, it is assumed that all of the files are potentially sensitive, though less because of death and destruction (which is common) than because of potentially sensitive ways in which the army represents itself in the documents. Certain archival documents might offend the dignity of known or recognizable persons and thus must not be made publicly available. Over time, his predecessors had marked certain boxes with a red sticker to signify potentially sensitive content. However, this has always been subject to revision over time and is without any official legal status, indicating rather a subjective and ethical point of view.

            Ultimately, the workshop was a valuable opportunity to exchange with the project’s archival partners. The team showed both the potential and the problems of these tools, which are sure to reshape archival work. At the same time, our archival partners were able to educate us about the current practical, legal, and scientific conditions that currently govern they ways in which they make photographic heritage available to researchers and the public.

Applying Pose Recognition to the World War One Valois Albums: Some AI Avenues for Photography History

Jonathan Dentler, EyCon Postdoctoral Research Fellow

The Early Conflict Photography 1890—1918 and Visual AI (EyCon) project is partnered with several UK and French archives and museums, including La Contemporaine, which notably holds the fonds des albums Valois, which contain the photographs produced by the Section photographique de l’armée (SPA) during World War One. Using both Application Programming Interfaces (APIs) for already-digitized collections, as well as scanning of collections that have not already been digitized, Eycon has assembled a very large corpus of images, and is now advancing to the computing stage. Computation involves training a neural network to recognize various features of the images in the corpus, with the aim of enriching their associated metadata, increasing their navigability, and making them more useful for researchers and the public. This short post will explain the project’s goals, with a focus on the fonds Valois. In particular, it will consider how machine learning holds the potential for new contributions to existing scholarship on photography, personal presentation, and emotion, as well as how the SPA attempted to shape the image of the war for domestic and international audiences.

Created in 1915 in order to counter German photo propaganda, the SPA was a collaboration between the École des Beaux-Arts, the Ministry of War, the Ministry of Public Instruction, and the Ministry of Foreign Affairs, which mobilized professional photographers to document the war for the purposes of propaganda as well as the creation of national patrimony and future public memory of the conflict. While the Minister of War charged the photographers with their general mission and assigned them to sections of the war front and the home front, the ministry of public instruction and the Beaux Arts took charge of logistics and funding, and the Minister of Foreign Affairs diffused the photos to the press in allied and neutral countries. The effort was organized on an industrial scale, and generated a vast number of photographic prints in a wide variety of formats (Guillot, 2010).[1] Digitization of this collection has allowed for a whole new set of uses by researchers and the public (Maurel and Morvan, 2016), and machine learning promises to build on this success in important ways.

Having assembled the EyCon database, the research team is now applying a set of deep learning tools to segment pages into images and associated text, estimate human pose and gesture, and to detect objects in the images. These techniques can be used to enrich searchable metadata for digitized archives, dramatically enhancing its navigability for researchers and widening the parameters by which one can search for a given set of images. While the fonds Valois already has quite detailed associated metadata for the photographs in its albums—including information such as date, caption, and location—the ability to automatically recognize objects and poses in the photo and produce new metadata can supplement this information, substantially enhancing the collection’s usefulness for researchers and the public.

Indeed, the fonds Valois’ relatively replete metadata makes it a vital collection for the EyCon project’s wider goals, because it allows us to assemble a training dataset for the deep learning network. In order to teach the network to recognize human poses such as “standing,” “sitting,” “lying down,” or “walking,” it is necessary to assemble at least ten thousand images tagged with this information as a training set. After computation, the OpenPose algorithm should be able to recognize such poses in the images without the assistance of preexisting metadata (Cao et. al., 2019). The fonds Valois’ existing searchable metadata allows us to find all the images that contain these terms in the captions, and automatically compile them into a training set. Inputting “assis” into Argonaute, for example, might return image VAL 413/091, which was taken on April 21, 1918 and depicts “French and British artillerymen sitting on a 75mm canon.” This image, along with all the others with “assis” in their metadata can be compiled into a training set.

Researchers have used the fonds Valois to pose questions about topics such as the relationship between photography, the war experience, emotion, and masculinity (Pichel 2021). However, scholars are limited by time and labor constraints to working with a fairly limited set of photos, often picking one battle or album (Pichel, 2012). Researchers who have applied quantitative methods to visual sources tend to confine their sample sizes to between four hundred and one thousand images (Männistö et. al., 2022). Information about pose can be extremely useful to the researcher asking questions about how experience of war, as well as the SPA’s official photographic representation of it, changed over the course of the conflict. In general, sitting connotes a more relaxed photographic model, and indeed in this instance the men are smiling and smoking. If a researcher could establish a change in the relative prevalence of photos displaying sitting soldiers over the course of the war, they could make an argument about changing photographic practices, which may have shifted from a stiffer and more official relationship to the model to a more relaxed attitude, emphasizing confidence, ease, and camaraderie among enlisted men (Marly, 2018).

Pose estimation will help to make such arguments by allowing the researcher to work at a quantitative scale that would be otherwise impossible, as well as by alerting the researcher to relevant photos for which existing metadata would not have been sufficient. Take, for example, VAL 320/045 dated January 15, 1917 and taken in the area of Alsace.

The photo shows a group of five tirailleurs sénégalais sitting down to a meal, and one soldier standing. Two of the soldiers are smiling, conveying a relaxed air of camaraderie behind the front lines. Yet the caption—“Billet of the Senegalese. The soup of the infantrymen (tirailleurs)”—contains no information on pose or facial expression, and would thus not come up in a search for “sitting” in Argonaute. EyCon will apply machine learning tools such as pose estimation and expression recognition to enrich such images’ associated metadata, opening up whole new avenues of research into emotion and photographic practice during World War One.

With a much higher degree of precision than is afforded even by the relatively replete metadata of the fonds Valois, researchers could ask questions such as “What is the relative prevalence of sitting vs. standing photos in representations of soldiers over the course of the war? How do such iconographic trends change over time?” These choices by SPA photographers could indicate a shift from early optimism to later disillusion and exhaustion, or a more relaxed stylistic approach. Or, they might point to changing propaganda strategies that shifted from a stricter adherence to an iconography of heroism to a limited admission of the hardship, sacrifice, and day-to-day routines and habitus of soldiers at the front. By calculating the distance of the subjects from the photographer and how it changed over time, researchers could study the relative prevalence of close ups, medium shots that show groups close enough to depict their actions, or more distant landscape shots. Close up shots tend to provoke identification with the subject, and a different emotional response in the viewer (Chumachenko et. al., 2020). Alternatively, using object detection a researcher could find photos that include smoking or certain types of military equipment and uniform, useful information for studies of material culture. The possibilities are nearly limitless.

Machine learning has the potential to reduce the time and labor required for annotating photographs, compiling large numbers of images for quantitative study around a wide variety of variables, and enables study of large image collections on a new scale. Such tools will shift the labor of quantitative visual analysis away from compiling data and toward the higher-level work of interpretation. These techniques are particularly interesting for the photographic archive, and will allow researchers to see this medium and its history anew; while traditional art historical methods were developed in the context of a relatively limited number of artists and works, machine learning allows researchers to take the industrial scale of photographic production into account for the first time.

References: 

Cao, Zhe, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. “OpenPose: Realtime Multiperson 2D Pose Estimation Using Part Affinity Fields.” IEEE Transactions on Pattern Analysis and Machine Intelligence 43.1, pp. 172-186.

Chumachenko, Kateryna, Anssi Männistö, Alexandros Iosifidis, and Jenni Raitoharju. “Machine Learning Based Analysis of Finnish World War II Photographers.” IEEE Access, vol. 8, pp. 144184-144196, 2020.

Guillot, Hélène. “La section photographique de l’armée et la Grande Guerre: De la creation en 1915 à la non-dissolution.” Revue historique des armées, 258, 2010, pp. 110-117.

Männistö, Anssi, Mert Seker, Alexandros Iosifidis, and Jenni Raitoharju.“Automatic Image Content Extraction: Operationalizing Machine Learning in Humanistic Photographic Studies of Large Visual Archives.” arXiv preprint arXiv:2204.02149 (2022).

Marly, Mathieu. “L’armée rend-elle virile ? Réflexions sur le “modèle militaro-virile à la fin du IXe siècle.” Clio. Femmes, Genre, Histoire, 47, 2018, pp. 229-247. 

Maurel, Lionel, Josselin Morvan. “Les circulations sur le web d’un corpus numérisé: l’exemple des albums Valois,” Rapport final du projet : “Le devenir du patrimoine numérisé en ligne: l’exemple de la grande Guerre.” 2016.

Pichel, Beatriz. “Making Sense of Verdun: Photography and Emotions During the First World War in France,” in On Suffering: An Interdisciplinary Dialogue on Narrative and Suffering. London: Brill, 2012.

Pichel, Beatriz. Picturing the Western Front: Photography, Practices, and Experiences in First World War France. Manchester: Manchester University Press, 2021.

EyCon: What we are doing with Machine Learning and Computer Vision.

Blog Post by Soumik Mallick

Objective:

Historical photographs held by archival institutions provide a huge source of information for researchers in several domains of computer science, image processing, and computer vision. Photographs have been analyzed in multiple fields, including archaeology, war history, post-phenomenological geography, propaganda research, post-colonial studies, and many more. However, the work required for methodical analysis of very large quantities of photographs is a time-consuming and painful task, which also necessitates research time. Nowadays state-of-the-art Machine Learning (ML), Deep Learning (DL) and Computer Vision (CV) algorithms have significantly sped up this process and are providing a new viewpoint for researching photo archives.

Within the EyCon project, we are contributing to historical photograph analysis through ML as well as DL. This is allowing us to take one step further towards automatically analyzing historical data and extracting valuable information, which can help historians to find deeper meanings. Currently, we are developing novel methods using ML and DL for historical layout analysis and object detection. Our early-stage experiential prototype shows how several historical photograph analysis research problems can be addressed by utilizing our techniques. Our in-house developed algorithm allows for automatically extracting layout and detection of the objects detailed in historical photographs. Our work is currently in its early stages and needs more refinement, but it is already demonstrating positive results.

Methods: 

Hand-crafted features are affected by the difficulty of capturing explicit knowledge about the attributes to be associated with archival photographs, such as newspapers layouts. DL and CV have opened innovative opportunities for computer science researchers to assist the wider research community with automatic tools, to analyse and further understand document layout and objects. They can recognize meaningful patterns in historical data that are intrinsically related to human perception, and can assist experts in document layout analysis, such as object detection in war photography, which is particularly useful for museum and art gallery websites.

Currently, our model has three separate stages 1) LayoutParser and 2) Object detection and 3) Human pose estimation. We will discuss them one by one.

  1. Layout-parser:

DL methods have played a significant role in the last decade in increasing the performance of CV tasks, including historical document analysis tasks and layout analysis. Yet, the drawback of common DL approaches is their enormous hunger for annotated data. This hunger is particularly problematic in historical document analysis and layout analysis since most tasks, like semantic labelling, require experts to label the document images correctly.

Figure 1: LayoutParser framework

In recent times several DL models and datasets have been developed for layout analysis tasks. Nowadays, we also see that object detection-based methods like Faster R-CNN and Mask R-CNN are also used for identifying document elements, as well as detecting tables. Most recently some research work shows that Graph Neural Networks have also been used in table detection. 

Nevertheless, drawbacks of those models are usually implemented individually and there is no unified framework to load, as well as use, such models. There have been a variety of document data collections to facilitate the development of DL models. Some examples include PRImA (magazine layouts); PubLayNet (academic paper layouts); Table Bank (tables in academic papers); Newspaper Navigator Dataset (newspaper figure layouts); and HJDataset (historical Japanese document layouts). So, the models trained on these datasets are already available in the LayoutParser model zoo to support different scenarios.

A Model in LayoutParser takes a document image as an input and generates a list of rectangular boxes for the target content regions. There are other methods, which depend on deep convolutional neural networks (CNN) rather than manually curated rules to identify content regions. It is formulated as an object detection problem and state-of-the-art models like Faster R-CNN and Mask R-CNN are used. There are three key components in the data structure, namely the Coordinate system, the TextBlock, and the Layout. They provide different levels of abstraction for the layout data, and a set of APIs (Application Programming Interfaces) is supported for transformations or operations on these classes.

In LayoutParser, coordinate supports generally two kinds of variation: TextBlock consists of the coordinate information and extra features like block text, types, and reading orders; a layout object is a list of all possible layout elements, including other Layout objects. They all support the same set of transformation and operation APIs for maximum flexibility. LayoutParser currently supports nine different pre-trained models, trained on five different datasets.

LayoutParser provides a unified interface for existing Optical Character Recognition (OCR) tools, and it supports the Tesseract and Google Cloud Vision OCR engines. In accordance with the authors of the research paper, we have already implemented a LayoutParser using Pytorch, Python >= 3.6, Detectron2 and CUDA.

LayoutParser uses Detectron2-based pre-trained models like Faster R-CNN, RetinaNet, and Mask R-CNN to detect the layout of our input document. This is basically an object consisting of a list of detected layouts.

(a)
(b)
(c)

Figure 2: Sample of output of LayoutParser

In each detected layout, you will get the following important information:

  1. Coordinates of the bounding box (x1, y1, x2, y2) of each detected layout;
  2. The type of detected layout (i.e text, image, table, list, or title);
  3. The ID of the detected layout;
  4. The text inside each detected layout;
  5. The confidence score of each detected layout;

 For EyCon, we fine-tuned our model of LayoutParser according to our dataset and achieved good results. Fig. 2 shows the outcome of our LayoutParser.

Historical Object Detection:

Detected objects are helpful in determining the context of historical photos when used in conjunction with appropriate classes, as well as the focus of each photographer. A photograph of a chair, for example, may have been taken indoors, whereas photos of horses, boats, cars, or trains may have been taken outdoors. Moreover, the presence of objects can help determine the time of the year, and even establish the period for unlabelled photographs. For example, a high number of chairs, ties, uniforms, and people in a photograph are likely to indicate some official event or group of soldiers. At the same time, photos of battle tanks, airplanes, artillery, sandbags and guns, would suggest a battle or near battle areas. Aside from identifying the context of each photo, such analysis can draw out the main focus of a photographer, evaluating which types of objects are most frequently present in their photographs.

Figure 3: The Mask R-CNN framework

The main goal of object detection is to predict a set of bounding boxes and category labels for each object of interest. In recent times detectors address this set prediction task in an indirect way, by defining surrogate regression as well as classification problems on a large set of proposals as anchors, or window centres. Most of their performances are significantly influenced by post-processing steps to collapse near-duplicate predictions, the design of the anchor sets, and by the heuristics that assign target boxes to anchors.

Secondly, we implemented object detection using Mask R-CNN. Mask R-CNN’s benchmark that is used in Detectron2 uses multiple frameworks. It is an overall two-stage procedure that uses parallelism in regard to the class and box. Every candidate object has two outputs: a class label and a bounding box offset. Building from Fast R-CNN, the first stage of a Regional Proposal Network (RPN) is adopted. Moreover, pixel-to-pixel alignment is added and features are extracted using RoIPool from each candidate box. RoIPool and RoIAlign are techniques that perform quantization for the pixels.

Mask R-CNN adopts the same two-stage procedure, with an identical first stage (which is RPN). In the second stage, in parallel to predicting the class and box offset, Mask R-CNN also outputs a binary mask for each RoI. This is in contrast to most recent systems, where classification depends on mask predictions. It follows the spirit of Fast R-CNN that applies bounding-box classification and regression in parallel (which turned out to largely simplify the multi-stage pipeline of the original R-CNN).  Mask R-CNN is also special because it outputs a binary mask for each RoI in contrast to most object segmentation systems where classification depends on mask predictions. For each RoI, a multi-task loss is defined as L = Lc + Lb + Lm. The mask representation is quite valuable to the performance of Detectron2/Mask R-CNN since it encodes an input object’s spatial layout and thus produces faster inference. Mask R-CNN also features a convolutional backbone for feature extraction (creates pixel-to-pixel correspondence provided by convolutions) and a network head for bounding box recognition as part of both classification and regression.

(a)
(b)

Figure 3a and 3b: Sample of output of object detection

Improved objection detection:

This simplifies the detection pipeline by dropping multiple hand-designed components that encode prior knowledge, like spatial anchors or non-maximal suppression. Unlike most existing detection methods, the object Detection transformer doesn’t require any customized layers and thus can be reproduced easily in any framework that contains standard CNN and transformer classes. Below Fig. 4 shows object detection architecture with transformer. 

Figure 4: Object detection architecture with transformer

The transformer encoder-decoder infers a fixed-size set of N predictions, in a single pass through the decoder, where N is set to be significantly larger than the typical number of objects in an image.

Next, an important step is finding a bipartite matching between these two sets, a permutation of N elements σ is searched with the lowest cost.

Where Lmatch (yi, ^y(i)) is a pair-wise matching cost.

The matching cost takes into account both the class prediction and the similarity of predicted is yi= {bi, ci}.

ci is the class probability.

Lmatch (yi, ^y (i)) is:

Compute the Hungarian loss, which is a linear combination of a negative log-likelihood for class prediction and a box loss defined as:

A conventional CNN backbone helps to learn a 2D representation of an input image and the backbone is ImageNet-pretrained ResNet-50 or ResNet-101 with frozen batch norm. After that the model flattens it and supplements it with a positional encoding before passing it into a transformer encoder. A Transformer decoder then takes as input a small, fixed number of learned positional embeddings, called object queries, and additionally attends to the encoder output. Each output embedding of the decoder is passed to a shared feed forward network (FFN) and that predicts either a detection with class and bounding box or a “no object” class.  Fig. 5 shows an output of the object detection we implemented with PyTorch and Detectron2. 

(a)
(b)

Figure 5: Example output of our improved Object detection

Human pose estimation:

A human pose skeleton denotes the orientation of an individual in a particular format. Fundamentally, it is a set of data points that can be connected to describe an individual’s pose. Each data point in the skeleton can also be called a part or coordinate, or point. A relevant connection between two coordinates is known as a limb or pair. However, it is important to note that not all combinations of data points give rise to relevant pairs.

(a)
(b)
(c)

Figure 6: Keypoints detected by OpenPose

The pipeline from OpenPose is actually pretty simple and straightforward. First, an input image is fed as input into a “two-branch multi-stage” CNN. Two branch means that the CNN produces two different outputs. Multi-stage simply means that the network is stacked one on top of the other at every stage.

Figure 7: Architecture of the two-branch multi-stage CNN of OpenPose

Two-branches: the top branch, shown in beige, predicts the confidence maps of different body parts location such as the right eye, left eye, right elbow and others. The bottom branch, shown in blue, predicts the affinity fields, which represents a degree of association between different body parts.

Multi-Stage: at the first stage the network produces an initial set of detection confidence maps S and a set of part affinity fields L. Then, in each subsequent stages’ predictions from both branches in the previous stage, along with the original image features F, are concatenated, represented, and used to produce more refined predictions. Finally, the confidence maps and affinity fields are being processed by greedy inference to output the 2D key points for all people in the image. We explain OpenPose first, which can help you to understand how we can get key points.    

Human pose from single image:

Estimating a 3D human pose from a single image is known to be a severely ill-posed problem because many different body configurations can virtually have the same projection. A typical solution consists in using discriminative strategies to directly learn mappings from image evidence (e.g., HOG, SIFT) to 3D poses. This has been recently extended to end-to-end mappings using CNNs. In order to be effective, though, these approaches require large amounts of training images annotated with the ground truth 3D pose. While obtaining this kind of data is straightforward for 2D poses, even for images ‘in the wild’ (FLIC or LSP datasets), it requires using sophisticated motion capture systems for the 3D case:  

(a)
(b)

Figure 8: 3D human pose estimation on historical images

With 3D body pose parameterization, most approaches use a skeleton with a number N of joints ranging between 14 and 20 and represented by 3N vectors in a Cartesian space. In order to enforce joint dependency during the 2D-to-3D inference and considered latent joint representations, obtained through Kernel Dependency Estimation and autoencoders. To gain depth-scale invariance, we have to first normalize the vertical coordinates of the projected 2D poses xi to be within the range [−1, 1]. 3D joint positions yi are expressed in meters with no further pre-processing. After, we can represent both 2D and 3D poses by means of Euclidean Distance Matrices. For the 3D pose y it is defined edm(y) to be the N × N matrix where its (m, n) entry is computed as:

Similarly, edm(x) is the N × N matrix built from the pairwise distances between normalized 2D joint coordinates. 

Retrieving the 3D joint positions

from a potentially noisy distance matrix edm(y) estimated by the neural network, can be formulated as the following error minimization problem:

As a regression between two Euclidean Distance Matrices encoding pairwise distances of 2D body joints and 3D body joints, a 3D human pose estimation problem can be formulated. Regression is carried out by a Neural Network, and 3D joint estimates are obtained via Multidimensional Scaling from the predicted 3D Euclidean Distance Matrices.  Above, Fig.8 shows the outcome of 3D human pose estimation.

Implementation:

Both our LayoutParser and object detection are based on PyTorch with Facebook Detectron2. For human pose estimation we used PyTorch. Currently, we are using a single NVIDIA RTX 3000 GPU and CUDA version 11.

Result:

Our work is still at an early stage. While we have achieved good results so far, it needs more refinement.

 Figure 9: Output result of LayoutParser

Figure 10: Output result of object detection

Figure 11: Human pose estimation                   

Difficulty and challenges:

We are facing some challenges with LayoutParser. Historical photographs contain lots of complex layouts and LayoutParser has failed to identify those layouts. Historical photographs also have poor image quality with high levels of noise. There can be several kinds of logos, symbols and stamps inside images as well as documents, which make it very difficult to perform LayoutParser on these records.

Conclusion:

In future, we aim to add more features to our platform, initially improving the detection of smaller-sized objects. In our updated version of the model, we will try to add person segmentation, face detection with facial expression analysis, group-level emotion recognition, gender estimation, and weapon detection. We believe that these updated features will help to provide us with a deeper understanding, which will enable further research by historians or within cultural heritage organisations. While our work is currently fragmented into different parts, soon we will be able to provide our data open source on our website through a Python SDK library with an integrated model, so you can also download all our code and database. We wanted to keep you updated on our progress so far, and to let you know that we are continuously working towards improving our model to provide a uniform system for the automated analysis of historical photography.

 References

Cai, Hongping, Qi Wu, Tadeo Corradi, Peter Hall, ‘The Cross-Depiction Problem: Computer Vision Algorithms for Recognising Objects in Artwork and in Photographs’, CVPR (2015).

Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko, ‘End-to-End Object Detection with Transformers’, CVPR (2020).

Castellano, Giovanna, Eufemia Lella, and Gennaro Vessio, ‘Visual Link Retrieval and Knowledge Discovery in Painting Datasets’, Multimedia Tools and Applications, 80 (2021).

Castellano, Giovanna, Giovanni Sansaro, and Gennaro Vessio, ‘Integrating Contextual Knowledge to Visual Features for Fine Art Classification’, arXiv, 2021.

Crowley, Elliot J., and Andrew Zisserman, ‘In Search of Art’, ECCV (2014).

Gonthier, Nicolas, Yann Gousseau, Said Ladjal, Olivier Bonfait, ‘Weakly Supervised Object Detection in Artworks’, ECCV (2018).

He, Kaiming, Georgia Gkioxari, Piotr Dollar, and Ross Girshick ‘Mask R-CNN’, ICCV (2017).

Francesc Moreno-Noguer, 3D Human Pose Estimation from a Single Image via Distance Matrix Regression, CVPR2016

Liu, Ziming, Guangyu Gao, Lin Sun, Li Fang, Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh, ‘Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields’, CVPR (2017).

Liu, Ziming, Guangyu Gao, Lin Sun, Li Fang, ‘IPG-Net: Image Pyramid Guidance Network for Small Object Detection’, CVPR (2020).

Shen, Zejiang, and Ruochen Zhang, Melissa Dell, Benjamin Charles Germain Lee, Jacob Carlson, and Weining Li, ‘LayoutParser: A Unified Toolkit for Deep Learning-Based Document Image Analysis’, arXiv (2021).

Strezoski, Gjorgji Marcel Worring, ‘OmniArt: Multi-task Deep Learning for Artistic Data Analysis’, CVPR (2017).

Review of the First EyCon Workshop ‘Photosensible’, 9 and 10 June 2022 – by Elise Clerteau (ENS de Lyon).

On 9th and 10th June, the EyCon Project (funded by Labex Passés dans le Présent, and the AHRC), the Musée du Quai Branly Jacques Chirac, and the Université de Paris Cité organised a workshop on sensitive photographs and their contemporary recirculations. Held at the Musée du Quai Branly Jacques Chirac (Paris), several specialists, brought together by the question of the ‘photosensitive’, were able to question collectively the specific problems caused by ‘controversial images’ that hurt, show human suffering or materialise domination, and consider the way historians, archive institutions and museums can approach this unstable material that resonates with the violence of the past. The discussions were articulated around several issues, relating to ‘gaze’; the reproduction/recirculation of the images; the legal delimitations; the difficult conciliation between the occultation of the past and the clash of sensitivities; and to the uses–notably digital uses–of these photographs. At the crossroads of ethics and legality, violence and sensitivity, visibility and invisibility, image and text, the ‘photosensitive’ has aroused artistic and historical reflections, but also philosophical, socio-anthropological and psycho-cognitive considerations to approach certain shocking and violent photos in accordance with the objectives of the EyCon project. Focused on the mass of war images and their treatment before the second half of the 20th century, this project aims at finding the tools to structure this material, make it more accessible, and solve the potential issues in its dissemination.

June 9th (morning)

The first hypotheses were put forward on the morning of June 9, as the representatives of the archives and museums gathered for the occasion, reported:

  • Musée du Quai Branly Jacques Chirac (Paris). Among the 700,000 photographs in the collection inherited by the MQB, many are sensitive and problematic both from an iconographic point of view (nudity, mutilation, torture, skeletons, medical photography, animal sacrifices, etc.) and from a contextual one. There is an existing digital library, but with restricted access — for example, some photographs appear without the image, and one has to request to see it.
  • Établissement de communication et de production audiovisuelle de la Défense/ Audiovisual section of the French Armed Forces (Ivry). The ECPAD holds 15 million photographs and approximately 94,000 hours of film. A set of rules has been established to determine a varying level of confidentiality of the photos according to their violence. Shocking terms or iconographic contents are mentioned as such in an informative and/or preventive note. The institution is also particularly attentive to offended relatives’ requests for the removal of images.
  • Division of Iconographic Archives of the Defence Historical Service (Vincennes). Public and private archives are treated in this military service whose priority is to make all (meta)data accessible to the public despite their explicit violence. However, the collections are not yet available online.
  • La Contemporaine (Paris-Nanterre). A library and ‘museum of contemporary worlds’, La Contemporaine documents the consequences of conflicts on civilians. In the management of its collections, there is a particular emphasis on questions of classification and on the psychological effects of war violence on populations. 
  • Archives nationales d’outre-mer (Aix-en-Provence). 37 kilometres of archives from colonising and colonised sources (1844-1970) are kept at the ANOM. Most of the photographs are accessible online, despite their sensitive nature.
  • Imperial War Museums (UK). Opened in 1917, the three IWMs now detain 33 million items documenting the events of both World Wars. The issues of sensitivity and digital dissemination trigger specific reflections aiming at bridging the gap between academic research and the public’s understanding of conflicts.
  • Search at Wellcome Collection (London). The Wellcome Collection museum’s search engine underwent a major development in 2015, optimising responses to the online audience’s expectations. The access to more than a million images is guaranteed despite gradual restrictions (open / open with advisory /restricted / by appointment / donor permission / temporary unavailable unavailable) depending on the sensitivity of the photographs.
  • Royal Museum for Central Africa (Belgium). Founded in 1960, this museum is an institute for research, valorisation and reinforcement of national capacities in Africa. A development of photographic metadata through digitalization is underway, but still partial.
  • Museum of Photography of Madagascar (Antananarivo, Madagascar). A museum of heritage conservation and education, this institution has received many financial and documentary donations since its creation in 2013. Digital technology has been put at the service of Madagascar’s heritage to give its photographic memory of the colonial period its visibility back. Many official photographs being paradoxically held by families, those shared by locals are often returned to them after having been digitised.

The presentations were structured around information about the different institutions; institutional (re)definitions of what ‘sensitive’ photographs are; policies for their management; the challenges of digital dissemination; online accessibility and its potential restrictions; additional metadata; and the establishment of prior warnings. Despite disparities in terms of the size of photographic collections and the progress of their digitalization, the origin of their sources, and the policies for exhibiting images considered ‘sensitive’, several common difficulties have emerged:

  • Definitory. The iconographic content is not systematically a criterion to define what a ‘sensitive photograph’ is. If nudity, ritual mutilation, corporal punishment, public executions, human remains, diseases and animal sacrifices can be identified as ‘sensitive categories’, it is no less true that the violence of certain photographs is due to their textual content (captions, notices, titles) and/or to their symbolically traumatic resonance in certain communities or cultures. Where an internal committee has been designated at the Musée du Quai Branly Jacques Chirac to discuss the problematic nature of certain images, a specific identification procedure was put in place at the IWM, from the acquisition of the photograph to its ‘encounter’ with the public.
  • Artistic and historical. What to show, what to correct? Should the captions appear as they were written? should they be corrected? should they be accompanied by an informative-preventive contextual caption? The choice could potentially obscure part of history. On the ECPAD website, in addition to the ‘sensitive image’ warning displayed on the screen, the captions judged to be shocking are explicitly labelled with their contextual origin, to clarify the distance between what was done at the time and the way it is presented today. Moreover, the cases of requests for the removal of images from relatives resolve an ethical issue, but also raise a historical problem concerning the erasure of evidence of past violence.
  • Legal and juridical. The uneven progress of the digitalization of the collections mentioned is certainly due to budgetary ties (the scale of the Wellcome Collection is largely due to the substantial investments that allowed the processing of 10,000 images per day in 2015 when the project was born), but also to legal constraints. Copyrights, classification issues, the duty to return sources, are factors that can complicate or even restrict the online or onsite exhibition of these images. In the iconographic archives of the Historical Service of Defence, where ‘violence, death and destruction cannot be hidden’ (Benjamin Doizelet), the greatest difficulty encountered by archivists is related to the classification of public archives.
  • Ethics and practices. These photographs raise ethical questions — both because of the sensitive nature of what they show and because of their potential effects on viewers’ sensibilities. This leads to theoretical and practical reflections on the criteria to be determined for their contemporary uses and (re)circulations. Dieter Van Hassel, head of the Archives-Information Department at the Royal Museum for Central Africa, spoke about the challenges of expanding online access to ‘sensitive photographs’: it is not only a question of legal constraints but also of fear of the public’s reactions. At the Musée du Quai Branly Jacques Chirac, some photographs are published online but without the image: only the informative note appears, thus testifying to the existence of the image while inviting one to contact the museum to access the image. However, we can assume that this policy of not showing also poses a problem: accessibility becomes a privilege when it implies a specific consultation process that is not within the reach of everyone. Circulation is essential in Madagascar, where the Museum of Photography is conceived as a depository-mediator of images and photographic archives. The digitalization of photographic documents and the creation of a database are similar to a ‘photographic heritage in the making’ (Helihantha Rajaonarison), which could remedy the occultation of the visual memory of colonisation in Madagascar where the main national museums are still closed to the public.

The link between photographic dissemination and ethical-legal problems superimposed on individual and/or collective traumas––of which the photographs are material and memorial traces––has thus been woven since the beginning of their exchanges. The ‘trauma-informed approach’, suggested by Helen Marvin, seems conducive to renewing the way we look at these photographs, to questioning and even redefining the uses and norms established in the past. As the eye of the specialist is not the only one to rest on these visual memorial objects, the role of the spectator is not negligible in the appreciation and the determination of the ‘photosensitivity’: one can go as far as supposing, like Heliantha Rajaonarison, that among sensitive photographs it is possible to distinguish photographs with positive and negative sensitivity, according to their reception. Indeed, the remediation of certain images can engender conversations on the collective recognition of forgotten and invisible violence. Generally speaking, it is the balance (still far from being reached) between a duty to make these collections public (both physically and digitally), freedom of expression, and privacy that is on the horizon.

June 9th (afternoon)

The afternoon of the first day was an opportunity for a collaborative and playful workshop of photographic confrontation(s), in every sense of the word. Photographs previously selected by each specialist according to their explicit or uncertain sensitivity were divided into two batches, just as the specialists were divided into two groups to encourage exchanges. Several images were thus discussed in light of their ‘sensitivity’, to see why and how they could be judged as sensitive, and to what extent it was possible to arrive at recommendations concerning their contemporary (re)circulations. The photographs were quite diverse but provoked comparable inquisitive reactions, starting from the context of production moving towards issues of reception and exhibition: 

  1. The context of the shooting of the photograph and past uses in question. What information do we have on the context of the shooting, on the identity of the photographer and on their intentions? What uses has this photograph been put to – propaganda, documentary, anthropology, medicine…?
  2. The iconographic content and composition in question. Does the photograph directly represent violence? To what extent does the photographic technique (framing, staging) influence the sensitivity of the image? Is the language of the caption, echoing the colonial rhetoric, possibly more violent than the image itself? 
  3. Reception and exhibition in question. Who to show these photos to and where? For whom are they sensitive? What can be done so that they do not offend sensibilities? Can they be exhibited (in a museum and/or online) and if so, how?

These interventions revealed the complementary plurality of approaches as well as an undeniable divergence between the sensitivity of the archivist or historian and that of the community––as one of the participants said, ‘we cannot put an archivist behind each reader’. Indeed, the status of photographs is also largely defined by the reactions of viewers, whose primary reaction is not necessarily as informed and objective as that of a specialist. From this observation, the caution for a respectful use of the photographs towards communities was agreed upon and accompanied by a recommendation to diffuse certain images online that undermine the dignity of certain people who were photographed in a condescending way, especially when their face is visible. Insofar as photographs are multicultural objects from their creation in a shared cultural-historical context, it seems difficult to decide unilaterally on their use(s). Adopting a ‘poly-cultural’ approach, according to Julien Schuh’s proposal, could help to avoid the risk of being out of step with those who have also participated in creating these objects, sometimes at their expense. More widely, the importance of visual cultures nowadays would advise us to pay particular attention to the education of the public. Providing adequate knowledge to understand what photography is and what it implies in the past, present and future, seems indispensable.

June 10th

Before welcoming the public to attend the conferences organised in the movie theatre of the Musée du Quai Branly Jacques Chirac during the afternoon of June 10, the specialists met once again behind closed doors. The historical management of sensitive visual documents in the era of mass digitization and the concomitant risks of misinformation; the difference between these documents and other types of sources; the methodologies to be adopted; and the apparent contradiction between emotion and the historical approach, were among the topics discussed. Three interventions brought together cognitive sciences and the consequences of the recirculation of sensitive images with the question of the public ‘in context’.

Grégoire Borst (professor of developmental psychology and educational cognitive neuroscience at the University of Paris) addressed the effects and consequences of ‘exposure to sensitive images on the brain and memorization’. The approach of the ‘photosensitivity’ by the cognitive processes showed that violent photographs tend to be memorisedmore quickly and durably, but especially that the sight of an aversive photograph is felt as a threat, involving a psychological process of defence. This is one of the reasons why photographs differ from videos, as it is easier to regain control of one’s reactivity at their sight. The colorization of photos could also be explained by neuro-psychological effects of identification, exacerbation of shock, distanciation, etc. As for the regular sight of violent or shocking images (as in video games), it is unlikely to lead to an excess of violence, but rather to a desensitisation, because it is less and less felt as threatening or shocking. Grégoire Borst also insisted on the fact that even though the spatio-temporal distance separating the photograph from its spectator can influence sensitivity, just as the reactions to violence are likely to vary according to the geographical areas and the socio-cultural groups, these reactions are cognitively identical. Photography appears as a medium provoking a specific reaction leading to an almost immediately constructed meaning. Certain collectively sensitive and contested images can, as the historian Susie Protschky reminded us, reflect an intergenerational trauma that gives a political coloration to the community’s emotional response.

This spectatorial dimension was then addressed ‘in context’ by Helihantha Rajaonarison (Musée de la photographie de Madagascar) in relation to sensitive photos from the colonial period in Madagascar. The initial idea––that the reader’s reactions make an image violent or not ––leads to an injunction to continually question the impact that photographs can have on those who view them (families, descendants but also specialists and archivists), and the extent to which they can remedy a violence that was previously hidden. In the case of Madagascar, colonial violence (including photographic violence) is to be understood through a cultural prism: certain symbolic, psychological and moral violence strategically perpetuated by Gallieni in the nineteenth century during the period of colonial pacification is represented in photographs — such as the submission of the chief Menalamba Rabozaka in front of the queen’s palace, later desecrated with its transformation into a training school for colonial administrators. Thus, the 2,500 photographs held by the Institution of Malagasy National Cartography, the personal collection of Grandidier Père & Fils, as well as private albums left by individuals preserved in the national archives, constitute traces of this violence to which the Malagasy people must have access, if only to decide for themselves whether they are positively or negatively sensitive.

Pierre Schill (Centre de Recherches Interdisciplinaires en Sciences Humaines et Sociales de Montpellier) finally evoked the potential for controversy and the unintended consequences in relation to digital recirculation and the reappropriation of sensitive images. Interested in the work of Gaston Chérau, a photographer-reporter during the Italo-Turkish War (1911), the historian analysed the recirculation of his extremely violent photographs (emaciated bodies, hanged, mutilated, etc.) in two paper works available online, and in the Getty images bank. While the contemporary politicization of photographs can be observed online in the comments made by users under the photographs published on social networks, the question of exhibiting certain photographs of the tortured bodies of foreign soldiers has arisen––not only in relation to their violence but also to their potentially dominant appropriation. Thus, the choice to display the photographs on a table, rather than on the wall, seemed a good solution to avoid erecting them as works of art while preserving their archival documentary dimension. The printed publication of Schill’s book, Réveiller l’archive d’une guerre coloniale (Revealing the Archive of a Colonial War), was likewise intended to protect these sensitive images from ‘documentary digital misuse’.

The public lectures that followed broadened several perspectives both on the sensitive photographic-historical material and on its exhibition and reception.

  • Nancy Rushohora (Senior Lecturer in the Department of Archaeology and Heritage Studies at the University of Dar es Salaam) looked at the demanding images and the memories of colonial violence in Tanzania with the example of the single portrait of sub-chief Nduna Songea Mbano. The photograph, subtitled ‘A Real Mgoni’ was taken from an anthropological perspective, but a false one, since this chief was not from this community. However, this image has been the object of countless circulations anduses in this region of Tanzania, which later became ‘a land of trauma’ because the body of the leader is buried there without its skull which has potentially been transported to Germany shortly after the killings. Representative of the evolution of forms and implications in the display and production of visual material from the colonial period to contemporary practices around this representation, this image testifies to the way in which a colonial photograph can be overturned in and through its sensitive effects.
  • Helen Mavin (Head of Photographs at IWM London) then presented the Provisional Semanticsdigitization project, which challenges the imperial gaze in IWM’s Second World War photographs. Underpinned by an ethical and collaborative methodology, both institutional and interpersonal, the project is dedicated to ensuring a transparent and critical reading of approximately 5,000 photographs. Starting from the premise that these documents are traces of Britain’s dominant presence in India, the titles and captions are scrutinised in order to discuss the socio-political context of creation and to suggest a contemporary recontextualization. The main issues at stake in this project are the continuous questioning of readings and attitudes towards the collections, as well as transparency towards the public. This is why biographies are available online alongside the photographs for researchers.
  • Susie Protschky (Associate Professor of History at Deakin University, Melbourne) communicated on ‘Late starters: The post-colonial afterlives of war photographs from Indonesia and neglected histories of colonial violence’. In support of several photographs, the historian speculates that they are often reproduced as historical evidence of the difficulty of Dutch history that few have been willing to take on. If the recent colorization of some of the images raises objections about the potential misuse of historical material, it explains to a large extent that the official apology of the responsibility of the Dutch state in these abuses occurred shortly after their publications on social networks. The current parallels made by viewers who are already aware of the extent of the colonial propaganda are distortions of the meaning that the photographs had at the time. However, they do testify to the difficulty of positioning between the object and the current national context. Thus, Susie Protschky concludes that photographs of colonial violence are ‘image-events’ circulating in a polyphonic public sphere where divergent discourses on these images develop. A constant readjustment of the interpretative framework of these photographs thus appears more than necessary.
  • The approach of Carine Peltier-Caroff (in charge of the photo library of the MQB) revolved around Helène Meissalloux’s 19th-century album of postcards. By questioning ‘the politics of naming and the ethics of showing’, Carinne Peltier-Caroff presented the process of digitalization carried out in the photographic section of the Museum and evoked the evolution of the standards of cataloguing allowing to fix codes. Questions must be asked from the context of production of the photograph to the documentary processing, including the uses and dissemination as well as the circulation, the development of links between data, the enhancement of the identities of photographed people, the revisitation of the colonial context of the collections, and the historicization of documentary practices. To resolve the ethical and legal issues encountered in relation to the management of the collections, an internal committee was set up to discuss the photographs’ iconographic content, the old titles and historical captions, as well as the conditions for putting them online.
  • ‘The most illuminating thing I have ever seen’, were Kim A. Wagner (professor at Queen Mary University of London)’s introductory words to the analysis of the photographic evidence of the 1906 Bud Dajo massacre. Starting with the observation that this massacre, among the least known in American history, is nevertheless one of its best documented, the professor wonders what a particular photograph (the image should be put online) has to say about a specific historical context. Revealing in several ways, notably because of its scenography, the image superimposes sensitive and historical layers whose significance can be discussed. At the time, it became a ‘visual war trophy’ circulating among soldiers on whose side this violence was ‘positively sensitive’ because it was victorious. In the newspapers, this photograph was used alongside other representations of wounded or dead American soldiers, in order to enhance a narrative in which the illustration of the massacre became an illustration of the white man’s burden. The image was thus implicated and (re)contextualised in various—and sometimes antagonistic—ways, as evidenced by the re-appropriation of the image by the anti-imperialist league for the purpose of colonial denunciations.

All in all, this thought-provoking workshop was an opportunity to further reflect on the ‘photosensitive’ engendered by the EyCon Project, and to lay down foundations for future collaborations between participating institutions. The questions which emerged (still far from being solved) testify to the stimulating richness of a multidisciplinary, polyphonic and polycultural topic, inviting us to renew our approaches and point of view.

AI & Cultural Heritage Eycon Team recording

GOOD NEWS!

The recording of the Eycon team’s presentation ‘AI and Contested Pasts’ at the ‘AI and Cultural Heritage’ event at the Loughborough University’s Institue of Advanced Studies on March 28th, 2022 is now available online!

Pour ceux qui n’ont pas pu assister à la présentation du projet Eycon ‘AI and contested pasts” lors de l’évènement ‘AI and Cultural Heritage’ le 28 mars 2022 à l’Université de Loughborough, l’enregistrement est disponible via ce lien :