PHOTO SYNTHESIS

MACHINE-ASSISTED VISUAL RESEARCH

An IDEO team had thousands of photos and a need to explore how uses of vehicles varied across cultural contexts. We set out to experiment with using machine learning techniques for design research, and ultimately built a tool to help them make sense of all those pictures.

In a typical time, project teams would travel to immerse themselves; however, with a pandemic in full-swing, research had to be conducted remotely. The team tried something new - hiring local photographers in a handful of different areas of the world to go out and take pictures of how vehicles are a part of their cultures.

The several thousand photos that came out of that is not necessarily an hugely overwhelming amount, but it is too many to appreciably hold in a human brain and make connections at scale. Thus, we got to work on several experiments to see what might be possible to build to make using this trove of pictures more approachable.

First, we created visual feature vectors by running all the images through a pre-trained image classification model with its output layer removed. This gave a numerical representation that we could use to understand images in a different way.

These feature vectors were used to create multiple interactive demos. One such demo facilitated clustering of images, which allowed users to see groups of images that are similar across different geographies.

Another allowed users to upload an image to see the most similar images contained within the dataset. In the example below, a face is uploaded and the images returned also contain faces. This functions as an intuitive visual search interface.

Yet another put all photos into a network graph which was created by creating connections between each image and the top 10 most similar other images. Users could choose two photos and traverse the network between one image to another, to be used as an inspirational story-generating device.

After the vehicle research team used these demos, we worked with them to understand which aspects could be useful and how to build upon them to make something they could use. In the next phase, we built a usable prototype which incorporated aspects of visual search, object detection, and ability to tag so that teams could build photo collections, organize around themes, and find new photos to add to their collections.

Special thanks to teammates Ross Weijer, Irmak Sirer, and Noah Shibley.