Google Image Search, with its simple interface and reasonable results, is the de-facto consumer-grade image search engine. Offerings from competitors are actually a little more feature rich, especially the MSN Live Image Search, but don’t resonate as loudly in academia or popular usage.

As an example, compare search results for “red corvette” from the big three: Google, MSN, Yahoo. MSN nails the exploratory task: no-refresh scrolling, quick access to filters such as “photos”, “black and white”, and image size options that feel a little more usable and natural than Googles. Yahoo! attempts some categorising and support for ontologies in their interface; while not perfect, it’s a direction highly praised in cutting-edge research.

Focus on a subset

Despite the simple Image Search interface, Google has been very active in image search research, particularly in hopes of making their results more relevant.

First off, there’s the neat little trick that allows you to search for faces.

In general, the goal is to reach a higher level of result-relevancy. In order to do that, a service need sto analyse images more deeply than by just reading the text that surrounds pictures. Image analysis that involves a computer “seeing” an image and extracting features and content is very expensive in terms of computation. So in the interest of progress, despite the heavy computing cost, the New York Times reports how Google plans to roll out some of their image analysis features on a subset of internet images, rather than try to analyse all the images on the whole of the internet. Google is focusing on the top several thousand topics of interest based on query popularity, and popularity in online shopping (iPods, Wiis, Air Force Ones, etc.)

This concept of focusing on a subset of internet images related to shopping isn’t new: like.com has been in the process of launching and relaunching (that’s a link to the CEO’s blog) its visual search service since early 2007. Their service helps you shop the way you do in real life – visually, with visual feedback and comparison of how things actually look compared to similar items in similar price brackets.

Community Tagging

While image analysis and retrieval based on image features is the future, in the mean time we have tags. Tags are human-contributed textual descriptions of images. There are a variety of problems with tags (inconsitencies, incompleteness, subjectivity, etc.), but their presence reduces the image search problem to a far more manageable text search problem.

With this in mind, Google has quietly launched the Google Image Labeler, a game in which two randomly matched people tag an image co-operatively, and score points. Using a games to entice people to contribute annotation information is cool, but it appears to suffer from over-simplification: a picture of a migrating albatross is far more likely to be tagged as “bird” rather than as “transatlantic migration” or “albatross.” Though I’ve read some promising research that focuses on developing better tools that aid in annotation of images, which will hopefully lead to tags that are more complete, categorised, and community edited.

Leveraging Recommendations

Google has done quite a bit of development and research on the topic of recommendations, especially recommendations related to their Google News service.

I can’t help but think of the possibilities of using recommendation algorithms enhance image search. A service can push new image content to the forefront during queries, inter-mixed with the most relevant results, and allow people searching to annotate the recommended images. This would quickly collect information on how relevant that new image is to that query. Users could dismiss the image as irrelevant by never clicking on it, despite how often it appears in the first row of the result set.

That kind of information would provide a feedback loop for the developers of the relevancy algorithms, and potentially also allow for very personal image results: if those recommendations and your responses to them were kept around and associated with your Google account, future results could take advantage of your previous feedback.

There is much progress to be made in image search, and based on how often I come across interesting and relevant Google research in the field, it’s obvious that Google will continue to be a leader as image search matures.

First, every semester I hope/pray to get a Graduate Teaching Assistant job, which luckily gets easier and easier as I accumulate “seniority points.”

Second, no funding means no specified project, which means freedom to choose any research topic I please, as long as my (very lenient/forgiving) advisor is OK with it. Well, it’s been about eight months since I’ve come back from India all ready to start researching, and only two days ago did I actually settle on a topic.

Eight months is a long time to pay tuition, and follow dead ends with literature reviews. Also, those months are expensive if you waste your time on partying, girls, video games, Union involvement, student government, keggers, new housemates, motorcycles, trips to Mississippi, Vancouver, Ottawa, and so on. Well .. maybe it wasn’t a complete waste, per se :)

Finally, I’ve settled on a topic that I’m truly interested in.

Usability and Image Search

The topic: usability of image retrieval interfaces for systems based on image content, rather than user-provided textual meta-data.

What does this mean? Image searching (like Google Image, or Flickr search) is actually really, really complicated stuff. The systems on the back-end of the process use a variety of properties of an image to catalogue it in their vast library: the file name, it’s dimensions, data size, human contributed tags and categories, and a whole slew of other things. That specific information is called the metadata. Most of major (Yahoo, Google, MSN, etc.) image search engines also scour the web-page content that surrounds an image to get more metadata that can be searched.

Human contributed metadata is particularly meaningful: tags, categories, and file names point to the content of the image, not just its properties. The problem with human contributed metadata is that it’s:

• very incomplete
• very subjective, thus possibly inaccurate
• very time consuming if you do want to make it accurate or complete
• there are no standard descriptive elements in markup languages like XHTML or HTML for people to tag their images on the web

If you have a photo of a man riding a bicycle in Greece, unless you’ve tagged that picture with “man”, “bicycle”, “greece” (in four or five of the most common languages in the world), the likelihood of that image being catalogued so it can be found by an interested party is quite low.

Researchers since the mid-1990′s have spent a lot of time and energy in a field that hopes to automate more and more of that descriptive process, so that images can be searched on more than their properties. The field is called Content Based Image Retrieval (CBIR). Rather than relying on metadata to describe images, one can have a computer “see” the image and record information about it: colour, texture, salient regions, shapes, layout, etc. This is the content of the image. Cataloguing all of the useful content of a very large number of images accurately, completely, and in a way that can be easily searched, is the holy grail of this field. We are decades away from such a system, but major strides in that direction have already taken place.

Humans + CBIR

A smaller consideration of all of this is: how is a person supposed to search for something based on content? As in, I know a dog is fuzzy, I know that the computer can find fuzzy textures, but how do I describe to a search engine that I want to find all images of a brown, fuzzy dog? It sounds simple, should be simple, but is not simple in the current state-of-the-art.

Usability is a field that concerns itself with how things are used, especially interfaces of computer programs. I’m very interested in usability of these image search systems; image search systems that already exist, and proposed systems that will exist once we have the technology to automatically identify brown fuzzy dogs in your Flickr photos from the cottage.

So that’s where I’m at. I have a general idea of the area I’m interested in, so I’m in the process of reading some review papers about the state of CBIR. Hopefully, a literature review is coming next.