If you’re interested in participating, take a look at my little recruitment page:

http://grabka.org/internet/flickrsearchtagging/

It’s a multi-step process that starts with you installing some software, and sending me an email with your Flickr username.  After that, it’s all simple :-)

Take a look at the study page if you are interested in participating.  I need about 50 people to do this.

Google Image Search, with its simple interface and reasonable results, is the de-facto consumer-grade image search engine. Offerings from competitors are actually a little more feature rich, especially the MSN Live Image Search, but don’t resonate as loudly in academia or popular usage.

As an example, compare search results for “red corvette” from the big three: Google, MSN, Yahoo. MSN nails the exploratory task: no-refresh scrolling, quick access to filters such as “photos”, “black and white”, and image size options that feel a little more usable and natural than Googles. Yahoo! attempts some categorising and support for ontologies in their interface; while not perfect, it’s a direction highly praised in cutting-edge research.

Focus on a subset

Despite the simple Image Search interface, Google has been very active in image search research, particularly in hopes of making their results more relevant.

First off, there’s the neat little trick that allows you to search for faces.

In general, the goal is to reach a higher level of result-relevancy. In order to do that, a service need sto analyse images more deeply than by just reading the text that surrounds pictures. Image analysis that involves a computer “seeing” an image and extracting features and content is very expensive in terms of computation. So in the interest of progress, despite the heavy computing cost, the New York Times reports how Google plans to roll out some of their image analysis features on a subset of internet images, rather than try to analyse all the images on the whole of the internet. Google is focusing on the top several thousand topics of interest based on query popularity, and popularity in online shopping (iPods, Wiis, Air Force Ones, etc.)

This concept of focusing on a subset of internet images related to shopping isn’t new: like.com has been in the process of launching and relaunching (that’s a link to the CEO’s blog) its visual search service since early 2007. Their service helps you shop the way you do in real life – visually, with visual feedback and comparison of how things actually look compared to similar items in similar price brackets.

Community Tagging

While image analysis and retrieval based on image features is the future, in the mean time we have tags. Tags are human-contributed textual descriptions of images. There are a variety of problems with tags (inconsitencies, incompleteness, subjectivity, etc.), but their presence reduces the image search problem to a far more manageable text search problem.

With this in mind, Google has quietly launched the Google Image Labeler, a game in which two randomly matched people tag an image co-operatively, and score points. Using a games to entice people to contribute annotation information is cool, but it appears to suffer from over-simplification: a picture of a migrating albatross is far more likely to be tagged as “bird” rather than as “transatlantic migration” or “albatross.” Though I’ve read some promising research that focuses on developing better tools that aid in annotation of images, which will hopefully lead to tags that are more complete, categorised, and community edited.

Leveraging Recommendations

Google has done quite a bit of development and research on the topic of recommendations, especially recommendations related to their Google News service.

I can’t help but think of the possibilities of using recommendation algorithms enhance image search. A service can push new image content to the forefront during queries, inter-mixed with the most relevant results, and allow people searching to annotate the recommended images. This would quickly collect information on how relevant that new image is to that query. Users could dismiss the image as irrelevant by never clicking on it, despite how often it appears in the first row of the result set.

That kind of information would provide a feedback loop for the developers of the relevancy algorithms, and potentially also allow for very personal image results: if those recommendations and your responses to them were kept around and associated with your Google account, future results could take advantage of your previous feedback.

There is much progress to be made in image search, and based on how often I come across interesting and relevant Google research in the field, it’s obvious that Google will continue to be a leader as image search matures.

In hopes of simplifying things, I’ve settled on a human-centred, conceptual definition of search:

Search refers to the process of a user developing a need, defining a query, retrieving information, viewing result(s), providing feedback, and refinements of those steps.

The end result does not have to be finding a single result. Occasionally, other steps in the search process, such as seeing a result set, can satisfy the users need. For example, if the need was to gain information (“What does a ’87 Oldsmobile Toronado FE3 look like?”) rather than find a specific image (“I need a picture of a black ’84 Cutty!”), viewing the result set may be enough.

Figure 1 is an illustrates the definition of search, in the human and interface context.

Figure 1 – Defining Search – View full size (54KB)

Search begins when a user develops a need for the search, and visits some image search service. The user then enters a query that they believe matches their search need (often is not the case). The form of this query is dramatically affected by the context of the interface: if the interface is just a keyword search bar, the user will enter keywords. If the interface offers a sample-image upload, the user may upload a sample image. If an interface offers domain specific search criteria, it will help the user find relevant images in that domain ex. an automobile search service that allows users to select “modified” vs. “stock” vs. “show” vehicles.

The query is then transformed into something that is useful for the information retrieval system. Information retrieval can include retrieving images as well as their metadata, their categories, or tags or whatnot. Here retrieval refers explicitly to the process done by the computers involved, rather than by the user. The bulk of the challenges that are tackled by researchers involved in CBIR occur around this step (query-to-retrieval-to-result-set).

Once the retrieval process is complete, a result set can be displayed back to the user. The format, order, and other information offered by the interface has a profound impact on a users ability to find a acceptable result, or otherwise navigate the result set. The result set can offer enough information for the user to then want to refine their query, or even refine their initial search need.

While interacting with the result set the user can choose to select an individual result for closer inspection, selection, or whatever they want. The user may offer feedback based on their reaction to either a single result, or the part of the result set they are interacting with. The interface hopefully allows for some sort of feedback mechanism which feeds into the relevance calculations of the image retrieval system. The result set then will update based on the feedback. Examples of this would be finding images “more like this one”, excluding small images, offering “yes, this is perfect!”

A users act of selecting an image out of a result set is probably important enough to provide at least some relevance feedback. This idea has been supported by research that claims that the vast majority of image searching work is done by interacting with the result set, well before a single image is selected.

The processing of feedback may not have an impact on the result set, but may improve search results in the future. An example of this kind of feedback would be tagging an image that the user viewed. ex. a photo of the Toronto skyline is viewed, and the user adds a tag or comment: “CN Tower at night”. Flickr, Facebook, and other image hosting services heavily leverage this process in their search process.

Anyway, that’s the taxonomy I will be working with. Feedback and commentary are most welcome.

CBIR is missing the point

As a side note, much of the technology mentioned in CBIR papers doesn’t give much credence to the human use element (“Why would people use this?”), nor the interface element (“How would a person make use of this, even if it was useful?”). There are many, many theses worth of papers to be written about: picking an element used in CBIR and studying the corresponding human and interface artifacts ex. searching with an image texture in mind (“fuzzy dog”), and studying what people are expecting to find, and how an interface can facilitate that.

First, every semester I hope/pray to get a Graduate Teaching Assistant job, which luckily gets easier and easier as I accumulate “seniority points.”

Second, no funding means no specified project, which means freedom to choose any research topic I please, as long as my (very lenient/forgiving) advisor is OK with it. Well, it’s been about eight months since I’ve come back from India all ready to start researching, and only two days ago did I actually settle on a topic.

Eight months is a long time to pay tuition, and follow dead ends with literature reviews. Also, those months are expensive if you waste your time on partying, girls, video games, Union involvement, student government, keggers, new housemates, motorcycles, trips to Mississippi, Vancouver, Ottawa, and so on. Well .. maybe it wasn’t a complete waste, per se :)

Finally, I’ve settled on a topic that I’m truly interested in.

Usability and Image Search

The topic: usability of image retrieval interfaces for systems based on image content, rather than user-provided textual meta-data.

What does this mean? Image searching (like Google Image, or Flickr search) is actually really, really complicated stuff. The systems on the back-end of the process use a variety of properties of an image to catalogue it in their vast library: the file name, it’s dimensions, data size, human contributed tags and categories, and a whole slew of other things. That specific information is called the metadata. Most of major (Yahoo, Google, MSN, etc.) image search engines also scour the web-page content that surrounds an image to get more metadata that can be searched.

Human contributed metadata is particularly meaningful: tags, categories, and file names point to the content of the image, not just its properties. The problem with human contributed metadata is that it’s:

• very incomplete
• very subjective, thus possibly inaccurate
• very time consuming if you do want to make it accurate or complete
• there are no standard descriptive elements in markup languages like XHTML or HTML for people to tag their images on the web

If you have a photo of a man riding a bicycle in Greece, unless you’ve tagged that picture with “man”, “bicycle”, “greece” (in four or five of the most common languages in the world), the likelihood of that image being catalogued so it can be found by an interested party is quite low.

Researchers since the mid-1990′s have spent a lot of time and energy in a field that hopes to automate more and more of that descriptive process, so that images can be searched on more than their properties. The field is called Content Based Image Retrieval (CBIR). Rather than relying on metadata to describe images, one can have a computer “see” the image and record information about it: colour, texture, salient regions, shapes, layout, etc. This is the content of the image. Cataloguing all of the useful content of a very large number of images accurately, completely, and in a way that can be easily searched, is the holy grail of this field. We are decades away from such a system, but major strides in that direction have already taken place.

Humans + CBIR

A smaller consideration of all of this is: how is a person supposed to search for something based on content? As in, I know a dog is fuzzy, I know that the computer can find fuzzy textures, but how do I describe to a search engine that I want to find all images of a brown, fuzzy dog? It sounds simple, should be simple, but is not simple in the current state-of-the-art.

Usability is a field that concerns itself with how things are used, especially interfaces of computer programs. I’m very interested in usability of these image search systems; image search systems that already exist, and proposed systems that will exist once we have the technology to automatically identify brown fuzzy dogs in your Flickr photos from the cottage.

So that’s where I’m at. I have a general idea of the area I’m interested in, so I’m in the process of reading some review papers about the state of CBIR. Hopefully, a literature review is coming next.