../[ dariusz grabka ]\..

I’m trying to wrap my head around all of the information I’m reading about image search. Search, retrieval, information, data, all of these terms are loaded, and used differently depending on whether I’m reading an HCI paper, a text analysis paper, or a blog post about search engine optimisation (SEO).

In hopes of simplifying things, I’ve settled on a human-centred, conceptual definition of search:

Search refers to the process of a user developing a need, defining a query, retrieving information, viewing result(s), providing feedback, and refinements of those steps.

The end result does not have to be finding a single result. Occasionally, other steps in the search process, such as seeing a result set, can satisfy the users need. For example, if the need was to gain information (”What does a ‘87 Oldsmobile Toronado FE3 look like?”) rather than find a specific image (”I need a picture of a black ‘84 Cutty!”), viewing the result set may be enough.

Figure 1 is an illustrates the definition of search, in the human and interface context.

Figure 1 - Defining Search - View full size (54KB)

Search begins when a user develops a need for the search, and visits some image search service. The user then enters a query that they believe matches their search need (often is not the case). The form of this query is dramatically affected by the context of the interface: if the interface is just a keyword search bar, the user will enter keywords. If the interface offers a sample-image upload, the user may upload a sample image. If an interface offers domain specific search criteria, it will help the user find relevant images in that domain ex. an automobile search service that allows users to select “modified” vs. “stock” vs. “show” vehicles.

The query is then transformed into something that is useful for the information retrieval system. Information retrieval can include retrieving images as well as their metadata, their categories, or tags or whatnot. Here retrieval refers explicitly to the process done by the computers involved, rather than by the user. The bulk of the challenges that are tackled by researchers involved in CBIR occur around this step (query-to-retrieval-to-result-set).

Once the retrieval process is complete, a result set can be displayed back to the user. The format, order, and other information offered by the interface has a profound impact on a users ability to find a acceptable result, or otherwise navigate the result set. The result set can offer enough information for the user to then want to refine their query, or even refine their initial search need.

While interacting with the result set the user can choose to select an individual result for closer inspection, selection, or whatever they want. The user may offer feedback based on their reaction to either a single result, or the part of the result set they are interacting with. The interface hopefully allows for some sort of feedback mechanism which feeds into the relevance calculations of the image retrieval system. The result set then will update based on the feedback. Examples of this would be finding images “more like this one”, excluding small images, offering “yes, this is perfect!”

A users act of selecting an image out of a result set is probably important enough to provide at least some relevance feedback. This idea has been supported by research that claims that the vast majority of image searching work is done by interacting with the result set, well before a single image is selected.

The processing of feedback may not have an impact on the result set, but may improve search results in the future. An example of this kind of feedback would be tagging an image that the user viewed. ex. a photo of the Toronto skyline is viewed, and the user adds a tag or comment: “CN Tower at night”. Flickr, Facebook, and other image hosting services heavily leverage this process in their search process.

Anyway, that’s the taxonomy I will be working with. Feedback and commentary are most welcome.

CBIR is missing the point

As a side note, much of the technology mentioned in CBIR papers doesn’t give much credence to the human use element (”Why would people use this?”), nor the interface element (”How would a person make use of this, even if it was useful?”). There are many, many theses worth of papers to be written about: picking an element used in CBIR and studying the corresponding human and interface artifacts ex. searching with an image texture in mind (”fuzzy dog”), and studying what people are expecting to find, and how an interface can facilitate that.