DFKI
Home
Selected Public Appearances
Current Work (Selection)
  VOICE Award 2006
  Project HALO 2
  GNAB
  juris
  Mobile Multimodal Systems
Recent Work (Selection)
  VOICE Award 2005
  Improved Internet Search
  WYSIWYM
  Document Manager
  Multitasking with Xybernaut
  Automatic Hyperlinking
  Speech-Based Home Cooking Assistant
  Sonicson Music Retrieval System
Equipment
  Tips and Tricks for Mobile Eye Tracking
Publications
Selected Videos
Services and Contact

German Competence Center for Speech Technology The other parts of the German Competence Center for Speech Technology:
Virtual Information Center LT-World
German Demonstration Center for LT-Systems

The COLLATE Project

Last modified: 2006-11-09

The DFKI Evaluation Center for Language Technology

 

Recent Work


 
 

Improved Internet Search

 
  Experiment 1, 2003:

Original version with modified order
Improved version with modified order, headings, and dots
Screenshot Original version Screenshot Improved version


This study was performed as part of Nadine Wirschum's diploma thesis (see Wirschum, 2003).

Two groups of 42 subjects were presented with a search results page that listed 25 web pages retrieved for a query about “assessment center building blocks”. For the second group, the summaries had been enhanced with headings. The task of all subjects was to answer a question about assessment centers by visiting at least some of the listed pages. The statistical analysis of the results showed that, with the enhanced summaries, a significantly higher percentage of the visited pages (54% vs. 37%) were ones that the experimenter had previously classified as being relevant to the to-be-answered question. This 44% relative increase in precision was not achieved at the expense of recall: In both conditions just about 67% of the relevant pages were visited. (Only pages whose title, URL, or summary the subject had looked at in the result list were counted in this analysis.) Subjects’ ratings and debriefing comments confirmed that the enhanced summaries gave them a clearer impression of the context in which the keywords appeared. But the presence of the headings did not increase the time that the subjects required to read or scan the summaries.

Further analyses of the eye tracking and mouse-click data yielded additional results that have implications for the design of search result pages. For example, about one-third of the subjects scanned part or all of the search list before going back to visit individual pages–a breadth-first strategy that is not very well supported by the leading search engines.

The study was performed using the ASL Model 504 remote eye tracker; see the videos recorded during the study.

 


Experiment 2, 2004:

Strictly Depth-First Search
Partly Breadth-First Search
Strictly Breadth-First Search
  
  

In Experiment 2, each of 27 subjects was asked to perform two tasks similar to those of Experiment 1, with 5 minutes allowed for each task. To create a situation in which breadth- first processing seemed relatively attractive, we allowed the subjects to open at most 10 of the 25 documents listed, rewarding them for each relevant document found (about half of all documents were relevant). Here again, contrasting strategies were identified: 52% of the subjects showed virtually no tendency to look ahead in the list. A minority of 11% used the extreme breadth-first strategy, scanning the entire list before opening any document; the remaining 37% applied a mixed strategy, looking ahead at an average of 2 to 6 documents within each list.

Publications on this subject: see Klöckner, Wirschum, and Jameson (2004).