etklab_logoWelcome to the Evaluation Toolkit Lab. We have created a toolkit designed to help you conduct image-based perceptual studies on your own visualizations. Using this toolkit, you can quickly construct and conduct studies that provide feedback on your visualizations, so that you can evaluate, understand and improve them. If you have a visualization, and want to know how effective it is, our toolkit can help.

The ETK consists of a set of modules that leverage the JavaScript API within Qualtrics Survey software ( to easily implement  and run different types of psychophysical experiments with images.   Each module consists of JavaScript, HTML and CSS scripts that are inserted into a Qualtrics question/survey.  The researcher needs to make only minimal changes (e.g., update the list of images to use in a study) in order to get a study question set up.

The ETK may not be able to design a user study for you.  But once you have an experimental design, the ETK makes it a snap to implement it within Qualtrics.

Currently, there are six modules:

  • 2AFC – a two alternative forced choice method
  • MoA – a method of adjustment method
  • RRC – a round robin comparison method
  • C2A – a method that compares each image in an array to its counterpart in a second array; could be used as a 2AFC variant or to hard-code specific comparisons
  • CC – a simple script to count the number of times an image is clicked.
  • KeyTask – a script that displays a stimuli image along with a set of answer keys; keys can be coded by color or by name

Funding for this project created under a grant from the US Department of Energy, Office of Science, Advanced Scientific Research Computing (Dr. Lucy Nowell, Program Manager).
Image created by Luis Prado from Noun Project