ECS 166: Evaluating User Interactions

Summary of Course Content:

As many applications and technologies demand quality user experiences and usable interactions between users and the systems, it is a critical step of the system development process to evaluate how well users interact with the systems. One challenge in evaluating how users interact with technologies resides at the complex dependencies between technological and human factors involved in such interactions, which raises the need for researchers to select and employ appropriate research methods to ensure the validity and reliability of user studies. In this course, students will learn about the characteristics, constraints and examples of different research methods for conducting user studies. Students will learn how to motivate user research, how to ask research questions, how to choose methods, how to design measures, and how to collect, analyze and report data. Topics will include:

1. Overview of user studies
2. Qualitative and quantitative research strategies
3. Experimental research and design
4. Fundamentals in quantitative measurement and analysis
5. Task analysis and heuristics
6. Self-report data collection -  surveys and diaries
7. Interactive data collection - interview and focus groups
8. Unobtrusive data collection - observations and logging
9. Qualitative and fields studies
10. Research ethics
11. Evaluating user interactions with non-human agents (e.g., chatbots, recommender systems)

Specific learning goals of this course include:

1. Understand the key principles, criteria and goals of user studies in technology evaluation
2. Understand the characteristics and trade-offs of different research methods
3. Being able to ask good research questions for user studies and identifying suitable methods and measures for the target questions
4. Being able to engage in the process of developing user study protocols for evaluating targeted systems
5. Being able to critically discuss the validity and reliability of specific study designs, and revising the study designs.
6. Possessing the awareness of emerging issues and challenges for evaluating new technologies.

Illustrative Reading

Research Methods in Human-Computer Interaction (2nd Edition) Authors: Jonathan Lazar Jinjuan Feng Harry Hochheiser

Potential Course Overlap

The course is to supplement ECS 164 Human-Computer Interaction with a different emphasis. This course focuses on designing user studies, controlled experiments, interview protocols, and data analysis etc. to evaluate how well users use and interact with the target systems. ECS 164 on the other hand is a general HCI course covering the breadth of HCI, and with a specific focus on using a standard user-centered design process for constructing new systems.