title: "Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos"
authors: Juho Kim, Phu Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, Krzysztof Z. Gajos
venue: ACM Conference on Human Factors in Computing Systems (CHI)
year: 2014
footer: "<font color=\"#AA0000\">Honorable Mention Paper</font>"

tweet: Crowdsourcing can effectively add annotations of step-by-step instructions to tutorial videos

abstract: >
  Millions of learners today use how-to videos to master new skills in a
  variety of domains. But browsing such videos is often tedious and
  inefficient because video player interfaces are not optimized for the
  unique step-by-step structure of such videos. This research aims to
  improve the learning experience of existing how-to videos with
  step-by-step annotations.
  <p></p>
  We first performed a formative study to verify that annotations are
  actually useful to learners. We created ToolScape, an interactive video
  player that displays step descriptions and intermediate result
  thumbnails in the video timeline. Learners in our study performed better
  and gained more self-efficacy using ToolScape versus a traditional video
  player.
  <p></p>
  To add the needed step annotations to existing how-to videos at scale,
  we introduce a novel crowdsourcing workflow. It extracts step-by-step
  structure from an existing video, including step times, descriptions,
  and before and after images. We introduce the Find-Verify-Expand design
  pattern for temporal and visual annotation, which applies clustering,
  text processing, and visual analysis algorithms to merge crowd output.
  The workflow does not rely on domain-specific customization, works on
  top of existing videos, and recruits untrained crowd workers. We
  evaluated the workflow with Mechanical Turk, using 75 cooking, makeup,
  and Photoshop videos on YouTube. Results show that our workflow can
  extract steps with a quality comparable to that of trained annotators
  across all three domains with 77% precision and 81% recall.

bibtex: >
  @inproceedings{KimCHI2014toolscape,
   author = {Kim, Juho and Nguyen, Phu Tran and Weir, Sarah and Guo, Philip J. and Miller, Robert C. and Gajos, Krzysztof Z.},
   title = {Crowdsourcing Step-by-step Information Extraction to Enhance Existing How-to Videos},
   booktitle = {Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems},
   series = {CHI '14},
   year = {2014},
   isbn = {978-1-4503-2473-1},
   location = {Toronto, Ontario, Canada},
   pages = {4017--4026},
   numpages = {10},
   url = {http://doi.acm.org/10.1145/2556288.2556986},
   doi = {10.1145/2556288.2556986},
   acmid = {2556986},
   publisher = {ACM},
   address = {New York, NY, USA},
   keywords = {crowdsourcing, how-to videos, video annotation.},
  }