CHI2013 RepliCHI Workshop Extended Abstract

 

RepliCHI 2013 - Proceedings

The RepliCHI 2013 Proceedings are now available online.
 

  • Didnt submit, but want to talk about RepliCHI?

We are meeting informally for drinks between the workshop and dinner (Dinner is TBA), in the Concorde LaFayette Hotel Foyer Bar, 5:30pm - 7pm on Saturday 27th April 2013. Meet us there to talk about RepliCHI.

 

RepliCHI 2013 (Draft) Schedule

Times Day 1 - Sat 27th April 2013 Day 2 - SunSun 28th April 2013
09:00 - 10:30 Session 1 - Framing the Workshop:
  • Welcome / Meet / Intro
  • Avoiding "It's JUST a Replication"
    Bonnie E. John
  • Is replication important for HCI?
    Christian Greiffenhagen, Stuart Reeves
Session 5 - Tandem Talks:
  • Re-start
  • Replicating an International Survey on User Experience: Challenges, Successes and Limitations
    Carine Lallemand, Vincent Koenig, Guillaume Gronier
    Original Authors Present: Effie Law, Virpi Roto
  • Replicating and Extending Research on Relations between Visual Aesthetics and Usability
    Noam Tractinsky
    Original Author Present: Masaaki Kurosu
Coffee Break
11:00 - 12:30 Session 2 - Differing Demographics:
  • NewsCube Replication: Experience Report
    Sidharth Chhabra, Paul Resnick
    Original Author Present: Souneil Park
  • Replicating Residential Sustainability Study in Urban India
    Mohit Jain, Yedendra B. Shrinivasan, Tawanna Dillahunt
  • Challenges of Replicating Empirical Studies with Children in HCI
    Quincy Brown, Lisa Anthony, Robin Brewer, Germaine Irwin, Jaye Nias, Berthel Tate
Session 6 - Tandem Talks & Planning:
  • Replicating and Extending a Facebook Uses & Gratifications Study: Five Years Later
    Tasos Spiliotopoulos, Ian Oakley
    Original Author Present: Adam Joinson
  • Replicating Two TelePresence Camera Depth-of-Field Settings in One User Experience Study
    Jennifer Lee Carlson, Mike Paget, Tim McCollum
  • Plenary and Planning (50 mins)
Lunch (1h may be optimistic)
13:30 - 15:00 Session 3 - Things Change Over Time:
  • Teaching HCI Methods: Replicating a Study of Collaborative Search
    Max L. Wilson
    Original Author Present: Chriag Shah (on Skype)
  • RepliPRI: Challenges in Replicating Studies of Online Privacy
    Sameer Patil
  • Re-testing the Perception of Social Annotations in Web Search
    Jennifer Fernquist, Ed H. Chi
Session 7 - Discussions:
  • Discussion 1 (45 mins)
  • Discussion 2 (45 mins)
Coffee Break
15:30 - 17:00 Session 4 - New Methods:
  • Do lab effects transfer into the real-world? And should we care?
    Petr Slovak, Paul Tennent, Geraldine Fitzpatrick
  • Replicating and Applying a Neuro-Cognitive Experimental Technique in HCI Research
    David Coyle
  • End of Day 1 Discussion (40 mins)
Session 8 - Planning for the future:
  • Discussion 3 (45 mins)
  • Closing Plenary (45 mins)
~17:30 - 19:00
Retire to discuss at leisure
at LaFayette Hotel Foyer Bar
 
19:00 onwards
Workshop Dinner
at location TBA
 

 

RepliCHI 2013 - Accepted Papers

 

Experience Reports

  • Replicating and Extending Research on Relations between Visual Aesthetics and Usability
    by Noam Tractinsky
  • Newscube Replication: Experience Report
    by Sidharth Chhabra and Paul Resnick
  • Do lab effects transfer into the real-world? And should we care?
    Petr Slovak, Paul Tennent and Geraldine Fitzpatrick
  • Teaching HCI Methods: Replicating a Study of Collaborative Search
    by Max L. Wilson
  • Replicating an International Survey on User Experience: Challenges, Successes and Limitations
    by Carine Lallemand, Vincent Koenig and Guillaume Gronier
  • Replicating and Extending a Facebook Uses & Gratifications Study: Five Years Later
    by Tasos Spiliotopoulos and Ian Oakley
  • Re-testing the Perception of Social Annotations in Web Search
    by Jennifer Fernquist and Ed H. Chi
  • Challenges of Replicating HCI Studies with Children
    by Quincy Brown, Lisa Anthony, Robin Brewer, Germaine Irwin, Jaye Nias and Berthel Tate
  • Replicating Residential Sustainability Study in Urban India
    by Mohit Jain, Yedendra Shrinivasan and Tawanna Dillahunt
  • Replicating and Applying a Neuro-Cognitive Experimental Technique in HCI Research
    by David Coyle
  • Replicating Two TelePresence Camera Depth-of-Field Settings in One User Experience Study
    Jennifer Lee Carlson, Mike Paget, Tim McCollum

 

Position Papers

Challenging RepliCHI <   <   >   > Supporting RepliCHI
Is replication important for HCI?
by Christian Greiffenhagen and Stuart Reeves
RepliPRI: Challenges in Replicating Studies of Online Privacy
by Sameer Patil
Avoiding "It's JUST a Replication"
by Bonnie E. John

 


Forms of Replication

Four common notions of replication, which have emerged from RepliCHI events and discussions so far, and from other disciplines, include:

Direct Replication consists of attempting to entirely replicate a study or system, using the same format and with the same tools, and experimental protocol. The aim of direct replications is often to replicate a specific finding. Direct Replication is often driven by the aspirations of strong science to confirm that results are true, are not created by an unseen bias, or that they apply in different contexts (geographic, cultural, topic, task) to the original study [2]. This method is often used as a teaching method for postgraduate students.

Conceptual Replications are systems and studies that focus on a certain principle or phenomenon and confirm findings using alternative methods. Of the three approaches, this is most common in HCI, in that multiple studies demonstrate the principles of direct manipulation. Many instances, however, are post-hoc reflections of their findings in the context of prior work. Through this approach we surmise heuristics about best practices for design or for evaluation.

Replicate+Extend is a common research method in which people first reach the level of prior research before investigating it further. This may involve reproducing a phenomenon before specifically investigating it further, or by building on the findings of the study. This form of research is often essential in understanding a form of interaction, after learning about the limitations of an initial approach, for example. However, Replicate+Extend is associated with the high risk of being described as ‘incremental’, and being rejected from prestigious peer-reviewed venues.

Applied Case Studies - One common form of replication is application – a special instance of conceptual replication. If HCI research produces a finding, and its application in real world contexts confirms it, then case studies are a form of replication. It is perhaps often one of your core desires that research findings are applied in real world contexts. Case studies of applying research findings, by practitioners in real world settings, are highly desirable, and this is an opportunity to engage with practitioners.

Reasons for Replication

For Research – Part of the maturity of a community involves reflecting and examining what has come from within it. We are currently driven towards novelty and impact, but part of impact is validating and understanding contributions. Having an archive of research findings that reflect directly on prior work would be highly valuable for our community.

For Practitioners - With an ever growing focus on integrating research and industry in within the HCI community, we can increase confidence in HCI findings by validating findings, whether our own or the findings of others. We can increase industry confidence, and thus our perceived value from outside our own community, of the work we together perform.

For Teaching – Replication of studies is commonly used in other disciplines as a teaching method for undergraduates or new postgraduate students. Trying to replicate a published study a) helps students to learn what is involved in performing a proper study, and b) reinforces the notions of properly reporting methods in papers (as they learn what is not reported in the papers describing the studies they have to replicate).

Workshop Key Dates

  • Submit: 1811 Jan 2013, 11:59pm PST
  • Notification: 8 Feb 2013
  • Camera-Ready: 18 Feb 2013
  • Earlybird Registion: 14 Mar 2013
  • Workshop: 27th & 28th April 2013

Register & Attend

RepliCHI 2013 is a workshop at CHI2013 in Paris

Find out more about registering and attending on the CHI2013 Attending Page.

If you have any more questions about attending, please contact Max.


Schedule

Repli CHI2013 is a 2-day workshop. The schedule is TBA, but we currently intend to focus on experience reports on the first day, and then go meta on the second day.


Call for Papers - CHI2013

We are inviting papers that describe experiences surrounding attempts to replicate HCI research. We believe that novel research findings should be published at venues equivalent to the magnitude of the results and their contribution. Experience Reports may be about a) previously published, b) unpublished, c) teaching, d) industry replications. Authors will retain copyright such that they can publish unpublished studies described in RepliCHI experience reports at a later time. The aim of these reports, therefore, is to help document the nature of replicating HCI systems and research, and provide a starting point for further discussions.

Consequently, we are primarily inviting experience reports (in 3-5 pages of ACM Extended Abstract format) that elaborate on:

  • the challenges you faced,
  • your successes,
  • the limitations of your replication
  • whether the results confirmed or challenged original findings
  • whether any differences in experimental protocols may have accounted for any differences in results.

We will also accept detailed well-considered position papers about the replication of HCI research, which contribute to our community’s understanding of the topic.

Reviewing

Papers will be reviewed according to

  • the insights into the challenges of replicating HCI research
  • the objective nature with which they discuss the possible reasons for differences in results
  • the professional tone in which they discuss the original work being replicated.

Submitting

3-5 pages - not anonymous - ACM Extended Abstract Format

Position papers will be discussed when we go meta in the second day

Experience reports may be about a) previously published papers, b) unpublished studies, c) teaching replications, d) applications of HCI principles in the real world, e) or any so-far unmentioned form of replication

Publishing experience reports will not prevent you from publishing the actual research elsewhere