-
Notifications
You must be signed in to change notification settings - Fork 7
Replication Assignment
In this project, you will work in self-forming groups of 2 or 3 to reproduce and extend a published research article. The primary goals of this project are to 1) provide you with hands-on understanding on how behavioral and social science experiments are performed; 2) show you, in a way that no homework assignment can, the complexity and joy of doing real experimental work; 3) provide a new understanding on how key social or behavioral theories and findings are relevant to modern Health Tech and Connective Media environments.
The steps to the project are:
Email Xiao at xiao@jacobs.cornell.edu by October 20 with the names of people in your group and attach a PDF of the selected paper.
You should pick a paper in an area that is interesting to you. The paper should use experimental methods in an area that is closely related to the topics we covered in class. You are also welcome to use one of the papers we directly covered in class, if the paper presents an experimental method, or use any of the papers cited from chapters or review articles we read in class. One way to choose a paper is to go back to your favorite reading (or class topic) and mine the relevant paper for experiments or experiment ideas. For example, you can build on one of the experiments described in Nudge (make sure to identify the original papers), or one of the experiments described in Ryan and Deci, or any other reading or related paper (Dan Ariely’s work mentioned in class is a good potential source). Make sure to review future topics and papers that we haven’t covered yet, as well, as all the weeks up to Week 10 have good potential for materials.
This paper you chose should have been published in a peer-reviewed scholarly venue. The paper should include an experiment: a treatment applied using a repeatable procedure to a subgroup of individuals. Many papers cover a set of experiments; you only need to reproduce one, not all experiments described in one paper. I recommend choosing a paper where the experiment is well-defined and well-articulated.
I strongly recommend that you think hard about the paper and the experiment you chose to replicate. When picking a paper it may be hard for you to tell if the experiment will be hard to reproduce. Make sure you think about it and consult the teaching staff if necessary.
Before getting too far into your project you must get your plan approved. Your project proposal must be written in a structured format, in 400 words or less, and must include each of the following elements:
- Citation information for the paper you want to reproduce, and link to the paper
- Short explanation for why your group picked this paper
- A brief summary of the experiment described that you wish to replicate. Make sure to also include: the hypothesis, the sampling/recruitment used, the size of the sample, the treatment(s), dependent and independent variables, and how they were measured.
- If you are planning to update the settings in any way, add a sentence or two explaining that. Transforming the experiment to a new Connective Media environment (mobile device; social network site; internet of things; etc.) will count for extra credit (see more below).
Submit the plan by email to Xiao at xiao@jacobs.cornell.edu in email attachment, in PDF format, including team member names and email addresses in the document.
Please start the document name and email title with [Replication-Plan].
Follow the steps described in the experiment to reproduce the study as carefully and as accurately as possible. A few guidelines:
- Try to stick to two experimental conditions (or one control and one treatment); if the experiment and paper you picked call for more, talk to the teaching team.
- Make sure to have at least 20 participants (10 in each condition) and no more than 30 in total in your experiment. If you think you need more, talk to the teaching team.
- Recruiting is hard: participants don’t just "show up". Start early.
- DO NOT email the entire campus or even all Cornell Tech students asking for participants. Find other ways to recruit that would not result in an excessive load on the inboxes of too many people.
- There is no budget to pay participants. If your sampling is not the best as a result, that’s fine.
- If you need a lab, book one of the rooms on the 3rd floor.
- Experiments are often time consuming: depending on the setup, they may take over 30 minutes for each subject. Start early.
- Feel free to change the manipulation in a way that is meaningful but still likely to generate results, if you are interested in doing that. Make sure you note it in your proposal.
- It is possible to use online platforms like Mechanical Turk to recruit individuals; we will not be able to reimburse for costs, though.
You could somehow extend, change or improve the paper you are reproducing to show that it applies in health tech or connective media settings. Run the experiment in new settings (e.g. online, on new devices or systems). For example, to follow BJ Fogg: are people polite to a mobile device?
At the end of the semester you will present your results to the class. The goal of your presentation is to teach the class about your replication project and explain what you have learned.
The presentation should last 10 minutes and include a summary of the writeup, touching of these (sometimes overlapping) points:
- summary of the paper you replicated (about 2 minutes)
- description of your replication and any problems you had (about 3 minutes)
- description of your extension if any (about 1 minutes)
- your results (about 2 minutes)
- Discussions/interpretations the results and what they mean; were they unexpected or new? (about 1 minute)
- the biggest things you learned from doing this project (about 1 minute)
Write up your experiment report in Medium-style prose (i.e. as if you are writing a blog post about your work). Include the material from your original proposal plus, in under 500 additional words:
- A description of the experimental procedure as it was performed.
- A description of the sampling technique and a short summary of the participant characteristics.
- Summary statistics that summarize the results of your experiment.
- One or more figures that demonstrates the outcome of your experiment.
- A paragraph summarizing the takeaways
There does not need to be any new theory or literature review in this document. It is just about reproducing the results. It should, however, refer to the original study and its methods/findings.
Note: A robust statistical data analysis is not required. Descriptive statistics are enough, and don’t worry about p-values and statistical significance. However, you should include graphs (e.g. histograms, scatter plots) and descriptive statistics that indicate whether your experiment produced the hypothesized results or not.
Submit the write up by email to Xiao at xiao@jacobs.cornell.edu as email attachment, in .docx or .pdf format, including team member names and email addresses in the document.
Please start the document name and email title with [Replication-Report].
All deadlines are by end of day 24:00.
- Oct 20th: Form a group and choose a paper
- Oct 25th: Submit the plan
- Nov 4th (suggested): Deadline to finish implementation/special setup if needed
- Nov 11th (suggested): Deadline to finish data collection (experiment execution)
- Nov 29th-Dec 1st: present
- Dec 5th, 11:59pm: submit final report
If you need code or any other experimental materials, you are welcome to use code written by other people, including snippets of code you find online and code written by people who are helping you. However, if you are using code from someone else that must be acknowledged.
In writing, you would not use someone else's sentences in a paper without attribution. Sometimes there may be a gray area, and if you are in doubt you should attribute.
What is an IRB? IRB is the acronym for Institutional Review Board for Human Participants. Any institution that receives federal funding to conduct research with human participants, such as Cornell University, is required to establish an IRB to review all research that directly or indirectly involves human participants, and to set forth institutional policy governing such research.
Note that according to Cornell's IRB guidelines, the experiments you will run do not need to be approved by the IRB, but the way you would use these results would be severely limited: "If the project is to be used in classroom setting only to teach research methods, the project may not constitute human participant research. However, this means that at no point during or after the conclusion of the course can the results or the data be used for publication, presentation or other research purposes."
If you are interested in following up with your experiment and study to pursue a publication, presentation or other research purposes, you should try to submit an IRB request; make sure to discuss this with the teaching team before you take any action. The deadline for doing so is October 14th.
Above all, IRB or not, when you run your experiment, be fair and reasonable; and don’t be weird.
Thanks to Matt Salganik for the inspiration, the help in shaping, and a lot of the text describing this assignment.