An undergraduate training in statistics tells us that sequential testing with optional stopping inflates the false positive rate, but exactly how much it inflates it is less intuitive. Sometimes, the metaphor of rolling a 20-sided dice every time a statistical test is run (assuming alpha = 0.05). However, whereas each roll of the dice is independent, running a new instance of a test on a slightly data set after adding a number of participants is not fully independent.

Ed Hagan recently wrote a piece titled “Academic success is either a crapshoot or a scam” in which he uses some simple maths about about the rate at which research is done to argue that “empirical social scientists with lots of pubs in prestigious journals are either very lucky, or they are p-hacking.” That is, regardless of intentions, a situation where research practices are inflating the false-discovery rate. I have no doubt that p-hacking occurs and is actively rewarded by our current incentive structures, but I’d nonetheless like to argue for an additional possibility: fabricated interest.

Most experiments using reaction time data separate scoring and analysis. That is, the study captures a large number of reaction times for each participant, then reduces this down to a smaller number of metrics (e.g., mean reaction time, D scores, etc), and then analyses these metrics. In doing so, we can throw away the variance associated with each participant’s performance and unnecessarily decrease the power of our analysis. Mixed effects modelling provides an accessible alternative that does away with data scoring.

Research questions What age would people chose to be if given the choice? How old is “old”? Do opinions about ‘how old is old’ change with one’s own age? Open data This data was collected on Project Implicit and is freely available under a CCO license on the Open Science Framework. Specifically, I took the data from the Age related Implicit Association Test collected in 2015. # dependencies ------------------------------------------------------------ library(tidyverse) library(psych) library(knitr) library(broom) data_trimmed <- read_rds("data/data_trimmed.

Perhaps it’s just my own learning curve, but it feels like internet resources are making it easier and easier to conduct and communicate research. LimeSurvey does away with the need for paper questionnaires, PsychoPy allows me to write training and testing procedures in a matter of hours, Google docs allows me to write collaboratively, Zotero references the manuscripts for me, Recite checks them for errors, ResearchGate takes care of distribution, and Dropbox covers my ass incase anything goes wrong.

If you were trained on SPSS you’re probably accustomed to the idea that running and analysis and reporting an analysis are entirely seperate stages in writing a paper. However, it doesn’t have to be so. The results returned by most stats packages have the important data spread out over a number of different tables. This always seemed inefficient to me, given that a huge proportion of users are all looking to produce APA styled output at the end of the day.

Experimental-clinical psychologists have finally started to capitalise on naturalistic internet data sources for research. For example, recent papers using data from the social news aggregator Reddit (e.g., Hipp et al., 2015), and the social networking site Tumblr (e.g., Cavazos-Rehg et al., 2016). The AskReddit subreddit is a particularly useful source. Here, hundreds or thousands of people provide answers to single questions. In many cases these data may be difficult to come by through other means.

Even if your data and code are publicly available, you can still fall at the last hurdle by not reporting the correct numbers in a manuscript. Luckily, the statcheck library can check this for you. Specifically, it: extracts all the analyses reported in your manuscript takes the test statistics and df and recomputes p values compares the recomputed p value with the p value reported in the manuscript to check for reporting errors, and also for APA compliance.