Questionnaires

Questions and Answers about the Entry and Exit Questionnaires.

README FILE

The first task - filling out the entry questionnaire - requires reading the data section of the article as well as any documentation in the supplementary data zip, in particular the README, attentively. In particular, check carefully if any data is provided in the ZIP archive - the presence of a ZIP archive does NOT imply that it contains data. When it does have data, please look carefully - BEFORE running any programs - what the README says, and if not all data are provided, distinguish, as we have tried to so far, between "input data" (often not provided) and "analysis data" (sometimes provided).

DATA THAT REQUIRES REGISTRATION

Q: If an article doesn't provide justification for why it doesn't include datasets, but you visit the database described in the article and it requires registration to access the database, then should we mark the datasets as missing data (no justification) or proprietary data in the entry questionnaire?

A: If you found the data, and the registration is trivial (I.e. After registration you can download the data), then the data is not missing, and the entry questionnaire should list the URL or doi for the dataset.

RESUBMISSION

Q: So I made a mistake with a past entry questionnaire, and realize that I based my answers on data for the wrong article. Can I just resubmit a new entry questionnaire?

A: Yes, please do resubmit, and let us know which DOI we should remove the faulty entry for.

RESULTS MISMATCH

Q: If we successfully run the code for an article, but some of the numbers don't match between the code-generated tables and the article tables, does that count as a full or partial replication?

A: This is a judgement call, to some extent. If the numbers don't match EXACTLY (i.e., there is some difference in the second or third decimal), then treat it as a "successful replication", but note the slight discrepancy in the comments of the EXIT questionnaire. If the numbers are really off, then you should mark it as a partial, or a non-replication, depending on how many tables are at fault (if some tables match, then it's a partial, if none match, then a non-replication).

MULTIPLE DATASETS

Q: The author provides Analysis, Input and Temporary data sets. How to classify Temporary dataset? The ReadMe states that it is data sets that are created during the data cleaning process. Additionally, each of these categories contains many datasets both as Excel and Stata files. How should I report this in the framework of the entry questionnaire?

A: Lump the datasets together as Analysis and Input, mark both Stata and Excel as format. You can probably ignore the intermediate - move them out of the way, and you can use them as a check that the data cleaning works (i.e., if you run the programs that read the Input and write the Intermediate, you should get the same datasets as the author provided). You do not need to describe the Temporary datasets in the Entry questionnaire.