This edition first published 2019
© 2019 John Wiley & Sons Inc.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.
The right of Leslie D. Rosenstein to be identified as the author of this work has been asserted in accordance with law.
Registered Office
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA
Editorial Office
111 River Street, Hoboken, NJ 07030, USA
For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.
Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand. Some content that appears in standard print versions of this book may not be available in other formats.
Limit of Liability/Disclaimer of Warranty
While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
Library of Congress Cataloging‐in‐Publication Data
Names: Rosenstein, Leslie D., author.
Title: Research design and analysis : a primer for the non‐statistician /Leslie D. Rosenstein, UT Southwestern Medical Center.
Description: Hoboken, NJ : Wiley, 2019. | Includes bibliographical references and index. |
Identifiers: LCCN 2019003539 (print) | LCCN 2019017653 (ebook) | ISBN9781119563624 (Adobe PDF) | ISBN 9781119563617 (ePub) | ISBN 9781119563594(hardback)
Subjects: LCSH: Medicine–Research–Methodology. | BISAC: SOCIAL SCIENCE /Sociology / General.
Classification: LCC R850 (ebook) | LCC R850 .R67 2019 (print) | DDC610.72–dc23
LC record available at https://lccn.loc.gov/2019003539
Cover image: © oxygen/Getty Images
Cover design by Wiley
I am grateful for my loving and supportive family who has helped me overcome some hefty obstacles. I would not have been able to write this book without the support of Jean, Marv, Dana, Shari, Kevin, Cory, and Caleigh.
I also want to acknowledge my students at UT Southwestern Medical Center for their patience and interest. Their reactions during class discussions, though hard to read at times, helped guide me in organizing and formulating the chapters of this primer. I especially want to thank Dr. Mallory Jacobs who inspired me to try to write something succinct and user‐friendly for busy physicians who want to be good consumers of clinical research.
| Figure 3.1 | Sample Latin square design completion 58 |
| Figure 3.2 | Sample double Latin square design 59 |
| Figure 5.1 | A roughly normal distribution 86 |
| Figure 5.2 | Positively kurtotic distribution 87 |
| Figure 5.3 | Negatively kurtotic distribution 88 |
| Figure 5.4 | A positively skewed distribution 89 |
| Figure 5.5 | A negatively skewed distribution 90 |
| Figure 5.6 | A bimodal distribution 90 |
| Figure 5.7 | Distributions with substantial overlap 92 |
| Figure 5.8 | Distributions with little overlap 92 |
| Figure 5.9 | Interaction between intervention and chronicity 105 |
| Figure 5.10 | Significant main effects of both intervention type and chronicity in the absence of an interaction 106 |
| Figure 5.11 | Significant interaction effect in the absence of main effects of gender or hand preference 107 |
| Figure 5.12 | Significant time × treatment group interaction in a repeated measures study 115 |
| Figure 6.1 | Illustration of a positively skewed distribution 141 |
| Figure 7.1 | A funnel plot of effect sizes 151 |
| Table 1.1 | Timeline of events and the evolution of research ethics 13 |
| Table 1.2 | Sample informed consent form 19 |
| Table 1.3 | Example of poor separation of investigator–clinician role 23 |
| Table 2.1 | Erroneous statements of causality 38 |
| Table 2.2 | Erroneous statements of generalizability 43 |
| Table 3.1 | Single-factor between-subjects design with two levels of the independent variable 49 |
| Table 3.2 | Single-factor between-subjects design with four levels of the independent variable 50 |
| Table 3.3 | Single-factor between-subjects design with three levels of the independent variable and three dependent variables 51 |
| Table 3.4 | A 2 × 3 multifactorial between-subjects design with two independent variables 51 |
| Table 3.5 | A within-subjects design with three levels of one within-subjects independent variable 53 |
| Table 3.6 | A 2 × 3 × 2 × 2 between–within subjects design with one dependent variable 56 |
| Table 3.7 | Single-case A-B-A-B research design 63 |
| Table 4.1 | The null hypothesis, power, and errors 69 |
| Table 4.2 | The relationship between error variance and the size of the F statistic 70 |
| Table 5.1 | Fictional illustration of mode, median, and mean 80 |
| Table 5.2 | Data illustrated in the normal distribution in Figure 5.1 87 |
| Table 5.3 | Data illustrated in the positively kurtotic distribution in Figure 5.2 88 |
| Table 5.4 | Data illustrated in the negatively kurtotic in Figure 5.3 89 |
| Table 5.5 | Dependent t test using raw and scaled scores 96 |
| Table 5.6 | Example of a t test for independent groups 100 |
| Table 5.7 | Example of a t test for matched pairs 100 |
| Table 5.8 | Sample study using ANOVA 103 |
| Table 5.9 | Example of a multifactorial ANOVA 108 |
| Table 5.10 | Example of a single-Factor ANCOVA 111 |
| Table 5.11 | Example of a multivariate analysis of variance study 113 |
| Table 5.12 | Example of a repeated measures study 118 |
| Table 5.13 | Sample study of a correlation between two variables 122 |
| Table 5.14 | Sample study using multiple correlation 123 |
| Table 5.15 | Multiple regression equation 123 |
| Table 5.16 | Sample multiple regression as an extension of the study in Table 5.14 125 |
| Table 5.17 | Example of a Logistic Regression Analysis 127 |
| Table 5.18 | An example of a study using Discriminant Function Analysis 129 |
| Table 6.1 | Hypothetical 2 × 2 contingency table for hand preference and gender 134 |
| Table 6.2 | Sample Chi-square table comparing groups 135 |
| Table 6.3 | Formula and sample calculation for X2 135 |
| Table 6.4 | Example of a study using the median test 138 |
| Table 6.5 | Example of a study using Phi 139 |
| Table 6.6 | Example of a study using a Mann–Whitney U test 140 |
| Table 6.7 | Example of a Wilcoxon signed-rank test 143 |
| Table 6.8 | Example of a Kruskal–Wallis test 145 |
| Table 6.9 | Example of a study with Spearman's rank-order correlation 147 |
| Table 7.1 | Sample layout of a meta-analytic table with standardized mean differences 154 |
| Table 7.2 | Sample layout of a meta-analytic table with r 154 |
| Table 8.1 | Abstract of a sample study 160 |
| Table 8.2 | Introduction section of a sample study 161 |
| Table 8.3 | Sample methods section 163 |
| Table 8.4 | Sample fictional results section 165 |
| Table 8.5 | Discussion section of a sample research report 166 |
| Table A.1 | Sample data set with SAS data step 180 |
| Table A.2 | A sample data step 181 |
| Table A.3 | Data from a within-subjects study 183 |
| Table A.4 | Larger data set with 1000 participants and 16 variables 184 |
| Table A.5 | Steps for creating a basic Microsoft Access database with forms 188 |
| Table A.6 | Data set created with Microsoft Access 189 |
| Table A.7 | Text file exported from Access database 190 |
| Table A.8 | SAS data step using pasted data exported from Access as a text file 190 |
| Table A.9 | SAS Proc Print output using pasted data from text file 191 |
| Table B.1 | Sample SAS program for conducting an analysis of variance 198 |
| Table B.2 | Sample SAS Log from an analysis of variance 199 |
| Table B.3 | Sample SAS output data for an analysis of variance 200 |
| Table B.4 | Sample SAS results for an analysis of variance (selected portions) 201 |
| Table B.5 | Sample t test with confidence intervals run using SAS 205 |
| Table B.6 | Sample Chi-square analysis run using SAS 207 |
| Table B.7 | Sample multiple regression analysis run using SAS 210 |
In this book, I set out to provide a hopefully, pain-free overview of research methods, design, and analysis. The intended audiences include those in the sciences who wish to conduct their own research without investing several semesters completing coursework in statistics and related fields, as well as those in the sciences, clinical fields, education, and the media who wish to read published research in an informed manner. In the former case, this manuscript will provide a general basis for designing and conducting research, though with the assistance of a statistical consultant. In the latter case, I hope this primer will provide a basis for reading, understanding, and critically evaluating research reports.
For health care providers who wish to read studies and make treatment recommendations to their patients based on study outcomes, I hope this book will be a good reference tool. Research publications can sometimes be full of nuances and jargon that are only meaningful to the trained researcher. Without a clear understanding of research design, validity, and interpretation, the results reported in publications can be misunderstood and applied improperly. Sometimes, the research may be poorly conducted or poorly reported, and a basic knowledge of research design and interpretation can be particularly useful in judging when that is the case. At other times, the research is well done, but difficult to understand without a basic knowledge of research methods.
Professionals working in the media are well aware of their great responsibility in reporting research findings to the public. The media has a special role in providing information to the public while avoiding harm as outlined in the Professional Journalists’ Code of Ethics (Society of Professional Journalists, 2014). That code also mandates that journalists are responsible for the accuracy of their reporting, including verifying the information before it is released.
Carrie Figdor (2017) points out the difficulty presented to journalists in their role of reporting and providing information that is accurate when the material is the product of scientific endeavors. Journalists cannot necessarily rely on authors of scientific reports to provide accurate and valid information, and this quandary has become exponentially worse with the evolution of mass communication tools. Non-peer-reviewed research reports are more readily available to the masses. Moreover, journalists cannot necessarily count on peer-reviewed journals to publish only sound research. Most do, but journalists must be careful, yet, to review and understand the research design as presented along with the results and conclusions.
Journalists must take care, for instance, to not translate a conclusion of an association between two events or variables into a claim of causality. Oftentimes, the correct language to that effect is included in a research publication, but it is incumbent on the journalist to read and understand such language. Otherwise, there is a real and great risk that the public will be misinformed and harmed as a result. In Chapter 9, I discuss this in more detail with respect to specific instances of marked harm being perpetrated unintentionally (e.g. the unsubstantiated fear of the measles vaccine, misinformation about the true risks of chronic traumatic encephalopathy, and misinterpretation of the Women's Health Initiative findings).
The chapters of this book are laid out into four major sections. In Section 1, I briefly review the purpose of research as well as ethics and rules guiding research involving human participants and animal subjects. In Section 2, I walk you through basic research designs and validity. In Section 3, I provide a cursory review of statistical techniques, just enough to make you conversant with your statistical consultant or to be able to comprehend the jargon you find in many research documents. I have also included a chapter on meta-analytic studies. The goal of that chapter is to help you in sifting through reports of meta-analyses, though I also provide some direction in case you ever consider conducting your own meta-analytic study. In the fourth section, I review the how-tos of disseminating research findings, including reporting and presenting research results. I discuss how to prepare a research paper for submission to a peer-reviewed journal. I also talk about the concept of poster presentations and how to submit research more quickly for presentation at a conference.
In Section 4, I also present my concluding remarks. There, I repeat what I emphasize throughout this primer; that is, research and research findings are only as good as the research design. Most importantly, it is crucial to avoid making statements of claims of causality between two conditions, or variables, when the research design does not permit drawing such conclusions with any degree of confidence. Accurate interpretation of research findings is of critical importance. This does not just apply to the authors of the original research but also to others who report about and share research findings and claims more broadly. In particular, I hope to underline the importance and responsibility carried by journalists and others who discuss research claims. Sadly, when research claims are reported and shared with the public without a critical eye or with misstatements about causality, harm may ensue.
Finally, I have prepared appendices with tools for those who are planning to conduct their own research. These contain information about data sets, databases, statistical software programs, and resources for those who want to learn more about inferential statistics. I have additionally included a glossary of many of the terms included in this primer; in the glossary, terms are alphabetized for quick lookup.