CONTENTS
PREFACE TO THE FIRST EDITION
PREFACE TO THE SECOND EDITION
ACKNOWLEDGMENTS
CHAPTER ONE: AN INTRODUCTION TO SURVEY METHODOLOGY
1.1 INTRODUCTION
1.2 A BRIEF HISTORY OF SURVEY RESEARCH
1.3 SOME EXAMPLES OF ONGOING SURVEYS
1.4 WHAT IS SURVEY METHODOLOGY?
1.5 THE CHALLENGE OF SURVEY METHODOLOGY
1.6 ABOUT THIS BOOK
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER TWO: INFERENCE AND ERROR IN SURVEYS
2.1 INTRODUCTION
2.2 THE LIFE CYCLE OF A SURVEY FROM A DESIGN PERSPECTIVE
2.3 THE LIFE CYCLE OF A SURVEY FROM A QUALITY PERSPECTIVE
2.4 PUTTING IT ALL TOGETHER
2.5 ERROR NOTIONS IN DIFFERENT KINDS OF STATISTICS
2.6 NONSTATISTICAL NOTIONS OF SURVEY QUALITY
2.7 SUMMARY
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER THREE: TARGET POPULATIONS, SAMPLING FRAMES, AND COVERAGE ERROR
3.1 INTRODUCTION
3.2 POPULATIONS AND FRAMES
3.3 COVERAGE PROPERTIES OF SAMPLING FRAMES
3.4 ALTERNATIVE FRAMES FOR SURVEYS OF THE TARGET POPULATION OF HOUSEHOLDS OR PERSONS
3.5 FRAME ISSUES FOR OTHER COMMON TARGET POPULATIONS
3.6 COVERAGE ERROR
3.7 REDUCING UNDERCOVERAGE
3.8 SUMMARY
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER FOUR: SAMPLE DESIGN AND SAMPLING ERROR
4.1 INTRODUCTION
4.2 SAMPLES AND ESTIMATES
4.3 SIMPLE RANDOM SAMPLING
4.4 CLUSTER SAMPLING
4.5 STRATIFICATION AND STRATIFIED SAMPLING
4.6 SYSTEMATIC SELECTION
4.7 COMPLICATIONS IN PRACTICE
4.8 SAMPLING U.S. TELEPHONE HOUSEHOLDS
4.9 SELECTING PERSONS WITHIN HOUSEHOLDS
4.10 SUMMARY
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER FIVE: METHODS OF DATA COLLECTION
5.1 ALTERNATIVE METHODS OF DATA COLLECTION
5.2 CHOOSING THE APPROPRIATE METHOD
5.3 EFFECTS OF DIFFERENT DATA COLLECTION METHODS ON SURVEY ERRORS
5.4 USING MULTIPLE MODES OF DATA COLLECTION
5.5 SUMMARY
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER SIX: NONRESPONSE IN SAMPLE SURVEYS
6.1 INTRODUCTION
6.2 RESPONSE RATES
6.3 IMPACT OF NONRESPONSE ON THE QUALITY OF SURVEY ESTIMATES
6.4 THINKING CAUSALLY ABOUT SURVEY NONRESPONSE ERROR
6.5 DISSECTING THE NONRESPONSE PHENOMENON
6.6 DESIGN FEATURES TO REDUCE UNIT NONRESPONSE
6.7 ITEM NONRESPONSE
6.8 ARE NONRESPONSE PROPENSITIES RELATED TO OTHER ERROR SOURCES?
6.9 SUMMARY
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER SEVEN: QUESTIONS AND ANSWERS IN SURVEYS
7.1 ALTERNATIVES METHODS OF SURVEY MEASUREMENT
7.2 COGNITIVE PROCESSES IN ANSWERING QUESTIONS
7.3 PROBLEMS IN ANSWERING SURVEY QUESTIONS
7.4 GUIDELINES FOR WRITING GOOD QUESTIONS
7.5 SUMMARY
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER EIGHT: EVALUATING SURVEY QUESTIONS
8.1 INTRODUCTION
8.2 EXPERT REVIEWS
8.3 FOCUS GROUPS
8.4 COGNITIVE INTERVIEWS
8.5 FIELD PRETESTS AND BEHAVIOR CODING
8.6 RANDOMIZED OR SPLIT-BALLOT EXPERIMENTS
8.7 APPLYING QUESTION STANDARDS
8.8 SUMMARY OF QUESTION EVALUATION TOOLS
8.9 LINKING CONCEPTS OF MEASUREMENT QUALITY TO STATISTICAL ESTIMATES
8.10 SUMMARY
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER NINE: SURVEY INTERVIEWING
9.1 THE ROLE OF THE INTERVIEWER
9.2 INTERVIEWER BIAS
9.3 INTERVIEWER VARIANCE
9.4 STRATEGIES FOR REDUCING INTERVIEWER BIAS
9.5 STRATEGIES FOR REDUCING INTERVIEWER-RELATED VARIANCE
9.6 THE CONTROVERSY ABOUT STANDARDIZED INTERVIEWING
9.7 INTERVIEWER MANAGEMENT
9.8 VALIDATING THE WORK OF INTERVIEWERS
9.9 THE USE OF RECORDED VOICES (AND FACES) IN DATA COLLECTION
9.10 SUMMARY
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER TEN: POSTCOLLECTION PROCESSING OF SURVEY DATA
10.1 INTRODUCTION
10.2 CODING
10.3 ENTERING NUMERIC DATA INTO FILES
10.4 EDITING
10.5 WEIGHTING
10.6 IMPUTATION FOR ITEM-MISSING DATA
10.7 SAMPLING VARIANCE ESTIMATION FOR COMPLEX SAMPLES
10.8 SURVEY DATA DOCUMENTATION AND METADATA
10.9 SUMMARY
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER ELEVEN: PRINCIPLES AND PRACTICES RELATED TO ETHICAL RESEARCH
11.1 INTRODUCTION
11.2 STANDARDS FOR THE CONDUCT OF RESEARCH
11.3 STANDARDS FOR DEALING WITH CLIENTS
11.4 STANDARDS FOR DEALING WITH THE PUBLIC
11.5 STANDARDS FOR DEALING WITH RESPONDENTS
11.6 EMERGING ETHICAL ISSUES
11.7 RESEARCH ABOUT ETHICAL ISSUES IN SURVEYS
11.8 ADMINISTRATIVE AND TECHNICAL PROCEDURES FOR SAFEGUARDING CONFIDENTIALITY
11.9 SUMMARY AND CONCLUSIONS
KEYWORDS
FOR MORE IN-DEPTH READING
EXERCISES
CHAPTER TWELVE: FAQS ABOUT SURVEY METHODOLOGY
12.1 INTRODUCTION
12.2 THE QUESTIONS AND THEIR ANSWERS
REFERENCES
INDEX
WILEY SERIES IN SURVEY METHODOLOGY
WILEY SERIES IN SURVEY METHODOLOGY
Established in Part by WALTER A. SHEWHART AND SAMUEL S. WILKS
Editors: Robert M. Groves, Graham Kalton, J. N K. Rao, Norbert Schwarz, Christopher Skinner
A complete list of the titles in this series appears at the end of this volume.
Copyright © 2009 by John Wiley & Sons, Inc. All rights reserved.
Published by John Wiley & Sons, Inc., Hoboken, New Jersey.
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008 or online at http://www.wiley.com/go/permission.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our Customer Care Department within the U.S. at (800) 762-2974, outside the U.S. at (317) 572-3993 or fax (317) 572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic format. For information about Wiley products, visit our web site at www.wiley.com.
Library of Congress Cataloging-in-Publication Data:
Survey methodology / Robert Groves… [et al.]. — 2nd ed.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-470-46546-2 (paper)
1. Surveys—Methodology. 2. Social surveys—Methodology. 3. Social sciences—Research—Statistical methods. I. Groves, Robert M.
HA31.2.S873 2009
001.4’33--dc22
2009004196
PREFACE TO THE FIRST EDITION
We wrote this book with a specific purpose in mind. We are all survey methodologists—students of the theories and practices of the various data collection and analysis activities that are called “survey research.” Surveys (in a form that would be recognizable today) are approximately 60–80 years old. Over the past two decades, a set of theories and principles has evolved that offer a unified perspective on the design, conduct, and evaluation of surveys. This perspective is most commonly labeled the “total survey error” paradigm. The framework guides modem research about survey quality and shapes how practicing survey professionals approach their work. The field arising out of this research domain can appropriately be called “survey methodology.”
We increasingly noticed a mismatch, however, between the texts related to surveys and how the science of surveys was evolving. Many survey research texts focused on the application of tools and deemphasized the theories and science underlying those tools. Many texts told students to do things that were no longer or never supported by the methodological research in the area. In short, there were books that emphasized “how to do” surveys but neglected the science underlying the practices that were espoused.
Most harmful we thought was the impression conveyed to those who read the texts that surveys were merely a recipe-like task; if step-by-step instructions were followed, high quality would be guaranteed. In contrast, we saw surveys as requiring the implementation of principles in unique ways to fit a particular substantive purpose for a particular target population.
These issues became particularly important to us when the demand for a one semester graduate level (and senior undergraduate level) course became obvious at the Joint Program in Survey Methodology (JPSM), a consortium graduate program funded by the U.S. Federal statistical agencies in which the authors teach. The students would often have advanced education in another field (e.g., economics, statistics, or psychology) but no formal exposure to the field of survey methodology. We planned a 14-week lecture course with exercises and examinations that began in the Fall of 1998, and we immediately suffered from the absence of a text that could accompany the lectures and motivate the exercises.
We began to envision a text describing the basic principles of survey design discovered in methodological research over the past years and the guidance they offered for decisions that are made in the execution of good quality surveys. We wanted to include exercises that would help integrate an understanding of the field. We wanted to convey that the field is based on experimental and other research findings and that practical survey design was not a mere matter of judgment and opinion but rather the result of a body of research findings.
We drafted this book over several years. After we wrote the first couple of chapters, we hit a dry spell, which was ended when our colleague Nancy Mathiowetz kicked us back in gear. We appreciated her energy in getting us going again.
The manuscript profited greatly from the critique of our student colleagues. The text had a dry run in the Summer of 2003 in a class at the University of Michigan Survey Research Center Summer Institute in Survey Research Techniques, entitled “Introduction to Survey Research Techniques,” taught by Maria Krysan and Sue Ellen Hansen. We thank these instructors for helping improve the manuscript. We learned much from the criticisms and ideas of both Krysan and Hansen and the students in the class: Nike Adebiyi, Jennifer Bowers, Scott Compton, Sanjay Kumar, Dumile Mkhwanazi, Hanne Muller, Vuyelwa Nkambule, Laurel Park, Aaron Russell, Daniel Spiess, Kathleen Stack, Kimiko Tanaka, Dang Viet Phuong, and Christopher Webb.
It is fair to say that this book strongly reflects the lessons taught by many of our own mentors. One deserves special mention. All of the authors were friends and students of Charlie Cannell (some formally; all informally). Charles F. Cannell began his survey career with Rensis Likert at the U.S. Department of Agriculture, Division of Program Surveys. Cannell later joined Likert and others in founding the University of Michigan Survey Research Center in 1946. He was the first director of field operations at the Center and had a long and distinguished career in survey methodology. In memory of Charlie and his work, the Institute for Social Research (the larger institute of which the SRC is part) established the Charles F. Cannell Fund in Survey Methodology. All royalties that result from the sales of this text will be contributed to this fund. The endowment from the fund is designated for support of young scholars developing their research careers in survey methodology. We can think of no better use.
We designed the text to be used in a class where the participants had taken one or more courses in statistics. The key relevant skill is the reading of statistical notation, including summation signs, notation for expected values, and simple algebraic manipulation of summed quantities. Some chapters present quantitative analyses using regression and logistic regression models, and students unfamiliar with linear modeling need some help in understanding these results.
This book has 12 chapters, in the order in which they are presented in the semester-length course on survey methodology called “Fundamentals of Survey Methodology” at the JPSM. We envision that instructors will want to assign additional readings, often from one or more review articles referenced in the chapters.
The first two chapters (“Introduction to Survey Methodology” and “Inference And Error In Surveys”) are conceptual in nature. Chapter 1 presents six example surveys that are used throughout the book to illustrate various principles and practices. The instructor can supplement the text by displaying the Web pages of these surveys in the class and leading class discussions about the key design features and products of the surveys.
The second chapter presents the key components of the total survey error paradigm. Again, at this early stage in the class, we have found that providing students with examples of key error components by referencing the example surveys aids in student understanding. A defining characteristic of surveys as we see them is that they are designed to produce statistical descriptions of populations. Although there are computer programs that will calculate statistics, we think it is critical that a survey methodologist understand the calculations that underlie those statistics. Hence, the book routinely presents statistical notation along with a conceptual discussion of what is being calculated.
The treatment of Chapter 2 would be a good time to devote a class to statistical notation, which, once it is learned, will help the students be more comfortable throughout the rest of the book
Starting with Chapter 3 (“Target Populations, Sampling Frames, and Coverage Error”), each chapter deals with a different component of total survey error and the methodological research discoveries that guide best practices. The focus of these chapters is deliberately the research on which best practices in survey research are based. We have often found that students beginning the study of survey methodology have the perspective that their opinions on a specific design feature are diagnostic of the best practices. The material that is presented in Chapters 3–11 attempts to show that there are scientific studies of survey methods that inform best practice; opinions are of little value unless they are research-based. Some of these studies do not have intuitively obvious findings. Hence, a student of the field must review the past methodological literature and at times do novel research to determine good design. There are two devices in the text that can help convey this perspective of the field. One is the set of embedded references to research in the discussions. The other is the presentation of illustrative boxes that give short descriptions of classic research in the domain covered in the chapter. These are summaries that describe the design, findings, limitations, and impact of the research. The full articles on this research can be used as supplementary readings, which could be discussed in class. There are also suggested supplementary readings at the end of each chapter.
Chapter 4 (“Sample Design and Sampling Error”) uses more statistical notation than most of the other chapters. When many participants in the course need remedial instruction in reading and understanding statistical notation, we have referred them to the small monograph by Kalton, An Introduction to Survey Sampling (Sage, 1983). In some editions of the course, we have spent three weeks on the coverage and sampling chapters.
Each of Chapters 5–10 is normally covered in one week of the course. We have found it useful to emphasize the parallels between equations expressing coverage error and nonresponse error. We have also emphasized how the basic principles of intraclass correlations apply both to sample clustering effects and interviewer variance.
Chapter 11 (“Principles and Practices Related to Scientific Integrity”) is included not just as sensitivity training but because it includes both conceptual frameworks underlying ethical treatment of human subjects and also recent theory and practice regarding disclosure analysis of survey data. Again, we describe how research, as well as judgement, can affect decisions related to ethical issues.
We wrote Chapter 12 (“FAQs About Survey Methodology”) in a very different style. It is a tradition in the course, in a review session prior to the final examination, to have an open question section. At this time, we found students asking the kind of questions that come from attempts to integrate their learning of specific lessons with their larger worldview. Hence, we constructed a “frequently asked questions” format including those global questions and offering our answers to them.
The manuscript was greatly improved by the editorial wisdom of Sarah Dipko and Sonja Ziniel. Adam Kelley assisted in computer-based processing of figures and tables. Lisa Van Horn at Wiley is a production editor with a wonderful sense of when intervention is needed and when it isn’t. We thank them all.
It was great fun writing this book, assembling our views on key research areas, and debating how to convey the excitement of survey methodology as an area of knowledge. We hope you have as much fun as we did.
Ann Arbor, Michigan | ROBERT M. GROVES | |
Boston, Massachusetts | FLOYD J. FOWLER, JR. | |
Ann Arbor, Michigan | MICK P. COUPER | |
Ann Arbor, Michigan | JAMES M. LEPKOWSKI | |
Ann Arbor, Michigan | ELEANOR SINGER | |
College Park, Maryland | ROGER TOURANGEAU | |
March 2004 |
PREFACE TO THE SECOND EDITION
We have been pleased by the acceptance of the first edition of Survey Methodology. It has now been used by instructors around the world and has been translated into several other languages. Some of these instructors and their students have graciously pointed out weaknesses and errors in some sections of the text. Some of them gave us great ideas to improve the text.
In addition, as survey methodologists actively conducting research in the field, we became increasingly aware that some of the lessons in the text were becoming out of date. This was most true of sections of the book that concern the role of survey nonresponse in the quality of survey estimates and the rapidly evolving new modes of data collection.
For those reasons, we assembled the group of coauthors and agreed to update parts of chapters that could most profit from changes. As the reader will see, there is increased discussion of sampling frame issues for mobile telephone and web surveys in Chapter 3. There is an integration of some of the example surveys into the presentation of sample designs in Chapter 4, along with a new section on selection of persons within households. The changes in Chapter 5 update the findings on mobile phone and web surveys. Chapter 6, on survey nonresponse, is radically changed, reflecting new insights into how nonresponse rates and nonresponse errors relate to one another. Chapter 8, on evaluating survey questions, highlights new research findings on effective questionnaire development techniques. Chapter 11, on ethical issues in survey research, is reorganized to emphasize the growing research results on privacy, informed consent, and confidentiality issues. The remaining chapters provide the reader with more recent methodological research findings, especially when they expand our understanding of survey errors. The chapters have about 50% more exercises, following feedback from instructors that such additions would benefit their use of the text.
Two assistants labored over this edition’s manuscript: Michael Guterbock and Kelly Smid. Some Ph.D. students at Michigan read draft chapters (Ashley Bowers, Matthew Jans, Courtney Kennedy, Joe Sakshaug, and Brady West). When we signed the contract with Wiley, we demanded that Lisa Van Hom continue as our production editor. All of the above went beyond the call to make this edition a success. We thank them.
As with the last edition, we want to use the royalties of the text to help persons newly entering the field of survey methodology. They will be given to the Rensis Likert Fund for Research on Survey Methodology, which directly benefits graduate students in survey methodology.
Ann Arbor, Michigan | ROBERT M. GROVES | |
Boston, Massachusetts | FLOYD J. FOWLER, JR. | |
Ann Arbor, Michigan | MICK P. COUPER | |
Ann Arbor, Michigan | JAMES M. LEPKOWSKI | |
Ann Arbor, Michigan | ELEANOR SINGER | |
College Park, Maryland | ROGER TOURANGEAU | |
March, 2009 |
ACKNOWLEDGMENTS
Reprinted tables and figures in Survey Methodology and their copyright holders are listed below. The authors appreciate permission to adapt or reprint them.
Figure 1.5c, from Mokdad, Ford, Bowman, Dietz, Vinicor, Bales, and Marks (2003) with permission of the American Medical Association. Copyright © 2003.
Table 5.1 from Groves (1989) with permission of John Wiley and Sons. Copyright © 1989.
Figure 6.5 from Groves (2006) and text surrounding from Groves and Peytcheva (2008) with permission of the American Association for Public Opinion Research. Copyright © 2006, 2008.
Table in box on page 185, from Merkle and Edelman in Groves, Dillman, Eltinge, and Little (2002) with permission of John Wiley and Sons. Copyright © 2002.
Figure 7.2, from Tourangeau, Rips, and Rasinski (2000) reprinted with permission of Cambridge University Press. Copyright © 2000.
Box on page 236, from Schwarz, Hippler, Deutsch, and Strack (1985) reprinted with permission of the American Association for Public Opinion Research. Copyright © 1985.
Figure 7.3, from Jenkins and Dillman in Lyberg, Biemer, Collins, de Leeuw, Dippo, Schwarz, and Trewin (1997) reprinted with permission of John Wiley and Sons. Copyright © 1997.
Box on p. 268, from Oksenberg, Cannell, and Kalton (1991) reprinted with permission of Statistics Sweden. Copyright © 1991.
Box on p. 293, from Schuman and Converse (1971) reprinted with permission of the American Association for Public Opinion Research. Copyright © 1971.
Box on p. 299, from Kish (1962) reprinted with permission of the American Statistical Association. Copyright © 1962.
Table 9.2, from Fowler and Mangione (1990) reprinted with permission of Sage Publications. Copyright © 1990.
Table 10.5, from Campanelli, Thomson, Moon, and Staples in Lyberg, Biemer, Collins, de Leeuw, Dippo, Schwarz, and Trewin (1997) reprinted with permission of John Wiley and Sons. Copyright © 1997.