First published 2017 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address:

ISTE Ltd
27-37 St George’s Road
London SW19 4EU
UK
www.iste.co.uk

John Wiley & Sons, Inc.
111 River Street
Hoboken, NJ 07030
USA
www.wiley.com

© ISTE Ltd 2017
The rights of Mikhail S. Nikulin and Ekaterina V. Chimitova to be identified as the authors of this work have been asserted by them in accordance with the Copyright, Designs and Patents Act 1988.

Library of Congress Control Number: 2017934786

British Library Cataloguing-in-Publication Data
A CIP record for this book is available from the British Library
ISBN 978-1-78630-000-3

Introduction

Chi-squared testing is one of the most commonly applied statistical techniques. The chi-squared test and its modifications are adopted for testing goodness-of-fit of various probabilistic models and provide reliable answers for researchers in engineering, quality control, finance, medicine, biology and other fields.

This book is devoted to the problems of construction and application of chi-squared goodness-of-fit tests for complete and censored data. We give examples of chi-squared tests for various distributions widely used in practice, and also consider chi-squared tests for parametric proportional hazards models and accelerated failure time models, which are widely used in reliability and survival analysis. Special attention is paid to the choice of grouping intervals and simulations.

Unfortunately, in many books on statistical data analysis, chi-squared tests are applied incorrectly or inefficiently. Classical chi-squared tests assume that unknown distribution parameters are estimated using grouped data, but in practice this assumption is often forgotten. In this book, we consider the modified chi-squared tests, which do not suffer from such drawback. Moreover, we describe the chi-squared goodness-of-fit tests for censored data with time-dependent covariates. Hence, the aim of this book is to demonstrate the application of the modified chi-squared tests in a wide variety of specific situations.

In the complete data case, a well-known modification of the classical chi-squared tests is the Nikulin-Rao-Robson statistic, which is based on the differences between two estimators of the probabilities to fall into grouping intervals: one estimator is based on the empirical distribution function, the other on the maximum likelihood estimators of unknown parameters of the tested model using initial non-grouped data [NIK 73b], [NIK 73a], [NIK 73c], [DZH 74], [KAP 58], [DZH 82], [DZH 83], [RAO 74], [VOI 93a], [VOI 96], [GRE 96], [DRO 88], [DRO 89], [VAN 98], [KEC 02], [VOI 13].

Li and Doss [LI 93] consider the modification of these results using wider classes of estimators from initial data. Goodness-of-fit tests for linear regression have been studied in [MUK 77], [PIE 79], [LOY 80], [KOU 84], [ZAC 71] and [VOI 07].

Habib and Thomas [HAB 86] and Hollander and Peña [HOL 92] considered natural modifications of the Nikulin-Rao-Robson statistic in the case of censored data without covariates. These tests are also based on the differences between two estimators of the probabilities to fall into grouping intervals. The first estimator is based on the Kaplan-Meier estimator of the cumulative distribution function, the second estimator is based on the maximum likelihood estimators of unknown parameters of the tested model using initial non-grouped censored data. Nikulin and Solev [NIK 99] considered the Pearson-type chi-squared goodness-of-fit test for doubly censored data.

The idea of comparing observed and expected numbers of failures in time intervals is due to Akritas [AKR 88] and was further developed by Hjort [HJO 90]. In the censored data case, Hjort [HJO 90], Royston & Lambert [ROY 11] and Pena [PEN 98a] considered goodness-of-fit for parametric Cox models, Gray and Pierce [GRA 85], Akritas and Torbeyns [AKR 97] and Zhang [ZHA 99] for linear regression models. Pena [PEN 98b] considered smooth goodness-of-fit tests for composite hypothesis in hazard-based models with possible covariates.

This book presents the recent innovations in the area as well as the important results previously only published in Russian. Different approaches to the choice of grouping intervals are considered. The chi-squared tests are compared with other goodness-of-fit tests (such as the Kolmogorov, Cramer-von Mises-Smirnov, Anderson-Darling and Zhang tests) in terms of power, when testing close competing hypotheses. The considerable contribution to the development of this direction belongs to Lemeshko (see, for example, [LEM 98], [LEM 00], [LEM 15b], [LEM 09c] and [LEM 10b]).

The book is intended for researchers interested in statistical testing on the basis of censored and complete data, as well as for university teachers and students. The book is also useful for representatives of industry and finance, in survival analysis and reliability, who deal with data analysis.

Table of Contents

Cover

Title

Introduction

1 Chi-squared Goodness-of-fit Tests for Complete Data

1.1. Classical Pearson’s chi-squared test
1.2. Joint distribution of and
1.3. Parameter estimation based on complete data. Lemma of Chernoff and Lehmann
1.4. Parameter estimation based on grouped data. Theorem of Fisher
1.5. Nikulin-Rao-Robson chi-squared test
1.6. Other modifications
1.7. The choice of grouping intervals

2 Chi-squared Test for Censored Data

2.1. Generalized Pearson-Fisher chi-squared test
2.2. Maximum likelihood estimators for censored data
2.3. Nikulin-Rao-Robson chi-squared test for censored data
2.4. The choice of grouping intervals
2.5. Chi-squared tests for specific families of distributions

3 Comparison of the Chi-squared Goodness-of-fit Test with Other Tests

3.1. Tests based on the difference between non-parametric and parametric estimators
3.2. Comparison of goodness-of-fit tests for complete data
3.3. Comparison of goodness-of-fit tests for censored data

4 Chi-squared Goodness-of-fit Tests for Regression Models

4.1. Data and the idea of chi-squared test construction
4.2. Asymptotic distribution of the random vector Z
4.3. Test statistic
4.4. Choice of random grouping intervals

Appendices

Appendix 1: Tables of Asymptotically D-optimal Grouping
Appendix 2: Data Sets

Bibliography

Index

End User License Agreement

List of Tables

1 Chi-squared Goodness-of-fit Tests for Complete Data

Table 1.1. Density functions and the references to the corresponding tables of AOG
Table 1.2. Optimal boundary points for k = 9
Table 1.3. Percentage points for the distribution in the case of testing normality (μ and σ are estimated) and using AOG
Table 1.4. Estimates of the power of Pearson’s χ² test with respect to hypotheses H₁, H₂, and H₃
Table 1.5. Estimates of the power of the Nikulin–Rao–Robson χ² test with respect to hypotheses H₁, H₂, and H₃

2 Chi-squared Test for Censored Data

Table 2.1. Distributions of censoring times in the case of the lifetime distribution W(2, 2)
Table 2.2. Power of the tests for the Weibull-gamma pair in the case of randomly censored data with Weibull distribution of censoring times
Table 2.3. Power of the tests for the Weibull-gamma pair in the case of randomly censored data with Type I beta distribution of censoring times
Table 2.4. Type I censored sample
Table 2.5. Intermediate calculation results
Table 2.6. Lifetimes of patients after surgical operation
Table 2.7. Intermediate calculation results
Table 2.8. Randomly censored sample of lifetimes
Table 2.9. Intermediate calculation results
Table 2.10. Randomly censored sample of lifetimes
Table 2.11. Intermediate calculation results
Table 2.12. Type I censored sample of failure times
Table 2.13. Intermediate calculation results

3 Comparison of the Chi-squared Goodness-of-fit Test with Other Tests

Table 3.1. Test power for the normal-generalized normal pair when testing the composite hypothesis
Table 3.2. Test power for the normal-Laplace pair when testing the composite hypothesis
Table 3.3. Test power for the normal-logistic pair when testing the composite hypothesis
Table 3.4. Test power for the lognormal-generalized Weibull pair when testing the simple hypothesis
Table 3.5. Test power for the lognormal-generalized Weibull pair when testing the composite hypothesis
Table 3.6. Test power for the exponential-Weibull pair when testing the simple hypothesis
Table 3.7. Test power for the exponential-Weibull pair when testing the composite hypothesis
Table 3.8. Test power for the first Weibull-generalized Weibull pair when testing the simple hypothesis
Table 3.9. Test power for the second Weibull-generalized Weibull pair when testing the simple hypothesis

Appendix 1: Tables of Asymptotically D-optimal Grouping

Table A1.1. D-optimal boundary points of grouping intervals for the exponential distribution and corresponding values of the relative asymptotic information A
Table A1.2. D-optimal probabilities P {X₁ ∈ I_j}, j = 1, …, k for the exponential and Rayleigh distributions
Table A1.3. D-optimal boundary points of grouping intervals for the Rayleigh distribution and corresponding values of the relative asymptotic information A
Table A1.4. D-optimal boundary points of grouping intervals for the Maxwell distribution and corresponding values of the relative asymptotic information A
Table A1.5. D-optimal probabilities P {X₁ ∈ I_j}, j = 1, …, k for the Maxwell distribution
Table A1.6. D-optimal boundary points of grouping intervals for the half-normal distribution and corresponding values of the relative asymptotic information A
Table A1.7. D-optimal probabilities P {X₁ ∈ I_j}, j = 1, …, k for the half-normal distribution
Table A1.8. D-optimal boundary points of grouping intervals for the Weibull and minimum extreme value distributions and corresponding values of the relative asymptotic information A
Table A1.9. D-optimal probabilities P{X₁ ∈ I_j},j=1,…,k for the Weibull and minimum extreme value distributions
Table A1.10. D-optimal boundary points of grouping intervals for the maximum extreme value distribution and corresponding values of the relative asymptotic information A
Table A1.11. D-optimal probabilities P{X1 ∈ Ij}, j = 1, …, k for the maximum extreme value distribution
Table A1.12. D-optimal boundary points of grouping intervals for the normal and lognormal distributions and corresponding values of the relative asymptotic information A
Table A1.13. D-optimal probabilities P{X1 ∈ Ij}, j = 1, …, k for the normal and lognormal distributions
Table A1.14. D-optimal boundary points of grouping intervals for the logistic and loglogistic distributions and corresponding values of the relative asymptotic information A
Table A1.15. D-optimal probabilities P{X1 ∈ Ij}, j = 1, …, k for the logistic and loglogistic distributions

Appendix 2: Data Sets

Table A2.1. The failure times of 76 electric insulating fluids, see [NEL 90]
Table A2.2. Survival data on 40 advanced lung cancer patients, see [LAW 03]
Table A2.3. The lifetimes of 586 lung cancer patients after operation, in months

List of Illustrations

1 Chi-squared Goodness-of-fit Tests for Complete Data

Figure 1.1. The distribution of the statistic
Figure 1.2. Probability density functions corresponding to the considered hypotheses H_i. For a color version of this figure, see www.iste.co.uk/nikulin/chisquared.zip

2 Chi-squared Test for Censored Data

Figure 2.1. Lifetime distribution. For a color version of this figure, see www.iste.co.uk/nikulin/chisquared.zip

3 Comparison of the Chi-squared Goodness-of-fit Test with Other Tests

Figure 3.1. The power of the χ² Pearson test when testing the simple hypothesis
Figure 3.2. The power of the χ² Pearson test when testing the composite hypothesis