0.010
                                                                          0.087
                                                                          0.284

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.003
                                                                          0.023
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.092
                                                                          0.251
                                                                          0.716
                                                                          0.929

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.015
                                                                          0.054
                                                                          0.342
                                                                          0.677

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.011
                                                                          0.109
                                                                          0.325

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.002
                                                                          0.005
                                                                          0.019
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.134
                                                                          0.345
                                                                          0.803
                                                                          0.957

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.019
                                                                          0.065
                                                                          0.421
                                                                          0.726

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.008
                                                                          0.140
                                                                          0.355

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.005
                                                                          0.020
                                                                             11
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.159
                                                                          0.411
                                                                          0.844
                                                                          0.977

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.021
                                                                          0.099
                                                                          0.523
                                                                          0.786

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.014
                                                                          0.181
                                                                          0.375

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.007
                                                                          0.017
                                                                             12
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.198
                                                                          0.467
                                                                          0.889
                                                                          0.985

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.020
                                                                          0.120
                                                                          0.603
                                                                          0.820

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.017
                                                                          0.220
                                                                          0.384

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.013
                                                                             13
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.169
                                                                          0.458
                                                                          0.915
                                                                          0.992

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.019
                                                                          0.114
                                                                          0.574
                                                                          0.829

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.006
                                                                          0.024
                                                                          0.233
                                                                          0.387

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.011

      Table A1.6: Study Design 5

        Binomial-Normal: PC=0.75, NPDay=100, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                                     Power

                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.297
                                                                          0.538
                                                                          0.851
                                                                          0.953

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.062
                                                                          0.180
                                                                          0.561
                                                                          0.780

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.010
                                                                          0.050
                                                                          0.239
                                                                          0.413

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.012
                                                                          0.019
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.408
                                                                          0.679
                                                                          0.909
                                                                          0.972

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.085
                                                                          0.259
                                                                          0.682
                                                                          0.825

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.011
                                                                          0.059
                                                                          0.309
                                                                          0.426

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.007
                                                                          0.013
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.513
                                                                          0.780
                                                                          0.939
                                                                          0.986

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.115
                                                                          0.363
                                                                          0.790
                                                                          0.877

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.015
                                                                          0.083
                                                                          0.360
                                                                          0.432

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.008
                                                                          0.010
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.655
                                                                          0.846
                                                                          0.961
                                                                          0.993

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.179
                                                                          0.491
                                                                          0.834
                                                                          0.895

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.024
                                                                          0.141
                                                                          0.394
                                                                          0.434

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.006
                                                                          0.006
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.749
                                                                          0.888
                                                                          0.987
                                                                          0.999

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.275
                                                                          0.603
                                                                          0.868
                                                                          0.918

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.045
                                                                          0.198
                                                                          0.410
                                                                          0.437

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.002
                                                                          0.006
                                                                          0.006
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.803
                                                                          0.912
                                                                          0.993
                                                                          0.999

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.377
                                                                          0.720
                                                                          0.907
                                                                          0.945

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.066
                                                                          0.262
                                                                          0.425
                                                                          0.447

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.003
                                                                          0.003

Appendix II: 	Mathematical derivation of the estimated efficacy using a binomial model with logit link function

Using the binomial model with the canonical logit link, the probability that an insect is blood-fed is given by the equations:
Logit Pbloodfed=logPbloodfedPnot bloodfed=μ+groupg+dayi+group*daygi
P is Probability. In this model, μ is the intercept, groupg is the effect for group g (either the treatment or the control group), dayi is the effect for day  i (1, 2, 3, or 4), and group*daygi is the group*day interaction term. The logarithms are natural logarithms. The control group is group 0 and the treatment group is group 1.

For day i, define the logits for the control group and treatment group by
      α=μ+group0+dayi+group*day0i,
      β=μ+group1+dayi+group*day1i.

For the SAS coding we also define the total
      τ=α+β
so that the covariance between the estimates of α and β can easily be calculated.
It follows that on day i,

      Pbloodfed for a control animal=11+e-α ,
      Pbloodfed for a treated animal=11+e-β ,
      Efficacy=1-Pbloodfed for a treated animalPbloodfed for a control animal=e-β-e-α1+e-β=f(α,β)

Let α,β, and τ be the estimated values of α,β, and τ, and let their standard errors be seα,seβ, and se τ . The estimated efficacy is f(α,β). To approximate the standard error of the efficacy we can apply the delta method, a first order Taylor series expansion using the partial derivatives of f (see Klein, 1953 or Oehlert, 1992, for example):

      Estimated efficacy= fα,β≈fα,β+(α-α)dfdα(α,β) +(β-β)dfdβ(α,β).
This gives:
      Estimated efficacy ≈fα,β+(α-α)e-α1+e-β +(β-β)-e-β1+e-α1+e-β2

Therefore, the variance of the estimated efficacy is approximately given by:

Variance Estimated efficacy≈Varαe-α1+e-β2 +Varβ-e-β1+e-α1+e-β22+2 Cov(α,β)e-α1+e-β-e-β1+e-α1+e-β2
Taking the square root and replacing the unknown true values of the parameters by their estimates gives

Standard Error Estimated efficacy≈Varαe-α1+e-β2 +Varβ-e-β1+e-α1+e-β22+2 Cov(α,β)e-α1+e-β-e-β1+e-α1+e-β2

In the last two equations, Var denotes the variance and Cov denotes the covariance of the estimated parameters. These can be estimated as follows:

      Varα=se(α)2
      Varβ=se(β)2
      Varτ=se(τ)2
      Varα+β=Varτ=Varα+Varβ+2 Cov(α,β)
      2 Covα,β=Varτ-Varα-Varβ

If the model with day effects and a group*day interaction fails to converge, then for each day, separate models are fitted of the form:
      Logit Pbloodfed=logPbloodfedPnot bloodfed=μ+groupg

For each day the logits for the control group and treatment group are defined by
      α=μ+group0,
      β=μ+group1

Appendix III: 	True blood-fed proportion vs. proportions of animals or datasets meet the required blood-fed proportion

The true proportion of blood-fed pests (or the true proportion of retained pests; for simplicity, only "blood-fed" terminology is used in this section) is one of the important factors that impact the power of the study design.  In reality, the proportion of blood-fed pests is determined by the natural behaviors of each specific pest species, and we don't have sufficient available data to reliably estimate these values yet.  One of the criteria for an animal to be selected for a study is that the observed blood-fed proportion of the animal from the screening test is greater than the required lowest observed proportion of blood-fed pests value to ensure the attractiveness of the animal to the arthropods.  It also requires that the observed blood-fed proportion of each control animal is greater than the required value during the study period.  This simulation (percent animals or datasets have at least a certain percent insect non-blood-fed.sas) is simply to determine the lowest blood-fed proportion such that a low probability for an animal to have an observed blood-fed proportion below a given required blood-fed proportion and a low probability for a dataset (single measurement/animal) to have at least 1 animal with observed blood-fed proportion below the required blood-fed proportion.  We determine the lowest true blood-fed for simulations because the power of the study will increase if the true blood-fed proportion is greater. It is worth mentioning that the lowest true blood-fed proportions obtained from these simulations are used in the simulations of determining power and sample size and used for the purpose of helping experts to determine whether the required blood-fed proportions are reasonable or not.

Two Tables A3.1 and A3.2 below present the results of simulations.  From these simulations, we conclude that the lowest blood-fed proportion are assumed to be:

    When the required blood-fed proportion for an animal is 0.25 and the number of arthropods/animal = 50, the lowest true value for blood-fed-proportion that can be used is 0.45

    When the required blood-fed proportion for an animal is 0.40 and the number of arthropods/animal = 100, the lowest true value for blood-fed- proportion that can be used is 0.60

    When the required blood-fed proportion for an animal is 0.60 and the number of arthropods/animal = 50, the lowest true value for blood-fed- proportion that can be used is 0.80

    When the required blood-fed proportion for an animal is 0.60 and the number of arthropods/animal = 25, the lowest true value for blood-fed- proportion that can be used is 0.85

    When the required blood-fed proportion for an animal is 0.60 and the number of arthropods/animal = 100, the lowest true value for blood-fed- proportion that can be used is 0.75
Note that if the true blood-fed proportion is greater than the assumed value in the simulations, the true power of the study design is greater than the power observed from the simulations.

Table A3.1: Proportion of animals with observed blood-fed proportion < require blood-fed proportion

                                          True             blood-fed proportion
                                         Required          blood-fed proportion
                                                   Number arthropods per animal
                                                                 Total animals
Proportion of animals with observed blood-fed proportion < require blood-fed proportion
                                                                           0.75
                                                                            0.6
                                                                             50
                                                                         460000
                                                                        0.01065
                                                                            0.8
                                                                            0.6
                                                                             50
                                                                         460000
                                                                        0.00073
                                                                            0.8
                                                                            0.6
                                                                             25
                                                                         550000
                                                                        0.01133
                                                                           0.85
                                                                            0.6
                                                                             25
                                                                         550000
                                                                        0.00129
                                                                            0.7
                                                                            0.6
                                                                            100
                                                                         460000
                                                                        0.01759
                                                                           0.75
                                                                            0.6
                                                                            100
                                                                         460000
                                                                        0.00060
                                                                           0.55
                                                                            0.4
                                                                            100
                                                                         460000
                                                                        0.00152
                                                                            0.6
                                                                            0.4
                                                                            100
                                                                         460000
                                                                        0.00005
                                                                            0.4
                                                                           0.25
                                                                             50
                                                                         460000
                                                                        0.01486
                                                                           0.45
                                                                           0.25
                                                                             50
                                                                         460000
                                                                        0.00209

Table A3.2: Proportion of datasets having at least one animal with observed blood-fed proportion < require blood-fed proportion

                                          True             blood-fed proportion
                                         Required          blood-fed proportion
                                                   Number arthropods per animal
                                                       Number animals per group
                                                             Number of datasets
Proportion of datasets having at least one animal with observed blood-fed proportion < require blood-fed proportion
                                                                           0.75
                                                                            0.6
                                                                             50
                                                                              8
                                                                           5000
                                                                        0.08320

                                                                              9
                                                                           5000
                                                                        0.08700

                                                                             10
                                                                           5000
                                                                        0.09940

                                                                             11
                                                                           5000
                                                                        0.11240

                                                                             12
                                                                           5000
                                                                        0.12300

                                                                             13
                                                                           5000
                                                                        0.12660

                                                                             14
                                                                           5000
                                                                        0.13780

                                                                             15
                                                                           5000
                                                                        0.15900
                                                                            0.8
                                                                            0.6
                                                                             50
                                                                              8
                                                                           5000
                                                                        0.00600

                                                                              9
                                                                           5000
                                                                        0.00560

                                                                             10
                                                                           5000
                                                                        0.00560

                                                                             11
                                                                           5000
                                                                        0.00820

                                                                             12
                                                                           5000
                                                                        0.00860

                                                                             13
                                                                           5000
                                                                        0.00960

                                                                             14
                                                                           5000
                                                                        0.01000

                                                                             15
                                                                           5000
                                                                        0.01280
                                                                           0.80
                                                                            0.6
                                                                             25
                                                                              5
                                                                           5000
                                                                        0.05780

                                                                              6
                                                                           5000
                                                                        0.06560

                                                                              7
                                                                           5000
                                                                        0.07540

                                                                              8
                                                                           5000
                                                                        0.08240

                                                                              9
                                                                           5000
                                                                        0.10040

                                                                             10
                                                                           5000
                                                                        0.11460

                                                                             11
                                                                           5000
                                                                        0.11160

                                                                             12
                                                                           5000
                                                                        0.12900

                                                                             13
                                                                           5000
                                                                        0.13980

                                                                             14
                                                                           5000
                                                                        0.14940

                                                                             15
                                                                           5000
                                                                        0.15320
                                                                           0.85
                                                                           0.60
                                                                             25
                                                                              5
                                                                           5000
                                                                        0.00760

                                                                              6
                                                                           5000
                                                                        0.00640

                                                                              7
                                                                           5000
                                                                        0.00880

                                                                              8
                                                                           5000
                                                                        0.00900

                                                                              9
                                                                           5000
                                                                        0.01040

                                                                             10
                                                                           5000
                                                                        0.01220

                                                                             11
                                                                           5000
                                                                        0.01500

                                                                             12
                                                                           5000
                                                                        0.01500

                                                                             13
                                                                           5000
                                                                        0.01700

                                                                             14
                                                                           5000
                                                                        0.02280

                                                                             15
                                                                           5000
                                                                        0.01660
                                                                            0.7
                                                                            0.6
                                                                            100
                                                                              8
                                                                           5000
                                                                        0.13320

                                                                              9
                                                                           5000
                                                                        0.14680

                                                                             10
                                                                           5000
                                                                        0.15900

                                                                             11
                                                                           5000
                                                                        0.18220

                                                                             12
                                                                           5000
                                                                        0.19080

                                                                             13
                                                                           5000
                                                                        0.20140

                                                                             14
                                                                           5000
                                                                        0.21840

                                                                             15
                                                                           5000
                                                                        0.23560
                                                                           0.75
                                                                            0.6
                                                                            100
                                                                              8
                                                                           5000
                                                                        0.00540

                                                                              9
                                                                           5000
                                                                        0.00420

                                                                             10
                                                                           5000
                                                                        0.00600

                                                                             11
                                                                           5000
                                                                        0.00660

                                                                             12
                                                                           5000
                                                                        0.00680

                                                                             13
                                                                           5000
                                                                        0.00780

                                                                             14
                                                                           5000
                                                                        0.00860

                                                                             15
                                                                           5000
                                                                        0.00980
                                                                           0.55
                                                                            0.4
                                                                            100
                                                                              8
                                                                           5000
                                                                        0.01260

                                                                              9
                                                                           5000
                                                                        0.01260

                                                                             10
                                                                           5000
                                                                        0.01340

                                                                             11
                                                                           5000
                                                                        0.01780

                                                                             12
                                                                           5000
                                                                        0.01680

                                                                             13
                                                                           5000
                                                                        0.01880

                                                                             14
                                                                           5000
                                                                        0.02160

                                                                             15
                                                                           5000
                                                                        0.02540
                                                                            0.6
                                                                            0.4
                                                                            100
                                                                              8
                                                                           5000
                                                                        0.00000

                                                                              9
                                                                           5000
                                                                        0.00020

                                                                             10
                                                                           5000
                                                                        0.00080

                                                                             11
                                                                           5000
                                                                        0.00040

                                                                             12
                                                                           5000
                                                                        0.00060

                                                                             13
                                                                           5000
                                                                        0.00080

                                                                             14
                                                                           5000
                                                                        0.00060

                                                                             15
                                                                           5000
                                                                        0.00120
                                                                            0.4
                                                                           0.25
                                                                             50
                                                                              8
                                                                           5000
                                                                        0.11520

                                                                              9
                                                                           5000
                                                                        0.12620

                                                                             10
                                                                           5000
                                                                        0.14180

                                                                             11
                                                                           5000
                                                                        0.14640

                                                                             12
                                                                           5000
                                                                        0.16220

                                                                             13
                                                                           5000
                                                                        0.17880

                                                                             14
                                                                           5000
                                                                        0.18600

                                                                             15
                                                                           5000
                                                                        0.19780
                                                                           0.45
                                                                           0.25
                                                                             50
                                                                              8
                                                                           5000
                                                                        0.01580

                                                                              9
                                                                           5000
                                                                        0.02000

                                                                             10
                                                                           5000
                                                                        0.02060

                                                                             11
                                                                           5000
                                                                        0.02380

                                                                             12
                                                                           5000
                                                                        0.02280

                                                                             13
                                                                           5000
                                                                        0.02720

                                                                             14
                                                                           5000
                                                                        0.02840

                                                                             15
                                                                           5000
                                                                        0.03140

Appendix IV: Values of SD_sub and SD_day used in the simulations

In the simulations, the random animal effects and the additional variation parameter were assumed to be normally distributed. Because of unavailable data from actual pet product efficacy studies to assess the values for parameters SD_sub of the variation between subjects or SD_day of the exhibiting of over dispersion of binomial data in pet product efficacy studies, SD_sub = 0.33 and SD_day = 0.166 were used in the simulations based on theoretical effects of these values on the variation of distributions of blood-fed proportion Ps.  The below table presents the 95% coverage of the distributions of the blood-fed pests between animals or between days when SD_sub = 0.33 and SD_day = 0.166.  For example, if the average proportion of proportions of blood-fed pests of the animals is 0.45, the 95% coverage of the distribution of true proportions of blood-fed pests of the animals is from 0.297 to 0.613.  If the proportion of blood-fed pests of all animals on a day (without exhibiting of over dispersion in the data) is 0.45, due to the additional effect associated with a SD_day = 0.166 (due to exhibiting of over dispersion in the data), 95% confidence limits of the proportions of blood-fed pests of the animals on the day would be from 0.370 to 0.533.

                                 Distributions
                                 population P
                                     logit
                                      SD
                                  logit - 2SD
                                  logit + 2SD
                                 95% CL of Ps
                                    Animals
                                     0.45
                                    -0.2007
                                     0.33
                                    -0.8607
                                    0.4593
                                     0.297
                                     0.613

                                      0.6
                                    0.4055
                                     0.33
                                    -0.2545
                                    1.0655
                                     0.437
                                     0.744

                                     0.75
                                    1.0986
                                     0.33
                                    0.4386
                                    1.7586
                                     0.608
                                     0.853

                                      0.8
                                    1.3863
                                     0.33
                                    0.7263
                                    2.0463
                                     0.674
                                     0.886
                                     Days
                                     0.45
                                    -0.2007
                                     0.166
                                    -0.5327
                                    0.1313
                                     0.370
                                     0.533

                                      0.6
                                    0.4055
                                     0.166
                                    0.0735
                                    0.7375
                                     0.518
                                     0.676

                                     0.75
                                    1.0986
                                     0.166
                                    0.7666
                                    1.4306
                                     0.683
                                     0.807

                                      0.8
                                    1.3863
                                     0.166
                                    1.0543
                                    1.7183
                                     0.742
                                     0.848

Appendix V: 	ICF's report of Evaluations of the applications of additional statistical methods for pet product studies

 MEMORANDUM

To:
David Miller, EPA
From:
Jonathan Cohen, ICF
Cc:
Kathryn Korthauer, EPA; Lori Brunsman, EPA; James Nguyen, EPA; Philip Villanueva, EPA; Matthew Crowley, EPA; Dave Burch, ICF
Date:
September 23, 2020
Re:
Estimation of Sample Sizes for Pet Product Studies

 Summary
In 2019, the Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) Scientific Advisory Panel (SAP) completed their peer review of EPA's proposed guideline for efficacy testing of topically applied pesticides used against certain ectoparasitic pests on pets. The review was completed at the June 11-14, 2019 FIFRA SAP meeting and the meeting minutes and final report were submitted to the docket. That review included various comments about the sample size determination which had been presented in the EPA document "Sample Size for Pet Product Studies." This memorandum presents the results of additional statistical analyses conducted by ICF for EPA in response to some of the FIFRA SAP comments.
The initial set of analyses repeated the original analysis in the EPA document "Sample Size for Pet Product Studies" for all 2000 simulations of one of the cases using data with normally distributed random effects. This provided the simulated data for all the analyses described in this memorandum as well as the power calculations for the probability of achieving a sufficiently precise efficacy estimate using the default method.
The next three sets of analyses evaluated the performance and power of the statistical models using the alternative Laplace and Quad estimation methods. Since the Laplace and Quad methods do not allow models with over- or under-dispersion, we first refitted the original model without the dispersion parameter. This approach reduced the percentage of cases that failed to converge from 10.01% in the original analysis to 1.59%. Replacing the RSPL estimation method by the Laplace and Quad methods further reduced the percentage of cases that failed to converge to 0.04% for the Laplace method and 0.43% for the Quad method.  These three approaches all gave very similar estimates of the power to the original set of analyses.
As another alternative approach we evaluated the Rao-Scott approach recommended by the FIFRA SAP. The Rao-Scott approach does power calculations for clustered binary data using a survey-adjusted variant of the standard chi-square test for homogeneity. Unfortunately this method does not provide comparable power estimates to the original statistical approach because the power for the Rao-Scott approach is defined as the probability that the blood-fed rates are statistically significantly different between the control and treatment groups and the statistical model assumes that the efficacy does not vary from day to day. In contrast, the power for the original approach is defined as the probability that the efficacy estimate is within a specified level of precision, that varies daily. The Rao-Scott approach produced power estimates that were uniformly equal to 1.
The final approach investigated in this memorandum was to use a bootstrap method to estimate the confidence intervals for the efficacy instead of the original delta method (a first order Taylor series approximation valid for sufficiently large samples). The bootstrap method resamples K sets of data from the fitted model multiple times and computes the efficacy estimates for each of the K samples. The K samples are called bootstrap samples. The 95% confidence interval for the efficacy is the interval from the 2.5[th] to the 97.5[th] percentile of the K efficacy estimates. Since the code takes several hours to run, the bootstrap confidence intervals were computed for the first 500 simulations only and used K = 250 bootstrap samples for each fitted model. As expected from the fact that the delta method is a large sample approximation, the power estimates from the original and bootstrap modeling differed most for small values of N, such as 5 and 6. The differences were also largest for days 1 and 2 and for the lowest desired precision (e.g. <= 0.025). For example, for N = 9 the power was within 0.1 of the original calculation in all cases except for day 2 with precision <= 0.025, day 2 with precision <= 0.030, and day 3 with precision <= 0.030. For N = 10 the power was within 0.1 of the original calculation in all cases except for day 2 with precision <= 0.025.
  Original Model
The original analysis in the EPA document "Sample Size for Pet Product Studies" can be summarized as follows. For more details see the EPA document. The analysis began by simulating 2000 sets of data for N animals each in the control and treatment groups where each animal was tested for 4 days. The simulated data for the treatment group assumed true adjusted survival rates of 0.95, 0.925, 0.9, and 0.85 for the four days and also had various required adjusted survival rates. The simulated data for the control group assumed certain true survival rates for each day. The numbers of insects that infest each animal on each day were drawn from a uniform distribution.  The simulated data used either normal or Weibull distributions for the random animal and day effects. The SAS generalized linear mixed model procedure GLIMMIX was used to estimate the statistical power of detecting a treatment effect.
The selected GLIMMIX model was fitted using the default RSPL estimation method. The selected GLIMMIX model assumes that the number of blood-fed insects has a binomial distribution where for each animal and day, the number of trials is the number of insects infesting an animal, with a probability of success p (i.e., the probability that an insect is blood-fed) of the form logit(p) = a + b*group + c*day + d*group*day + e*randomday. The logit function is logit(p) = log(p/1 - p)). "Group" is the fixed treatment effect (control or treated), "day" is the fixed day effect (days 1, 2 ,3 and 4), and "randomday" is the random day effect, which is a multivariate normally distributed four-component vector for each animal. The four components are for the four days, allowing for possible correlations between blood-fed rates for the same animal on different days. The model allows for overdispersion or underdispersion so that the variance of the number of successes is φnp(1-p), where n is the number of trials, p is the probability of success, and the dispersion parameter is  φ. For the standard binomial distribution,  φ equals 1. The binomial distribution is overdispersed if φ is greater than 1 and underdispersed if φ is lower than 1.; these overdispersed or underdispersed models do not fully define a valid probability distribution. The estimated probability of success is p(group, day) = invlogit(a + b*group + c*day + d*group*day), where invlogit(x) = 1/(1 + exp(-x)). The efficacy is then defined as 1  -  p(treated, day)/p(control, day). The power is defined as the probability that the lower bound of the 95% confidence interval for the efficacy is at least the required adjusted survival rate and the precision (estimate minus lower bound) is at most the required precision level.  If the GLIMMIX model fails to converge, then in the second step a simpler model with separate parameters for each day and no random animal effects was fitted.
The analyses described in this memorandum are for the example set of cases where N is between 5 and 10, the mean number of insects per animal per day is 100, the true survival rate of the control group is 0.6 for each day, and the required adjusted survival rates are 0.9 for each day. Daily required precision levels are 0.025, 0.030, 0.040, and 0.050, respectively. The same SAS programs and methods can easily be applied to other cases of interest. For these analyses, the simulated data were for the normal distribution and not the Weibull distribution, based on the FIFRA SAP's response to Charge Question 9 related to the fact that the SAS GLIMMIX procedure does not allow random effects that are not normally distributed.
The first set of analyses described in this memorandum were to simulate the data and refit the original set of models, using 2000 simulations. The same simulated data set was used for all the alternative analyses described in this memorandum. Note that 4,804 (10.01%) of the 48,000 fitted models failed to converge in the first step where all four days were modeled together. (There were actually 12,000 models fitted at the first step since there are 6 values of N, each model gives estimates for all 4 days, and each data set is simulated 2,000 times.) All the models fitted in the second step with separate models for each day converged. Also note that only 57.56% of the models that converged in the first step had overdispersion (where φ is greater than 1). The primary SAS program "Sample size for Pet Product Studies (normal)_Binomial.original.2000.sas" is attached. Also attached are the SAS programs used to count the numbers of models with overdispersion: "Sample size for Pet Product Studies (normal)_Binomial_original_2000.overdispersionparam.sas" and "overdisp.sas". The power calculations are shown in the following Table 1.

                Table 1. Power Calculations for Original Model
        Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166

                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.212
                                                                          0.442
                                                                          0.798
                                                                          0.938

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.035
                                                                          0.118
                                                                          0.444
                                                                          0.738

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.007
                                                                          0.031
                                                                          0.182
                                                                          0.391

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.009
                                                                          0.026
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.309
                                                                          0.564
                                                                          0.886
                                                                          0.956

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.050
                                                                          0.170
                                                                          0.581
                                                                          0.789

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.005
                                                                          0.035
                                                                          0.231
                                                                          0.412

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.023
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.394
                                                                          0.699
                                                                          0.935
                                                                          0.977

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.067
                                                                          0.238
                                                                          0.700
                                                                          0.858

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.008
                                                                          0.045
                                                                          0.313
                                                                          0.435

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.013
                                                                          0.017
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.526
                                                                          0.795
                                                                          0.958
                                                                          0.983

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.101
                                                                          0.332
                                                                          0.787
                                                                          0.874

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.013
                                                                          0.078
                                                                          0.381
                                                                          0.445

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.010
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.646
                                                                          0.885
                                                                          0.978
                                                                          0.996

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.168
                                                                          0.487
                                                                          0.860
                                                                          0.909

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.021
                                                                          0.126
                                                                          0.433
                                                                          0.460

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.008
                                                                          0.009
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.738
                                                                          0.928
                                                                          0.988
                                                                          0.998

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.214
                                                                          0.594
                                                                          0.899
                                                                          0.926

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.027
                                                                          0.164
                                                                          0.441
                                                                          0.458

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.005
                                                                          0.005

 Original Model Without Dispersion
The second set of analyses refit the original models without allowing for overdispersion or underdispersion. This step was necessary for comparison purposes since the alternative analyses for evaluating the Laplace and Quad integration methods do not allow the fitting of models with over- or under-dispersion because the marginal models need to be well-defined. The refitted models in this second set also used the RSPL estimation method. 764 (1.59%) of the 48,000 fitted models failed to converge in the first step. The SAS program "Sample size for Pet Product Studies (normal)_Binomial_nodispersion_2000.sas" is attached. The power calculations are shown in the following Table 2. The power estimates are quite similar to those shown in Table 1 for the original model.

       Table 2. Power Calculations for Original Model Without Dispersion
        Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166

                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.146
                                                                          0.401
                                                                          0.854
                                                                          0.983

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.012
                                                                          0.072
                                                                          0.449
                                                                          0.769

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.008
                                                                          0.164
                                                                          0.411

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.005
                                                                          0.023
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.254
                                                                          0.592
                                                                          0.956
                                                                          0.996

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.017
                                                                          0.126
                                                                          0.626
                                                                          0.845

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.000
                                                                          0.017
                                                                          0.246
                                                                          0.438

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.022
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.381
                                                                          0.750
                                                                          0.990
                                                                          0.998

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.035
                                                                          0.208
                                                                          0.772
                                                                          0.899

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.032
                                                                          0.327
                                                                          0.456

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.013
                                                                          0.020
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.559
                                                                          0.871
                                                                          0.997
                                                                          0.998

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.073
                                                                          0.326
                                                                          0.857
                                                                          0.902

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.006
                                                                          0.053
                                                                          0.403
                                                                          0.461

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.009
                                                                          0.012
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.690
                                                                          0.948
                                                                          1.000
                                                                          1.000

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.124
                                                                          0.479
                                                                          0.907
                                                                          0.922

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.010
                                                                          0.105
                                                                          0.443
                                                                          0.465

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.009
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.800
                                                                          0.981
                                                                          1.000
                                                                          1.000

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.190
                                                                          0.616
                                                                          0.931
                                                                          0.933

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.011
                                                                          0.148
                                                                          0.462
                                                                          0.466

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.005
                                                                          0.005

 Laplace and Quad Methods for Models Without Dispersion
The third and fourth sets of analyses evaluate the Laplace and Quad estimation methods. In those cases, models with over- or under-dispersion are not permitted in SAS because marginal distributions of the data need to be available, hence "R-side" covariance structures and overdispersion are not available for those two methods.
For the Laplace method, using the GLIMMIX model without over- or under-dispersion, 0.04% of the 48,000 fitted models failed to converge in the first step. For the Quad method, using the GLIMMIX model without over- or under-dispersion, 0.43% of the 48,000 fitted models failed to converge in the first step. Also note that a concern for the Laplace and Quad methods is that frequently the error message "At least one element of the gradient is greater than 1e-3" was reported even though the convergence criterion was satisfied. These results show that the Laplace and Quad estimation methods substantially reduce the problem of non-convergence but at the expense of being unable to model over- or under-dispersion. For the Laplace method, the SAS program "Sample size for Pet Product Studies (normal)_Binomial_laplace_2000.sas" is attached. The corresponding power calculations are shown in the following Table 3. For the Quad method, the SAS program "Sample size for Pet Product Studies (normal)_Binomial_quad_2000.sas" is attached. The corresponding power calculations are shown in the following Table 4. The power estimates are quite similar to those shown in Table 1 for the original model.

                      Table 3.  Power for Laplace Method.
        Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166

                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.197
                                                                          0.504
                                                                          0.906