UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                         WASHINGTON, D.C. 20460

                                                	OFFICE OF CHEMICAL SAFETY AND
                                                           POLLUTION PREVENTION

MEMORANDUM

DATE: 	January 6, 2021

SUBJECT:	Supplemental Materials for The Product Performance Test Guidelines OCSPP 810.3300: The Efficacy of Topically Applied Pet Products Against Certain Invertebrate Pests

PC Code: NA
   DP Barcode: NA
Decision No.:
   Registration No.: NA
Petition No.: NA
   Regulatory Action: NA
Risk Assessment Type: NA
   Case No.: NA
TXR No.: NA
   CAS No.: NA
MRID No.: NA
   40 CFR: NA
		 									Ver.Apr.08

FROM:	James Nguyen, Mathematical Statistician
      Chemistry and Exposure Branch
		Health Effect Division (7509P)

THROUGH:	David J. Miller, Chief
		Chemistry and Exposure Branch
		Health Effects Division (7509P)

TO: 		Autumn Metzger, Biologist
		Invertebrate &Vertebrate Branch 1
		Registration Division

Please find attached per request supplemental materials for the Product Performance Test Guidelines OCSPP 810.3300 which relate to the efficacy of topically applied pet products against certain invertebrate pests.
Briefly, the attached supplemental materials were produced following the June 11-14, 2019 FIFRA SAP meeting peer review of EPA's proposed guideline for efficacy testing of topically applied pesticides used against certain ectoparasitic pests on pets. The SAP review included various comments about the sample size determination which had been presented to the SAP in the EPA document "Sample Size for Pet Product Studies". The attached Supplemental Materials address the issues, concerns, and questions raised by the SAP during this June 2019 FIFRA SAP meeting as well as their requests for additional information/analyses.
Supplemental Materials for The Product Performance Test Guidelines OCSPP 810.3300: The Efficacy of Topically Applied Pet Products Against Certain Invertebrate Pests

                               Table of Contents

    Executive Summary......................................................................................3

    Simulations to determine sample sizes for pet product efficacy studies...........................4

    Conclusions and suggested sample sizes...............................................................9

    An example of data analysis of a pet product efficacy study......................................11

    Appendices...............................................................................................14

   Appendix I: 	Results of power vs. sample size simulations.......................................14

   Appendix II:	Mathematical derivation of the estimated efficacy using a binomial model
                  with logit link function....................................................................18

   Appendix III:	True blood-fed proportion vs. proportions of animals or datasets meet the required blood-fed proportion..........................................................20

   Appendix IV: 	Values of SD_sub and SD_day used in the simulations..............................24

   Appendix V: 	ICF's report of Evaluations of the applications of additional statistical
                  methods for pet product studies .........................................................25

   Appendix VI: 	SAS codes....................................................................................37

			SAS code of simulation to determine the lowest blood-fed proportions............37

            SAS code of power vs. sample size simulations.......................................39

            SAS code of an example data analysis...................................................46

    Executive Summary

This document is directly related to the guideline for Product Performance Test Guidelines OCSPP 810.3300: The Efficacy of Topically Applied Pet Products Against Certain Invertebrate Pests.  This supplemental document will (1) present more detail about how the sample sizes provided in the guideline were calculated and (2) provide an example (with SAS code) of data analysis for pet product efficacy studies using the statistical methods recommended in the guideline.

This document provides statistical simulations based on sections (j)  -  (l) of the draft guideline, which illustrate the potential power of several different experimental designs to detect efficacy of 90% or greater based on the behavior of different types of arthropods. Adequate power is typically considered 80% power to achieve an estimated effect at a given level of precision. For the studies in the guideline, EPA determined the sample size which results in 80% power to achieve an estimated efficacy of >=90% with a precision of <= 5%, given that the true efficacy of the product is 92.5%.  Specifically:

 For studies assessing efficacy against ticks (study design 1), 11 animal subjects (if using 50 ticks per animal) would be appropriate per treatment.
 For studies assessing efficacy against fleas, 7 animal subjects (with 100 fleas per animal) per treatment would be appropriate for tests on cats (study design 2), and 6 animal subjects (with 100 fleas per animal) per treatment would be appropriate for tests on dogs (study design 5).
 For studies assessing efficacy against mosquitoes, 6 animal subjects (with 100 mosquitoes per animal) per treatment would be appropriate if testing against one species (study design 5), and 8 animal subjects (with 50 mosquitoes of each species per animal) per treatment if testing against three mosquito species simultaneously (study design 3).
 For studies assessing efficacy against stable flies, horse flies, and deer flies, 8 animal subjects (with 50 pests per animal) per treatment (study design 3) or 12 animal subjects (with 25 pests per animal) per treatment (study design 4) would be appropriate.
 For studies assessing efficacy against sand flies, black flies, or biting midges, 6 animal subjects (with 100 pests per animal) per treatment would be appropriate (study design 5).

Note that the suggested sample sizes above are based on the simulations where the delta approach is used in which an asymptotic (only appropriate for large sample size) standard error was used to obtain the 95% confidence interval of the estimated efficacy.  While the bootstrapping technique is more appropriate to obtain the 95% CI of the estimated efficacy in the small sample size studies, it is not practical to use bootstrapping technique in simulations to evaluate sample size and power given the required computing power and the time required to run all the simulations.  Therefore, the bootstrapping technique was used only for one scenario with small numbers of datasets to compare the results of power and sample sizes between simulations using the delta approach and simulations using bootstrapping technique.  The results show that the power from simulations using bootstrapping technique was slightly higher than that from the simulation using delta approach; however, there was no evidence that the sample sizes would substantially different  (Appendix V).

To demonstrate how data from a pet product efficacy study are analyzed, this document also provides an example of data analysis of a synthetic dataset, using the statistical methods suggested in the guidelines.  SAS codes of this data analysis are also provided.

    Simulations to determine sample sizes for pet product efficacy studies

Background

Purpose

Historically, the Agency has recommended using a minimum of 6 animals per study group for efficacy testing of topically applied products on pets, however studies have been submitted with 2-12 animals per group. Previously no statistical simulation or calculation had been conducted to evaluate the statistical power associated with the sample size of these experimental designs in detecting a treatment-related effect. Therefore, these simulations have been conducted to determine the sample sizes and associated power of the study designs in the proposed guideline.

Study design

    2 groups (1 control and 1 treatment) with X animals per group
    Data are collected at 4 different days over the study duration
    Each day, Y arthropods (+-10%) per animal
Objective of study

    Evaluate the efficacy or effectiveness of the treatment where efficacy or effectiveness of the treatment is defined as Abbott's formula:
           (retained or blood-fed proportion of control group  -  retained or blood-fed proportion of treated group)/retained or blood-fed proportion of control group for repellency evaluation.
           (Survival proportion of control group  -  survival proportion of treated group)/survival proportion of control group for mortality evaluation.
Objective of Simulations

Determine the sample size and the power to obtain an observed efficacy >= 90% with a given (suitable) pre-stated level of precision in studies to evaluate repellency of pet products.

Assumptions

    The data from pet product efficacy studies are in the form of n events/N trials (e.g., n blood-fed pests/N pests used) and are reasonably assumed binomial distributions.

    In addition to the variation of the arthropods (described by the variation of binomial distribution), the difference between animals is also a factor influencing the variation in the data.  There may be similarities between measurements of the same animal compared to measurements of different animals.  This factor of variation in the data is called the "random animal effect".

    Also, there may be similarity between measurements from the different animals on the same testing day compared to measurements from different animals on different testing days.  The similarity between measurements on the same testing days is called the "random day effect".  Because the pet product studies are conducted in laboratorial environments, the random day effect is believed to be very minimal and is assumed to be zero in order to simplify the simulations.

    Another factor that may often occur in binomial distribution data is over- dispersion.  Over dispersion in binomial distribution data generally affects confidence intervals of the estimates.  Since the precision of the estimates is one of two criteria that significantly affects the power of a study and the regulatory decision-making  surrounding a product, it is important that the  sample size is adequate to  account for over-dispersion in the data.  Synthetic or artificial over-dispersion in the data was created by adding small additional variation (the parameter SD_day) to the variation of the binomial data of each animal on each day in the step of data generation.

    The random effects of animal and SD_day are assumed to follow normal distributions.

    The simulations are used to determine the sample size based on the efficacy of repellency (i.e., the simulated data are the counts of pests used and counts of blood-fed pests on the animals). However, for application in the guideline, the words "mortality" and "repellency" can be used interchangeably for each model when determining sample size.

Parameters in simulations

The following are parameters included in the simulations:

 NminSub = the smallest number of animals per group;
 NmaxSub = the largest number of animals per group;
 NPDay = number of arthropods per animal per day;
 RNdiff = allowable proportion of deviation in number arthropods/animal;
 CP = true blood-fed proportion of control group;
 TE = true efficacy of treatment;
 ReqTE = required observed efficacy of treatment;
 ReqPrecision = required maximum precision value;
 SubVar or SD_sub = degree of variation between animals;
 DayVar or SD_day = degree of additional variation introduced to create the exhibiting of over dispersion in the binomial data;
 Nsim = number of simulated datasets per scenario (set to be 2000); and
 seed = seed to regenerate the results of simulations.

As the power of a study design is defined as the proportion of datasets that have the estimated efficacy >= 0.90 and the precision of the estimated efficacy <= ReqPrecision, the power is impacted by the values of parameters:

    The required precision (ReqPrecision) of the estimates: a smaller value of required precision would result a smaller number of datasets that have the precision of the estimated efficacy less than the required precision. This means the study design would have lower power to achieve a lower uncertainty in estimated efficacy from the accepted studies.

    Number of animals per group (ranges from NminSub to NmaxSub): larger number of animals in each group would result in better precision for the estimated efficacy and a larger proportion of datasets that have the precision of the estimated efficacy smaller than the required precision (i.e., higher power).

    Number of pests per animal (NPDay): larger number of pests per animal would result in better precision for the estimated efficacy and a larger proportion of datasets that have the precision of the estimated efficacy smaller than the required precision (i.e., higher power).  However, practical and humane reasons generally determine how large the number of pests per animal could be.

    The proportion of blood-fed in the control population (CP): the variance of a binomial distribution becomes greater as the value of true proportion or rate approaches 0.5.  However, the proportions of blood-fed pests are determined by natural pest behaviors.  Based on the required level of the lowest observed proportion of blood-fed pests on an animal in order for the animal to be qualified to participate in the study, the proportion of blood-fed pests on the (qualified) animal population is estimated such that a very high proportion of the animals would be qualified to participate in the study and a low probability that a study would be rejected due to the control animals failing to meet the required lowest proportion of blood-fed pests.  (The true proportion of blood-fed pests are estimated using the SAS file "percent animals or datasets have at least a certain percent insect non-blood-fed.sas", Appendix III).

    The required observed efficacy of treatment (ReqTE): a smaller value of required observed efficacy would result a larger number of datasets that have the estimated efficacy greater than the required observed efficacy (i.e., higher power).

    True efficacy (TE) of the product: If the study can be replicated thousands of times, we would have a distribution of estimated efficacy values and expect half of the estimated efficacy values to be less than the true efficacy TE and half of the estimated efficacy values to be greater than the true efficacy TE.  When the true efficacy TE is exactly equal the required observed efficacy ReqTE, half of the estimated efficacy values will be >= the true efficacy value TE =ReqTE, and therefore the power of the study design will be <= 50%.  A greater difference between the true efficacy TE and the required observed ReqTE would result in a greater proportion of datasets with observed efficacy values >= the required observed ReqTE, and therefore a study with greater power.  This is why the selected true efficacy TE of a product has to be greater than the required observed efficacy ReqTE in order to have acceptable power for the studies.
Data generation

Step 1: calculate the proportion of blood-fed in the treatment group (TP)

From Abbott's formula:

    - 		TE = (CP - TP)/CP,

where TE is true efficacy and CP and TP are the proportions of blood-fed pests in the control group and treated group, respectively.

Then the proportion of blood-fed in the treated group is calculated as:

      TP =CP x (1  -  TE)

Step 2: generate number of pests used and number of blood-fed pests on each animal

Assume P is the true blood-fed proportion of a group on a specific day (P = CP for the control group and P=TP for the treated group).  The logit of the group is calculated as:

      Q = log(P/(1-P))

Then, the logit of the of animal s is generated as:

      Qs= Q + rands,

where rands is randomly generated from a normal distribution with mean = 0 and SD = 0.33.

Similarly, the logit of the animal s on a specific day i is generated as:

      Qsi = Qs + randi
            = Q + rands + randi,

where randi is randomly generated from a normal distribution with mean=0 and SD=0.166.

The blood-fed proportion of pests on the animal s on the specific day i is

      Psi = exp(Qsi)/(1+exp(Qsi)).

The number of arthropods used on animal s on day i is randomly generated from uniform distribution

      Nsi = U [ NPDayx(1  -  RNdiff), NPDayx (1  -  Rndiff) ],

where NPDay is the target number of pests used on each testing day, and RNdiff is the allowed factor for the number of pests used on each animal to be deviated from the target number of pests

The number of blood-fed arthropods on animal s on day i is randomly generated from binomial distribution of Nsi trials and success proportion = Psi.

Step 3: generate 2000 datasets

For each dataset, step 2 above is repeatedly used to generate the numbers of pests used and numbers blood-fed on each animal for each of four testing days for n animals of each control and treated groups.

The process above is repeated to create 2000 datasets.

Analysis of simulated datasets

Generalized linear mixed effects models (SAS PROC GLIMMIX), with logit link function for binomial distribution data is used to analyze each of simulated datasets.  In the models, the animal is experimental subject (RANDOM Day/subject = animal type = cs;).  Over dispersion in the data is accounted in the model (RANDOM _residual_;).  For those datasets that have a computational issue (non-convergence) in the generalized linear mixed effects model (GLMM), a generalized linear model is used to separately analyze the data by each Day.

The ratio of blood-fed proportions between treated and control groups and its 95% CI are estimated for each Day, using the delta approach (Appendix II).  The efficacy of the treatment group = 1  -  (blood-fed proportion of treated group/blood-fed proportion of control group).  The precision of the estimated efficacy = estimated efficacy  -  lower limit of 95% CI.

The power of a study design is the proportion of datasets that have observed treatment efficacy >= ReqTE = 0.90 and observed precision of the estimated efficacy <= ReqPrecision.

Results of power vs. sample size simulations

Results of simulations are presented in Appendix I.

    Conclusions and suggested sample sizes

Conclusions

Given the results of sample sizes, a required precision ReqPrecision = 0.05 is selected since a smaller value of 0.03 or 0.04 for the required precision parameter would require much larger sample sizes in the studies in order to obtain a sufficient power.

A true efficacy value of 0.925 is selected since it is the lowest value such that the studies would still have acceptable power with reasonable sample sizes (not too large).  When the true efficacy of a product is 0.85 (i.e., less than the required efficacy of 0.90), there is a very low probability that the product would be incorrectly accepted (i.e., low probability that the estimated efficacy from a study of this product >= 0.90 and the precision <=0.05). When the true efficacy of a product is 0.925 (slightly greater than the required efficacy of 0.90), there is a high probability that the product would be correctly accepted (i.e., there is at least 0.80 power that the study would result an estimated efficacy >= 0.90 and a precision <=0.05) for reasonable sample size studies.

Note that there was a high proportion of datasets for which the GLMM model did not converge, and the generalized linear model is subsequently used to analyze those datasets for each separate Day.  Simulations using a GLMM model (with link function for binomial data) without overdispersion are performed for one scenario (PC = 0.60 and NPDay = 100).  The results from the simulations of this scenario indicate GLMM model without overdispersion significantly alleviates the non-convergence issue (only 1.59% of the datasets that do not converge).  The power results are quite similar to those from GLMM model with the option of overdispersion; therefore, the simulations with GLMM without overdispersion has not been replicated for all scenarios or presented in this document.  (Appendix V)

Suggested sample sizes

Below is the summary table including simulation results of all the scenarios.  EPA determined that the sample size which results in at least 80% power to detect the true efficacy of >=90% with a precision of <= 5% using the simulations where random effects were randomly generated from normal distributions and the data were analyzed using GLMM for binomial distributions for the studies in sections (k)-(m) of the guideline. Specifically:

 For studies assessing efficacy against ticks (study design 1), 11 animal subjects (with 50 ticks per animal) would be appropriate per treatment.
 For studies assessing efficacy against fleas, 7 animal subjects (with 100 fleas per animal) per treatment would be appropriate for tests on cats (study design 2), and 6 animal subjects (with 100 fleas per animal) are appropriate for tests on dogs (study design 5).
 For studies assessing efficacy against mosquitoes, 6 animal subjects (with 100 mosquitoes per animal) per treatment would be appropriate if testing against one species (study design 5), and 8 animal subjects (with 50 mosquitoes of each mosquito species per animal) per treatment if testing against three mosquito species simultaneously (study design 3).
 For studies assessing efficacy against stable flies, horse flies, and deer flies, 8 animal subjects (with 50 pests per animal) per treatment (study design 3) or 12 animal subjects (with 25 pests per animal) per treatment (study design 4) would be appropriate.
 For studies assessing efficacy against sand flies, black flies, or biting midges, 6 animal subjects (with 100 pests per animal) would be appropriate (study design 5).

                       GLMM for Binomial with logit link

                                 Study design
     Required lowest observed blood-fed proportion on each control animal
                  True blood-fed proportion in control group
                         No. pests per animal per day
                                    True TE
                         Required Sample Size (power)
 Ticks on cats or dogs, (k)(2)
                                       1
                                     0.25
                                     0.45
                                     50+-5
                                     0.925
                                      11
                                    (0.806)
 Fleas on cats, (j)(2)

                                       2
                                     0.40
                                     0.60
                                    100+-10
                                     0.925
                                       7
                                    (0.858)
 Mosquitoes or flies on cats or dogs using 50 pests, (l)(2)
                                       3
                                     0.60
                                     0.80
                                     50+-5
                                     0.925
                                       8
                                    (0.822)
 Biting flies on animal subjects using 25 pests, (l)(2)
                                       4
                                     0.60
                                     0.80
                                     25+-2
                                     0.925
                                      12
                                    (0.815)

                                     0.85
                                     25+-2
                                     0.925
                                      12
                                    (0.820)
 Fleas on dogs, (j)(2)
 Mosquitoes or flies for cats and dogs using 100 pests, (l)(2)
                                       5
                                     0.60
                                     0.75
                                    100+-10
                                     0.925
                                       6
                                    (0.825)

    An example of data analysis of a pet product efficacy study

Data Example1

This dataset Example1 is a synthetic dataset which mimics the data of a pet product study.  There are 5 variables (columns) in the dataset:

 Treatment: values = 0 (for control group) and 1 (for treated group)
 Animal: 12 different animal IDs per group in this study
 Day: values = 1, 2, 3, and 4 in this dataset
 Ntotal: each animal's total number of insects used in the data analysis
 Nblood: number of blood-fed insects (this can be the number of retained insects or number of survival insects)

                                                                      Treatment
                                                                         ANIMAL
                                                                            Day
                                                                         Ntotal
                                                                         Nblood
                                                                              0
                                                                            0-1
                                                                              1
                                                                             24
                                                                             21
                                                                              0
                                                                            0-2
                                                                              1
                                                                             23
                                                                             19
                                                                              0
                                                                            0-3
                                                                              1
                                                                             27
                                                                             26
                                                                              0
                                                                            0-4
                                                                              1
                                                                             27
                                                                             22
                                                                              0
                                                                            0-5
                                                                              1
                                                                             26
                                                                             20
                                                                              0
                                                                            0-6
                                                                              1
                                                                             23
                                                                             15
                                                                              0
                                                                            0-7
                                                                              1
                                                                             24
                                                                             20
                                                                              0
                                                                            0-8
                                                                              1
                                                                             23
                                                                             22
                                                                              0
                                                                            0-9
                                                                              1
                                                                             23
                                                                             18
                                                                              0
                                                                           0-10
                                                                              1
                                                                             25
                                                                             22
                                                                              0
                                                                           0-11
                                                                              1
                                                                             26
                                                                             26
                                                                              0
                                                                           0-12
                                                                              1
                                                                             25
                                                                             13
                                                                              1
                                                                            1-1
                                                                              1
                                                                             27
                                                                              0
                                                                              1
                                                                            1-2
                                                                              1
                                                                             26
                                                                              1
                                                                              1
                                                                            1-3
                                                                              1
                                                                             25
                                                                              0
                                                                              1
                                                                            1-4
                                                                              1
                                                                             26
                                                                              0
                                                                              1
                                                                            1-5
                                                                              1
                                                                             27
                                                                              2
                                                                              1
                                                                            1-6
                                                                              1
                                                                             26
                                                                              1
                                                                              1
                                                                            1-7
                                                                              1
                                                                             25
                                                                              1
                                                                              1
                                                                            1-8
                                                                              1
                                                                             27
                                                                              1
                                                                              1
                                                                            1-9
                                                                              1
                                                                             27
                                                                              6
                                                                              1
                                                                           1-10
                                                                              1
                                                                             26
                                                                              2
                                                                              1
                                                                           1-11
                                                                              1
                                                                             27
                                                                              0
                                                                              1
                                                                           1-12
                                                                              1
                                                                             23
                                                                              2
                                                                              0
                                                                            0-1
                                                                              2
                                                                             24
                                                                             23
                                                                              0
                                                                            0-2
                                                                              2
                                                                             27
                                                                             24
                                                                              0
                                                                            0-3
                                                                              2
                                                                             23
                                                                             22
                                                                              0
                                                                            0-4
                                                                              2
                                                                             26
                                                                             25
                                                                              0
                                                                            0-5
                                                                              2
                                                                             25
                                                                             20
                                                                              0
                                                                            0-6
                                                                              2
                                                                             26
                                                                             21
                                                                              0
                                                                            0-7
                                                                              2
                                                                             25
                                                                             22
                                                                              0
                                                                            0-8
                                                                              2
                                                                             25
                                                                             22
                                                                              0
                                                                            0-9
                                                                              2
                                                                             23
                                                                             22
                                                                              0
                                                                           0-10
                                                                              2
                                                                             26
                                                                             20
                                                                              0
                                                                           0-11
                                                                              2
                                                                             26
                                                                             20
                                                                              0
                                                                           0-12
                                                                              2
                                                                             26
                                                                             21
                                                                              1
                                                                            1-1
                                                                              2
                                                                             27
                                                                              0
                                                                              1
                                                                            1-2
                                                                              2
                                                                             27
                                                                              1
                                                                              1
                                                                            1-3
                                                                              2
                                                                             24
                                                                              4
                                                                              1
                                                                            1-4
                                                                              2
                                                                             24
                                                                              0
                                                                              1
                                                                            1-5
                                                                              2
                                                                             23
                                                                              2
                                                                              1
                                                                            1-6
                                                                              2
                                                                             23
                                                                              0
                                                                              1
                                                                            1-7
                                                                              2
                                                                             24
                                                                              3
                                                                              1
                                                                            1-8
                                                                              2
                                                                             26
                                                                              1
                                                                              1
                                                                            1-9
                                                                              2
                                                                             26
                                                                              2
                                                                              1
                                                                           1-10
                                                                              2
                                                                             27
                                                                              2
                                                                              1
                                                                           1-11
                                                                              2
                                                                             27
                                                                              2
                                                                              1
                                                                           1-12
                                                                              2
                                                                             23
                                                                              2
                                                                              0
                                                                            0-1
                                                                              3
                                                                             24
                                                                             20
                                                                              0
                                                                            0-2
                                                                              3
                                                                             26
                                                                             23
                                                                              0
                                                                            0-3
                                                                              3
                                                                             24
                                                                             23
                                                                              0
                                                                            0-4
                                                                              3
                                                                             25
                                                                             24
                                                                              0
                                                                            0-5
                                                                              3
                                                                             27
                                                                             21
                                                                              0
                                                                            0-6
                                                                              3
                                                                             27
                                                                             18
                                                                              0
                                                                            0-7
                                                                              3
                                                                             27
                                                                             23
                                                                              0
                                                                            0-8
                                                                              3
                                                                             27
                                                                             21
                                                                              0
                                                                            0-9
                                                                              3
                                                                             24
                                                                             22
                                                                              0
                                                                           0-10
                                                                              3
                                                                             24
                                                                             18
                                                                              0
                                                                           0-11
                                                                              3
                                                                             25
                                                                             20
                                                                              0
                                                                           0-12
                                                                              3
                                                                             25
                                                                             21
                                                                              1
                                                                            1-1
                                                                              3
                                                                             23
                                                                              1
                                                                              1
                                                                            1-2
                                                                              3
                                                                             24
                                                                              0
                                                                              1
                                                                            1-3
                                                                              3
                                                                             27
                                                                              2
                                                                              1
                                                                            1-4
                                                                              3
                                                                             26
                                                                              1
                                                                              1
                                                                            1-5
                                                                              3
                                                                             23
                                                                              4
                                                                              1
                                                                            1-6
                                                                              3
                                                                             27
                                                                              4
                                                                              1
                                                                            1-7
                                                                              3
                                                                             25
                                                                              1
                                                                              1
                                                                            1-8
                                                                              3
                                                                             24
                                                                              3
                                                                              1
                                                                            1-9
                                                                              3
                                                                             27
                                                                              4
                                                                              1
                                                                           1-10
                                                                              3
                                                                             24
                                                                              1
                                                                              1
                                                                           1-11
                                                                              3
                                                                             26
                                                                              3
                                                                              1
                                                                           1-12
                                                                              3
                                                                             24
                                                                              1
                                                                              0
                                                                            0-1
                                                                              4
                                                                             25
                                                                             23
                                                                              0
                                                                            0-2
                                                                              4
                                                                             25
                                                                             19
                                                                              0
                                                                            0-3
                                                                              4
                                                                             23
                                                                             22
                                                                              0
                                                                            0-4
                                                                              4
                                                                             25
                                                                             24
                                                                              0
                                                                            0-5
                                                                              4
                                                                             26
                                                                             22
                                                                              0
                                                                            0-6
                                                                              4
                                                                             27
                                                                             24
                                                                              0
                                                                            0-7
                                                                              4
                                                                             26
                                                                             21
                                                                              0
                                                                            0-8
                                                                              4
                                                                             23
                                                                             22
                                                                              0
                                                                            0-9
                                                                              4
                                                                             27
                                                                             19
                                                                              0
                                                                           0-10
                                                                              4
                                                                             25
                                                                             19
                                                                              0
                                                                           0-11
                                                                              4
                                                                             27
                                                                             19
                                                                              0
                                                                           0-12
                                                                              4
                                                                             24
                                                                             20
                                                                              1
                                                                            1-1
                                                                              4
                                                                             27
                                                                              2
                                                                              1
                                                                            1-2
                                                                              4
                                                                             27
                                                                              4
                                                                              1
                                                                            1-3
                                                                              4
                                                                             26
                                                                              5
                                                                              1
                                                                            1-4
                                                                              4
                                                                             23
                                                                              1
                                                                              1
                                                                            1-5
                                                                              4
                                                                             25
                                                                              5
                                                                              1
                                                                            1-6
                                                                              4
                                                                             24
                                                                              0
                                                                              1
                                                                            1-7
                                                                              4
                                                                             25
                                                                              3
                                                                              1
                                                                            1-8
                                                                              4
                                                                             24
                                                                              1
                                                                              1
                                                                            1-9
                                                                              4
                                                                             26
                                                                              8
                                                                              1
                                                                           1-10
                                                                              4
                                                                             25
                                                                              3
                                                                              1
                                                                           1-11
                                                                              4
                                                                             24
                                                                              3
                                                                              1
                                                                           1-12
                                                                              4
                                                                             26
                                                                              0

Statistical methods

Generalized linear mixed models (GLMMs) with logit link function for binomial data are used to analyze the data using the following procedure:

 A GLMM model with over/under-dispersion (model 1) is first used to analyze the data.  If model 1 converges, the point estimated efficacy of the treatment is calculated using the proportions of blood-fed insects (of the groups) estimated from this model 1.
 If model 1 fails to converge, the data are then analyzed using a GLMM model without the over/under-dispersion option (model 2). If model 2 converges, the point estimated efficacy of the treatment is calculated using the proportions of blood-fed insects (of the groups) estimated from this model.
 If model 2 fails to converge, a GLMM with Laplace estimation method and without the over/under-dispersion option (model 3) is used to analyze the data.  If model 3 converges, the point estimated efficacy of the treatment is calculated using the proportions of blood-fed insects (of the groups) estimated from this model.
         Notes:
 The probability of all three models 1, 2, and 3 above failing to converge for a dataset is very small (probably less than 0.5%).
 Since only the point estimates (not the 95% CIs of the estimates) from the models 1, 2, and 3 are used, a compound symmetric (CS) variance-covariance matrix is used in each of models to increase the possibility of convergence of the models.

 If model 3 fails to converge, a generalized linear model (model 4) is used to analyze the data on each day separately. The point estimated efficacy of the treatment is calculated using the proportions of blood-fed insects (of the groups) estimated from this model.
 95% CI of the point estimated efficacy is obtained using bootstrapping technique.  1000 bootstrap datasets are created using resampling technique with replacement.  Each of bootstrap dataset is analyzed as the procedure (models 1, 2, 3, and 4) above.  The 95% CI of the estimated efficacy is the 2.5th and 97.5th percentile of the efficacy estimates obtained from 1000 bootstrap datasets.

SAS code

The SAS code of the data analysis of dataset Example1 are included in the Appendix VI.

Results of data analysis of dataset Example1

                                                   Estimated_    lower     upper
                  day    probtreat    probcontr     Efficacy     95% CI    95% CI

                   1       0.047        0.832         0.943       0.898     0.978
                   2       0.060        0.875         0.931       0.897     0.966
                   3       0.079        0.840         0.906       0.870     0.944
                   4       0.109        0.846         0.872       0.809     0.928

    Appendices

Appendix I: 	Results of power vs. sample size simulations

      Table A1.1: Study Design 1

         Binomial-Normal: PC=0.45, NPDay=50, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                                     Power

                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.110
                                                                          0.266
                                                                          0.673
                                                                          0.901

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.010
                                                                          0.046
                                                                          0.285
                                                                          0.626

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.006
                                                                          0.090
                                                                          0.320

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.005
                                                                          0.031
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.124
                                                                          0.330
                                                                          0.774
                                                                          0.953

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.020
                                                                          0.073
                                                                          0.372
                                                                          0.720

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.000
                                                                          0.007
                                                                          0.114
                                                                          0.355

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.035
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.166
                                                                          0.414
                                                                          0.843
                                                                          0.975

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.018
                                                                          0.076
                                                                          0.441
                                                                          0.773

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.000
                                                                          0.013
                                                                          0.171
                                                                          0.424

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.005
                                                                          0.029
                                                                             11
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.198
                                                                          0.479
                                                                          0.881
                                                                          0.978

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.029
                                                                          0.111
                                                                          0.555
                                                                          0.806

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003
                                                                          0.018
                                                                          0.196
                                                                          0.424

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.026
                                                                             12
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.261
                                                                          0.562
                                                                          0.940
                                                                          0.984

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.035
                                                                          0.138
                                                                          0.644
                                                                          0.840

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003
                                                                          0.017
                                                                          0.256
                                                                          0.438

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.012
                                                                          0.024

      Table A1.2: Study Design 2

        Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                                     Power

                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.212
                                                                          0.442
                                                                          0.798
                                                                          0.938

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.035
                                                                          0.118
                                                                          0.444
                                                                          0.738

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.007
                                                                          0.031
                                                                          0.182
                                                                          0.391

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.009
                                                                          0.026
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.309
                                                                          0.564
                                                                          0.886
                                                                          0.956

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.050
                                                                          0.170
                                                                          0.581
                                                                          0.789

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.005
                                                                          0.035
                                                                          0.231
                                                                          0.412

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.023
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.394
                                                                          0.699
                                                                          0.935
                                                                          0.977

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.067
                                                                          0.238
                                                                          0.700
                                                                          0.858

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.008
                                                                          0.045
                                                                          0.313
                                                                          0.435

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.013
                                                                          0.017
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.526
                                                                          0.795
                                                                          0.958
                                                                          0.983

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.101
                                                                          0.332
                                                                          0.787
                                                                          0.874

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.013
                                                                          0.078
                                                                          0.381
                                                                          0.445

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.010
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.646
                                                                          0.885
                                                                          0.978
                                                                          0.996

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.168
                                                                          0.487
                                                                          0.860
                                                                          0.909

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.021
                                                                          0.126
                                                                          0.433
                                                                          0.460

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.008
                                                                          0.009
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.738
                                                                          0.928
                                                                          0.988
                                                                          0.998

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.214
                                                                          0.594
                                                                          0.899
                                                                          0.926

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.027
                                                                          0.164
                                                                          0.441
                                                                          0.458

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.005
                                                                          0.005

      Table A1.3: Study Design 3

         Binomial-Normal: PC=0.80, NPDay=50, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                                     Power

                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.121
                                                                          0.268
                                                                          0.636
                                                                          0.858

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.016
                                                                          0.063
                                                                          0.280
                                                                          0.581

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.006
                                                                          0.016
                                                                          0.108
                                                                          0.287

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.001
                                                                          0.004
                                                                          0.023
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.166
                                                                          0.361
                                                                          0.762
                                                                          0.920

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.022
                                                                          0.085
                                                                          0.383
                                                                          0.694

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003
                                                                          0.017
                                                                          0.142
                                                                          0.350

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.005
                                                                          0.025
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.208
                                                                          0.448
                                                                          0.833
                                                                          0.953

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.025
                                                                          0.110
                                                                          0.514
                                                                          0.777

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.004
                                                                          0.019
                                                                          0.196
                                                                          0.393

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.010
                                                                          0.024
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.308
                                                                          0.582
                                                                          0.900
                                                                          0.980

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.051
                                                                          0.174
                                                                          0.635
                                                                          0.822

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.005
                                                                          0.036
                                                                          0.273
                                                                          0.419

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.007
                                                                          0.012
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.380
                                                                          0.685
                                                                          0.941
                                                                          0.992

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.067
                                                                          0.258
                                                                          0.728
                                                                          0.860

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.006
                                                                          0.053
                                                                          0.339
                                                                          0.434

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.012
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.454
                                                                          0.761
                                                                          0.967
                                                                          0.995

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.074
                                                                          0.344
                                                                          0.801
                                                                          0.896

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.005
                                                                          0.071
                                                                          0.372
                                                                          0.438

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.007
                                                                          0.008

      Table A1.4: Study Design 4

         Binomial-Normal: PC=0.80, NPDay=25, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.112
                                                                          0.242
                                                                          0.633
                                                                          0.889

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.014
                                                                          0.039
                                                                          0.261
                                                                          0.627

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003
                                                                          0.008
                                                                          0.079
                                                                          0.292

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.001
                                                                          0.004
                                                                          0.022
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.106
                                                                          0.254
                                                                          0.676
                                                                          0.900

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.014
                                                                          0.056
                                                                          0.324
                                                                          0.648

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.009
                                                                          0.094
                                                                          0.321

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.001
                                                                          0.004
                                                                          0.023
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.140
                                                                          0.378
                                                                          0.843
                                                                          0.973

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.017
                                                                          0.068
                                                                          0.462
                                                                          0.809

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.004
                                                                          0.009
                                                                          0.160
                                                                          0.435

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.008
                                                                          0.026
                                                                             11
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.150
                                                                          0.376
                                                                          0.833
                                                                          0.968

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.013
                                                                          0.069
                                                                          0.462
                                                                          0.779

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.004
                                                                          0.012
                                                                          0.159
                                                                          0.391

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.001
                                                                          0.006
                                                                          0.019
                                                                             12
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.195
                                                                          0.454
                                                                          0.874
                                                                          0.981

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.018
                                                                          0.114
                                                                          0.543
                                                                          0.815

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.017
                                                                          0.211
                                                                          0.395

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.006
                                                                          0.016
                                                                             13
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.227
                                                                          0.513
                                                                          0.911
                                                                          0.989

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.021
                                                                          0.131
                                                                          0.628
                                                                          0.827

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.014
                                                                          0.253
                                                                          0.419

                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.009
                                                                          0.013

      Table A1.5: Study Design 4

         Binomial-Normal: PC=0.85, NPDay=25, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.087
                                                                          0.227
                                                                          0.621
                                                                          0.869

                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.011
                                                                          0.040
                                                                          0.249
                                                                          0.589

                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003