And the W(t) is a seasonal wave, right?  A seasonal wave is a function.  We don't see the form of that function; we're just told that it's indexed to two parameters, h, which is a decay rate, measured in decimal months, and s, which is a phase shift, which is also measured in decimal months.
                  And together, with all these different combinations --
                  UNIDENTIFIED FEMALE:  And also m.
                  DR. KENNETH PORTIER:  Huh?
                  UNIDENTIFIED FEMALE:  There is also m.
                  DR. KENNETH PORTIER:  Oh, and then there's also m; that's right.  And so, there's three parameters.  It represents over 1,000 forms that they're looking at.
                  And when the model is fit to certain data, it looks through all these 1,000 forms to pick the one that's closest to it.  And the W(t) form is basically a concave function that asymptotes out to a certain point and then it decays kind of exponentially from that asymptote.  So, it's kind of a hump that goes down.
                  So, that's the seasonal wave component.  And then there's this -- the AMT or the midterm flow anomaly function, which is kind of a measure of long-term streamflow averaged out over about a month.  It's a 30-day rolling average, so it gives you some idea of kind of long-term average flow.  And it's weighted by these B2 terms.
                  And then you've got the short-term flow anomaly, which is kind of the residual flow that's taken away from the average long-term flow.  So there's a lot of variability in STFA.  And MTFA is kind of just a nice smooth function.  So, you've got those two functions that actually are the covariates that are used to explain Log concentration.
                  And then you've got this (t-tm) term, which it's taken me a little while to figure out, but that's kind of, I think, best described as a triangular function that goes up linear to tm and then goes down linear to tm, as long as the b4 term is positive.  And then you've got an error term.
                  And the difference between the Q and the QEX model is that they added a little bit more to the error term.  So, they looked at the residual, the e(t) term, and said, well, this is a time series, so there's probably autocorrelation that's occurring.  You know, tomorrow's residual is probably correlated with today's residual to some extent, so let's model that in.
                  But they also looked at the residuals and said, well, we get more variation when concentrations are high and less variations when concentrations are very low; so this SSD term here on the right attempts to take into account that heterogeneity that occurs with increasing or decreasing concentration.
                  And then the bottom term correlation takes into account that autocorrelation, which they're modeling as an exponential decay curve.  Okay?  So all of that put together is what they call the SEAWAVE-QEX model.
                  So, it's got a lot of moving parts.  When you open the hood, the engine is full.  There's a lot in there.  It shines, but there's a lot in there.  And you don't want to be messing with any part because it might mess the whole thing up.
                  Next slide.  So, I said there were a bunch of data steps.  In the sense of a regression, it's not a simple regression, where you kind of throw it in, you do a least squares, and there it is.  Because you're pulling out these correlates with streamflow, you're having to kind of take it into account, compute residuals, and then go back and figure out correlation structures and apply that in.
                  So it takes this long data -- you know, five-step process where you do data preparation.  Then you do a regression and try to figure out the shape of the seasonal wave.  Then you compute the residuals and you try to correlate that with the streamflow data.  And then you put that back in and you compute some new residuals.
                  Next one.  And then you try to figure the correlation function.  And that gives you different residuals.  And then at the end of the day, you have a fitted model called SEAWAVE-QEX that's applied to a certain set of data.
                  Go back to the first -- the second slide set.  Oh, you got the -- just leave -- next one.  So, I'm going to leave this up as I continue with my discussion.
                  So, SEAWAVE-QEX is a process for imputing missing values and pesticide concentration time series.  And I have a note to myself.  It does not predict -- and we use the word yesterday, a couple of people used the word yesterday to predict.
                  And I don't think it predicts.  It's for simulating missing values.  That's what imputation does; it kind of fills in with a possible value.  And so, this model is just a fill-in model that imputes.
                  In step 4 of the fitting process, random values are generated for censored normalized residuals; and random residual values are generated for days with no observations.  These are added to the mean trend values generated in the previous steps to produce a complete time series.
                  As is typical with imputing, more than one realization for missing values are generated.  In this case 100 values are generated to create 100 equally likely conditional traces.  So, that's kind of what they're doing.  And this is very standard imputation methodology.
                  In the first use of SEAWAVE-QEX described in the White Paper, the method is used to fill-in concentrations to complete concentration time series of selected USGS and NAWQA sites.
                  In this study, we can examine how the one-day series mean can change between observed time series and the mean of equally likely conditional traces.  And we can see that it's not -- you know, the conditional traces are -- produce maximums that are not far from the maximum of the observed time series.
                  And again, this is to me a classical imputation study.  And the findings that they report are what one would expect from imputation.  And when you have most of the data and you're just imputing a few points, the maximum is going to go up a little bit, because some of your random realizations are going to jump above the previously recorded maximum, most of them are going to be below.
                  You know, when we have the roughly 100 USGS datasets that are almost complete, you don't see a whole lot of difference between the observed maximum and the long-term imputed maximum.
                  Okay.  As pointed out by some of the public commenters, there are other methods of imputing values into the near-complete USGS and NAWQA time series, for example, kriging.
                  One of the reasons SEAWAVE-QEX may be superior to other imputation methods is its explicit handling of temporal trend, incorporation of streamflow information and predicting both temporal trend via the MTFA, and day-to-day variation via the STFA.  And then the use of temporal autocorrelation in the generation of residual variation.
                  The temporal trend components constrain the infill values, following the general pattern of concentrations prior to and following the missing days.  The variation components allow for the generation of concentration that exceed those recorded on other days in the monitoring of time series, while not allowing these days to have more variation than has been observed prior to or following the missing days.
                  In the second use of SEAWAVE-QEX reported in the White Paper, the process is used to impute values in time series with large gaps of missingness.  The question being addressed is, how well does SEAWAVE-QEX perform with sparse monitoring data?
                  To answer that question, measured datasets or subsample to create smaller datasets using a sampling protocol designed to mimic typical practices employed by various monitoring programs across the country.  By generating the sparse dataset from the complete monitoring data, the study is able to compare findings from fitting SEAWAVE-QEX to sparse data to the actual series maxima.
                  The sparse data study also examines the performance of SEAWAVE-QEX process when input datasets that have more intensive sampling, for example, seven-day sets, are compared to datasets with longer subsampling windows, for example, 14-day sets.
                  Two subsampling protocols are used to generate five realizations each of monitoring time series with data recorded on 7- and 14-day sample time intervals.  Each of the synthetic monitoring time series had a minimum of 12 samples per year to meet the SEAWAVE-QEX minimum requirements.
                  Each synthetic monitoring time series has input to SEAWAVE-QEX -- is input into SEAWAVE-QEX, and model results compared to full-measured datasets.
                  This is a large study involving four NCWQR sites assessed for two pesticides, atrazine and metolachlor, two subsampling protocols, the 7- and 14-day replicated each five times, and two covariates, a total of 160 runs.
                  And then there's two AEMP sites are also assessed for one pesticide, atrazine, two subsample protocols, 7 and 14 days, replicated each five time, and two covariates for a total of 40 runs.  Together, 200 runs are examined.
                  The White Paper in Chapters 3 and 9 -- Chapter 9 is Appendix B -- reports fitting results.  A total of 100 conditional traces are generated for each of the 200 synthetic datasets.
                  For each dataset, SEAWAVE-QEX outputs were condensed to the annual maximum estimate.  That is for each site, there are 100 estimates for the maximum for daily data, 100 estimates for the maximum of a 4-day average time series, 100 estimates for 21-day and for 365-day average.
                  The maximum of these 100 values, i.e. the maximum of the maximum, was then compared to the true value computed from the original complete dataset.  In addition, summary statistics means mediums, upper, mid, and lower percentiles of the 100 distributions were examined and summarized as well.
                  As an aside, one can argue that allocation of effort in this study might have been better balanced between imputation variability, estimated by the 100 equally likely conditional traces, and sparse sample to sample variability estimated by the five replicated datasets.
                  Five replicates does not really provide the best understanding of sample to sample variability.  With 100 traces, one can examine individual percentile estimates as alternatives to the maximum.  So, I can understand why they use the 100 there.
                  In addition to examining the maximum of the 100 maximum, other percentiles of interest were examined and are discussed in Appendix E.  These include the maximum of the annual 99th percentile concentrations, the 99th percentile of the annual 99th percentile concentrations, and the 90th percentile of the -- so there were other alternate estimates for estimating the maximum.
                  The White Paper provides a written summary of findings and evaluates performance with broad trends in the estimates.  For example, the 365-day average was underestimated more often than for other averaging times.
                  The magnitude of underestimation was mentioned.  For example, the 365-day average went underestimated at estimates that were at least 50 percent of the true average, about 40 percent of the 4-day, averaging 60 percent of the 21-day, and 20 percent of the 1-day series had those -- the -- oh, I'm sorry; that's -- the 50 percent true average -- it was about a 40 percent true average for the 4-day averaging, 60 percent for 21-day, and 20 percent for the 1-day series.
                  The discussions of findings from both the imputation study, and the sparse data study could and should've, included more details and a more rigorous and logically organized presentation.
                  The White Paper presents some diagnostic plots.  And the Panel was provided access to all diagnostic plots for each imputed site and much of the sparse data study outputs.
                  When discussing the quality of model fits, the accuracy and performance target was not clearly stated.  So the discussion kind of implied that the method performed well, but it was never very clear, performed well against what criteria?
                  While the White Paper does not go into detail, the flow of SEAWAVE-QEX tool include the eight diagnostic plots, that together allow users to get a very good understanding of the model performance and the quality of the fit through a particular set of data.  And I've listed in my notes here the eight diagnostic plots, simulation and summary of the output graph, a seasonal wave summary, an adjusted concentration, regression against a midterm flow, adjusted concentration versus the short-term flow, the trend for measured concentrations over the entire simulation.
                  Each of these graphs allow you to look at specific components of the model to see how well it's going.  So, you can actually visualize, well, how well is this b2Amt(t) doing in the model?  You know, is that b2 parameter very big or is it close to zero; which means it's not adding much to the explanation of the time series.
                  And then there's a couple of residual plots, one by month and one by year that allows you to look and see whether the model has captured all of the residual pattern, so that there's not any leftover temporal between your pattern that hasn't been captured in the model.  And then there's an important graph, which is the correlation function of the normalized residuals; which depicts the fitted exponential correlation function curve, plotted to the normalized residuals versus time.
                  And on that graph, you actually see the estimated correlation time scale, the CTS term, which is important, which estimates the average time between successive observations.  And then given acceptable fit criteria for the correlation function, that the 95 percent confidence interval for empirical autocorrelations, with time steps lower than the CTS should overlap the fitted correlation function curve.
                  The White Paper in Section 9.3 attempts to present the results of this assessment.  Because the assessment covers so many different issues, it's difficult reading, and often the key findings are not immediately obvious.  In some places there's too much detail and in others too little.
                  Nevertheless, reviewing the White Paper along with the SOP document and the Vecchia paper, the following strengths and weaknesses can be determined.  At least this is my take on the strengths and weaknesses.
                  So, the strengths related to using SEAWAVE-QEX to estimate pesticide concentrations with near-complete monitoring data for both concentrations and streamflow.
                  SEAWAVE-QEX appears to logically deconstruct the concentration time series into components that describe major sources of variation.  The multistep model fitting process is a logical approach to a difficult parameter estimation problem.
                  SEAWAVE-QEX performs well in describing near-complete monitoring data -- and this, I got out of the Vecchia paper.  It seems able to adequately simulate time series scatter of pesticide concentrations in near-complete monitoring data.
                  When fit to this near-complete monitoring time series, it provided an estimate of maximum concentrations for various averaging schemas that tend not to underestimate the true maxima, and which also tends to not overestimate the maxima by more than an order of magnitude.
                  SEAWAVE-QEX performs well when applied to sparse monitoring data scenarios, when the monitoring data meet general requirements of greater than three years of data, 12 or more samples per year, spread across the application season and less than or equal to 70 percent censored observations.
                  The estimated maxima and 80 percent error bounds are less than two orders of magnitude greater than the true series maxima in most of the examples that they had.
                  As expected, the more samples per year and the more uncensored samples, the better are the SEAWAVE-QEX maximum in estimating the true maximum.
                  SEAWAVE-QEX performance does not significantly degrade when missing streamflow data are estimated with precipitation data.  SEAWAVE-QEX seems to work best when the model time period does not significantly extend beyond the min and max sampling times, and when the annual or seasons-limited time series is best described by one seasonal wave.
                  I didn't see any scenarios where multiple seasonal waves seemed to do any good.  And I see a lot of that seems to be just artifact of the fitting process rather than a real deficit of the model.
                  Weaknesses related to using SEAWAVE-QEX to estimate pesticide concentrations with near-complete monitoring data for both concentrations and streamflow.
                  SEAWAVE-QEX is a complex model and will be difficult to understand by the uninitiated.  This may be less of an issue if use of the model is limited to tier 4 assessments as suggested, and when it's supported by staff that truly understand the model and the complexities of fitting it.  So, this is not a turnkey system.  It's going to need a little bit of expertise the two or four times it's applied each year.
                  SEAWAVE-QEX is a mix of empirical and theoretical constructs.  But at its core, it's a regression.  As such, it's better at estimating mean patterns and less competent at estimating stochastic or residual patterns.  In addition, the important stochastic patterns may not be fully captured with less than 100 monitoring data spread over multiple years, and when fully 70 percent of these data points are below detection.
                  Also, because of this, it's difficult for regression models -- and I would say in most statistical models for that matter -- to predict with any accuracy extreme events such as maximum.  This was a point I made in the 2012 panel report.
                  You know, estimating extreme owner statistics like the maximum is very difficult to do, because there's a lot of variability in the maximum value.  And it's a very skewed distribution, highly -- I have to always remember the tail right skewed, right?  It's always the tail goes way out, which means the maximum of the maximum has the potential of being a very high overestimate.  But I think in this study there's a realization that that can occur.  And there's been some attempts to avoid going way out in the tail where you're going to get overly conservative estimates.
                  SEAWAVE-QEX performance very much depends on the situation.  As mentioned on page 141 of the White Paper, "The summary statistic that performs best, differ somewhat for each pesticide combination, and depends on the definition of best.  Whether the objective is for the subsample to always overestimate the highest annual maximum, or overestimate as often as underestimate the highest annual maximum, or all annual maxima.  This means that use of the model can never be automatic.  It'll require an educated user to properly fit and use.
                  And finally, I have a quote that I started typing right as the meeting started about the long-term 365-day average maximum estimate -- estimation of it.  I think the reason the 365-day average underestimation is more frequent than we see in the other scenarios, is that it's more likely that the seasonal wave, the W(t) component in the model, is not well-captured in that annual data.  And the regression on the MTFA and the STFA don't kind of step up and take care of it.  It doesn't come in and compensate for the loss of that seasonal wave curve.
                  And that produces -- and that puts a lot of the effort in the modeling on the variability of residuals to kind of give you a high maximum.  And so the model is just not capable, in that situation, of going high enough to really capture the average annual estimate.  But I think I need to think a little bit more about that.
                  And with that, I'm done.  I'm sorry.  It's all written though, Bob.  Questions?
                  DR. RAYMOND YANG:  Ray Yang.  I thank Ken for giving me these three pages of this writeup and really clarify things.  And I want to echo that.
                  I'd want to tell the EPA colleagues that you cannot talk about modeling using 255 pages without talking about the model.  Therefore, this stuff had to go into your White Paper; okay?  In fact, you will do yourself a great favor if you attach Vecchia 2018 paper as an appendix.
                  DR ROBERT CHAPIN:  All right.  Thank you very much.  Let's see.  Dr. Berrocal, do you have stuff to add?
                  DR. VERONICA BERROCAL:  Yes.  Not very much, because he already has covered a lot.  But I wanted to start by saying that I think that this problem of estimating or imputing pesticide daily concentration data, and the response migrating data, is actually a very hard problem.  It's not very easy since there are many factors that play a role in our factors besides concentration.
                  Yesterday, I was asking lots of questions about this model.  I would actually like to commend Dr. Vecchia for tackling these complex issues and having propose a model that is in my mind elegant, robust, and as simple as possible, even though somebody believes it's not too simple.
                  I'm not going to go over the details of the model, but I think as it was said, that it captures the main model variation in daily concentration, pesticide concentration data.  I also believe that the fact that now there is introduction of these serially-autocorrelated adverse terms, that allows to account for lingering temporal correlation in the daily pesticide concentration that cannot be explained by the seasonal wave and the streamflow.
                  The introduction of these serially-correlated adverse teams with seasonal variance standard deviation has been sought to allow the model to provide a more flexible characterization of the viability in the daily pesticide concentration, that one could do with a kriging model, which typically assumes a constant variance.
                  Also, I would like to point out that without the addition of these residuals that have seasonal variance standard deviation, the SEAWAVE-QEX model can actually be thought as a specific case of universal kriging.  So, the SEAWAVE-Q is an example of universal kriging, while the SEAWAVE-QEX is a step above universal kriging, which I think it's more adapt for this situation.
                  I believe that the model has several strengths.  It's an elegant and not overly-complexed model that allows, through conditional simulation, to reproduce the statistical characteristic of the daily pesticide concentration data.  And thus, it can be used to infer upon the distribution of certain summary statistics from an incomplete time series of pesticide concentration measurements.
                  The parameters in this model have an interpretation that can actually be useful for designing future sampling strategies.  For example, the correlation temporal skill is important for one designing the sampling frequency.
                  The assessment that Vecchia performed of the SEAWAVE-QEX model in the paper show that the model has an overall good performance in real pesticide concentration data.  And the simulation experiments that Vecchia performed showed that when the assumptions of the model are satisfied, the SEAWAVE-QEX model actually performs pretty well in recovering the true daily concentration data.
                  And this is particularly true when the sampling rate is low, and the sampling frequency is lower than the correlation temporal scale.  And I really want to underline this because I think this is an important point.
                   Also, for long-term in the data evaluation that EPA performed, it seems as SEAWAVE-QEX is able to neither consistently underestimate, and not always overestimate, long-term pesticide concentration when it is characterized in terms of extremes.  So, either the maximum of the 95th or maximum of the 99th percentile.
                  In terms of weakness, I think that as it was mentioned, the SEAWAVE-QEX model might not be easy to implement for somebody that has not experience with this type of modeling or with this type of data.  Also, the evaluation of the SEAWAVE-QEX model discussed in the White Paper in appendix B, Section 9(2), showed instances where the two season waves model was selected, as the best model to fit the data, but the data did not support the presence of the two waves.  Hence, a user and assessor should be exercising care when feeding the SEAWAVE-QEX model to data and determine whether the model should be fit in the four modalities using all data or using only a partial record.  So, that's another complexity that somebody who doesn't have the expertise might not understand.
                  I was a little bit disappointed, once after I had read the Vecchia paper; in seeing that in terms of evaluation of the performance of the SEAWAVE-QEX to derive summary statistics for characterizing short-term and long-term pesticide concentration, the results were not as good as the ones presented in Vecchia's paper.  And there was not an easy generalization since the results varied by pesticide and site, as it was already mentioned.
                  Particularly, for each site pesticide combination subsample we did, a 7-day or 14-day samplings frequency were derived with pesticide concentration data.  There was subsamples that include only 12 measurement, which was the lower limit for the number of samples indicated by Vecchia.
                  So, given that 12 was the lower threshold indicated by Vecchia's number of observation, it seems here that the utility of the SEAWAVE-QEX model has been tested in the worst-case scenario.  I think it would be useful to assess whether the capacity of SEAWAVE-QEX to adequately capture pesticide concentration, was assessed when a slightly larger subsample of data was used.
                  I'm not yet ready to describe the SEAWAVE-QEX model as a model that can capture short-term concentration just because I feel that the subsampling was really put in the model in the worst-case scenario of only 12 samples.
                  Another point that was mentioned, but I think that this was not highlighted.  It was that the empirical correlation that is produced by the SEAWAVE-QEX diagnostics, seems to indicate basically that there is independence in the residuals.  When instead, the fitted exponential correlation indicated there should be a correlation because that's what the model assumes.
                  I believe that this is a result of the subsampling scheme that has been used, that has made the data less correlated.  I understand that the subsampling scheme was performed to generate data that was similar to what is observed in real datasets.
                  But I wonder if the performance of the SEAWAVE-QEX model was also affected by the fact that the data that was provided was basically independent, based on the critical correlation reality the model assumes dependence.
                  I think that for the long-term concentration, it was already mentioned that SEAWAVE-QEX has problem in characterizing long-term concentration, particularly when we're thinking about annual average, 365 days.  While instead, characterization of short-term pesticide concentration does not present the same problem of underestimation.
                  And I had one question that maybe this is more a clarifying question than a comment on the charge questions.  I don't know whether I should ask it now or --
                  Dr. ROBERT CHAPIN:  Sure, go ahead.
                  DR. VERONICA BERROCAL:  SO, in the evaluation, it was assessed whether SEAWAVE-QEX would work if instead of streamflow, precipitation was used.
                  I'm just confused as in this equation that is put up here on the screen, we have Amt and Ast that are derived from the streamflow.  So, are those -- when you're using instead precipitation, are you computing the anomalies with respect to precipitation?  It was just confusing; there was no mention explicitly in the paper.  Or it was just -- you're just throwing away the AMT or the AST.
                  DR. SARAH HAFNER:  The model uses the precipitation to do that, the same as it does for streamflow.
                  DR. VERONICA BERROCAL:  Mm-hmm.
                  DR. ROBERT CHAPIN:  All righty.  Excellent.  Thank you very much.  Let me see.  I think what we'll do is we'll go through the leads and associates for hydrology and then for modeling.  And then open it up to -- and then get Dr. Sobrian's comments.  And then open it up to anybody else who wants to make comments on this question about strengths and weaknesses.  We'll go to Dr. Baffaut.
                  DR. CLAIRE BAFFAUT:  Thank you.  My name is Claire Baffaut.  So, SEAWAVE-QEX estimates a pesticide concentration during days that no sampling was done.  So, that is a set of statistical distribution of concentration simulated by SEAWAVE-QEX, is similar to the observed distribution of pesticide concentrations.
                  The nice thing about SEAWAVE-QEX is the fact that it takes flow as a covariate, in my opinion.  Because by doing that, we take care of some of the hydrological landscape processes that take place and that affect pesticide transport.
                  So, run off and flow are an outcome of precipitation.  And run off and flow, and infiltration, drive the pesticide transport.  So, by linking the pesticide concentration to the flow, we take care of other processes that take place on the landscape, runoff generation, infiltration, evapotranspiration; that all affect that flow which is a driving motor of pesticide transport.
                  So, that's what I like about it.  And in the equation of the model, all those terms correspond in some way to some physical processes.  The wave represent the input.  The short-term anomaly represents the short-term flow, which would be what's driven by runoff and quick subsurface flow.
                  The long-term flow anomaly would be what's represented by the ground water flow.  And then the last term would be degradation, I think. So, that's -- it's nice to -- even though it's a statistical model, it's nice to bring it back to what's really happening.
                  In terms of evaluation, I don't know what the pesticide water calculator does.  But in any places in the White Paper, it's mentioned that it improves the results of what is available now.  But since I don't know what the water pesticide calculator does, I would have liked to see, for all those cases, what would have been used instead of SEAWAVE-QEX.
                  Another thing that I thought was a limitation of the evaluation is that it's evaluated at four sites.  All of those are in northeast Ohio.  Two of the sites, Rock Creek and Honey Creek, are nested within the larger Sandusky River water shed.
                  All have a dominance of agricultural land that varies between 72 and 81 percent.  And the main crops are the same in four watershed, corn, soybean, wheat, and some hay.
                  Most agricultural land in those four watersheds is artificially drained with subsurface tile.  And those tiles speed up the transport of many things, including pesticide, from the fields to the stream.  And the rate of subsurface drainage in those four watersheds is about the same; that's about 70 to 80 percent of the agricultural land, which represent roughly 50 percent of the watershed.  So, as intensive hydrologic transports, there is not a lot of difference between those watersheds.  They kind of works the same way.
                  When it comes to the simulation of hydrologic variables, including pesticide concentration, it's important to test them all under different sets of condition; different water, different -- that includes amount of rainfall and intensity of rainfall.  Different crops because it affects residues, they affect ET, they affect other things.  Different soils and different management practices.
                  The biophysical property of a region do have an impact on the vulnerability of the land to transport a contaminant.  Thus, it seems important to evaluate the tool across a range of its biophysical characteristic, rainfall amount, timing, intensity, has a strong impact on contaminant transport from agricultural areas.
                  If datasets with daily data are not available in other biophysical settings, I think maybe EPA should consider the next best option, which may be near-daily data.  And it may not be a whole dataset that has several watersheds.  There may be -- you know, one watershed with a good dataset in that region and another dataset in another region.  But I think it's very important to test this in systems that function differently in terms of hydrology.
                  SEAWAVE-QEX assumes constant pesticide rates from year to year.  This is very large assumption because crop distribution, crops choice change from year to year, and therefore pesticide application.
                  I know that in some watershed that are well-drained, for example, in Iowa, that may be more stable.  In other areas where soils are more vulnerable to excess moisture, then that excess or moisture in the weather will affect the crop choice.
                  So, even though the main crops are corn and soybean, I can guarantee you that in northeast Missouri this year, there was much less corn and much more soybean planted because it was very, very wet in the spring.  And that affects the amount of pesticides -- the choice of pesticides and the amount of pesticides that are applied.
                  I'll get back a little bit on the evaluation criteria.  And Dr. Portier raised the point, is that there is no clear criteria set in the White Paper that tells us what is good.
                  So, in the Vecchia 2018 paper there is actually some criteria.  For example, let's say that if the bias between an estimated and observed metric -- whatever it is -- was between minus 10 and 25 percent, then the bias was considered low, which is good.  If it was between 25 and 50 percent, it was moderate.  If it was greater than 50 percent, it was high.  I didn't see similar criteria in the evaluation that was run for the four sites.
                  I recommend EPA to define such criteria to evaluate SEAWAVE-QEX.  What constitutes acceptable results?  What does that mean that the peak concentration is not underestimated, or that it's not grossly overestimated?  Is it a 10 percent difference?  Is it a 25 percent difference?  Is it a 50 percent difference, or in order of magnitude or what?  But we don't know.
                  And same thing in the appendix.  When comparing all the distribution, the concentration distribution, what is an acceptable thickness of the colored band that represents 100 simulations for each subsample?  Some of those, it's like maybe -- it's definitely less than an order of magnitude, maybe half of one.  And some cases it's more than one.
                  So, what is acceptable, what is not acceptable, I think, would make it easier to decide whether this is an acceptable model.  It would also make it easier to decide what sampling strategy is to be selected.  And it would be based on something rational.
                  There is the issue of missing or zero flow data.  So, I went through a little thing about what could cause missing flow data.  So one is that stages higher than the range of the rating curve.  And so, there is no basis to extrapolate the stage value to a discharge value.  But in that case, streamflow estimate can be obtained by extension of the reading curve, by first physical principles or by correlation with another trusted stream gauge.
                  Another case, we may be missing flow data because there was an equipment failure.  Values can be interpolated, or values can be obtained by correlation with another trusted stream gauge.  The stream bed went dry for a period of time.  Note that during that time if it went dry there is no transport of pesticide, and those concentration are zero as well.  So, that's not the most critical part of what this tool is made for.  So, I think that the idea of replacing by .1 or points was fine.
                  If there is no stream gauge, streamflow value from a different watershed that functions similarly to the one on the study may be useful as a covariate.  This would be particularly useful for smaller watershed like a 12 size or less than 100 square kilometers.  It's not likely to work for larger ones because it's more difficult to find an ecohydrologically similar watershed for larger ones.
                  But I would like also to say that hydrological model have made some progress.  And now, there's -- I've seen more and more model application that show good simulation of flow values, even with minimal calibration or even sometimes without any calibration.  And same thing, the model can be calibrated for a similar watershed and then applied to the one where there is no stream gauge to get some flow values.  And that's it.  I think I'm done.
                  DR. ROBERT CHAPIN:  All right.  Thank you very much.  And we'll turn to Dr. Green.
                  DR. TIMOTHY GREEN:  Tim Green.  I'm actually associated on, I think, the next three, and I'd like to defer to, I think, 1(c) for my next comments.
                  DR. ROBERT CHAPIN:  Thank you.  All right.  And now, we'll -- the spotlight swings far west to Dr. Zhang.  Yes?  And we'll hope that your caffeine has kicked in.  Dr. Zhang, are you with us?
                  DR. XUYANG ZHANG:  Yes, I am.  I've been drinking black tea in these couple days, and it's important.  Well, I'd like to start with a big thank you to the colleagues in EPA.  I think this is no trivial work.  It takes a lot of hard work to put things together.  And excellent presentations yesterday.  That really helped me clear lots of questions that I've had.
                  I will go straight forward to the strengths.  Dr. Portier had a great introduction on the SEAWAVE-QEX model already.  I think there are a few strengths of the model that I really like.  The first one is that it can generate daily times zero chemograph that can be used to estimate uncertainties in the margin data.
                  The second point is that it can generate daily concentration estimates that are higher than the measured maximum, and consequently reduce the chance of underestimating the short-term average pesticide concentration.
                  This is a large advantage compared to the traditional imputation methods where, you know, the maximum value can never be exceeding the maximum of this value.
                  Number three is that it provides the diagnostic plot that really helped USGS evaluate the goodness of fit of the model.  As it can be shown in the SOP and the White Paper after model simulation, the SEAWAVE-QEX generates eight different plots that really provide lots of information about a model fitting.  And that also helps the following simulation, and it helps the users to improve based on what they've seen on this diagnostic plot.
                  Another advantage is that the development of the model has been well-documented in my opinion.  In Vecchia 2008, the questions were crystal-clear.  And like Dr. Portier and Dr. Berrocal had mentioned, I really like the model being simple and elegant.
                  And then, the next strength bullet is that based on EPA's evaluation, using the daily or a near-daily monitoring data of atrazine and metolachlor, the model seems to be able to generate daily concentrations from subsample data with reasonable accuracy.  That is, as mentioned in the White Paper, there's a low percent of underpredicting as shown in Figure 9.18 and 9.19, except for the results in the 365-day average period.
                  And in the section of sampling bias factors, it was also mentioned that the resulting sampling bias factors developed using SEAWAVE-QEX, estimated a chemograph where it's very close to those developed using the full record of data.  So, that's -- it has good applicability to enter the data then used to calculate a sampling bias factor.
                  And lastly, when running the model, there are a minimum number of parameters that require user input.  And speaking based with my experience of a hydrological model, which often involves hundreds of parameters that the models and model users have to tweak.  So in comparison to that, SEAWAVE-QEX is relatively simple to use.  And this ensures, in my mind, the consistency in modeling results and minimizing human error.
                  With that, I also realize that there are some drawbacks of the model, especially when we try to use it in pesticide drinking water assessments.
                  The first one is the data requirement of the model.  So, the minimum requirement is more than three years of data and less than 30 percent of censorship.  Which in my mind are a stringent criteria and relative to the typical monitoring data that are currently available.
                  Many monitoring datasets do not need even the lowest of requirements, especially on the sentinel rates.  So, that results in a very limited use of the model.  So again, the significant errors could occur when model assumptions are violated.
                  So, I want to state the following two modeling assumptions.  And I think there's a big chance that the assumptions could be valid, if nothing paying close attention, in many application cases.
                  So, the first one is that the model -- when the variation in flow and concentration during the day is large.  And the model really assumes, you know, fitting the daily concentration using largely available graph samples, which is not representative of the daily peaks under systems that have big variation during the day.
                  And also, the model uses the daily average flow.  So, in these kinds of systems, the daily average flow might not also be representative of the peak flow within the day.  So, in some hydrological systems, when the variation in both flow and daily concentration is large within a day, the model may not fit very well.
                  And secondly, the relationship between the pesticide transport and flow, according to this model, is linear.  So, the model assumes a linear relationship between pesticide transport and flow.  But in reality, in some hydrological systems, the relationship could be nonlinear.  So, when these assumptions are valid, I think we could expect large errors in model simulation.
                  The model -- it seems that the model cannot be applied beyond the years with available sampling data.  As Dr. Portier had mentioned, that the model essentially is an imputation method, it cannot be used for prediction.  So, I just want to make a point of that.
                  And also, about the time trend.  The model assumes a leaner trend over time.  And a scientist who worked in the pesticides for many years, I know, that changes in regulation and consequently the use patterns of pesticide may results in abrupt changes in pesticide concentration trends.  For example, chlorpyriphos with the cancellation of use in urban areas.
                  Around year 2000, 2002, there's a significant change in the use pattern.  So, under these conditions, the sample data should be subdivided into different periods and the SEAWAVE-QEX model should be seeded separately for each period.  I think EPA is aware of that and it's been already mentioned in the White Paper.
                  The next point is that the model assumes constant pesticide input from year to year, which may not be true in areas with saturating pesticide usage due to climate change and crop management.
                  I think Dr. Baffaut had just mentioned that also in areas of Ohio, and I know in California too, pesticide usage changes from year to year.  And the assumption of constant pesticide usage from year to year can easily be violated.
                  The next point is that for the two-season model, pesticide removal is assumed to be no longer than 60 days.  I think this may not be true, especially for persistent pesticides, which take much longer then to dissipate.
                  And so users should avoid using the two-season model, especially for persistent pesticides.  And looking from the White Paper anyway, it seems that simulation is being done by EPA, the one-season model seemed to work better on many occasions, in the two-season model anyway.
                  So, the next point is that I think the model worked pretty well for near-complete datasets.  But the accuracy of the model, when applied to (inaudible) datasets, remains uncertain in my opinion, due to the following two reasons:
                  The first reason is that in this evaluation and as documented in the White Paper, only three pesticides were used in the evaluation.  And the pesticides were actually very similar, regarding to the case of chemical property.  And the size are very close to each other.
                  This datasets, although, I think it's very valuable, but they are not representative of the wide array of pesticides that are currently being used in the nation.
                  Take Koc for example.  I think among the three pesticides that are relatively having low Koc, compared to hydrophobic pesticides, for example, pyrethroids, which may behave very differently compared to the three pesticide.
                  So, the dataset also does not represent the variability in watershed properties and pesticide use patterns that are observed across the nation.
                  Again, Dr. Baffaut said it'll be beneficial if we can continue evaluating the model in areas with different hydrological conditions than the existing four sites.
                  In EPA's evaluation, the SEAWAVE-QEX generated results using the subsample data and would compare it to the log integrated (inaudible).  So this comparison, I think it's a valid approach to evaluate a model performance.  But this -- it did not provide an independent validation.
                  This is perhaps a very high requirement of the evaluation -- of the person conducting evaluation.  But in an ideal world, to evaluate the performance of the model, it'll be best to be conducted if there's an independent dataset.
                  But I understand that atrazine is probably the best data available.  And obtaining a completely separate, independent dataset might be not feasible.
                  But given that, I think a cross-validation approach could be used to include the evaluation.  In which a subsampled data could be set aside, excluded from model fitting.  And the set aside dataset can then be used for model validation.  Instead of using all the subsample dataset for fitting the model, we can keep part of it for validation.  So, I think with this approach, we could have more confidence in the results regarding the performance of the model.
                  And I second Dr. Portier's opinion, that the model  --  in terms of evaluation criteria.  I was also looking for that in the White Paper and I didn't see it.  I recommend that EPA put that in the White Paper and calculate statistics that can be used to evaluate the model performance.
                  And another point about the weaknesses of the model is that to some users it may have a steep learning curve.  And the model also has limited use in nonflowing water bodies; and flow is not really correlated with concentration.
                  That's all for me.  Thank you.
                  DR. ROBERT CHAPIN:  Okay.  Thank you very much, Dr. Zhan.  All right.  Dr. Miglino.
                  DR. ANDREW MIGLINO:  Hi.  So, I'm going to keep this brief because I think a lot of what I had written down has already been said.  But I'm going to stress a couple things in here.
                  So, the strengths have been well-established here.  I think the model actually is quite elegant for how nice and tidy it is.  It captures what you need to capture.  It gives you relevant data out of it that you can make regulatory decisions from.
                  And as long as the site meets the data criteria, it seems to perform well.  But that, I think, is one of the major points that has been brought up a couple times, is how many sites really do meet these daily criteria because they are quite stringent; especially since you need both concentrations, and you need some flow characteristics or some covariate.
                  And that's where I started to kind of think about difficulties in applying this model in the future to newer chemicals that don't have nearly as much data as the few that you've shown here, or maybe are a little stranger.
                  So, you had simazine -- I think that's the name of it.  Simazine in there, which did not perform very well because it was more of an intermittent use.  And I don't exactly know how you can capture an intermittent-use pesticide with this model.  But that's certainly another limitation on top of all the data limitations you already have.
                  There's also, I think, something that I'm going to echo in a couple other comments as we move forward, which is there needs to be some bright lines for when to switch covariates or when to switch your analysis procedures.  Because in some of the test cases, you have a flow covariate performs very well and is excellent.  But the same covariate for a different chemical no longer performs as reasonably as precipitation.
                  And it didn't seem to make sense why that would occur.  Because they -- as it was just said -- they are pretty similar chemicals overall, at least the ones that you could use.  So that's something, I think, to look into, maybe just rules for proceeding down that or chemical properties that define what covariate.  I don't know, something to that effect.
                  Something that was brought up also yesterday by, I believe, Ian, was repeatability.  And I know it's probably not actually an issue of, is the number going to change by an order of magnitude.  But it is probably an issue of, hey, how come my number is different than that guy's number?  And so again, kind of defining that rule set to say this is how I chose my number, it's the same number every time, or extremely close to that same number every time, would be advised.
                  And I think that's probably everything I had that wasn't already said.  Yeah.  I'll stop there.
                  DR. ROBERT CHAPIN:  Brevity is the soul of delight sometimes.  Dr. Sobrian?
                  DR. SONYA SOBRIAN:  I'm going to be even briefer.  At this point, all the concerns that I wrote and some of the strengths have already been covered.
                  DR. ROBERT CHAPIN:  Okay.  Anybody else who hasn't been assigned?  Tom.
                  DR. TOM POTTER:  I'll try to be -- I will be brief, but I wanted to echo a few things that some of the previous discussants had highlighted.  And that includes identifying model weakness relative to flow and/or the absence of it.  I think that whole area needs further development.
                  And I'm still wondering what the actual equation is that exists in the White Paper, that shows the form in which precipitation data is used in the model.  Now, maybe it exists and perhaps I missed it, but if I did, I would appreciate your input on that.  So again, I see that as a major weakness, and we'll discuss that in some detail later on.
                  I also want to highlight the issue of potentially incorporating pesticide use data into the model.  I think several of the prior discussants have highlighted this in different ways; for example, indicating that cropping patterns change, there are new products on the market, et cetera, et cetera.
                  There's also the problem of intermittent use.  And those can be dealt with to some degree by incorporating a flow -- a covariate in the model that actually addresses the actual pesticide use.
                  And there's certainly data out there that could be used for that purpose.  The State of California is a leader in this area.  I think we heard that from Dr. Zhang in terms of compiling use data on a year to year basis.  And certainly, there's a rich source of information there.  And I think we heard yesterday mention of the USGS program that uses sales and NASS data to basically estimate pesticide use on a national scale by county.
                  Now, I do recognize that county data can't be directly imported into making an estimate on a watershed scale.  But again, I think there have been efforts applied in that area.  There are indeed publications, which I will provide as citations.  And with advancements in our GIS and our -- also our landcover databases, I think tremendous progress has been made, where I think actual pesticide use can be used in the model.  And to me, that's critical in terms of having confidence in the result.  So I offer that as a thought.
                  With regard to applications of the model, I realize, you know, we're in the early stages here.  But we didn't see any that represented cases where pesticide degradates that would be of concern or evaluated.  Certainly, that needs to be perhaps a next step in the evaluation process.  It may not be critical; depends again on the active ingredient you're looking at and whether or not you have difference of concern.
                  I will note that there was a paper -- again, I'll provide the citation -- that was published, I believe, in 2011 or '12 by the USGS team to look (inaudible), herbicide degradates in Iowa rivers.  And actually, the model seemed to fit the data quite well.  So, I thought that was really encouraging.
                  I also would like to see some more effort in looking at pesticide scenarios where there are multiple applications during the season.  There are lots of active ingredients that fall into this category, especially fungicides.  We didn't see a example on fungicides, and I realize fungicide surface waterData is pretty limited, so it becomes a challenge.  But I think that's a class of chemicals that definitely needs more work in SEAWAVE-QEX; in particular, looking at how wave forms respond to situations where you have multiple applications for a growing season.
                  I worked for many years with peanut growers, so I'm quite sensitive to that.  They have their tractors in the field every two weeks, literally spraying their crops with the same or different active ingredients, again, to grow plant diseases.  So you have season-long applications rather than a single application such as the case of a Post-Emergent herbicide.  So, absolutely critical that we look at some scenarios that deal with that.
                  Finally, I would like to, you know -- I'm not -- I wasn't quite sure where to shoehorn this in, but I think it's important to say, so I'll put it in here in my comment.  And that has to do with the constraint that SEAWAVE-QEX handles only daily data and can't deal with composite sampling data.  So, in essence, we can't use the valuable sampling approach of compositing if the intention is, down the stream, to use something like SEAWAVE-QEX.
                  I think there needs to be some thought put into that, particular with regard to the design and construct of monitoring progress.  And I greatly appreciated the talk by our colleague from Washington State yesterday, who was using the tool to evaluate and assess future monitoring programs.
                  Because obviously, that's something that we need to take away from this meeting.  Not necessarily the use of the tool today, but where we're headed down the road in the future in terms of the data collection process.  I certainly commend you for doing that and I think there needs to be more work in that area.
                  I will note that regard to this issue of a grab and composite sample, I saw a really interesting paper that was published maybe a couple months ago, in Environmental Science and Technology by a Swedish research group that monitors pesticides in surface waters there.  And they had this neat little tool where they were basically collecting weekly composite samples in rivers by using a microscale solid phase extraction system deployed in the river.  I think they called it a TIMPHY (phonetic) sampler.
                  But what was interesting about the results, and that seemed to validate the sampler quite well, is that they showed that they got almost twice as many pesticide hits when they composited as when they grab sampled, and they did side-by-side sampling.
                  So, compositing certainly has some potential for -- because again, because of the concentration effect in this case, to identify more active ingredients in the surface water samples above detection thresholds.  So, that's important, as we know, in terms of our dealing with other issues of censoring.
                  And the other thing they noted was that grab sampling appeared to underestimate pesticide peaks; which again, I thought was interesting and certainly worthy of consideration as we go forward in designing monitoring programs that mesh well with the use of this tool.
                  DR. ROBERT CHAPIN:  Excellent.  Thank you.  So Ken, do you have a --
                  DR. KENNETH PORTIER:  Yeah.  Two paragraphs.  And I promise after this not to talk that long or to bring up this slide.  I want to bring it up one more time.
                  While the Panel has not asked to comment on the specifics SEAWAVE-QEX model, there is one issue with the model that, I think, should be addressed as it represents a potential weakness.
                  In the model the t-tM, the tM term in the model is specified as the midpoint of the time period being analyzed.  So, somewhere in the fitting process, the tM term is set.  And it's not estimated, and it's clearly a function of how the season is specified.
                  And in addition, it's clear that there should be a relationship between this tM term and the time at which W(t) maximizes; right?
                  So you get the tM term, obviously -- let me see what I said -- okay.  If the beta 4 term is positive, the t-tM regression component is designed to add linear increasing trend from the beginning of the season up to time tM, and then subtract linear trend to the end of the season.
                  If the season is specified wrong and we start too early, or we end too late, the tM term has shifted.  Right?  So they end up getting some kind of added peak before the W(t) or some after the W(t).  And I think this may be kind of one source of situations where the model doesn't seem to work.
                  I'm really concerned with that particular term and exactly what it's trying to capture.  It's almost like capturing a quadratic added peak that doesn't show up in the seasonal wave function.
                  The next step of the procedure development, or development of the model, you might want to look into estimating tM.  So, rather than just specifying it as a function of the middle of the season, actually let the model try to set it and see how that relates to the middle of the season.  You might find that that tM term is close to the time at which W(t) is maximized; in which case that tells me something about the wave function may not be capturing exactly what's happening in the seasonal pattern.
                  Or the tM term may be far from that time, in which case you've kind of got two peaks, two potential peaks within the season, again, that the wave form is not capturing.  That's one of those things that came up.
                  And the other one was the issue of --
                  DR. ROBERT CHAPIN:  Ken, let me just interrupt for clarification.  Do you think this is a got to do right now or --
                  DR. KENNETH PORTIER:  No, no, I think, this is the next step of the development.
                  DR. ROBERT CHAPIN:  Okay.  Thank you.
                  DR. KENNETH PORTIER:  You know, everything we see in the model fits well.
                  DR. ROBERT CHAPIN:  Go ahead.  That's fine.  Go ahead.
                  DR. KENNETH PORTIER:  You know, one of the things they asked us to do is, where's the weakness?  And to me, this represents a weakness in the model; this is not completely tied down.
                  And then the second issue was the use of the model for intermediate use chemicals that Dr. Miglino talked about.  But I think potentially, this can be seen as a failure of the process to properly define season for the chemicals of interest.  As currently used, season is broadly defined as the potential window of use, but also the potential window of monitoring data.
                  For intermediate use chemicals, there may be multiple windows of use, high removal and high removal of the chemical between use.  Which this is not going to pick up.  This model is not going to -- if you define season widely, it's just not going to pick that up.  And it ties in with this tM and max W(t) term, trying to accommodate that kind of thing and not doing a good job.  So, this may require different monitoring programs to really address these kinds of uses.
                  The nice thing is, though, you have complete daily data for some of these sites.  You can actually experiment.  Well, what if I monitored closer to the season of use, higher.  You know, what does the model do with multiple seasons in the year, but monitoring that's targeted to the intermediate use chemical rather than target it to the whole season?
                  And one of the benefits of this model is that you can explore that kind of thing.  They've got it all in hand; it's just a little side experiment.  Thank you.
                  DR. ROBERT CHAPIN:  Ray?
                  DR. RAYMOND YANG:  Ray Yang.
                  DR. ROBERT CHAPIN:  Strengths and weaknesses?
                  DR. RAYMOND YANG:  What was that?
                  DR. ROBERT CHAPIN:  Strengths and weaknesses is what we're addressing now.
                  DR. RAYMOND YANG:  Yeah, yeah, yeah, I understand.  First of all, I want to compliment EPA colleagues for utilizing computational technology in your scientific and regulatory process, okay.  This is a really, in many ways, advancement of the TOX 21 initiative and so on.
                  And also, I want to compliment the effort you put in to put this White Paper together.  So not all criticism, okay?  As long as you put the model in your paper front center, like the Vecchia 2018 paper.
                  Now, I have one recommendation.  An imminent statistician, George Box made this statement.  All models are wrong, some are useful; and therefore, from that perspective, your model has weaknesses, and you need to improve it.  And from the history, since 2007, you have eight peer review -- and this is the ninth one -- is indication of that sort of process, of slowly improving.
                  And this is very, very much in the spirit and essence of Bayesian approach.  This is one of my recommendation, okay.  For EPA to consider Bayesian approach and coupling with technology such as Markov chain Monte Carlo, okay.
                  Because I don't know how these parameters are set up, whether they have probability distribution or not and so on.  But one thing is certain, there's uncertainty, there's variability.  Okay.  If you need to deal with these issues, Bayesian approach, coupled with Markov chain Monte Carlo, is one very important aspect to consider.  And thank you.
                  DR. ROBERT CHAPIN:  Dr. Green?
                  DR. TIMOTHY GREEN:  Tim Green.  My comment -- oh, good, Ken's coming back in -- is a follow up to Ken Portier's comment.  It appears to me that SEAWAVE-QEX may be over-parameterized.  So, I wonder with the statisticians in the room, why you wouldn't throw out this term that Ken discussed; or just a thought even from my EPA colleagues.
                  DR. XUYANG ZHANG:  I actually wanted to be on record to say that I don't necessarily agree with the overestimating tM, because I don't think that if tM now becomes something that is estimated, the model is even identifiable, since you have b4 times tM, that is to estimate.  You also have an intercept term, so how can you distinguish what is the intercept and what is this b4 times tM.  So, I'm not sure that the overestimating tM is possible.
                  I think that in some ways there was discussion about this issue of how to determine the time period.  The data that needs to be fed to SEAWAVE-QEX in the White Paper; when there was a discussion about the fact that for certain sites and pesticides, if you fed the whole data to SEAWAVE-QEX, then this wave was estimated to have two peaks.  When in reality, the data showed that there should be only one peak.  And you could remedy that by just specifying maybe I'm only fitting to SEAWAVE-QEX, the four months or three months when the pesticide was applied, rather than use the whole year of data.
                  I would just not embark into this exercise of doing model fitting, adding more complication to this model.  The other comment I wanted to make --
                  DR. ROBERT CHAPIN:  You're suggesting removing a parameter, right?  Right?
                  DR. TIMOTHY GREEN:  Tim.  Yes, that's what I'm suggesting, but I'm asking my colleagues for their opinion.  To me, you have redundant trend parameters.
                  DR. XUYANG ZHANG:  I also want to say that when the model is fit, it's all together, one.  It's not that you're estimating b1, and then you go a second time and you estimate b2.  You estimate everything together.  So, there is going to be some balance that is going to be reached between the seasonal wave and the linear trend to best fit the data.
                  So, I wouldn't be playing with the model as it is.  I think that the expression for the mean or the expected daily concentration is following what we expect from the hydrological processes that govern daily concentration, being as simple as possible.  Of course, you can add more parameters, but that's making your model overly parameterized.
                  DR. ROBERT CHAPIN:  I'm sorry, George, did you -- were you waiving your hand?  Rebecca?
                  DR. REBECCA KLAPER:  Rebecca Klaper.  Just a short comment, because a lot of the things that I was thinking about have been mentioned.
                  But one, going along with what I had brought up yesterday and Dr. Zhang brought up again today, about evaluating this with compounds that are not of similar chemical characteristics.  The four compounds are all relatively insoluble in water and have very similar chemical characteristics.
                  So, it's unclear -- although, I think this is a really interesting way of solving the problem and trying to estimate pesticides, it isn't clear from the evaluations that you did that this would be applicable to a water-soluble compound.  And I understand that there are a lot of pesticides that are not water-soluble; and you're more interested in the pots because they have the greater potential for cancer, et cetera, over a long period of time.
                  But it would be very important to eventually have an evaluation of how well this works for a more water-soluble pesticide from a risk assessment perspective.  Just to demonstrate that this also works, in that case, since you're trying to tie it to 1-, 4-, 21-, and 365-day exposure periods.
                  And that was just my comment from that.
                  DR. LISA NOWELL:  Lisa Nowell.  I wanted to go back to the trends term for a moment, because it's  my understanding that that's what that term is.  And this originally was the SEAWAVE-Q model.  And I mean I believe that was what this was designed to do, was to assess trends over time.
                  And with respect to something that Dr. Baffaut said, I actually would like to solicit more input on this.  Do you think this model actually does assume constant use?  Because I thought one of the functions of that trends term was to address this in an empirical way.
                  My tendency is to agree with Veronica that it's dangerous to start pulling pieces out of the model.  And I believe Dr. Portier also said that this was -- the model is complex and if you start messing with pieces of it, it's not going to work as well as it does necessarily.
                  And I also just want to mention one more thing.  I believe there is the -- of the four test compounds, there are some water-soluble ones.  Fipronil is the least so.  It's probably more different from the others -- well, I guess chlorpyrifos actually also has a fairly high Koc.  But there is some variability among the four, maybe not as extreme as if we had one of the pyrethroids included.
                  DR. REBECCA KLAPER:  It's not variable enough for --
                  DR. LISA NOWELL:  Yeah.  And it would be great to have a fungicide in there too.
                  DR. REBECCA KLAPER:  Yeah.
                  DR. LISA NOWELL:  I did notice that azoxystrobin was listed in one of the tables as having sufficient sites and data to do SEAWAVE calculations, so that would be an interesting one to take on.  However, I believe you probably used the datasets that were in the Vecchia paper, so; I don't blame you for that.
                  DR. KENNETH PORTIER:  With regard to Dr. Green's question.  In 2012, the Panel spent a lot of time talking about this term.  And Dr. Vecchia had a very good reason for putting it in the model, and I can't think of that reason right now.  So, I don't really want to recommend getting rid of the term.
                  But when you look at that tM component, you can estimate it, but it becomes a nonlinear regression.  It ups the fitting process at least in order of magnitude.  You know, you got to go to something like an (EM) algorithm or something like that to estimate it.  And I haven't had enough time to think about whether it's non-estimable.  I have to think about that.  I think it is.
                  DR. VERONICA BERROCAL:  No.
                  DR. KENNETH PORTIER:  I think it is.  So we agree to disagree.  Our first disagreement on the first question.  I think it is with the full data.
                  But I'm worried as to what it's estimating, you know, what it's capturing.  And I'd have to go back to Dr. Vecchia's paper to try to understand exactly what aspect of the curve.
                  The issue about use, I think this is more about release.  I think the idea is that use in this spatial is attenuated over time.  Not every farmer is putting it out on April 1st.  And so, application is attenuated.  And so, it's presence in this stream is kind of long-term.
                  And I think that's what this model is just trying to catch, the fact that it's used over a period of time.  It finds its way into the stream, over time, concentration builds up and then it decays out.  And that's all it's trying to capture here, is that time pattern.  Not get to too much details of mechanism.  I mean that's my impression.
                  This is a regression model.  And you know, that's what regression models do, regress to the mean.  So it's mostly focusing on mean pattern rather than on the details.
                  The unfortunate part is we're really interested in the maximum.  So, the stochastic component becomes as important here as well.  And that's where the difficulty comes.
                  DR. ROBERT CHAPIN:  Okay.  I'm sensing that we're starting to eat our own tail here.  So let me turn to our EPA colleagues and say, do you have any questions for us?  Do you have any questions for clarification that you want to go back to various parts and pieces and get clarification?
                  DR. ANNA LOWIT:  We've heard a lot on, a good hour-and-a-half on the first question.  I was hoping maybe we could take a five-minute bio break and check with my team and see if they have anything specific.  If that's okay.
                  DR. ROBERT CHAPIN:  Something quick?
                  DR. CLIFFORD WEISEL:  Well, this is a question to Dr. Berrocal.  If I heard you correctly, you said that you thought the data -- the examples in the Vecchia project paper was very good, less so here.  And you said something about maybe 12 is not the right number.  Did I hear that correctly, or was I missing something?
                  DR. VERONICA BERROCAL:  No, I didn't say that 12 is not the right number.  I said that Dr. Vecchia had in the paper different simulation experiments; where he played with how much observation do you need to have and what is the frequency, and things like that.  And basically arrived to the conclusion that if you had at least one observation, you were able to do a good job at interpolating the pesticide data concentration.
                  But 12 is the lower bound.  So, if you have more than 12 it would be even better.  And so, my comment was that I guess maybe this was a wave to test how good the SEAWAVE model was; that when they tested they use the lowest number of observation possible.
                  DR. CLIFFORD WEISEL:  And do you think they should be going -- I don't know if -- do you think they should be probably pushing it a little higher for what they're trying to use?  Or are you dissatisfied with what they have?
                  DR. VERONICA BERROCAL:  I'm hearing that maybe this is the next question; so maybe we should wait.
                  DR. ROBERT CHAPIN:  My gavel.  Bam.  Look at your watch; we'll be back in 10 minutes.
                  [BREAK].

                  DR. ROBERT CHAPIN:  So should I wait for the rest of the EPA chairs to fill in, or are you ready to go ahead and do this?
                  DR. ROCHELLE BOHATY:  Yeah.  This is Rochelle.  I think we can go ahead and go.  We were satisfied with 1(a).
                  DR. ROBERT CHAPIN:  Excellent.  All right.  Go for it.
                  DR. KENNETH PORTIER:  I was just going to say, we're no longer in disagreement.  In 30 seconds she showed me why I was wrong.  The tM term just centers the fitting of the model, so that the competence intervals on the estimates are as tight as they can be.
                  It's like in linear regression, if you set the set point at zero and you're trying to estimate the slope, you don't get the best term.  But if you set it zero at the middle, you get -- you know what I'm saying?  Okay.
                  DR. ROBERT CHAPIN:  It reminds me so much of a pre-calc teacher I had one time.  We started off January 2nd, he picked up his chalk and said, now, as I was saying, and just kept on going.
                  MS. JESSICA JOYCE:  I just want to clarify that you're saying that we shouldn't be changing the way the tM is handled.  Thank you.
                  DR. ROBERT CHAPIN:  Okay.  So, we're done with question 1(a).  Whew.  Okay.  Question 1(b).  And we'll let Dr. Bohaty read the charge question to us.
                             CHARGE QUESTION 1(b)

                  DR. ROCHELLE BOHATY:  Okay, 1(b).  EPA subsampled daily or near-daily measured concentration data to generate SEAWAVE-QEX input data that are more reflective of available non-targeted surface water monitoring data.  These subsample data were used as inputs in SEAWAVE-QEX, and the results were compared to the original source data.  Discuss the soundness of the approach for evaluating the use of SEAWAVE-QEX to develop reliable daily kymographs.
                  DR. ROBERT CHAPIN:  Okay, excellent.  I haven't been told to change the order of who talks when, so we'll just start off and work our way down.  And so, Dr. Berrocal?
                  DR. VERONICA BERROCAL:  Yes.  As it was mentioned, daily pesticide concentration of atrazine and metolachlor measured the monitors within the NCWQR were subsample by EPA to perform auto-sample evaluation of SEAWAVE-QEX.
                  The sampling frequency used to create a daily subsample was either 1 in 7 days or 1 in 14 days.  At first reading of the White Paper, I was not entirely clear what type of procedure was used to derive the subsample.  Probably also because of the fact that the White Paper described another subsampling procedure at another point in the document when describing the sampling bias factors.
                  At a second reading and during yesterday's EPA's presentation, it appeared that the strategy adopted was to use a systematic sampling approach with only the start day chosen randomly within a range of possible dates.  Between the first sampling date and the midpoint of the pesticides monitoring period.  In both situations, once the start date was selected, subsequent pesticide concentration data was sampled every 7 or 14 days, as long as 12 daily measurements were subsampled.
                  The charge of the sampling procedure was selected to obtain pesticide concentration data that are more representative of a typical sampling schedule for pesticide monitoring sites.  Five datasets of daily pesticide concentration sampled every 7 and 14 days were used as input to SEAWAVE-QEX.
                  While I understand the sampling interval of 7 or 14 days is representative of real pesticide concentration data, I believe that the inter-frequency, so sampling every 7 or every 14 days, might be disruptive of the dependent structure.
                  Daily pesticide concentration data is not an independent series of observation.  And subsampling observation should be done in a way that it preserves the original dependent structure in the data.
                  In the 2018 paper, Vecchia showed in the simulation experiments the bias and 80 percent upper-bounds were smaller when the pesticide concentration data was sampled with an inter-measurement time interval that is smaller than the concentration time scale.  And it also showed that the concentration timescale for many USGS sites was estimated to be below 5 days as it's shown in Figure 16 in Vecchia's 2018 paper.
                  I do believe that the idea of subsampling is a sound idea.  Vecchia himself, in the 2018 paper, suggested to (inaudible) the original time series of daily pesticide concentration data to avoid the SEAWAVE-QEX model is overfit.  However, I believe that Vecchia (inaudible) the data so the data was subsampled every 3 days, again, making sure that the sampling frequencies were smaller than the estimated CTS.
                  Another issue with the subsampling procedure was that the subsample data typically has lower daily pesticide concentration values as the original daily data, as we see in Figure 9.2 in the White Paper.
                  Given the smaller magnitude and concentration of the subsample pesticide data, there's no surprise that the interpolated daily pesticide concentration obtained from applying SEAWAVE-QEX might be underestimating, particularly when looking at the 365-day average concentration.  I already made a comment on this sampling frequency for charge question 1(a).
                  It was also not clear to me why, for one side in the AMP program, random daily sampling was allowed.  However, it was not performed for the NCWQS sites.
                  Finally, I believe that EPA should investigate whether the results obtained in terms of characterization of long- and short-term pesticide concentration using the SEAWAVE-QEX, just depend on the fact that only five subsamples were generated.  Or is this really a presentation of the actual performance of the SEAWAVE-QEX?  So, it was just chance, or it is something that you would see if you were doing more than five subsamples per site on pesticide?
                  DR. ROBERT CHAPIN:  Okay.  Dr. Portier?
                  DR. KENNETH PORTIER:  Thank you.  There's a little bit of overlap between -- I'll just go ahead and read my comments.
                  DR. ROBERT CHAPIN:  I hoped there would be some.
                  DR. KENNETH PORTIER:  The soundness of the approach depends on fully understanding the stratified sampling process.  Unfortunately, the sampling process is poorly described in the White Paper; with some key details glossed over, making a firm assessment of the soundness of the approach difficult.
                  Page 111 of the White Paper describes the process of creating a stratified subsample from the NCWQR data.  And I quote:
                  "To meet the suggested SEAWAVE-QEX minimum number of samples, each dataset contain 12 samples per year.  The first concentration for each subsample dataset was from a random date, between the first sampling date of the year and the midpoint of the sampling period (the start date limited to Spring) and samples were taken every 7 or 14 days, until 12 samples were subsampled for the year".
                  And to me, the statement wasn't clear, although clarified yesterday in the presentation by the EPA staff before the Panel.  The goal seems to be to create a sparse subsample that exactly captures the original observed concentration timeseries, at its selected dates that occur exactly 7 or 14 days after the initial randomly selected sample date.  These are in effect systematic samples.
                  The study might be what is called a "what if" study; designed to explore questions, such as, what if we only had weekly grab samples that started at a random day during the first halves of the season?  How does the estimated maximum concentration change if one only has biweekly grab samples?  Oh, and don't forget that one has the complete flow time series to help recreate the concentration time series.
                  This last part is critical to the effective use of SEAWAVE-QEX, since both the MTFA and the STFA are very important in constructing trend and adding variability information to the simulated traces.
                  Also important is the definition of the characteristics of the minimum sparse dataset.  Three years of data, 12 data points a year, no more than 70 percent no detects.  This means that the smallest usable dataset might only have 36 sample points, of which 30 percent or 4 or 5 datapoints are actual detects.
                  The SEAWAVE-QEX model has about seven parameters that need estimating.  While some of the parameters do get support from the flow time series, the covariates, it's still asking a lot of 4 or 5 real datapoints to estimate this much model.
                  Note that for the AEMP data, a random sampling strategy is also examined.  The White Paper, on page 14, states that the random process selects 13 values from each year but fails to indicate how this is done.  It's assumed this means selecting 13 dates, from each year at random and retaining associated concentrations; since the White Paper talks about random samples, including neighboring dates, which requires dropping some samples because of the minimum three-day spacing rule.
                  An alternative could be selecting 13 concentrations from the statistical distribution of concentrations, developed from the full dataset for that year, and then assigning those concentrations to randomly selected dates.  The two methods have the potential of generating quite different results.  And I think they should be further clarified in the White Paper.
                  The ultimate goal of the subsampling is to create synthetic sparse monitoring datasets from a complete dataset where the truth of maximum concentration is known.  Using five subsamples allows us to see how limited monitoring, combined with SEAWAVE-QEX modeling, produces variability in maximum concentration estimates and allows us to look at some confidence intervals.
                  For each subsample, one can assess the closeness of maximum concentration estimates to the true maximum, but more importantly assess the extent to which estimated confidence intervals include the true maximum.  Looking across the five subsamples gives one an idea of how far off the point estimates can be from the true values, and how often the estimated confidence bands do not include the true value.
                  Using only five subsamples limits how well this sample-to-sample variability in the estimates can be determined.  It would've been better to examine 20 or 30 subsamples and perform a more rigorous analysis of within and between variability, in maximum concentrations values generated, and report that in the White Paper.
                  The assessment of point estimates is performed by looking through the figures presented in Chapter 12, Appendix E of the White Paper.  The dot plots allow assessment of how different point estimates of maximum concentration, compare to measured maximum or true values for different pesticides, locations, and years.  The summaries are provided in Section <IP_ADDRESS>, and Figures 918 to 920.
                  It's difficult to understand the take-all message from this study.  The summary concentrates on the percent of time the SEAWAVE-QEX estimates from the limited monitoring subsamples are lower than the true measured maximum concentration.
                  The White Paper offers that this occurs approximately 15 percent of the time, which does not sound very bad.  But Figure 920 also suggests that when the SEAWAVE-QEX estimates from the limited monitoring exceed the true maximum, it's typically less than twice the true maximum, but can be as high as 19 times the true maximum.
                  No effort seems to have been applied to exploring which site or sample characteristics correlate to better or poor estimates.  Such information might further inform when SEAWAVE-QEX could be used or could be effective.
                  And we pointed out during the break that -- a couple of us were talking -- that in the White Paper, there's no real definition of what's the target for this estimate.  Is 15 percent underestimate an acceptable level?  Are you targeting 5 percent?  You know, do you want overestimates to be below 2 times the true, or is 19 times acceptable?
                  So, there's no specification of what the target for the estimation is.  And that kind of makes it a little harder for us to be able to come back and say, you did a good job.  You know, because we don't know what you were trying to accomplish at that point.
                  The difficulty of applying SEAWAVE-QEX to sparse monitoring data is tied to the general difficulty of properly estimating the seasonal standard deviation, or the SSD term, in associated autocorrelation coefficient or the correlation time lag from so few data.  And I think Dr. Berrocal mentioned this.  Getting these statistics estimated well is key to producing maximum estimates that have good performance.  The results summary suggest that when the minimum data requirements are met, the maximum estimates are within some acceptable limits of over and underestimation.  But it's not clear what happens when this doesn't occur.  You know?
                  And I suspect -- and I think Dr. Berrocal probably would agree -- that there's something wrong with the autocorrelation term.  In the way that particular sample replicate is set up, you're just not able to capture that autocorrelation term.  Which means you're either underestimating the variability, or you're grossly overestimating the variability.
                  I can't tell from this where that's happening.  And I'll leave it at that.
                  DR. ROBERT CHAPIN:  I'm trying to crystalize out a do it sooner rather than do it later thing from your collective comments.  And maybe a useful one might be to crystalize out the performance characteristics that they'd be happy with and do that sooner; right, the criteria for being a good estimation.
                  DR. KENNETH PORTIER:  So, they asked to address soundness; and we feel like it's a sound approach.
                  DR. ROBERT CHAPIN:  Okay.
                  DR. KENNETH PORTIER:  The next step is probably to understand, what's the characteristic of that small sparse subsample that caused that 19-fold maximum estimate?  There's going to be some -- something in the characteristics of that sample that's going to flag why -- and they haven't gone to that step.
                  And that's going to require generating more than five replicates.  You're going to have to generate a lot more replicates to understand where is this failing?  And that's going to then go back to inform which monitoring datasets look good that's going to work and which ones don't look good.  They don't have that information yet.  So it's going to inform the SOP --
                  DR. ROBERT CHAPIN:  And so, my sense is that might be a medium-term deliverable?
                  DR. KENNETH PORTIER:  I think that's the next step.  Yeah.
                  DR. ROBERT CHAPIN:  Okay.  Dr. Berrocal?
                  DR. VERONICA BERROCAL:  Yeah.  I wanted to add that I think, since you guys have the daily data for some of the sample -- at least almost the daily data -- I would look also into subsampling that was done not just every 7 days or 14 days, but even a shorter time interval just to see whether the performance improves when you're looking at shorter time intervals.  That might be informative for how monitoring should be done for certain pesticides and in certain sites.
                  DR. ROBERT CHAPIN:  All right.  Working our way back down the list of folks.  And we just move down the table; Dr. B?
                  DR. CLAIRE BAFFAUT:  All right.  This is Claire Baffaut.  I don't have that much to add to what has been said.
                  I would like to bring to the attention of EPA that the American Society of Agricultural and Biological Engineers, ASABE, really had a long thought process about model performance.  And we were focusing on process-based model.
                  But I think some aspect of a discussion, and the papers that came out of it, applied to statistical models as well.  There is a model calibration and validation guideline; it's called EP-621 -- and I can send it to you if you want.  And then there is a few peer-reviewed papers that go with it.
                  And in it they advise to set criteria.  And they advise to set performance criteria as a function of the model's objectives, how the model is going to be used.  And with the idea that if the model is used for regulatory or house protection purposes, then the criteria should be more stringent than if the model is used for planning purposes, for example.  I would recommend looking at those discussions and then reemphasize a need to set this criteria for this case.
                  I would like also to emphasize the need to look at the overestimation as much as the underestimation.  I understand the underestimations are needed for protecting health.  But I think the overestimations are going to be needed to get by in the model itself, for whatever it's going to be used for.  And that's important as well.  And it'll make your life much easier, I think.
                  And finally, for the purpose of comparing all those sampling strategies, personally, I would like to have the model parameters that come out of each sampling strategies, and sample all that so that I can see the difference in those model parameters.  And I can start seeing, you know, do those differences make sense or not?  Is there some rationale behind it?  And that's it.
                  DR. ROBERT CHAPIN:  All right.  Thank you.  Dr. Green?
                  DR. TIMOTHY GREEN:  Tim Green.  Just one small comment on sampling frequency.  And I certainly appreciate and acknowledge all the work that my colleagues in EPA have put into synthesizing a lot of information in the report and in your oral presentations yesterday.  Thank you.
                  This is about kind of the educational nature of your function.  What was shown in sampling frequency was, I believe, a weekly sampling that missed all of the peaks.  And that can actually provide kind of misinformation or a lack of confidence from savvy members of the public who would say, well, you chose the period where you didn't hit any peaks.
                  So that you can have a large sampling bias, instead, I would recommend that you shift your time, your start time, by a day, with the same frequency, and illustrate what happens over a range of start times.  And then you would have a more reasonable example, I think.  Thank you.
                  DR. ROBERT CHAPIN:  All right.  Dr. Zhang?
                  DR. XUYANG ZHANG:  Yes.  I -- so first of all, I'd like to acknowledge that the sampling, the current subsampling approach as representative of the typical monitoring programs.  And I think overall, the evaluation is sound.
                  In the current EPA evaluation, EPA used the two types of results in model evaluation, the summary statistics and then the full distribution.  I think these two types of model output are appropriate and representative of the exposure parameters typically used in drinking water assessments.
                  However, there are a few aspects of the approach, I think, that could be further improved.  The current approach, which compared the SEAWAVE-QEX generated results using subsamples to the logs integrated full dataset.  It's across model evaluation methods, but I think there is a risk of overfitting of the model for mainly two reasons.
                  First, the SEAWAVE-QEX always tried to generate the best fit to the data, to the particular dataset of data that the users provide.  And the second is that the step sample dataset, they're essentially from the same population, which is the full dataset.  And this does not speak to the performance of the model when applied to datasets that have different seasonal waves and trends from the current dataset.
                  In another word, there's a lot of independent validation.  I think all subsample of the datasets were used for model fitting, as I mentioned in my comments in the previous question.
                  So, I'm seeing some independent validation dataset is likely not available, I propose a cross-validation approach that could be used to include the evaluation in which a subset of the subsample dataset of set-a-side excluded from model fitting and then the set-a-side dataset are then used for model validation.
                  So the cross-validation approach, it seems the way to mitigate the (inaudible) risk, considering a more definite to the evaluation, because the datasets may not be feasible.
                  My second point is that regarding to the starting date of the sampling.  So, the current sampling approach with 7-day and 14-day intervals, they -- it'll force the starting date in the Spring.  I think that would result in a lack of data from the winter months.
                  As I've mentioned yesterday, I did ask a question about, if we force the starting date always in the springtime, the latest date that we can sample, probably around August or September time.  So the samples in October or in the winter months are not really fully represented.
                  Now, this may result in a better model fitting as the majority of sample did occur during the use season.  It doesn't reflect pesticide concentration in winter months.  And in some situations, the winter storms, it's actually the first flush after the dry period, it could often generate peaks in pesticide concentration.
                  And I think the current subsampling method may result in a lack of testing of the model for this kind of scenario.  And this is further confirmed by the artifacts of results that, as mentioned in the White Paper, that the 14-day sampling interval -- that result on the 14-day sampling interval outperforms the 7-day interval.
                  This doesn't mean that the 14-day interval samples are better than the 7-day sampling interval.  It's purely because the 14-day sampling approach covers a longer period of the year than the 7-day.  So, I think that's one of the drawbacks and that should be improved if we do not force the beginning day in the springtime.
                  In addition to the stratified sampling strategy, this is sort of related to the previous point, and also the point that Dr. Green just made.  I think a random sampling approach, which allowed flexible starting date, should be considered in the future work.
                  That's all my comments to these question.  And I'd like to get back to Dr. Nowell, her comment about the assumption of constant pesticide use.  But I'm not sure if I should address it now or later.
                  DR. ROBERT CHAPIN:  Well, you've got the floor, why don't you do it now?
                  DR. XUYANG ZHANG:  Okay.  All righty.  Yeah.  Well, Dr. Nowell mentioned that we should not assume constant usage in the model.
                  And I agree.  I didn't mean to say that the model assumes constant usage.  It's just that the use trend from year to year, when fitting the model across multiple years, because the model assumes the same parameter for that long-term trend, then it means that the model assumes the year-to-year valuation of the pesticide use follows a linear trend.  So when things happen in the pesticide use from year to year, that trend became now linear, then that would become an issue.
                  And I also wanted to acknowledge that the SEAWAVE-Q model is a very elegant and simple model.  And I really appreciate Dr. Vecchia's effort in developing the model.  My pointing out of the drawbacks, hopefully trying to bring to attention that the model should be used in situations where the assumptions can be met.  And using the models out of the context of the model assumptions could be very dangerous.
                  That's it for now.  Thank you.
                  DR. ROBERT CHAPIN:  Excellent.  Okay; thank you very much, Dr. Zhang.  Dr. Miglino?
                  DR. ANDREW MIGLINO:  Again I'll be quite short, because I think pretty much everything has been said.
                  The one thing that I'm curious about, and that I think I'd like to see investigated in the future though, is the repeatability from these graphs that you create.  Does the -- so you have five replicates in this case, right?  Can we compare them each other to say we're getting similar numbers, because we make similar trends, we make similar graphs?
                  I think that would be at least useful to determine whether it really is highly dependent on when I start this sampling frequency.  Or if I sampled well enough, I get the same numbers.  If that makes sense?
                  DR. ROBERT CHAPIN:  Is that a short-term requirement, or a midterm, or a longer --
                  DR. ANDREW MIGLINO:  I'd say midterm.  I mean, I don't think it's necessary to say, well, if the model works; I do think it is necessary to validate that the model is acceptable in the future.
                  DR. ROBERT CHAPIN:  Excellent.  All right.  And that brings us to Dr. Weisel.
                  DR. CLIFFORD WEISEL:  So again, I agree with what was said.  I just want to reemphasize one or two small things.
                  One is Dr. Zhang's point about your sampling window that you're doing is not the whole year.  And for most of the compounds, you're probably okay.  But a couple places, you talk about multiple seasons for some compound, some pesticides.  If you actually have one, then you're going to have to expand that frame.  Now, the question is, is 12 the right number?  Which I asked before.  I don't know, but that may be something to look at.
                  The other thing I found confusing is, on page 112, you talked about this; you indicated that the 14-day sample can go for 3 months, and the 7-day sample can go for 6 weeks.  And I don't understand those  numbers.  I think the 7-day sample goes for 3 months, and the 14-day sample goes for 6 months, because when I added up 12 times -- yeah.
                  So, take a look at things like that and make sure that there isn't something that I'm missing there.  But I think the approach is right.
                  And the randomized, I have less concerns, but that's probably about whether you do it every 7 days or you randomize that as was suggested because of autocorrelation.  Because I don't know enough about the autocorrelation in the water system.  But from a sampling perspective, I know more about air sampling; and what we do in air sampling is actually every 6 days.  Because we're worried about weekends and utilization.
                  I don't know if that's a problem here; whether the 7 days may always put you at the same day of the week and whether that matters in this or not.  But you should take those into consideration.
                  DR. ROBERT CHAPIN:  Okay.  Dr. Portier?
                  DR. KENNETH PORTIER:  Hi.  Ken Portier.  It's wonderful on these panels, where you think you've said everything; and then somebody mentions something, and synapses go off.
                  Dr. Baffaut mentioned this issue of estimated model parameters.  And as one looks through the diagnostic plots, the impression is that often some of the regression parameters are not significant.  And at the same time, the estimated values are quite small.  In regular regression, we drop these nonsignificant components and we refit the model.  And that small bit of variability that was explained gets dumped into the residual.
                  And as I said in the last question, that SSD term is really important in estimating the maximum.  So, when you've refit by dropping a component, you've added variability, which means you increase the probability of seeing a higher maximum.
                  So I think that's something, in the short-term, that they should look at.  Is what happens when you drop -- and in particular, some of them, the nonsignificant term is the STFA.  Which means the STFA adds a lot of daily variability to the model, and now you're taking that out of a regression component, a weighted value, and you're throwing all of that variability into the residual term.  I think you will significantly impact the maximum.  You don't think so?
                  Dr. Berrocal?
                  DR. VERONICA BERROCAL:  Yes.  I would not drop anything.  The model is fit, has a specific expression.  And when you're fitting it, it's true some of the parameters are not significant, but they're not significant maybe because for that particular dataset, the regression coefficient was not significantly different from zero.  But if you remove it, you're changing the expression of what is the expected daily concentration.
                  I would just leave the model as it is and not change the model.  And do this ad hoc where the model changes depending what pesticide and site you are considering.
                  DR. KENNETH PORTIER:  Isn't it fun when statisticians start talking?  I would argue not changing -- I'm changing the model for that site chemical, not the model in general.
                  I'm saying, in that situation, STFA doesn't add to the explanation of the temporal pattern over those years.  So, what is the use of a statistical test on the parameters if you're not going to act on that test?
                  You're still explaining some of the variability with a parameter estimate that statistically you've told yourself is not important.  So, why wouldn't you act on that, drop that term, and add that variability to the estimate?
                  DR. VERONICA BERROCAL:  No.  So, statistical models can be designed for two purposes.  One is to study association, and one is to predict.
                  So, when you're doing a predictive model, as I think the SEAWAVE was designed, because it was designed to fill in whether daily sampling data is not available, you don't just throw away pieces because they don't turn out to be significant.
                  You look at significant terms when you are interested in association.  But here, we're not interested in association in determining are these particular terms in the model significantly affecting daily pesticide concentration.
                  We're trying to figure out with these terms in the model, how should I pull up one value or pull down another value so that I get the best prediction possible?  So, if you are changing the structure of the model, you are changing the prediction.  And that's the game that we're playing here.
                  DR. ROBERT CHAPIN:  Okay.  I'm going to break in and say that I'll get you two a table for lunch.  This is totally -- reminds me of my thesis committee.  Where the prayer was that they would get in an argument among themselves and just run out the clock.
                  All right.  Other people's comments on the soundness of this approach, this subsampling approach?  Anybody else?
                  All right.  Hearing none, I'll turn to my EPA colleagues and ask if you guys are happy or if you have questions for clarification from us?  Or if you think you understand everything you've heard, statistics aside.
                  DR. ROCHELLE BOHATY:  We do.  Thank you.  That was enough.
                  DR. ROBERT CHAPIN:  Life is good.  All right.  Okay.  Shall we move on to the next question?  Let's do it.  Let's move on to the charge question 1(c).
                             CHARGE QUESTION 1(c)

                  MR. DANA SPATZ:  Please comment on the use of precipitation and stream stage data as inputs into SEAWAVE-QEX, to provide reasonable estimates of pesticide concentrations.
                  Also, please discuss the suitability and limits of using methods for in-filling missing streamflow data.  For example, waterData R package, Ryberg and Vecchia, 2017.
                  DR. ROBERT CHAPIN:  And this time, we get to start off with a hydrologist.  Dr. Baffaut?
                  DR. CLAIRE BAFFAUT:  About the using stage as a covariate, the relationship between the flow and the stages of power function, which goes well with the Log linear model.  So, mathematically, it looks pretty good.
                  The power of the relationship between flow and stage is usually not the same over the whole range of stages.  But it shouldn't matter too much because we are interested in the high concentrations, which are usually associated with high flow.
                  Note that stage will not work wherever flow doesn't work; meaning, where there is backflow effect, where there's tidal flows and so on.  In addition, it is difficult to use stage for reservoir -- and that's noted in the White Paper -- because stages may change minimally in response to flows coming in.
                  In addition, a reservoir stage is often managed and operated on by water releases from the reservoir, which have nothing to do with hydrological processes.  So, yes for flowing streams, no if there is back water effects, and probably not in reservoirs.
                  Using precipitation as a covariate.  So my problem with that is I said what I liked -- at the beginning I said what I liked about it; is that flow is an outcome of precipitation.  And as such, it integrates a lot of the processes that take place in a landscape.  Where if we use precipitation, we lose that big advantage.
                  So for example, if we use precipitation, there is nothing that differentiates precipitation on wet soils from the same amount of precipitation with the same intensity on very dry soils.  And yet, one may cause a lot of runoff in pesticide transport and the other one may cause no transport at all.  So that's the problem.
                  I realize it's a test run.  And the four Ohio watersheds show that precipitation could be used as a covariate.  However, again, on cultural land, in those four watersheds, is tile-drained, for the most part.
                  This makes a transport of pesticides from field to streams rapid.  The tiles kind of shortcut a lot of processes.  This may give an advantage to using precipitation as a covariate.
                  So, I would like to see use of precipitation tested in areas without artificial drainage and with high infiltration, where pesticides with high solubility and low absorptions have a chance to reach the aquifer and get back to the streams with the ground water.  And we would have totally different transport processes.  They would be slower.  And then I'm not sure what precipitation as a covariate is going to do in that case.
                  And for each of those tests, I would like to see the parameters obtained with precipitation stage and flow, so that we can assess the changes and whether they make sense or not.
                  And that's it.  That's all I have to say about this.
                  DR. ROBERT CHAPIN:  Excellent.  Dr. Green?
                  DR. TIMOTHY GREEN:  Yes.  Brief comment.  I retract my comment on 1(a).  I understand it's a long-term trend, so two different things.
                  On 1(c), I'm again acknowledging the work put into the White Paper, written for a broad audience.  It was a little overwhelming to process in a short time, so caveat is that I haven't read all of the supplementary information.
                  But that was useful, so I do welcome your feedback.  And what I'm focusing on here is results not really the method.  So I'll just read this.
                  Covariates, daily streamflow and precipitation, noting not irrigation -- at this point, I understand, which in Washington also may be important -- are used to both interpolate and extrapolate measured pesticide concentration to daily time series.
                  Importantly, the values of interest to estimating extremes, maximum of daily concentrations are driven by covariate maximum used to -- I'm calling this "extrapolate" -- concentration values.
                  In the White Paper, on page 123, Figure 9.10, which is an apparent worse case, this illustrates an issue of estimating extreme values.
                  In this case, the maximum measured concentration, visually from the figure, is less than 10 micrograms per liter.  One realization is shown in the figure, with a maximum of 100 micrograms per liter, which is not near the medium.  And then the estimated annual maximum concentration from all 100 realizations is approximately 10,000 micrograms per liter.
                  So, all these were estimated visually from the figure, but this illustrates kind of the issue of estimating extremes.  Extremes from covariate relationships, in particularly, that impact these Log normal distributions, where the 80 percent confidence interval spans the full order of magnitude of estimated maximum daily concentrations.
                  As noted, Figure 9.10 is likely a worst-case scenario, which is affected by data sampling and censoring noted in the document.
                  Even so, a much better, maybe a best case, is shown in Figure 9.5 on page 118.  And that illustrates the same issue, but to a lesser degree.  That is the maximum measured concentration always falls below or maybe at the lower limit of the 80 percent confidence interval for annual maximum concentrations estimated using SEAWAVE-QEX.  Which, again, is driven by the covariate data streamflow, I believe in that case.
                  Likewise, all of the plots, which I reviewed in the supplemental files -- thank you again for those, and I can give you the folder where those are -- display these features, and its measured concentration maximum in each year fall below the 80 percent confidence interval of estimated annual maximum concentrations.
                  So, these comments are made in the context that estimation of extreme values is a very difficult statistical problem.  Other regression-based methods for extrapolating extreme values beyond measured values of concentration will likely face similar issues.
                  These comments are not intended to exclude use of streamflow as a covariate, but to caution -- and note that caution is warranted -- when none of the measured concentrations fall within the annual competency intervals; which seems to be a common case.
                  So those are concerns about, I guess, short-term interpretation.  Longer-term advances will likely demand more biophysically-based flow -- which Dr. Baffaut has referred to -- and transport models, which would only be feasible maybe in tier 4, or if you invented a new tier 5 approach to some data-rich watersheds as a proof of concept to improve -- again, it's all about confidence in the limits of estimating maximum concentration.  That's all I have.
                  DR. ROBERT CHAPIN:  Okay.  Excellent.  Dr. Kennedy?
                  DR. IAN KENNEDY:  I tried to review some of the things.  And I found it not as clear as I was hoping in this case.
                  One thing is that because precipitation on probably most days in most regions of the country is zero, the Log transformed that is done by SEAWAVE-QEX doesn't work.  And as a result, you changed it, basically, into a cubed root.  I'd kind of like to see some sort of explanation about why that was done over say, just putting in a default value; which I guess SEAWAVE-QEX has built in of .1 -- was it?  I don't remember exactly.
                  I didn't really understand why that was done over some other method.  And why a cubed root was chosen instead of something else.
                  I guess my other comment is that a lot of these were done on the NCWQR sites.  And those, as Dr. Baffaut mentioned earlier, are all in the same general area.  They're all going to have the same precipitation characteristics, similar flow.  I'm not sure how well the precipitation would work, in a more arid environment, where you have even more of those zeros in the precipitation.
                  If you use the dayshift concept and kind of move the precipitation along more slowly, to generate a kind of pseudo-flow, then that's going to remove some of those.  But it probably will leave some zeros still there.  And that's going to depend on where you are.  Also, areas that have more sudden precipitation might behave differently.
                  Finally, you asked about the infilling some of the data, which was missing, using the waterData R package.  And that package has a function called fill/miss, that will fill missing data in the time series.  And it seems to do it mostly just by linear -- something close to linear interpolation.
                  Maybe someone else can answer this better than me.  But it seems that maybe for a few missing values, that would work.  But the default block of missing values that it will handle is set for 30, which seems like it's way too much for that sort of interpolation.
                  And it think that's all I have to say right now.
                  DR. ROBERT CHAPIN:  Okay.  Dr. Miglino?
                  DR. ANDREW MIGLINO:  You're probably used to it; this will be brief.  I agree with what's been said so far.
                  Dr. Baffaut and I seem, I think, to have a brainwave thing going on here because she basically said exactly what I had written down.  And that complexity of you've analyzed only really one set of sole characteristics and drainage characteristics in one small area of the country.  That's a real potential issue down the road, right?
                  If we had a large dataset we could really determine what is going on.  And maybe it performs very adequately everywhere you apply it.  It's just not known to us.
                  And beyond just arid and Ohio and Idaho, some mixed-use watershed kind of data, right, the PCA, Percent Cropped Area, crops up in all your previous tier work, that may be applicable here as well; to say that there is some level of urbanization in the watershed, which would have vastly different characteristics than cropland.
                  So, the only other thing I'll say is about the inflow filling -- or infilling missing streamflow data.  I agree with what Dr. Portier has said.  It seems fairly reasonable over shorter timespans than 30 days; although, perhaps there's some justification out there for waterData to be applicable to those large timespans of a month.
                  I'd be a little more hesitant to kind of reach out to those extremes.  But it seems reasonable over shorter terms for filling things in there.  And I think a lot of this is going to carry over into the next discussion, the next question, where maybe things are a little more sparse.
                  DR. ROBERT CHAPIN:  All righty.  Excellent.  Dr. Potter?
                  DR. THOMAS POTTER:  Okay.  Well, again, thanks to the colleagues, the discussants who have spoken before me because they've obviously covered a lot of ground.  And I think they've emphasized the critical points regarding the variability of rainfall runoff relationships.  And those certainly need to be taken into account in evaluating the potential for precipitation for use as a surrogate.
                  There appears to be some cases where it worked quite well.  And so, further analysis of why it worked would appear appropriate.  It may have to do with the shape and size of the watershed, et cetera, a whole series of other factors, including tile flow.  So, certainly, again, a detailed assessment there would seem appropriate.
                  With all of that said, I will say that the use of precipitation as a surrogate for flow appears to reduce confidence in results.  So, I guess I would ask the question, is it appropriate to use?
                  Your confidence is the lowest among -- lower than, for example, stage and certainly less than flow.  So, I think it presents a communication problem, among other things, that you really need to take into account as you go forward with this.
                  DR. ROBERT CHAPIN:  Let me just interrupt and say, what they are looking for from us are recommendations, not questions.  So, if you've got a recommendation not to use it, that would be more useful --
                  DR. THOMAS POTTER:  That's my recommendation.  Yeah.  Yeah.  I think I stated that right up front when I said it reduces the confidence in results, and therefore you need to question why you're using it.
                  Okay?  And that's my position on it.  We can get into all the reasons why.
                  DR. ROBERT CHAPIN:  That's helpful.
                  DR. THOMAS POTTER:  Yeah.  And I will say that there are other regression models out there.  And I don't know if we've discussed them at all during this meeting.  But USGS has a WARP model, a watershed area regression model.
                  And interestingly, in that model, they found that Dunne overland flow, which is a percent of rain discharge as runoff, was very informative when describing atrazine losses; also, the presence of the sole restrictive layer, and a whole bunch of other factors.
                  I would encourage you to take a look at the experiences of -- that were gained by that modeling effort, because I think it further informs the use of precipitation in this case in the model, and ultimately may guide decisions regarding the function that's used to describe precipitation.
                  And I understand at this point that it's a cube root.  I'll make a note that regarding use of precipitation, you may want to consult with a SEAWAVE model, with Dr. Vecchia and others who have worked on this project.
                  I read in his report and he states -- I can give the exact page -- but he states that if daily streamflow are not available, surrogate value variables, computed using estimated precipitation from the watershed, may be considered in place of streamflow.  Yeah.  I mean, that sounds reasonable.
                  And he goes on to cite Johnson et al.  So I actually went back and dug up the paper -- again, I'll provide the citation.  And in this paper, SEAWAVE-Q was used to describe pesticide monitoring data in a number of sites in the western US, in particular California.
                  And in the model, transformed precipitation was used, but was used in conjunction with flow.  And there was a parameter that basically -- describing flow that took into account the 30-day antecedent precipitation, divided by the 5-day antecedent precipitation.
                  To me, that makes sense because it gets back at the question that Dr. Baffaut -- or point that Dr. Baffaut made regarding whether or not you have rain on wet or dry soils.  And antecedent precipitation certainly plays an important role there.
                  I suggest you -- again, maybe you can go back, and there's some further insight that would be gained by talking to the SEAWAVE developers, in particular, Dr. Vecchia, and also, Johnson et al. and that research group in California that used an alternate transform for precipitation.  It seemed to work quite effectively, again, combined with flow, not used separately.
                  We've brought up irrigation several times.  You know, again, I think it's an important factor for a lot of different reasons; including the fact that irrigation can have a strong impact on runoff in subsequent storms.  Again, rain on wet soil, et cetera.  So, I think that needs to be factored in, in some way.  Perhaps there can be a -- I hate to say this, but another parameter that can be added or it could be lumped in some way with either flow or precipitation to gain some insight in there.
                  I will say that well-timed irrigation incorporates pesticides into soil and reduces the amounts available for runoff.  And we led research effort in that area for a number of years.  And certainly -- it's actually a good conservation practice.  So, it's something to consider and contemplate.
                  What else do I have to say here.  Oh.  I will say that, you know, if we're, again, looking at this we ought to focus efforts on static water systems.  So, we're going to get into that into the next section.  But, you know, whether or not it's appropriate.  Either stage, which we've heard is probably a challenge to use in a static water system because of the small variation in stage.  Or the precip, indeed, is appropriate to use in a static water system.
                  I think priority needs to be given to that area.  And we'll talk about that, I think, in the next question.  I agree that the infilling approach is useful.  And there are a lot of ways we can attack it, but I think you've -- you certainly have a technically defensible approach there and it seems reasonable.
                  You know, one of the things we have to remember is that there's a lot of data out there that you may want to use.  Whether it's flow data or what not, that's provisional.  And so you're going to have to make a decision about, you know, whether or not provisional data is used.  The geological survey has a very nice disclaimer, which they say, all data is provisional until it's not.  So, I think that's reasonable to take into account.
                  And I think I'll stop there.
                  DR. ROBERT CHAPIN:  Wonderful.  Please do.
                  DR. REBECCA KLAPER: Tom, since one of your points is that the precipitation doesn't necessarily -- it's just not good in the model, probably for a variety of reasons, right, it goes through all sorts of different landscapes before it gets to that river.  Wouldn't irrigation, then, also be bad to add to that model?  Because essentially, it's false precipitation, it doesn't necessarily add -- it's streamflow and it would capture what kind of irrigation ended up in that stream.
                  DR. THOMAS POTTER:  Yeah, I admit I was going out on a limb on that one.  I'm not suggesting that we add to the model.  I'm just saying that somehow, if we indeed, we're going to consider precipitation in the model, we have to consider other water inputs on the landscape.
                  And that's irrigation.  So, certainly that needs to be taken into account in some form.
                  DR. ROBERT CHAPIN:  Like my professor said, it's complicated.  Okay.  Mr. Councell?  Go ahead.
                  DR. LISA NOWELL:  Hi.  It's Lisa Nowell.  Without disagreeing with all of the things that you said, all of the complicated real factors, I'm just sort of being devil's advocate here, I think.
                  If we don't have flow, you can't use it?  Is that absolute?  I mean that's basically the question, because we don't have flow most places.  So, if they can't use it with precipitation, then it'll make a big difference in how usable the model is.  Just from a practical point of view.
                  And in terms of the work model, I just want to add one qualifier; which is that that generates a 4- or 21-day moving average.  So, it's not going to get those high estimated concentrations, which is what this whole effort is kind of pushing at.
                  I know that they used their work model or have used it before.  So, it's a good suggestion, but it's also got limitations.  They all do.
                  MR. TERRY COUNCELL:  All right.  And Dr. Nowell stole part of my thunder.  And that was what I was going to say, is back when I was with USGS, the number of sites where they were measuring flow and discharge was shrinking.  And I understand with frustration and further budget cuts, that those number of sites are continuing to reduce and get smaller and smaller.
                  So therefore, the usability of your data, and being able to use this model and have that flow data, is going to keep getting smaller, and smaller, and smaller.  And so, you have this nice neat model, and you're going to be restricted even further with what you can use it on.  So having something to fall back, maybe precipitation or something else, is good.
                  One of the challenges that you have is that our country spans an entire continent; and we have all these ecoregions and climate regions, and it's a huge mess.  And it's very difficult to deal with.