DR. ANNA LOWIT:  Anna Lowit.  Just a couple of things from a few minutes ago.
                  So Dr. Yang, when you asked us about the opinion of Crop Life, I think that's a question for them and I don't think we'll cover at any point what Crop Life is going to think.  So I just -- make sure he that wasn't expecting use to comment on their comments.
                  And as a federal regulator, it makes sense to start asking implementation questions, like, how are going to use it?  The number is going to change ever time, you know.  There are a lot of policy and implementation questions that become obvious when you think about this, but understand that where we are right this second is that we've been working with the developers of SEAWAVE-QEX and the thought of moving more fruitfully towards sampling bias factors and so we're really in the methods development with this evaluation and how that would actually in a real assessment through public comment and to do a risk mitigation decision using these tools is really not part of what we're talking about today, although it's obvious in your mind of how it would actually work in process that, you know, a lot of the conversations about the, you know, the way the model is working and the number of simulations.  These are the kind of recommendations we're really looking for, but, you know, we're not going to have a lot of answers on how you're going to use it and what's it going to look like in the risk assessment because, you know, I think we're still working through that, you know, if we're honest.
                  DR. VERONICA BERROCAL:  Yes.  Dr. Berrocal.  So, I have a question about the chemograph.  I understand that the model is fit to the data, the parameters are estimated so those regression coefficients and then the standard deviation of the residuals and the correlation approximation.
                  And this simulated condition of simulation are generated by simulating the residuals, right?  But there is uncertainty in the parameters of the models.  Do you generate also random values for these parameters using the, you know, distribution that you get when you do maximal likelihood; when you estimate the parameter there is some standard data attached to it, or you just keep them fixed and you just simulate the residuals?  Am I being too technical?
                  DR. CHRISTINE HARTLESS:  This is Christine Hartless and no, we're not doing any kind of a nested Monte Carlo in going back and evaluating the distributions and simulating from the distributions of the estimated parameters within SEAWAVE.
                  And I also want to add, further clarification, in that when we're evaluating the SEAWAVE and you fit the data from one particular site using the SEAWAVE-QEX model, and this component is included in the SOP and there's also additional information in the Vecchia paper about the steps that we go through and it is recommended to go through to evaluate the fit of the model and determine whether or not these data are appropriate for using SEAWAVE-QEX.  There are some monitoring sites where we decide these data just don't fit and we're not going to be able to utilize SEAWAVE-QEX and there's some additional tools and guidance provided in those.  And I don't think that any of that was incorporated in today's presentations because it's getting very down into the weeds and we wanted to try to present to you all the bigger picture of what we're doing.  But that information is included in all of the affiliated SAP documentation.
                  DR. ROBERT CHAPIN:  Okay.  I think we're kind of going to be done with questions for now, and we'll turn it back over to Dr. Hafner to lead us through the second part of her business here.

EVALUATION OF SEAWAVE-QEX AS AN IMPUTATION TECHNIQUE FOR ESTIMATING DAILY PESTICIDE CONCENTRATIONS FROM PESTICIDE MONITORING DATA PART 2

                  DR. SARAH HAFNER:  Okay.  So I'll now continue with part two of the SEAWAVE-QEX evaluation which is the result.
                  So this is the outline for the second part.  First, we will go over the results for the accuracy evaluation and next discuss the alternative covariate evaluation and the use of SEAWAVE-QEX for alternative waterbodies.  We'll end with a summary of the results, discussion of the challenges the EPA faces in adopting SEAWAVE-QEX, and EPA's proposed approach for using the model.
                  So as described earlier in the methods of data preparation, there were three parts to the accuracy evaluation of SEAWAVE-QEX.  First, to determine how frequently concentrations are underestimated by the model compared to the measured concentrations.  Second, determine summary statistics of estimated concentrations that correspond with measured concentrations.  And third, plot the distributions of estimated concentrations with the measured concentrations.
                  First, the underestimation rate was viewed by combining replicates and determining the number of years that SEAWAVE-QEX underestimated the measured maximum for each year.  For this presentation, I'm going to walk through a few comparisons of the data showing the underestimation rate for the 1-day and 365-day averages at two sites, Honey Creek and Maumee River.
                  Although the 4-day and 21-day averages are also presented in the White Paper as well as data for Rock Creek and Sandusky River sites, the data are truncated for the purposes of this presentation.
                  The first comparison is between using streamflow and precipitation as covariates.  The 1-day average is shown in blue and the 365-day average is shown in black with streamflow on the left in solid and precipitation on the right in cross-hatch.  This slide is showing metolachlor results.
                  When a bar appears to be missing, as is often the case with precipitation here on the right, it means that the underestimation rate is zero percent or that SEAWAVE-QEX never underestimated these values.  This data suggests that SEAWAVE-QEX is underestimating somewhat more frequently with streamflow than precipitation for this set of data although the results are more similar between the two covariates for atrazine as is shown on this line, where the rate of underestimation for streamflow on the left is similar to that for precipitation on the right.
                  The next comparison I'm showing is the 1-day against the 365-day averages with flow as covariate for atrazine on the left and metolachlor on the right.  As before, the one-day average is in blue and 365 is in black.
                  A broad trend can be seen in the data that can be seen through these figures that the 365-day average seems to be underestimating more frequently than the other averaging periods which is seen by the black bars being larger than the blue bars more frequently; however, this may be an artifact of the interpolation method that was used to derive a 365-day average for the measured data since the data was robust but not completely daily year-round.
                  While the 365-day average underestimated more frequently than other average periods, the magnitude of underestimation was at most 50 percent or half the concentration of the measured maximum.  This is similar to the magnitude of underestimation that was observed for the 4-day and 21-day averages.  When the daily average was underestimated, the estimated concentrations were above 20 percent of the measured maximum.
                  The final comparison here is between the 7-day stratified sampling on the left and the 14-day stratified sampling on the right shown here for metolachlor.  The 14-day stratified sampling strategy is somewhat less likely to underestimate than the 7-day which may be related to the 14-day sampling strategy representing a greater part of the year since both strategies were capped at 12 samples per year regardless of the sampling interval, meaning that the 7-day sampling strategy would span 12 weeks each year whereas the 14-day sampling strategy spanned 24 weeks in a year.  It is important to note that this difference would not be expected if the sampling windows were a fixed length rather than fixing sample number.
                  So viewing all the data combined broadly, the rate of underestimation was less than 30 percent for all average periods for both atrazine and metolachlor across sites using either flow or precipitation as a covariate.  With few exceptions, the rate was less than 15 percent.  This suggests that SEAWAVE-QEX is capable of capturing peak concentrations most of the time.  The magnitude of overestimation is not considered at this point as overestimation is considered more easily corrected than underestimation.
                  The next part of the accuracy evaluation was to overlay summary statistics from SEAWAVE-QEX estimates with maximum concentrations from the measured data.  The following slides will be showing an example of the measured maximum concentrations for each year compared to the summary statistic described earlier.
                  DR. LISA NOWELL:  This is Lisa Nowell.  Sorry to backtrack just a little bit.  I wanted to ask for clarification of something I think you said.  Correct me if I'm misstating this.  It had to do with -- I think you said that the underestimation was worse for the 365-day average than the 1-day and that that may be an artifact of the method you used to interpolate your measure data.  Is that what you said?
                  DR. SARAH HAFNER:  Yes.
                  DR. LISA NOWELL:  And would you mind clarifying that?
                  DR. SARAH HAFNER:  Yes.  So if you -- well, so for the data that we used for this, it's daily sampling in the summer months, but in the off-season it's biweekly sampling.  So for the 1-day, 4-day and 21-day averages, the maximum for year is generally in the summer during the peak season and so that's calculated mostly from daily sampling.  But to calculate the 365-day average, we did have to do some infilling.  So we did log linear interpolation for the 365-day measured concentration that we're comparing SEAWAVE against.
                  Does that answer your question?
                  DR. LISA NOWELL:  Thank you.
                  DR. SARAH HAFNER:  Okay.
                  DR. THOMAS POTTER:  Oh, question over here.
                  DR. SARAH HAFNER:  Where?
                  DR. THOMAS POTTER:  Tom Potter.  I want to go back.  I don't really know a whole lot about the particular watershed you're looking at.  I assume they're quite runoff prone since precipitation is seemingly performing equally to flow so there must be a reasonably strong relationship there.  Is there data out there that describes what the hydrologic characteristics are of these watersheds are and we can delve into that in terms of seeing why this relationship exists here and whether it could be extended elsewhere?
                  DR. SARAH HAFNER:  There is a lot of information out there on these sites.  The Heidelberg NCWQR.
                  DR. THOMAS POTTER:  Yeah, I'm aware of where they're coming from and, you know, I guess part of it is let me ask a question.  Did you look in some detail here to kind of do some ground truthing here as to why you're seeing this, what appears to be a very good relationship with precipitation, which I have to say, I remain a skeptic.  So --
                  DR. SARAH HAFNER:  No, we didn't.
                  DR. THOMAS POTTER:  Okay.
                  DR. SARAH HAFNER:  We just used the best available data.
                  DR. THOMAS POTTER:  And we'll get into this later on, but, you know, best available data for rainfall is really, you know, I have to say, a can of worms.
                  DR. SARAH HAFNER:  Oh, I meant pesticide data.
                  DR. THOMAS POTTER:  Yeah.
                  DR. ROBERT CHAPIN:  Okay.  Here we go.  Dr. Hafner.
                  DR. SARAH HAFNER:  Okay.  So in the next slides I'll be walking through an example of the measured maximum concentration for each year compared to the summary statistic that was described earlier at the maximum of the 99th percentile for all subsample data sets.  A model that estimates concentrations well should neither consistently over or underestimate measured concentrations and so the most reasonable estimate of the true concentration could be viewed as having the most balance between over and underestimation of the measured annual maximum across replicate data sets with few outliers.  This is demonstrated for the maximum of the 99th percentile concentrations and generally observed in the data for other sites covariates in both chemicals.
                  So I'm going to briefly walk through an example figure that demonstrates this concept.  In this figure, concentrations on the Y-axis and years of the simulation on the X-axis.  The yellow bars are the maximum measured concentration for each year.  Each blue circle within each year represents a different seven-day stratified subsample data set based on the subsample process described earlier so that there are five total seven-day subsample data sets overlaid for each year.  The same is true for the 14-day stratified subsamples shown as green diamonds.
                  When the annual summary statistic for all of the subsample data sets are overlaid on the maximum measured concentrations, there is not consistent over or underestimation of the measured maximum as there are estimated concentrations both above and below the yellow lines.
                  It should be noted that this approach looks broadly at the patterns of estimated versus measured concentrations for many subsamples in the same data set and that a risk assessor would only get one data set from a site.  And so in some cases using this approach, a single data set may miss the peak concentration.
                  Similar figures were developed for rolling averages; that is, 4-day, 21-day, and 365-day averages.  These are presented in the White Paper.
                  The last part of the accuracy evaluation was comparing the full distribution of raw SEAWAVE-QEX estimates to the measured concentrations.  This figure shows an example distribution for metolachlor at Maumee River.  The orange-colored band is made up of 100 distributions of SEAWAVE-QEX estimates from one of the subsample data sets.  The black line overlaid on top of the colored band is the single distribution of the measured concentrations.
                  Estimated distributions that fall to the left of the measured concentrations are underestimated and those to the right are overestimated relative to the measured.  To walk through an example, when looking at the one microgram per liter concentration on the X-axis, this corresponds to the 70th percentile on the distribution for the measured concentrations.  This indicates that 70 percent of the concentrations in the original measured data set were less than one microgram per liter.
                  Next, look at one of the SEAWAVE-QEX estimated distributions that falls to the right of the measured distribution and you can see that these cross the one microgram per liter line below the 70th percentile line.  For example, the distribution indicated here with an arrow is at the 60th percentile indicating that only 60 percent of those estimated concentrations were less than one microgram per liter and so overestimating concentrations relative to the measured.
                  Conversely, for a distribution to the left of the measure, the one microgram per liter line crosses above the 70th percentile meaning that more than 70 percent of the estimated concentrations are less than one microgram per liter and so overall, the concentrations are underestimated relative to the measured.
                  For the distribution analysis, all five subsample data sets for each chemical, site, and covariate combination were overlaid with the measured data distribution.  This is shown here with each colored band representing one of the five subsample data sets in each subsample data set representing distributions from 100 SEAWAVE-QEX chemographs.  Generally, the distributions matched well with the measured concentrations across chemical sites and covariates.  Often when there was variability between the five subsample data sets, it occurred at low concentrations such as below the limit of detection.
                  Both flow and precipitation were found to be suitable covariates.  Occasionally, there was some extreme tailing in the distribution as seen here.  Both flow and precipitation were subject to tailing on some distributions though using precipitation tended cause a few more extreme concentrations.
                  In addition to the more extensive quantitation comparison of flow and model precipitation, the diagnostic plots were evaluated when using various other covariates available.  The two plots that were evaluated in particular relate to mid-term and short-term flow anomalies to the estimated concentrations in SEAWAVE-QEX.  Note that the concentration on the Y-axis is in log base 10-scale and the flow anomaly on the X-axis is a unit-less value.
                  We compared the similarity of these plots created when using flow as a covariate to those when stream stage or precipitation was used as a covariate.  For model precipitation, which was a more extensive comparison because of the number of data inputs, we found that both flow anomalies behaved similarly as with streamflow.  This supports the quantitative evaluation that model precipitation is a viable covariate.
                  DR. RAYMOND YANG:  Ray Yang.  What are the open circle and solid circle?
                  DR. SARAH HAFNER:  The closed circles represent the measured data inputs to SEAWAVE-QEX and the open circles are still the censored data points.
                  DR. RAYMOND YANG:  The open circle is what?
                  DR. SARAH HAFNER:  The censored data that SEAWAVE is --
                  DR. RAYMOND YANG:  Censored data.  An the model is the straight line?
                  DR. SARAH HAFNER:  That's just a regression line.
                  DR. KENNETH PORTIER:  In the material I gave you -- this is Ken Portier, Dr. Yang.  In the material I gave you, so that the slope of that line is the Beta 2 or Beta 3 term in that regression.  That's all they're showing here is that part of the decomposition of the whole SEAWAVE model.
                  DR. RAYMOND YANG:  All right.
                  DR. SARAH HAFNER:  Stream stage and measured precipitation were also evaluated, though on fewer data sets.  Stream stage matched with streamflow very well which is expected as streamflow is often correlated with stage measurements.  Measured precipitation was also similar to flow in this evaluation, although deriving the data is much more difficult than monitoring stage or estimating precipitation.
                  The use of SEAWAVE-QEX on waterbodies with intermittent flow or no flow was assessed with a few data sets from the atrazine monitoring program or AMP data.  These sites are of particular interest to use since water monitoring in reservoirs would be highly relevant to drinking water assessments.
                  For this assessment, the diagnostic plots were evaluated for an appropriate model fit.  Despite the lack of seasonality in some of the data sets, SEAWAVE-QEX diagnostic plots were reasonable for most of the sites.  Sites that did not have reasonable plots typically had an underlying uncertainty in the data that might explain poor fits.  This is primarily because the samples were collected in such a way that the source water watershed is not always known as several sampling sites have multiple intakes from which treatment plants may draw at a given time.
                  It is unclear at times from viewing the data if a sudden change in concentration is real or if the source is switched which can affect the model results; however, for some sites, it's clear that SEAWAVE-QEX fits the data well and could be used to estimate concentration on these types of waterbodies.
                  Overall, throughout our analysis, SEAWAVE-QEX proved to be a robust tool with minimal data inputs.  The model does not frequently underestimate the measured maximum in each year and the magnitude of underestimation is low for most averaging periods.  Our method of selecting summary statistics from the SEAWAVE-QEX chemographs shows promise for estimating the true concentration, although when choosing a summary statistic, there is a tradeoff in the frequency of overestimating versus capturing the peak in every data set.  The distributions also matched most of the measured data while with some tendency to overestimate at the tails, particularly with modeled precipitation.
                  As for choosing a covariate, streamflow is always preferable if it is appropriate.  Cases when streamflow would not be appropriate may include waterbodies with periods of stagnation or backflow.  Stream stage would be the next best alternative covariate followed by measured precipitation and model precipitation.
                  While SEAWAVE-QEX can be used on waterbodies with irregular flow patterns, this is less straightforward than flowing systems and in systems with higher frequency sampling with low or no flow, there may not be much added value to using the model to produce daily concentrations if there is high correlation between sampling events, for example, with persistent pesticides in low flowing systems.
                  While SEAWAVE-QEX is a powerful tool, there are several challenges to adopting it, particularly in a regulatory setting.  SEAWAVE-QEX requires more pesticide data than is available for most monitoring sites needing three or more years of data from individual sites with roughly 12 or more samples per year at 30 percent detection rate.  Additionally, while SEAWAVE-QEX is simple to run, there's a steep learning curve to analyzing the diagnostic plots and optimizing the input parameters which leads to subjectivity and its use for risk assessment.
                  SEAWAVE-QEX was also designed for use with specific types of data such as flowing systems and daily grab samples which is not representative of all water monitoring sources available for inclusion in drinking water assessments.  Monitoring databases include a range of waterbody types including flowing and non-flowing sources such as reservoirs which are highly relevant for drinking water, and a range of waterbody flashiness.  Monitoring programs also vary in sample collection methods and complimentary data such as the availability of streamflow data.
                  A large obstacle to consider when utilizing any monitoring data is the time and resources allocated to cleaning up the data.  This includes filtering and formatting large data sets without excluding valuable data by filtering too broadly or including inappropriate data such as non-surface water samples, passive samples, duplicate data sets that are reported by different agencies, and data with incorrect units.  This data processing is made more challenging when trying to determine if a data set can meet the minimum data requirements for SEAWAVE-QEX since there is flexibility in these requirements.
                  There's additional time used for processing or deriving covariate data before actually running the model and ultimately, time is needed to describe the site relevance based on the vulnerability of the site to a pesticide of interest.
                  Because SEAWAVE-QEX would be a valuable tool to have available for use in drinking water assessments, we've taken several steps to address the challenges that we expect to most hinder our use of the model.  First, to address the challenge of subjectivity in using the model, we've developed a standard operating procedure for running SEAWAVE-QEX and interpreting the diagnostic plots.
                  We have also developed a weight of evidence approach for determining site relevance for use in drinking water assessments which will be discussed in a later presentation by Christine Hartless.
                  In order to maximize data that could be used in SEAWAVE-QEX, the model was only evaluated on flowing systems with streamflow as a covariate, but also using alternative covariates to streamflow including precipitation and stream stage.  The model was also tested on systems with irregular flow including low flow and non-flowing systems.
                  Our use of the model showed that although SEAWAVE-QEX requires some effort to use, it can be adapted to the risk assessment fairly easily with summary statistics from daily concentrations used in deterministic assessments and distributions used in probabilistic assessments.  Additionally, surrogate covariates can be used when streamflow is not available for historical data or when streamflow is not appropriate.
                  Overall, due to the level of effort and spatial resolution require, SEAWAVE-QEX is considered a tool for use in Tier 4 drinking water assessments for pesticides needing the highest level of refinement.
                  And I can now take any more clarifying questions.
                  DR. CLIFFORD WEISEL:  This is Cliff Weisel.  I have two that I want to pose.  One is I thought I heard you say that was easy to deal with when you overestimate rather than underestimate.  Is that easier to deal with from a perspective that you care less about those or is it easier to deal with mathematically?
                  DR. SARAH HAFNER:  More mathematically.
                  DR. CLIFFORD WEISEL:  Why is it easier to deal with?
                  DR. SARAH HAFNER:  By considering the summary statistic, it's easier to choose a different percentile, but you can't get higher than the highest concentration that you have.
                  DR. CLIFFORD WEISEL:  Okay.  The other question I have has to do with you're talking about when you have multiple sources and how that might affect the data.  All right, so this is a -- I'm beginning to understand what's going on as a statistical fit of your measurement data trying to incorporate something else that would help you evaluate that.  So if you do have multiple sources, that's a real situation, and I don't think it's a rare situation.  So what do you think might be the approach?  Would more monitoring data be useful?  Would it be data that helps understand why you have multiple sources and what might you need to do to try to get at that better?
                  DR. ROCHELLE BOHATY:  So that particular comment comes up for the monitoring data set that was relevant to community water systems.  So it's not uncommon for community water systems to have multiple intakes.  And so from my understanding, that data set, the sample is taken at like the point of entry for the plant.  And so maybe for a better use of our tool would be to sample at the intake location so that that way you know exactly where the data were coming from.
                  DR. CLIFFORD WEISEL:  Yeah, but many streams -- I mean, you're looking at surface water in general as opposed to the surface water at the plant, intakes for plants, or is the latter really what you want to think about more?  Because they're not the same and if you're trying to -- the way this was presented was to look at surface water and then as a secondary situation when you reach the point that there may be a problem, okay, let's also see is it really impacting what's going through the drinking water system?  So those are slightly different questions.
                  DR. ROCHELLE BOHATY:  So we pulled in that extra data source because a lot of times the questions that we get relevant to the other sources is what is the relevancy to community water systems.  So that data set is specific for community water systems.  So we started looked at that data set.  We looked at the flowing.  It seemed like they worked well for those flowing systems and then we moved on to looking at those, like non-flowing systems and low-flow.
                  And so you're right.  When the model may not be appropriate and likely isn't appropriate for use if we are getting measured values at the intake location in terms of fitting the model, but the relevancy to drinking water would remain for those measured concentrations.
                  DR. TIMOTHY GREEN:  I was essentially going to ask a similar question, but a follow-up --
                  DR. ROBERT CHAPIN:  Your name, Tim, your name.
                  DR. TIMOTHY GREEN:  Tim Green, sorry.  Many drinking water sources are coming from reservoirs or low-flow and non-flowing sources.  So to what extent have you tested -- you've shown cases of primarily rivers and streams.  What's happening with reservoirs and lakes, and what is your confidence in SEAWAVE, I guess?
                  DR. SARAH HAFNER:  So that data set, the AMP data, is the only one that we've used for the non-flowing system.  And unfortunately, it does have those issues where the samples were taken at the treatment plant so there is some uncertainty in that data.  But for the ones that seem to work, and it doesn't have this concentration, or it doesn't look like the source is being switched, the model does seem to work, but it's a very flat wave.  You can look in the supplemental data to look at it if you'd like.  So it almost seems that you may not, at least for the frequency of that sampling -- for that program, it's weekly sampling -- that the tool may not, for a persistent chemical, be as relevant.
                  DR. TIMOTHY GREEN:  Thank you.  So is there, going forward, a strategy that differs between flowing and non-flowing if we call them that?
                  DR. SARAH HAFNER:  I believe we have charge question asking if you guys know of any more tools.
                  DR. TIMOTHY GREEN:  Well, that's why I'm asking the question is to understand your direction and what your plans are before we make up input to you.
                  DR. VERONICA BERROCAL:  Dr. Berrocal here.  So I wanted to understand a little bit more how SEAWAVE-QEX enters into this Tier 4.  So my understanding is that you have a site, you construct this chemograph, and you get your summary statistics.  You're going to do this for several sites where the pesticide data is good enough or it fits the criteria for using SEAWAVE-QEX and then how does this then move on onto the Tier 4?   I guess I want to know what is the long-term goal here.  I understand what happens side by side.  I just don't understand how that then gets all together to make a decision about the pesticide itself.
                  DR. SARAH HAFNER:  If you could hold on a little bit, we will talk about that in the case studies.  There's an example of how the SEAWAVE sites would be used in a drinking water assessment.
                  DR. ROBERT CHAPIN:  Terry.
                  MR. TERRY COUNCELL:  All right.  Terry Councell here.  So the two examples that you gave are atrazine and metolachlor you used.  From the data sets that you got that data from, those both compounds have significant metabolites, oxalinic acids or ethane sulfonic acids and all that.  So did you use just the parent compound?  Did you stoichiometrically add the metabolites back in or how did you deal with that?
                  DR. SARAH HAFNER:  We just used the parent compounds for this.
                  DR. ROBERT CHAPIN:  Cliff.
                  DR. CLIFFORD WEISEL:  Cliff Weisel.  This might be too broad right now, but what's happening with our water stream systems is they're being more and more controlled as to when things are being released intentionally in some places which will change our streamflow rate quite a bit.  So I have two edges to that question.  One is, is that being considered and how do you consider that?  And the second edge is reverse from what you're predicting for certain areas, do you think this might be a tool for helping advise how that might be done in the future?
                  DR. SARAH HAFNER:  Well, I can answer your first question.  Mr. Bischof will be talking about some of his sites later today that have some of this issue with streamflow and that's where he was able to use measured precipitation as a covariate for those sites that worked for him.
                  As for if this could be used --
                  DR. CLIFFORD WEISEL:  It's just a thought.  Looking at how we control our water systems in the future so I'm just putting this out as a thought.  If this really works, that might be another thing to be using, to worry about, because this is really trying to predict what is - the measured value and as Dr. Klaper pointed out, that's not always the real value that's there, but that's what we have.  But that's getting more and more tenuous as people try to control water systems.
                  DR. REBECCA KLAPER:  Rebecca Klaper.  I just had one question about variability too.  I understand -- maybe you can get around this with -- by considering the order of the stream, but with the climate change issues happening now where we have differences in variability of rainstorms and flow rates, is there an evaluation of when you have -- as it's gone through time.  I guess you only have like four years there, but, you know, especially over the last five, there's been a lot more variability in water flow.  How does the model handle when you have more consistent versus a variable rain flow events, you know, large storms and overflows and things like that?
                  DR. SARAH HAFNER:  Yeah, I guess we didn't do that analysis, but it would impact the short-term flow anomaly probably more than the other parts of the model, but we have focused on more recent data and if the run indicated, you could split it up into different runs based on that intensity.
                  DR. ROBERT CHAPIN:  So let me just point out -- Dr. Yang, first, let me just point out that we're eating into our lunch hour.  We're going to be rejoining here at 1:00 so if you've got important questions that --
                  DR. RAYMOND YANG:  Very brief, okay?
                  Ray Yang.  Once again, thank you for your presentation.  With Tom's help, I have a far better understanding of what you're doing, and, in fact, the model is, in fact, a polynomial equation with a variety of coefficients.  And I do have one question for Tamue.  Tamue, are we going to continue to get all these slides and so on?  These are very important information.
                  MS. TAMUE GIBSON:  I've already emailed them to the panel.
                  DR. RAYMOND YANG:  Okay.  Thank you.
                  MS. TAMUE GIBSON:  Yes, yes.
                  DR. ROBERT CHAPIN:  Andrew.
                  DR. ANDREW MIGLINO:  Andrew Miglino.  This will be, I hope, very quick.  So to get back to something Cliff said earlier about overestimation and choosing or fixing overestimation, in the real world, right, we're not going to know if we're overestimated.  So how do you make the decision to pick a different percentile in that case?
                  DR. SARAH HAFNER:  So that decision would be made based on these test data sets and based on the feedback from today.
                  DR. ANDREW MIGLINO:  So you see the four-day is consistently overestimated at the 99th, you pick 75th or something like that and that's now the de facto standard for the four-day?
                  DR. SARAH HAFNER:  That's the idea.
                  DR. ANDREW MIGLINO:  Okay.
                  DR. RAYMOND YANG:  May I make a recommendation very quickly?  Okay.  Ray Yang.
                  I would recommend to EPA colleagues, this equation that Tom gave me actually is from the paper of Vecchia 2018.  And because SEAWAVE-QEX is a very, very important aspect of this whole meeting, you should have this information, the equation, which is model, and the explanation of all the parameters and so on, in your White Paper and your presentation, okay.
                  DR. KENNETH PORTIER:  You stole my first comment.
                  DR. RAYMOND YANG:  I'm sorry.
                  DR. ROBERT CHAPIN:  All right.  Are we done with this part?  Okay.  We'll come back and do SBFs after lunch.  We're going to reconvene at 1:00.  It's 12:15.  See you in 45 minutes.
                                    [BREAK]
                  DR. ROBERT CHAPIN:  All right.  Thank you all.  Thanks for being back here on time.  And now let's start doing some SBFs.
                  Dr. Hartless, I think you're up.

DEVELOPMENT AND EVALUATION OF A SAMPLING BIAS FACTOR PROGRAM PART 1: SHORT-TERM

                  DR. CHISTINE HARTLESS:  Good afternoon, everyone.  I'm Christine Hartless and I'll be presenting talking about the development and further optimization of our sampling bias factors.  For this presentation, I will be focusing on the short-term sampling bias factors.  Mr. Peck, next to me, will be presenting on the long-term sampling bias factors next.
                  My presentation is structured to start with the purpose of this methodology, move to the development and evaluation approach, results, and finally implications for pesticide drinking water assessments.
                  So this is a flow diagram showing the overall picture of what we'll be presenting to you today in terms of our method development and evaluation of SEAWAVE-QEX and sampling bias factors that Dr. Bohaty presented earlier today.  And just to kind of remind you where we are, Dr. Hafner presented earlier about the SEAWAVE-QEX and that evaluation and so what she has done there is look at the monitoring data that were available and used SEAWAVE-QEX as tool, a regression-based tool, to help fill in those missing data and create realization such that we can have good picture of what's going on over a daily time step for those particular sites.
                  And there was a lot of discussion this morning about some optimization and what we were doing and there are some tools, diagnostic plots, and guidance that are included in the SOP documentation as well as in Dr. Vecchia's report that helped us to make sure that we are making the best use of our data and not over-using the data, not driving the tools too hard, and making sure that we're understanding the uncertainty that are present in our data.
                  So at this point, we're now going to be working with sampling bias factors, so I will assume that for the purposes of this presentation, what the -- the data that will be going into the development of these general sampling bias factors will be data that came out of SEAWAVE-QEX.  It had gone through all of the appropriate evaluation and that we have fair confidence in what those estimates are telling us about the specific sampling sites.
                  So these data, like from SEAWAVE-QEX, like daily monitoring data can be used in the development of sampling bias factors.  This provides this opportunity for us to greatly expand the number of pesticides for which sampling bias factors can be developed.
                  Short-term sampling bias factors allow the EPA to use sparse data sets, for example, those that may be composite samples collected on a weekly to a monthly basis over the course of a year, to be used quantitatively in drinking water assessments where short-term exposures would be of concern.  Additionally, they allow EPA to use the available monitoring data to the maximum extent possible permitting us to include more sites than if we were only able to use data from SEAWAVE-QEX alone.
                  Although many of you may be familiar with the use of sampling bias factors as they have been previously presented in several science advisory panels, I'm going to start with the basic definition.  A sampling bias factor is a protective multiplier for monitoring data to account for the uncertainty associated with reductions in sampling frequency.  As you've seen in some of the earlier presentations with Dr. Hafner, a reduction in the sampling frequency can potentially have a huge impact on various summary statistics calculated from the data, most specifically if we end up missing those peak observations -- peak concentrations.
                  So the sampling bias factor can be applied to summary statistics like the annual daily maximum or the annual 21-day rolling average derived from less than daily monitoring data to ensure that for some percentage of time, the sampling bias factor adjusted monitoring concentration is equal to or higher than the true unknown parameter.  Currently, the sampling bias factor has been developed such that the adjusted values will cover the true unknown maximum 95 percent of the time.
                  The impetus behind this approach was to allow for the use of monitoring data that is limited by the number of samples while still being applicable to the largest possible number of monitoring locations and maintaining a simple, straightforward methodology.
                  To build on our work from earlier Scientific Advisory Panels with short-term sampling bias factors, we used the USGS surface water monitoring data from the four pesticides listed here, the atrazine, carbaryl, chlorpyrifos, and fipronil.  These pesticides represent different use patterns across the United States and different environmental fate and transport properties.  These data were released as part of the SEAWAVE-QEX publication from USGS and included additional QAQC steps compared to the data directly available in the water quality portal.  These USGS data then were processed through SEAWAVE-QEX to obtain the daily chemographs.
                  And this is where I noted earlier, we've gone through and evaluated those data based on all of the various SOPs and the additional diagnostic plots in order to develop the best regression models for each of those particular sites and chemicals included in this analysis.
                  The SEAWAVE-QEX chemographs from atrazine USGS data were used to evaluate the different sampling and mathematical methods within the bootstrapping algorithm for the short-term bias factors.  The root mean square error was used to evaluate quality of fit and the data were visually evaluated.  Based on those results, we then moved on to estimate the short-term sampling bias factors for three additional pesticides based also on this USGS data set.
                  Finally, we did an independent evaluation of the sampling bias factor methodologies using monitoring data from the National Center for Water Quality Research data set.  This is the same data set that Dr. Hafner presented earlier which is also known as the Heidelberg data set.
                  This graphic here shows a high-level overview of the steps for generating a short-term sampling bias factor from the daily pesticide concentration data.  Over the next few slides, I will be going over each of these steps in more detail.  It is important to note that the methods here are only for the short-term sampling bias factors as the long-term sampling bias factors were developed with a different approach and will be discussed by Mr. Peck in the next presentation.
                  So first, we'll start with one year of daily concentrations of data in this upper left-hand graph in this figure -- in this slide.  This could be data from a monitoring program where samples were collected daily or it could be output from the SEAWAVE-QEX program.  In this case, we were utilizing output from SEAWAVE-QEX.  This graph displays the concentration on the Y-axis in micrograms per liter over the course of one calendar year.
                  The next step here is to calculate the parameter of interest from this full one-year data set.  Say, in this example, I'll be looking at the on-day annual maximum and here it's 18.4 micrograms per liter as you can see depicted on the slide.
                  Next, to create the subsample data, we're going to first choose a sampling intensity and then choose a sampling methodology.  Here, the sampling intensity that we selected was 13 samples per year in this figure and a random sampling methodology.  The other sampling methodologies that we -- the other sampling frequencies that we evaluated were 52, 26, 17, and 13 samples per year which roughly equilibrates to weekly, bi-weekly, and monthly.  And then we also could choose between either the random or the stratified.  So on the right-hand graphic, I have randomly drawn 13 daily samples from that year, from the graph on the left.
                  Next, we choose one of the three methodologies to infill these daily data.  This is the same graphic that we saw on the previous slide, to infill the daily data between the 13 sample time points in out bootstrap sample.  The methods that we chose from were linear, log linear, or stairstep.
                  The first of these methods we evaluated here was the linear interpolation.  In non-technical terms, we're just basically connecting the dots.  This is shown in this example here.  And what this means is that we assume a linear relationship between neighboring points and use that simple equation to calculate a concentration for any of the dates that happen to fall in that interval.  So in this example, the infilled concentration following the red line on day six would be 1.25 micrograms per liter.
                  The second approach that we used was log linear which is very similar; however, first, the concentration data are transformed to the log scale and the linear relationship is estimated in that transformed scale.  After the values are estimated for the non-sampled days using that same approach as with the linear data, those estimates are then quote, back-transformed, to the original scale.  Please note that there is no figure for the log linear interpolation shown on this slide.
                  Finally, for stairstep, the third interpolation method that we evaluated, daily concentrations on the non-sampled days are determined by using the concentration on the most recent preceding sample day.  For this example shown in the lower figure, days two through nine would have concentrations equal to the concentration measured on day one of 0.75 micrograms per liter.
                  An important note for all of these methodologies is that a limitation is that the infilled concentrations will never be higher than the highest of the highest sample concentrations from that annual chemograph.
                  So now we're moving to the next step in our bias factor flow chart.  Remember that the graph on the left here represents the 13 monitoring values that we sampled from the full initial chemograph.  And now in this figure here, we've used log linear interpolations to infill the values between those 13 sample dates as shown on the figure on the right.
                  So now based on this new time series, which is, again, seen on the left-hand graph, one calculates a summary statistic of interest, say for example, the 1-day, the 4-day, or the 21-day rolling averages.  For here for this one iteration of the bias factor development, the maximum one for 21-day rolling averages were 8.2, 8.1, and 7.9 micrograms per liter.  Then this whole process, the subsampling, the developing the new chemograph, and infilling between those sampled points, calculating the summary statistics is repeated 10,000 times.  One year of data from that site, from a given site, then results in 10,000 values estimated for a maximum 1-day, 4-day, and 21-day rolling average.
                  So now we've got this distribution and again, I'm going to stick to talking about the one-day annual maximum values for this example as we step through.  We have a distribution of 10,000 one-day annual maximum values.  And here in this hypothetical example, the one-day annual maximums for the first three simulations that we happen to have run were 8.2, 9.3, and 15.9.  And remember then, for the initial daily chemograph, the one-day maximum measured value was 18.4 micrograms per liter.
                  So now to calculate the one-day sampling bias factor for that particular site-year combination, then we're going to take the one-day maximum measured value, which was here at 18.4 micrograms per liter, and divide it by the fifth-centile from the distribution of the one-day annual maximums.  In this example, the fifth-centile was determined to be 9.5 micrograms per liter and therefore the one-day sampling bias factor was estimated to be 18.4 divided by 9.5.  So the sampling bias factor then results as 1.9.
                  This process was done for each of the available site-year combinations of data remembering that for each site-year for utilizing data from SEAWAVE-QEX, we would have multiple realizations, so it would be actually for each site-year realization of data that we're calculating and estimating a bias factor.
                  So now what do these bias factors look like once they're calculated?  Here are the distributions for three sites based on some of the atrazine data.  A reminder here that each of the box plots that you see is contracted from the sampling bias factors that were generated from the multiple realizations of SEAWAVE-QEX data.  For this component of the work, we were using just 50 realizations of each site-year of data.  So each box and whisker plot that you see here represents 50 realizations.  So along the X-axis is the year for each of those three sites and along the Y-axis is the one-day sampling bias factor and that value is a unitless value.
                  So several things to note here as you're looking at this figure is that while all these sites had some years of overlap, some sites in our world of sites that we use from the USGS data only have three years' worth of data.  And sometimes there was less temporal overlap than as seen here.  It's also important to note that the Y-axis here is reported in log base ten scale and if these data were plotted in the original scale, you would see how strongly right skewed they are, meaning that vast majority of the data would be in the lower end of the distribution, but there would still be a significant percentage of data points with very high values.
                  Finally, when looking at the first site as shown in red, one can see that the variation in sampling bias factors within each year is quite large and that the median value, the horizontal bar in the middle of the colored box tends to vary quite a bit from year to year.  The bottom and the top of the colored boxes here represent the 25th and 75th percentiles respectively.
                  When one looks at the second site shown in green, the within year variation is much smaller, but there still is a moderate level of year to year variation on that site as well.  Finally, the third site shown in blue, tends to have the lowest and most consisting variation from year to year, and there is little change in the median or any measure of central tendency over time.  These data just represent a subset of what we see when we're looking across all of the different sites.
                  So for each of these realizations of daily chemographs that are available for each site year, all combinations of sampling intensity, sampling method, and interpolation method were used to generate sampling bias factors.  Concurrent to calculating the sampling bias factors, the root mean square error, the formula that is shown on the slide here, was also calculated.  This provides a measure of goodness of fit of the daily chemograph generated within the sampling bias factor program with the original data and here these data were from the SEAWAVE-QEX chemograph.
                  Interpolation methods and sampling methodologies were compared using a generalized linear mixed model and the resulting interpretations from that analysis were that the log linear interpolation performed better than linear or stairstep interpolations when looking at the root mean square error as an evaluation term, and that there were minimal differences amongst sampling methods when evaluating the fit of the data for the bias factors. These results held across all of the different sampling intensities.
                  Oh, is there a question?
                  DR. ROBERT CHAPIN:  Quick question.
                  DR. VERONICA BERROCAL:  Yeah, sorry.  Can I understand the rationale behind this procedure?  I'm just very confused about the steps that have been implemented here.  So there is minor pesticide concentration data that is not available for every day.  So you apply SEAWAVE-QEX and you get this daily chemograph which is one of many possible data chemograph.  Then you subsample values from those so -- which might be observed values but might also be estimated values that are caught into that daily chemograph and then you interpolate with either a line or this step function.  Why do that?  Why not just use the daily chemograph several times?  I just don't understand it.  Moreover, there is obvious limitation that with the step or the line you would never be able to estimate the concentration greater than what you sampled.
                  DR. CHISTINE HARTLESS:  The idea behind using the kind of development of the sampling bias factors was to utilize site data from sites where we do have adequate data to develop those sampling bias factors to get a feel for if we have really good data and then subsample from that, we can get a kind of -- true isn't the right word, but a more confident kind of level of what that is, if we were to generate the sampling bias factor.  Then what we would do is we would take those sampling bias factors, which is basically kind of looking at a ratio of what we would expect as that parameter or choice, say a one-day maximum, if we have the optimum data set, which is daily data, versus data that was not an optimum data set, and we would kind of know what that ratio was.
                  Then let's say, for example, we have data from another site -- monitoring data from another site, where we might have 13 samples across the year, but we can't use SEAWAVE-QEX on it because we only have one or two years' worth of data.  SEAWAVE-QEX in the best of situations requires a minimum of three years' worth of data.  So this allows us to use that very minimally sampled data of, say, 12 observations per year, to be able to kind of estimate and come up with a better feel for where our uncertainty would be if we had better data to adjust for that uncertainty there.  Does that help?
                  DR. VERONICA BERROCAL:  I'm still very confused because the SEAWAVE-QEX model is supposed to try to represent what is the trajectory of this pesticide concentration.  So I don't understand one, why the sampling bias factor could not have been computed by using one daily chemograph over all the possible of 10,000 values that you got from 10,000 chemographs.  You get your sampling bias factor for the site that has enough data and then extrapolate to the site that don't have enough data.  I just don't understand where the -- maybe there is another logical reason why?
                  DR. KENNETH PORTIER:  So I think there's a misunderstanding and I had the same thing reading the White Papers.  If you go back to your slide where you describe -- you're using SEAWAVE --
                  DR. CHISTINE HARTLESS:  Which?
                  DR. KENNETH PORTIER:  Right there is fine.  Go back.  There.
                  So you're not using SEAWAVE here.
                  DR. CHISTINE HARTLESS:  Correct.
                  DR. KENNETH PORTIER:  You only use SEAWAVE in an almost full data set to kind of fill in the missing -- it's as an interpolation tool.  And then SEAWAVE is done.  They take the complete time series and draw a subsample from that time series, interpolate it, calculate their statistic, their five percentile, that's one, and then do it again.  Draw another sample, interpolate, get an -- do that 10,000 times.
                  DR. VERONICA BERROCAL:  So I understand that, but I guess -- sorry, but those 13 dots, where all 13 measure concentration, I think I wouldn't have this issue, but it's the fact that of these 13 dots, maybe 9 of them have been estimated using the data chemographs.  So why do this estimation twice and in one case using a function that you had -- you have some logical reason to believe that the pesticide concentration has that wave shape versus this one that is like a step function.  I'm just very confused.
                  DR. KENNETH PORTIER:  But they only SEAWAVE once.
                  DR. ROCHELLE BOHATY:  So --
                  DR. KENNETH PORTIER:  Yeah.  I think they're just using it once.
                  DR. ROCHELLE BOHATY:  So maybe just to add a little bit of context is that we have very few pesticides out there where we have daily data.  So in order to develop bias factors for more pesticides, we need to fine a best way to infill the concentrations between non-daily data, right?  And so we're trying to use SEAWAVE to do that.  And then from that, because we're only going to, as you'll see in the case studies, have maybe 30, 100 sites depending on the pesticide, right, where we can develop sampling bias factors, but we have maybe a thousand other sites where we could apply bias factors.  And so we're trying to leverage as much of the available data that we have to make some sort of conclusions about the potential exposure.
                  DR. JAMES SADD:  Jim Sadd.  So you extract the points and then interpolate them.  And the three interpolation techniques that you use are all deterministic.  Did you think about or explore the possibility of using non-deterministic interpolation method which would better recognize the uncertainty in each of the estimated measurements just from sampling and analysis and also, perhaps, uncertainty from the way in which they were selected and that would, perhaps, be a better way to recognize that the sample points you have or the estimate sample points you have don't capture the peak.
                  DR. CHISTINE HARTLESS:  We did not include any additional methodologies in this piece of our work to look at different methods of interpolation for the development of these bias factors.
                  And the other thing that I'll add in too, because -- and it took me a while to get to understand where the differences between when you're using SEAWAVE and when you're using the bias factors -- remember, the SEAWAVE-QEX, if you think about kind of a general framework presentation that was presented earlier, on a chemical by chemical analysis when we're going through and doing risk assessments, SEAWAVE-QEX would typically only be used at the Tier 4 level for an individual chemical when you're looking at individual sites and you're doing very highly refined analysis.  That's a fairly big level of lift in terms of level of effort to go through and do that.
                  The develop -- what I'm -- with the bias factors what we're developing here is to trying to develop something that's perhaps a bit more generic in that we're going to be eventually -- we're looking at it across a couple of chemicals to look at how much variation there is in these bias factors amongst chemicals to start looking at that.  And this is something that we're hoping that we can be able to apply at the Tier 3 level which would impact more chemicals and it's a lesser lift to, if we've got these bias factors, to use those.
                  So I think you have to kind of put in context what we're doing here.  And this is really gut-wrenching detailed work as was mentioned earlier of why are we doing all this subsampling when we have perfect data?  We're doing all of this so that we can develop something to use on a broader scale in Tier 3 when we have less than perfect monitoring data.
                  DR. JAMES SADD:  This is Jim Sadd again.  I understand.  If you look at the graph on the right, the interpolated data never goes above one of those sample points and we know in reality it probably does.  So my suggestion or my question is, if you don't use a deterministic interpolator, you could have a curve that goes above or below your sampling points.  And I'm wondering if that would be a more realistic way to interpolate the estimate?
                  DR. CHISTINE HARTLESS:  We've not looked at that specifically.  What we did to account for that was run those 10,000 Monte Carlo simulations where we're sampling a different set of 13 points each of those 10,000 times.  And so hopefully by doing that, we would capture that broad range of distributions and be able to capture where all those peaks were and then using a percentile off of that distribution.
                  DR. ROBERT CHAPIN:  Okay.  Question for clarification only, right?  Clarifying questions.  We're here for clarifying questions if we're going to interrupt their presentation.
                  DR. CLIFFORD WEISEL:  All right.  Well, the reason I brought it is there was an initial statement that you're doing these so you can do it for other chemicals, and you clarified that you're doing -- when you look at other chemicals -- you're trying to understand something about the properties.  Are also trying to understanding something about the properties of the site, how that interacts with the chemicals because that's --
                  DR. CHISTINE HARTLESS:  Yes.  Stay tuned for later this afternoon.
                  DR. ROBERT CHAPIN:  Okay.  Go.
                  DR. CHISTINE HARTLESS:  All right.  Let me -- All right.  I'm trying to remember where I am.  That's okay.
                  DR. ROBERT CHAPIN:  I think you were back one.
                  DR. CHISTINE HARTLESS:  All right.  There.  Okay.
                  So to summarize what we have so far, the distribution of these sampling bias factors that we've developed within a site-year combination are right skewed.
                  Sampling bias factors can be highly variable within a site year and among site years within a site.  Log linear interpolation performs better than linear or stairstep interpolation and minimal differences in sampling bias factors were identified when looking at either random or stratified sampling methodologies.
                  So for further work in this stage of the process, we're going to limit ourselves to looking only at the log linear interpolation and random sampling methodologies.  Our next step then was to take the sampling bias factors that we generated and obtain a summary statistic for each site.  Remember that for each site, a sampling bias factor was generated for each year realization of the SEAWAVE-QEX data and that's quite a few values if you remember from the previous box plot figures.  For example, if there were eight years of data for a given site and SEAWAVE-QEX was parameterized to generate 50 realizations, then we have 8 times 50, or 400 sampling bias factor estimates for that one site based on the data that we have available.
                  So to simplify this down to look at one value per site while accounting for the skewed distributions, varying time windows, both in the number of years represented at a site and the starting and ending of the sampling windows for the monitoring data, the following approach was used.  The first step was to calculate the median sampling bias factor for each year within a given site.  So in this example here, we would have a median sampling bias factor for 2009 for 2010, 2011, et cetera.  Then using those annual medians from the first step, we would calculate an overall site median to obtain one sampling bias factor per site.
                  So now that we have evaluated and honed in on some of the methods that we will be using to estimate the short-term sampling bias factors and have an approach to summarize data for each site, we went back to the SEAWAVE-QEX data generated earlier for the other three chemicals from the USGS data set and developed sampling bias factors for those pesticides as well.  As a reminder, we were focusing on the log linear interpolation and random sampling for the sampling bias factors.  And what is pictured here is the results based on 26 samples per year.
                  Each data point here represents one value per monitoring site and each box and whisker plot here contains a single value for each site.  So remember that atrazine had approximately 100 sites within that USGS data set so there are approximately 100 points within that -- the dark blue box and whisker plots that are presented up above.  The other three pesticides had approximately 30 sites for each pesticide.  And again, these sampling bias factors had right skewed distributions noting that the Y-axis here is in log scale and we also see that the differences amongst the pesticides were consistent across the various different averaging windows looking at 1-day maximum, 4-day rolling average, or 21-day rolling averages.  And then as the averaging window increasing, the sampling bias factor will decrease.
                  These relationships amongst the pesticides in the averaging windows that you see here are similar to those as if a different sampling intensity was used.  We've done those work, but I just did not present them in this slide.
                  So lastly, we're going to do some further evaluation of these sampling bias factors.  We're going to compare in two different examples.  We're going to compare the short-term sampling bias factors derived from this USGS sites to those developing using the National Center for Water Quality Research data set.  Again, these are the Heidelberg data, which were subsampled to create a sparse data set which to better represent the data typically available to assessors when completing drinking water assessments.
                  So there were four watersheds that were used for this evaluation.  This is Honey Creek, Rock Creek, Maumee, and Sandusky.  All four of these sites had data for atrazine.  So the data were run through SEAWAVE-QEX to create annual chemographs for use in the short-term sampling bias factor program.  Sampling bias factors were derived for 13, 17, 26, and 52 annual sampling intensities using the random sampling method and log linear interpolation and then were compared to the estimates derived from the atrazine from those USGS sites, that will be the hundred sites that were shown on the figure from the previous slide.
                  The minimum, median, and maximum values were plotted for the USGS data set represented by the circles on the figure here and the values for the Heidelberg data set are represented by the diamonds.  It's hard to distinguish there, but the diamonds are the more orange-red color.  All the Heidelberg values fell within the values derived from the USGS sites meaning that they fell between the maximum and the median except for the flashiest system when the minimal sampling intensity was selected.  This provides some general indication that the method for developing short-term sampling bias factors appears to work for these particular sparse data sets.
                  Finally, to --
                  DR. XUYANG ZHANG:  Xuyang Zhang here.  I have a question.
                  DR. ROBERT CHAPIN:  Go ahead.
                  DR. XUYANG ZHANG:  Can you go back to the previous slide?  Yes.  So you mentioned that the -- looks like both USGS data sets and the NAWQA data set has atrazine.  Is that -- do they have any overlap in terms of, you know, the same sampling site and the same time between the two data sets for the same chemical?
                  DR. CHISTINE HARTLESS:  There is overlap in the temporal construct.  The USGS data sites -- the USGS did not include the same sampling sites as were included in the Heidelberg data set.
                  DR. XUYANG ZHANG:  Okay.  So they are different?
                  DR. CHISTINE HARTLESS:  Yes, they are different.
                  DR. XUYANG ZHANG:  Okay. Thank you.
                  DR. CHISTINE HARTLESS:  Finally, to compliment the analysis done earlier as part of the SEAWAVE-QEX evaluation, sampling bias factors were compared using chemographs derived from flow or precipitation as a covariate in SEAWAVE-QEX.  Here you see the data for metolachlor in Rock Creek.  The closed circles are the sampling bias factors when flow was used as a SEAWAVE-QEX covariate for those five sparse data sets that Dr. Hafner discussed earlier and the open circles represent the sampling bias factors when precipitation was used as a SEAWAVE-QEX covariate.  The straight line represents the sampling bias factor that was estimated using the full metolachlor data set for that particular site.
                  We recognize that this is a small exploratory analysis as only one site was evaluated, but it does demonstrate that use of flow or precipitation as a covariate can provide comparable sampling bias factors.
                  So to present some of our -- to summarize with some results and conclusions, the analysis using the USGS monitoring data showed that the log linear interpolation provided the best fit for the sampling bias factor estimation.  There were small differences in sampling bias factor estimates between the random and stratified sampling, but we chose to move with the random sampling methodology.  The site-specific sampling bias factor was calculated as a median within a year and then median across years.
                  Use of the Heidelberg monitoring data provided an independent evaluation of the sampling bias factor estimation methodology and as few as 13 samples per year were needed to apply a sampling bias factor to an estimated upper end short-term average concentration.
                  So finally to introduce some implications for drinking water assessments, the short-term sampling bias factors were derived for four pesticides at between 30 to 100 sites and they permit the use of a value, the maximum across sites, to be used as a screen at Tier 3 assess potential exposure concerns.  The pesticide-specific sampling bias factors then can be used at Tier 4 to capture the range of potential concentrations not measured between sampling events.  Sometimes these large sampling bias factors -- sometimes, large sampling bias factors are obtained suggesting uncertainty in the available data and more data and further exploration into the specific factors for that particular site may be warranted.
                  Thank you, and I'll take any clarifying questions at this point.
                  DR. ROBERT CHAPIN:  Okay.
                  DR. CLAIRE BAFFAUT:  On your --
                  DR. ROBERT CHAPIN:  And your name?
                  DR. CLAIRE BAFFAUT:  -- table where you had --
                  DR. ROBERT CHAPIN:  Claire, just state your name.
                  DR. CLAIRE BAFFAUT:  Oh, Claire Baffaut.  On your table where you had explained how you combined the factors for each year to calculate one for the site --
                  DR. CHISTINE HARTLESS:  Yep.  I went too fast.  That one.
                  DR. CLAIRE BAFFAUT:  Okay.  So you explain here for the median and you take the median for each year and then the median of the median basically.
                  DR. CHISTINE HARTLESS:  Correct.
                  DR. CLAIRE BAFFAUT:  But in the White Paper, you say you used the same method for all the other percentiles, the 90th percentile, 99th percentile, so I can understand how you calculate the 99th percentile, for example, for each year, but then how do you combine all of those values into one?  So if you have ten years of data, that gives you ten 99th percentile.
                  DR. CHISTINE HARTLESS:  Correct.
                  DR. CLAIRE BAFFAUT:  How do you combine those?
                  DR. CHISTINE HARTLESS:  You would probably -- you would use the median of those 99th percentiles.  You would not use the 99th percentile.
                  DR. CLAIRE BAFFAUT:  Well --
                  DR. CHISTINE HARTLESS:  I understand where the confusion would be.  Does that help?
                  DR. CLAIRE BAFFAUT:  Yeah.
                  DR. CHISTINE HARTLESS:  Okay.
                  DR. CLAIRE BAFFAUT:  Thanks.
                  DR. ROBERT CHAPIN:  Ray.
                  DR. RAYMOND YANG:  Ray Yang.  I have two questions.  One is how realistic is the risk of one-day and four-day exposure?  Do you actually have data to indicate that somebody have one-day exposure and get sick?  I remember earlier on, I think maybe Rochelle's presentation, there was a large percentage of the sample or below one part per billion.  So are we talking about overkill?
                  DR. ANNA LOWIT:  So the rolling -- so the days, the number of days that they're doing, the one-day, the four-day, come from the different durations of exposure we derive from the Human Health Risk Assessments that are largely derived from the toxicology from the pesticides.  So in a standard Human Health Risk Assessment, we would do acute dietary.  So we do one-day assessments for food, and that would include food plus water.  We would also do what we call short-term which is generally 30 days, approximately, but in some cases like atrazine and the triazines, for example, we actually had a series of SAPs on this back around 2011, 2012 as we talked about this morning, we did five in 18 months.  It was an insane period, but the four-days applied to the triazines is derived from that work of how LH attenuation in the rats is then brought over to the human experience.  That's where the four-day is actually applied specifically to the triazines.
                  So, what we're doing here is to try to match the mode of action in toxicology to the exposure duration, so we have a seamless duration across.  So for example, in methylcarbamate where we do only single-day peak assessments.  So, in methylcarbamate, cholinesterase peaks after about an hour and a couple hours later it's mostly recovered.  So, all of our assessments are only one-day peak assessments.  So we need to match that highest peak in the water to the peak in the cholinesterase inefficient.  That's where the numbers come from.
                  DR. RAYMOND YANG:  Okay.  Well, I think, you know, the point I raise this -- the reason I raise this is that maybe EPA should think about something like this rather than, you know, this is the way we've been doing for the last decade and so on and so forth, okay.
                  So second question, if we go back to slide 102, this is, again, for education purposes.  Okay, 102.  Okay.  You got this bracket, the three-step process, all right.  I'm interested in the bootstrapping method of resampling, all right.  Am I correct that, you see, you've got this profile there.  That represents the actual sampling monitoring data, right?  And now you resampled it certain times, certain -- once a month or what have you.  When you do that bootstrap resampling, are you sampling from a probability distribution for that point, from earlier results and so on?
                  DR. CHISTINE HARTLESS:  I'm going to slide back to this slide here where I'm actually presenting the bootstrap sample.  So the figure -- the graph on the left represents the daily monitoring data, be it actually observed out the field and measured or be it generated using SEAWAVE-QEX.  And then what we'll do is sample from that and that's where those 13 samples come from and we walked that all the way through that process to get one value at the end which say, for example, might be the one-day maximum value and the four-day rolling average.  And then I'm going to resample again from the original complete daily chemograph that's shown on the left to come up with a new set of 13 values.  That's the piece that I'm doing the bootstrapping with and sampling 100 times, but it's not based on a probability distribution.
                  DR. RAYMOND YANG:  It's not?
                  DR. CHISTINE HARTLESS:  No.
                  DR. RAYMOND YANG:  All right.  Thank you.
                  DR. ROBERT CHAPIN:  Tom.
                  DR. THOMAS POTTER:  Hey, Tom Potter.  A couple of clarifying questions.  One, I recall reading in the White Paper, and perhaps I'm having some tunnel vision here right now, but that you only use SEAWAVE-QEX with flow as a covariate in this analysis?
                  DR. CHISTINE HARTLESS:  For this piece of it, yes.
                  DR. THOMAS POTTER:  Yeah, okay.  Now, was there -- what was your rationale there?  Is that -- you have greater confidence in the result?
                  DR. CHISTINE HARTLESS:  It was really a function of time.
                  DR. THOMAS POTTER:  Okay.
                  DR. CHISTINE HARTLESS:  And we did do some bits and pieces where we looked at some, and one of the very last slides that I presented was a small subset -- was a small look at comparing bias factors that were generated using SEAWAVE-QEX with flow as a covariate versus bias factors that were generated using SEAWAVE-QEX with precipitation as a covariate.  I believe that was probably the last figure slide that I presented.
                  But that was, again, just a very small --
                  DR. THOMAS POTTER:  In this presentation here as opposed to the White Paper or the --
                  DR. CHISTINE HARTLESS:  It's in the White Paper.  I don't remember the page number.
                  DR. THOMAS POTTER:  I'll -- maybe I'll dig in there again tonight and take a look, but I just wanted to be clear on that.
                  So the second clarifying is, these are all flowing systems; is that correct?
                  DR. CHISTINE HARTLESS:  All the USGS --
                  DR. THOMAS POTTER:  All your analyses were based on flowing system only.  You didn't look at any of the static system scenarios.
                  DR. CHISTINE HARTLESS:  No.  There would have been quite a range in the amount of flow that each of these systems may have had just based on what's in the USGS data set, yes.
                  DR. THOMAS POTTER:  Okay.  And finally, just to wrap it up, and again, maybe we'll defer to later on this, but there seemed to be some real differences in responses across sites.  So I'm assuming you folks are looking at this and trying to gain insight as to what drives that, whether it's the size of the watershed, the flashiness, et cetera.  Is that ongoing work?
                  DR. CHISTINE HARTLESS:  Actually, I'll speak some to that after Mr. Peck's presentation.
                  DR. THOMAS POTTER:  Okay, super.  Thank you.
                  DR. ROBERT CHAPIN:  Last one.
                  DR. VERONICA BERROCAL:  Yes.  Can I --
                  DR. XUYANG ZHANG:  Xuyang Zhang here.
                  DR. VERONICA BERROCAL:  Dr. Berrocal.  Can you explain again how you do the sampling bias factor evaluation?  I guess you went through the slide and I didn't see it.  Can I -- can you clarify?
                  DR. CHISTINE HARTLESS:  Which -- the whole --
                  DR. VERONICA BERROCAL:  No, no, no.  Like, there was a slide where you explained how you evaluated the sampling bias factor.  You had a statement at the end that said, the log linear random sampling was the best, so I just wanted to know how did you determine best?  That's what I was talking about.
                  DR. CHISTINE HARTLESS:  I was using the variable that I was evaluating there, was the root mean square error term and I used a mixed -- generalized linear mixed model to evaluate that.
                  DR. ROBERT CHAPIN:  All right.  Dr. Zhang.
                  DR. XUYANG ZHANG:  Yeah, I have a question on slide 108.  Yes.  So here, we -- I think you guys are deriving the median value from the 50 different chemograph simulations for each year.  And my question is, how variable, you know, what's the variation among these different chemographs because atrazine data is probably one of the best data sets that we can find for now and I'm curious about their ability among the 50 chemographs that it generates.
                  DR. CHISTINE HARTLESS:  There is a fair bit of variation amongst the chemographs.  One of the things that I looked at in developing that -- the generalized linear mixed model was incorporate variants terms to account for that variability amongst the chemographs in years.  I don't remember what those variant parameter estimates are off the top of my head though.
                  DR. XUYANG ZHANG:  Okay.  That's a good idea.  Thank you.
                  DR. ROBERT CHAPIN:  Okay.  We done with the short-term bias factors?
                  DR. CLIFFORD WEISEL:  I think it's the next slide or the one that you showed the comparisons of the different water.  Just, no.  Keep going.  We have the NCWRQ (sic) and can you tell me what you're using this for because I see you have a range?  What are you trying to explain out of this?
                  DR. CHISTINE HARTLESS:  The intent of why we looked at this was to look at doing -- it's a very small.  I wouldn't consider it a full validation of the methodology, but a very small validation of the methodology where I looked -- we developed quite a large data set using the USGS data where I had roughly a hundred sites.  And then I said, well, is that going to kind of potentially encompass the world of other sites?  And we had data available for four sites that we then calculated comparable --
                  DR. CLIFFORD WEISEL:  Well, that's -- it says minimum, max, and medium.  That's three, not different sites, but --
                  DR. CHISTINE HARTLESS:  That's from the USGS data where I had the hundred sites.
                  DR. CLIFFORD WEISEL:  Right.
                  DR. CHISTINE HARTLESS:  And so I took the minimum when I looked at --
                  DR. CLIFFORD WEISEL:  These are minimum.  That's the --
                  DR. CHISTINE HARTLESS:  The minimum from one site.
                  DR. CLIFFORD WEISEL:  Oh, okay.
                  DR. CHISTINE HARTLESS:  So I took --
                  DR. CLIFFORD WEISEL:  That's what I was trying to --
                  DR. CHISTINE HARTLESS:  So I had my 100 sites --
                  DR. CLIFFORD WEISEL:  Okay.  Now I understand.  That's what I was trying to understand where --
                  And I have a similar question for, I think, it's the next slide, where you're looking at the bias.  Yeah.  Again, how are you trying to -- explain to me what you're trying to say from this data so I can see a number of things on things on here.
                  DR. CHISTINE HARTLESS:  Again, it's a very, very small, it's not a full validation, of the methodology.  But looking at comparing, if I was to develop sampling bias factors using data from SEAWAVE-QEX that were developed using flow as a covariate or if I used precipitation as a covariate for that same monitoring data, do the bias -- are the bias factors comparable?
                  DR. CLIFFORD WEISEL:  Between the two of them?
                  DR. CHISTINE HARTLESS:  Correct.
                  DR. CLIFFORD WEISEL:  Not compared to your full record?
                  DR. CHISTINE HARTLESS:  And this one I also looked at -- this one we also have the full record for that one site.  I did not look at this with relative to the entire -- all of the USGS data set.
                  DR. CLIFFORD WEISEL:  All right.  The reason I ask that is the range I'm seeing is in order of magnitude across these and -- yeah, pretty -- well I see -- if we look on the left-hand side, I don't know what the lower value is, but it looks like it's closer to one to me.  And --
                  DR. CHISTINE HARTLESS:  This one -- remember, this one is not presented in log scale.  This is --
                  DR. CLIFFORD WEISEL:  No, I realize it's not a log scale, but I'm trying to understand a variation of the order of the levels that you're seeing here in these factors.  What do they mean and how do they translate to the variation in predicting what the concentrations would be?
                  DR. CHISTINE HARTLESS:  All right.  So the set of five points that you see in a vertical column there are the sampling bias factors that would be developed from each -- one of those from each of those five sparse data sets that Dr. Hafner presented -- talked about earlier.  And so you can get a feel for the variability based on that prescribed subsampling that was done in terms of how that impacts the sampling -- how that carryover impacts the sampling bias factors.
                  DR. CLIFFORD WEISEL:  All right.  So --
                  DR. CHISTINE HARTLESS:  Is that what you're trying --
                  DR. CLIFFORD WEISEL:  You know, this gives a -- then you would always take the maximum you would get out of this or the ---
                  DR. CHISTINE HARTLESS:  One would not typically do this in practice because this is --
                  DR. CLIFFORD WEISEL:  Well, that's my concern because now you've done it.  In practice, you would only do one.  You get a single value.
                  DR. CHISTINE HARTLESS:  So this is just to show that there is variation in those and so that's part of why we're accounting for that uncertainty.  And I think one of our questions to you is in looking at the various percentiles that we're proposing for these -- for the sampling bias factors kind of at the end stage of our process --
                  DR. CLIFFORD WEISEL:  Right.
                  DR. CHISTINE HARTLESS:  -- is that going to adequately cover the variability that we see in looking at all of these different pieces of the variant.
                  DR. CLIFFORD WEISEL:  And that's what I see when I'm looking at it and that's what worries me.  I don't know enough to answer that question, but hopefully -- that's why I'm bringing it up is something that hopefully --
                  DR. ROBERT CHAPIN:  Okay.  Last one.  Speak into the mic please and identify yourself.
                  DR. IAN KENNEDY:  Ian Kennedy.  I just would like you to possibly confirm that what I think you're doing is what I think you're doing, and that is you're taking a SEAWAVE-QEX-generated chemograph and you're taking random samples from that repeatedly and trying to kind of the fifth percentile worst possible random sample that you would have and comparing that to the maximum.
                  DR. CHISTINE HARTLESS:  I don't like the characterization of it.
                  MR. CHARLES PECK:  So we're taking the chemograph.  We're subsampling it.  We're generating another chemograph using log linear, linear, whatever.
                  DR. IAN KENNEDY:  Right.
                  MR. CHARLES PECK:  And then from that we're calculating a statistic, whether it's the maximum 1-day concentration, 4-day, 21-day.
                  DR. IAN KENNEDY:  Right.
                  MR. CHARLES PECK:  We're setting that aside.  We're going that 10,000 times using the fifth percentile and coming up with --
                  DR. IAN KENNEDY:  Right.  It's the fifth percentile of those 10,000 times.
                  MR. CHARLES PECK:  Correct.
                  DR. IAN KENNEDY:  So you're taking 10,000 possible -- if that was the real chemograph, these are 10,000 possible ways that someone might have sampled it.
                  MR. CHARLES PECK:  Correct.
                  DR. IAN KENNEDY:  And then you're basically taking the fifth percentile of all the maximum --
                  MR. CHARLES PECK:  Correct.
                  DR. IAN KENNEDY:  -- of those.  Okay.  Thanks.
                  DR. ROBERT CHAPIN:  I'm going to declare victory and we'll move on to Mr. Peck and talk about long-term sampling bias factors.
                  And may the Lord have mercy on you.

DEVELOPMENT AND EVALUATION OF A SAMPLING BIAS FACTOR PROGRAM PART 2: LONG-TERM

                  MR. CHARLES PECK:  Hopefully, this will be a little bit easier, but good afternoon.  Again, my name is Chuck Peck, and I'll be discussing the development of the long-term sampling bias factors and how they will be applied in drinking water assessments.
                  For this portion of the SAP, we'll look at why we're developing long-term sampling bias factors, how they were developed and evaluated, and their implications in drinking water assessments completed by the Office of Pesticide Programs.
                  So just to reorient and remind folks where we're at, at this point, we have daily chemographs, either generated by SEAWAVE-QEX or that luckily, somebody's gone out and monitored daily for at least a year and developed a chemograph.  We then take that, and we put it into a sampling bias factor program to develop a long-term sampling bias factor.  And that sampling bias factor program and algorithm is what I'm going to talk about here.
                  So just a little bit about why we're doing this.  Okay, certain pesticides at EPA valuate have long-term exposure durations.  For these pesticides, EPA typically uses modeled estimates, annual averages for chronic concerns, and 30-year averages for pesticides with cancer concerns in our drinking water assessments.  And the long-term sampling bias factors would allow EPA to develop annual and 30-year averages using monitoring data that we could then use in the human risk assessments.  They allow EPA to use sparse data sets comprised of four to ten samples per year.  As many community water systems typical conduct monitoring on a quarterly basis, it would be really desirable for EPA to use long-term sampling bias factors on these data sets to adjust the samples and obtain an annual average for use in a drinking water assessment.
                  Lastly, long-term sampling bias factors allow EPA to use the available monitoring data to the maximum extent possible permitting us to include more sites than would have been considered using SEAWAVE-QEX alone.  Specifically, they provide EPA with an easy method for estimating an annual average and a 30-year average concentrations that would account for temporal uncertainty in non-daily surface water monitoring data.
                  So as Dr. Hartless had presented, I'm going to sort of walk you through my process here.  Okay, and this is sort of a visual representation of what we're doing.  And I just need to emphasize that this process is going to be different than Dr. Hartless' for a couple different reasons.  One, I mean, for short-term sampling bias factors, after we do sampling and interpolation, you take the maximum rolling average from that for the exposure duration you're in interested in.  But when you're looking at an annual average, you're only going to get one value each time you do that.  And so trying to develop, you know, the maximum rolling average, you're just going to get an annual average.
                  The other thing is that when we're doing this, the tails.  When we start to subsample, you lose the tails in that.  So quite honestly, you're not able to estimate a 365-day average using the preceding technique.  So what we've done is we've come up with somewhat simpler method to develop a long-term sampling bias factor.
                  So the process starts with a chemograph here on the left for a chemical and depending on the number of samples you want to develop the sampling bias factor for, either four, six, eight, or ten, and the sampling method you want to use that you want to evaluate, whether it's a random or a stratified sampling method, you would then go in and you would do what's called estimating the 90th percentile upper confidence value around the mean of those values.
                  You would then repeat this process 10,000 times, okay, getting 10,000 90th percentile upper confidence values around the mean, and then you would then take the fifth percentile of that distribution and you would divide that into the actual 365-day average that you have to develop your long-term sampling bias factor.  In this case, the 365-day average from the chemograph was 1.8 micrograms per liter.  When we did our distribution, our 10,000 runs, the fifth percentile was 1.4 so you would end up with a long-terms sampling bias factor of 1.3 and that would be the sampling bias factor for that site, for that realization, and for that year of data.
                  So this slide here presents the equation that we use to estimate the 90th percentile upper confidence value around the mean using the number, the average, standard deviation of the samples, and the one-sided student T-value at an alpha 0.1.
                  It was discussed earlier by Dr. Hartless to simplify this down to one value per site while accounting for the skewed distributions and the varying time windows, EPA calculated the median sampling bias factor across all 50 equally probable annual chemographs for a site and year, then calculated the median sampling bias factor for each year within a given site, then calculated the median sampling bias factor for that site.
                  To determine the feasibility and use of long-term sampling bias factors, EPA used the existing SEAWAVE-QEX analysis presented earlier for the four pesticides from the USGS data set to, one, evaluate the number of samples that would be compatible for use in a long-term sampling bias factor process.  Two, develop long-term sampling bias factors for both the random and stratified sampling methods to three, see how different the results would be, and four, compared these long-term sampling bias factors to those using sparse or minimally in-field sampling derived from the National Center for Water Quality Research Data Set commonly referred to as the Heidelberg data set, to evaluate the impact of different data sets on long-term sampling bias factors.
                  So the first step was to determine how many samples we would need in order to effectively use the long-terms sampling bias factors.  So using 502 site years of USGS chemographs for atrazine, EPA estimated the 90th percentile upper confidence value around the mean 10,000 times for between 2 and 15 samples and created a distribution for each of the number of samples.  EPA then compared the value for N samples to the value of N plus one samples to estimate the percent gain in adding a sample.  This graph depicts the results as one moves from an N of 2 to an N of 14.
                  For two samples, the percent gain moving from two to three samples range from minus 30 to 169 percent with an average gain of 111 percent.  So moving to three samples is a significant improvement.  Once you get to four samples, the average gain falls to about 16 percent.  At 10 or more samples, the average gain is five percent or less so there isn't much gain in developing long-term sampling bias factors for more than 10 samples.
                  It's important to note that this figure does not mean that the long-term sampling bias factors are below one.  Okay.  It simply means that the increase in the number of samples doesn't impact the value calculated for the sampling bias factor.  Based on EPA's analysis, four samples were selected as the lower bound for the long-term sampling bias factors and 10 samples were selected as the upper bound.
                  Next, EPA developed long-term sampling bias factors for four USGS pesticides using four, six, eight, and ten samples and the random sampling method and compared them across pesticide sites and years.  An example of this analysis is provided in the box and whisker plots for three sites for carbaryl.  Please note that the Y-axis is on the log scale.  This can be seen in the figure long-term sampling bias factors can vary across sites as can be seen in the differences between the values for sites one and two, the pink and green boxes respectively, compared to the values for site three, the blue boxes, and across years as can be seen in the spread of the box plots for sites one and two.  Although it is not shown in this figure, there's also differences in the long-term sampling bias factors across the four pesticides that we evaluated.
                  Lastly, EPA compared the long-term sampling bias factors derived for the USGS sites to those developed using the Heidelberg data sets which were minimally infilled using SEAWAVE-QEX.  Three watershed sites were selected for assessment, Honey Creek, Rock Creek, and Sandusky River.  All three sites had data for atrazine, so the minimally in-field data were run through the long-term sampling bias factor program and sampling bias factors were derived for four, six, eight, and ten samples using the random sampling method.  The results were then compared to the estimates derived for atrazine at the USGS sites.  The minimum, median, and maximum values were plotted for both data sets and the values for the three Heidelberg sites, the orange diamonds, fell within the values derived for the USGS data set, the hollow circles, indicating the method for developing long-term sampling bias factors appeared to work for other data sets.
                  Based on our valuation, EPA believes that using SEAWAVE-QEX pesticide concentrations, long-term sampling bias factors can be developed to quantify the range of potential concentrations occurring between samples.  And as few as four samples per year are needed to apply a sampling bias factor to estimate upper end long-term average concentrations.
                  With regards to using long-term sampling bias factors in drinking water assessments, EPA believes that sampling bias factors can be used to quantify the uncertainty in available surface water monitoring data.  Specifically, EPA has developed a range of long-term sampling bias factors for four pesticides in 30 to 100 USGS sites.  These values can be used to adjust monitoring data as part of the screening assessment conducted at the Tier 3 level on a drinking water assessment which can then be compared to the drinking water level of concern to see if more refinements are needed at the Tier 4 level.
                  EPA also believes that the pesticide specific long-term sampling bias factors can be developed and used at the Tier 4 level of a drinking water assessment to adjust concentrations and capture the range of potential concentrations that were not measured during sampling events.
                  This concludes my presentation on the long-term sampling bias factors, and I'm happy to consider any of your clarifying questions at this time.
                  DR. KENNETH PORTIER:  Ken Portier.  This might be an easy question.  In calculating the mean and the standard deviation for your average, how did you handle no-detects or censored observations?  I'm trying to get at if you did half the detection limit or something like that?
                  MR. CHARLES PECK:  So, because we were using the SEAWAVE-QEX sampling data set, we didn't go back in and adjust the values that may have been below a detection limit.  We just used the values that came out of the chemographs.
                  DR. KENNETH PORTIER:  So the whole experiment assumes no no-detects.  It assumes everything's detected down to infinitely small?
                  MR. CHARLES PECK:  That's correct.
                  DR. ROBERT CHAPIN:  Tom.
                  DR. THOMAS POTTER:  Tom Potter.  On your comment of regarding the use in Tier 3, I'm really not clear as to how you're approaching this.  Are you taking the maximum sample bias factor for one of the pesticides or -- I mean, you're making some intercomparison between pesticides of different classes, et cetera, so you know, can you kind of amplify that a little bit?
                  MR. CHARLES PECK:  So this is sort of into the regulatory how we apply things, at this point, we're just trying to see that can we use them, but I would envision -- and this is me speaking personally -- I would envision that we would look at sort of the chemical properties of the chemical being evaluated, look at the four chemicals we have, potentially see how the fate properties are consistent with them, and then possibly use it that way.  But I think also we may end up talking about this in the next presentation when we look at the regressions that we did as well as the way to evidence approach that we're considering as well.
                  DR. THOMAS POTTER:  Well, I'm kind of remembering this storm again.  I'm kind of remembering the outcome of those regressions and they weren't very encouraging.
                  MR. CHARLES PECK:  Exactly.
                  DR. THOMAS POTTER:  Wade into that.
                  MR. CHARLES PECK:  So one other thing when we get to the case studies, you'll sort of see an indication of how we intend to use it as well there.
                  DR. ROCHELLE BOHATY:  This is Rochelle.  We're also asking that question on that.
                  DR. VERONICA BERROCAL:  This is Dr. Berrocal.  Can I ask a clarifying comment on the previous slide where you said something about, yes, as few as four samples per year are needed to apply a sampling bias factor to estimate upper end long-term average concentration?  Why upper end?  You're just -- can you explain why you're talking about upper end?
                  MR. CHARLES PECK:  So again, so the bias factors are designed to account for the uncertainty where you haven't sampled and so you're always going to be applying a number and trying to get a higher number with the bias factor.  And so generally when we do our drinking water assessments, we're looking at sort of, upper end, upper bound concentrations.
                  DR. ROBERT CHAPIN:  Okay.
                  DR. REBECCA KLAPER:  This is Rebecca Klaper.  Just a clarifying question.  So the factors that you're calculating, would they be then used on a watershed by watershed basis or is it an aggregate one that gets used then across all of them?
                  MR. CHARLES PECK:  So we're going to start to talk about that in the next section I think.
                  DR. ROBERT CHAPIN:  All right.  So we've had three introductions -- three mentions of the next talk so it's time to do that.
                  No, too late.  All right.  That's right.  Dr. Hartless.

WATERSHED EXTRAPOLATION AND WEIGHT-OF-EVIDENCE APPROACH

                  DR. CHISTINE HARTLESS:  Thank you, Mr. Peck.
                  Again, I'm Christine Hartless and I'll be talking about the watershed regression analysis that we looked at and the weight of evidence component of our work today.
                  I'll start by talking about the watershed regression, its goals, methods, and outcomes, then move to introducing the weight of evidence approach and its implications for drinking water assessments.
                  As a reminder of where we are in the overall method development process, after looking at and evaluating our short-term and long-term sampling bias factors, we're looking now to utilize that information as well as available watershed characteristics and fate characteristics of the pesticides that we have the sampling bias factors for to develop prediction equations for sampling bias factors that could then be used at sites with little monitoring data.
                  In following up from previous science advisory panels, our goal here was to identify watershed characteristics and possibly fate characteristics that could be reliably used to estimate sampling bias factors for watersheds.  Then those identified characteristics and corresponding regression equations would be used to predict sampling bias factors for those watersheds with inadequate monitoring data for site-specific bias factor estimation.
                  The work here focused on the data from the USGS sites that were used for earlier sampling bias factor development and evaluation.  Atrazine, carbaryl, chlorpyriphos, and fipronil had 98, 33, 30, and 28 sites, respectively.  There were 102 unique monitoring sites as some sites had multiple pesticides measured.  This is a slightly reduced number relative to the sampling bias factor analysis because there was some limited availability of some of the regression -- the regressor variables for a few of sites.
                  Watershed, catchment, landscape, and environmental characteristics were obtained from the StreamCat database.  These included such variables as watershed area, percent area classified as agriculture, urban area, and open water, et cetera.  We also included a few variables from outside the StreamCat data set such as average precipitation during May and June.
                  Prior to any regression analysis, we noted that there was a very large number of potential regressors.  We had 66 variables total and many of them were highly correlated with each other or had little to no variation across all of the selected sites.  Preliminary work was done using principle component analysis, correlation analysis, and graphics to reduce the initial set of potential regressor variables for use in the step-wide regression to a set of only 36 variables.
                  The pair-wise correlations between the 1-day, 4-day, and 21-day sampling bias factors and the pair-wise correlations of the sampling bias factors across all the different sampling intervals of 13, 17, 26, and 52 samples per year were all very high.  All of these pair-wise correlations were greater than 0.9; therefore, we elected to start with just one set of bias factors for the initial regression explorations.  We started with one-day sampling bias factor developed from random sampling and log linear interpolation with 28 samples per year which is comparable to a 14-day sampling interval.
                  For each of the four pesticides, step-wise regression was used to initially -- to identify initially important predictor variables.  Further exploration of the initial regressions was done using plots and influence statistics.  Final regression equations were focused on trying to find the best fitting, most robust model.  And note, for this stage of the regression analysis, we're only looking at the short-term bias factors.
                  For example here, the best fitting regression with the highest R² was for atrazine with two predictor variables, percent agriculture in the catchment, as calculated in 2006, and the average precipitation in May and June in the catchment.  The figures show these two variables on the X-axis and the one-day sampling bias factor on the Y-axis, each of the points on the graph represents one of the USGS atrazine sites that were included in the regression analysis.  The orange point here is one site that was identified as being highly influential in the regression analysis with a very high sampling bias factor.  The blue line approximately represents the fitted regression line for these data with the influential site removed from the analysis.  The R² for this resulting model after removing the influential site was only 0.26.
                  Of the four evaluated pesticides, atrazine results as shown here in the regression, resulted with the highest R²; however, we did not have confidence that these regressions were adequate for prediction purposes.  We're looking for feedback from the panel on this conclusion as well as recommendations on how and if it is worthwhile to continue to explore this process.
                  Another idea was to look across both sites and pesticides and use the pesticide fate properties for prediction of sampling bias factors.  These data sets here were very limited as the analysis was restricted to those site years where all four pesticides had sampling bias factors.  There were only five of 102 sites where this occurred.  If we relax the restriction to include those site years with any three of the four pesticides, we were able to include an additional 15 sites.
                  This exploratory work here shows that there is some potential that the fate characteristics may be useful to predict sampling bias factors, but the data are too limited for strong confidence at this time.  The two characteristics with the highest potential for use in regression equations were aerobic aquatic metabolism half-life, which was negatively correlated with the bias factors as shown here, and the minimum reported terrestrial field dissipation half-life which was positively correlated with the bias factor.  But again, we don't have confidence in using any of these at this point.
                  Given the limited usefulness of the regression-based watershed characteristics and the regression based on fate characteristics for the one-day sampling bias factors, we chose not to explore these regressions for other short-term or long-term bias factors at this time.  We're looking for feedback from the panel on this conclusion as well as recommendations on how and if it is worthwhile to continue to explore watershed and fate characteristics as a means for predicting sampling bias factors.
                  Specifically, some of the concerns that we would like for you to contemplate are sampling bias factors for each site were summarized over a several year window and those windows may not overlap across the sites included in the regression analysis.  In addition, the windows of data from each site may not overlap with some of the potential regressor variables that were included, specifically landscape type percent cover variables were based on data from 2006 imagery.  While some classes of landscape may not change much, like percent open water, other classes may change a meaningful percentage, a meaningful amount over a 20- to 30-year time span.  Some variables like average annual rainfall or average rainfall during May and June were based on data over a range of years.  In this analysis, we utilize data on average from 1981 to 2010.
                  Finally, for this preliminary work with pesticide fate properties, is it promising enough to warrant the development of sampling bias factors for additional pesticides at these USGS sites to further explore these relationships.
                  With no quantitative methods, meaning the regression analysis, to use to predict the sampling bias factors at sites with insufficient monitoring data, EPA moved to develop a more qualitative weight of evidence approach to evaluate the monitoring data.  This approach could be used for evaluating the spatial relevancy of monitoring sites and the associated monitoring data determined if it was applicable for use in estimating drinking water pesticide concentrations, applicability and usefulness of developing the SEAWAVE-QEX chemographs, and in the use of sampling bias factors to adjust existing monitoring data.  This weight of evidence approach is intended to be used as an interim process until more quantitative approaches are developed for sites with these minimal monitoring data.
                  In short, this weight of evidence approach was designed to answer these three questions when considering monitoring data in a drinking water assessment.  Are the sufficient monitoring data available to quantify concentrations in high usage areas?  This may include addressing questions such as, are the monitoring data collected from areas of known usage?  Are they collected during the timeframe when runoff into the waterbodies would be expected?  And are the data collected with a frequency such that the timeframe of concern could be adequately represented?
                  Second, can sites with monitoring data be used as surrogates for areas without monitoring data?  Here questions might be that are they adequate -- are there adequate data to developing sampling bias factors over that relevant timeframe?  And finally, are spatially relevant sampling bias factors available for use in adjusting these monitoring data?  Here one might ask if existing sampling bias factors from other sites may be relevant.
                  As a result, EPA feels that the following concepts or lines of evidence can be used to focus the evaluation of spatial relevancy of monitoring sites.  First, potential pesticide use sites and usage information can help EPA focus on areas where the pesticide is likely to be applied and potentially enter waterbodies.  EPA distinguishes pesticide use as the sites where the pesticide can legally be applied while pesticide usage as sites were observed pesticide applications have occurred.
                  Pesticide usage data can be obtained based on review -- excuse me.  Pesticide use data can be obtained based on review of the pesticide labels and geographic data identifying where different crops are grown.  Pesticide usage data can be obtained through survey databases or national and state public resources.
                  Second, EPA will consider a watershed and waterbody properties conducive to a pesticide reaching the drinking water intake.  Such evaluation could include the proximity of the monitoring site to the community water system, hydrologic connectivity of the monitoring site to the drinking water intake, soil properties and geology of the monitoring site that might impact the likelihood of runoff, and finally climatic factors such as precipitation.
                  Lastly, EPA will also consider the properties of the pesticide including how likely it is to be dissolved in water and move as opposed to how likely it is to bind to the soil and sediment.  In many cases, EPA will be able to employ available GIS technology and databases to help assist assessors in this weight of evidence approach.
                  So here's a hypothetical example of how EPA might be able to employ this weight of evidence approach.  The figure to the right depicts a typical community watershed with the drinking water intake depicted by the yellow dot.  Two green dots depict the monitoring sites identified within the watershed boundaries where the pesticide has been detected.  Next, we can evaluate these data and this information by stepping through those various lines of evidence and questions outlined on previous slides.
                  First, we looked at the pesticide use and usage.  For this hypothetical example, the red depicts the agricultural areas where the pesticide can legally be applied according to the label.  Pesticide usage data, which was not shown in the figure, indicated the pesticide applications occurred in close proximity to those green monitoring sites.
                  Second, evaluation of the NHD plus flow lines indicate that the water flows from the two sites -- the water from the two sites flows directly to the drinking water intake.  The estimated time of travel indicates that suspended pesticide in the flowing could reach the intake in a relatively short time, say, two to three days.
                  Third, for this hypothetical scenario, pesticide fate information indicated that half-life and water would be estimated at five to seven days.  That combined with the time of travel indicated that the pesticide in the water at the monitoring location could reach the drinking water intake location.
                  Given this information, EPA would conclude that the two monitoring sites are spatially relevant with regards to evaluating pesticide concentrations at that drinking water intake.  Further evaluation of any available monitoring data from these sites would be done at the Tier 4 of the drinking water framework.
                  A second example, again, here is the drinking water intake and a new monitoring site was identified in the watershed boundaries where different pesticide of interested was detected, down in the very bottom of the figure, of the graphic.
                  So next, we can again evaluate these data by stepping through the various lines of evidence justifications as outlined in the previous steps.  So the red here depicts the agriculture areas where this pesticide can be legally applied according to the label and pesticide usage data indicated that the pesticide -- however, if you'll see here that this one -- that the pesticide is not used in near immediately adjacent to that location.
                  Second, the water from this new monitoring location does flow into the drinking water intake, but the travel time is expected to be much longer based on the flow data, estimated seven to eight days.
                  And finally for this pesticide, fate information indicates that the half-life in water is estimated at only one to three days.  That combined with the time of travel indicated that the pesticide in the water at the monitoring location is less likely to reach the drinking water intake.
                  Given this information then, EPA would conclude that this monitoring site is not likely spatially relevant with regards to evaluating pesticide concentrations at the drinking water intake.
                  In conclusion, with regards to the implications of using watershed regressions and weight of evidence in drinking water assessments, the watershed regressions are not ready for use quantitatively in drinking water assessments; however, the weight of evidence approach can be used immediately to assist EPA in better understanding the spatial relevancy of the available surface water monitoring data.  This approach would be used at the Tier 4 level due to resource requirements and would allow EPA, using GIS databases and tools, to expand the use of monitoring sites and their data in drinking water assessments, whether, the data were used directly in SEAWAVE-QEX or in the development or application of sampling bias factors.
                  After walking through this weight of evidence approach and a couple simplified examples, we're asking for the panel to provide feedback on ways our posed approach could be implemented in our drinking water assessments.
                  Thank you and I'll take any clarifying questions.
                  DR. ROBERT CHAPIN:  Or any other questions.  Cliff.
                  DR. CLIFFORD WEISEL:  Cliff Weisel.  So a regression analysis, the way I think I saw it, assumes linearity.  A lot of the properties of these chemicals and the absorption properties are not going to be linear.  Did you consider how you might start using what you're learning from the way of evidence as to what might be some of the properties and see if you can do a semi-mechanistic understanding of what you should be doing in the regression?
                  DR. CHRISTINE HARTLESS:  I think that that would be a valid approach.  At this time, we only have the sampling bias factors developed for pesticides so there's not -- we don't have a lot to go on at this point.  So I think one the questions that we're asking to the panel is, do you think that there is enough going on with this approach and enough that we need to do more work and perhaps look at more pesticides to further look at that or are we spending a lot of resources for something that's not going to gain us much in the end?
                  DR. XUYANG ZHANG:  Xuyang Zhang.  I have I question.
                  DR. LISA NOWELL:  I'm Lisa Nowell.  Oh.  Ms. Zhang, want to go?
                  DR. ROBERT CHAPIN:  Hold on a second, Dr. Zhang.
                  Go ahead.
                  DR. XUYANG ZHANG:  Okay.  Sure.
                  DR. LISA NOWELL:  Lisa Nowell speaking.  I just wondered if you had tried -- I realize there are only four pesticides, but did you look at multivari- you tried look at more than one variable at a time, combined, not just simple univariate correlations but look for -- try to build a multivariate model?
                  DR. CHRISTINE HARTLESS:  With the fate properties, no, because we only had four pesticides.  You're basically connecting the dots at that point.
                  DR. ROBERT CHAPIN:  Okay, Dr. Zhang.
                  DR. XUYANG ZHANG:  In your regression analysis, I noticed that the pesticide usage is not included as one of the factors while pesticide usage was mentioned in the weight of evidence.  So is there a particular reason that you don't include that in your regression?
                  DR. CHRISTINE HARTLESS:  We do not have pesticide usage for specific pesticides on a watershed or catchment scale across the country for all of the different monitoring sites.  That's available on fairly -- some states have much better data than other states, and we also have that data available in general for overall pesticide usage, but not for specific pesticides at a national scale.
                  DR. XUYANG ZHANG:  Have you tried to -- instead of conducting regression analysis, using pie (phonetic), which is located in a watershed that has pesticide usage data, like for example, in California?
                  DR. CHRISTINE HARTLESS:  We did not look at that specific regression analysis.  We were trying to focus on looking at more broader scale national available -- looking at a broader scale.  I think that would be something that we could look at in the future in the panel suggests that is a path forward.
                  DR. XUYANG ZHANG:  Okay.  Thanks.
                  DR. ROCHELLE BOHATY:  I think it -- Rochelle Bohaty here.  I think it's also important to add that when we get down to a state level where we might have more detailed use information, like from the state of California, we have very few sites to include in that analysis.
                  DR. TIMOTHY GREEN:  Tim Green.  So the proximity of the monitoring station to water supply to me seems to come out of the blue.  If you had a good regression analysis, would you consider proximity?  It seems like you have two different things going on so I'm not sure why one replaces the other instead of somehow complimenting.
                  DR. CHRISTINE HARTLESS:  I think our initial hope was that the regression analyses would be able to be used.  And as we move through the work, that was not playing out the way we had hoped in terms of having good strong predictive abilities for sites with less data.  So we were trying to figure out what are the best ways to move forward and make the most use of the data that we had and this was kind of our approach trying to figure out how to do that and fitting it into the broader -- in something that would also fit naturally into the broader framework of drinking water assessment, that tiered approach that we have -- that was presented earlier.
                  DR. TIMOTHY GREEN:  Okay.  And to follow up, so if you had a really nice regression relationship with SEAWAVE, would you then look at proximity in addition to that?
                  DR. CHRISTINE HARTLESS:  So the regression work that I presented here was using the sampling bias factors, not SEAWAVE.  I think that had that regression analysis panned out really well that the weight of evidence would be -- I think as we kind of develop through our process and I think the weight of evidence would certainly have a lot of added value, even if the regression equations were more predictive and we felt more comfortable using them.
                  DR. LISA NOWELL:  Lisa Nowell.  I know you already know this, but I just wanted to mention that the USGS has pesticide agricultural use estimates on the county's fate scale that could, with a fair amount of work, be adapted.  You could overlay the base boundaries, et cetera, but I think the fact that it requires a lot of work is part of the problem.  Also, it's just ag use so it leaves a big portion of the non-agricultural applications unaddressed.
                  DR. REBECCA KLAPER:  Rebecca Klaper.  I'm just wondering if there's some way to improve your regression model, considering or dividing up the streams by the type of, you know, what order they are or base flow conditions?  I don't see those in there in the list of things that you did the regression against each of those things, the partial regressions.
                  DR. CHRISTINE HARTLESS:  I think the only chemical that we had adequate data for to kind of do any subdividing would be atrazine.  The other chemicals we did not have.  We only had roughly 30 sites for each of those other three chemicals.  We wanted to try to find something that was more broadly applicable, but I think that looking at some further subgrouping for the atrazine may be a way to try to tease something out of these data.
                  DR. REBECCA KLAPER:  Yeah, just because the stream characteristics can be so different even with all of the other factors that you're talking about based on the base flow of the stream.  Or have you tried something like decision tree analysis or something like that instead of doing the partial regression analysis?
                  DR. CHRISTINE HARTLESS:  I had done some decision tree analysis early in the work and I think that's where I was discovering that how many factors were so highly correlated.
                  DR. REBECCA KLAPER:  Right.
                  DR. CHRISTINE HARTLESS:  And that kind of approach allowed me to minimize my data set such that I had much less issues with correlation.  I did not go back and utilize the decision tree at that point.  That could be something that could be done with some further work.
                  DR. REBECCA KLAPER:  Thank you.
                  DR. THOMAS POTTER:  I probably could find this out by looking back on your list of regression parameters, but I thought it's just better to ask you because I think it's an important issue that will come up later on.  Irrigation.  Was that one of your regression factors?
                  DR. CHRISTINE HARTLESS:  I don't remember that one off the top of my head.
                  DR. THOMAS POTTER:  It hasn't been discussed yet in our dialogue.  We're talking about flow and rain, but we haven't got to irrigation and I think we need to.  And we'll talk about that, I guess, in the answer to the charge questions.  But the other part is a management issue, a land management issue was tile drainage.  A lot of what we're looking at is in the Midwest where a lot of the acreage is tiled drained and that certainly could be a dominating factor in some watersheds and I just wondered if you'd taken a look at that?
                  DR. CHRISTINE HARTLESS:  I did not break the variables --
                  DR. THOMAS POTTER:  Yeah.
                  DR. CHRISTINE HARTLESS:  I did not break it down on tile drainage.  If you, as part of the panel, have good data sets that we could utilize to grab that information, we'd really appreciate it.
                  DR. THOMAS POTTER:  Yeah, I think probably some of it can be excavated somewhere.
                  DR. ROBERT CHAPIN:  All right.  Look at your watch.  We have 10-minute bio-breaks.  So 10 minutes from right now we come back and dive in.

                   [BREAK]

                  DR. ROBERT CHAPIN:  All right ladies and gentlemen.  Let's do this thing.