SUPPORTING STATEMENT INFORMATION COLLECTION REQUEST for Public Involvement Activities Feedback Surveys of The United States Environmental Protection Agency January 2004 ii Table of Contents Title Page 1. Identification of the Information Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1( a) Title of the Information Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1( b) Short Characterization/ Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. Need for and Use of the Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2( a) Need/ Authority for the Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2( b) Practical Utility/ Users of the Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Non duplication, Consultations, and Other Collection Criteria . . . . . . . . . . . . . . . . . . . . 6 3( a) Non duplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3( b) Public Notice Required Prior to ICR Submission to OMB . . . . . . . . . . . . . . . . . . . . 6 3( c) Consultations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3( d) Effects of Less Frequent Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3( e) General Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3( f) Confidentiality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3( g) Sensitive Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4. The Respondents and The Information Requested . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4( a) Respondents/ SIC Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4( b) Information Requested . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 ( I) Data items, including record keeping requirements . . . . . . . . . . . . . . . . . . . 8 ( II) Respondent Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5. The Information Collected Agency Activities, Collection Methodology, and Information Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5( a) Agency Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5( b) Collection Methodology and Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5( c) Small Entity Flexibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5( d) Collection Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 6. Estimating the Burden and Cost of the Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 6( a) Estimating Respondent Burden . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 6( b) Estimating Respondent Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 ( I) Labor Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 ( II) Capital and Operations and Maintenance Costs . . . . . . . . . . . . . . . . . . . . . . 18 ( III) Capital/ Start up vs. Operations and Maintenance ( O& M) Costs . . . . . . . . . . 18 ( IV) Annualizing Capital Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 6( c) Estimating Agency Burden and Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 6( d) Estimating the Respondent Universe and Total Burden and Costs . . . . . . . . . . 22 6( e) Bottom Line Burden Hours and Cost Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 ( I) Respondent Tally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 ( II) The Agency Tally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 ( III) Variations in the Annual Bottom Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 ( IV) Reasons for Change in Burden . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 ( V) Burden Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 iii List of Tables Table 5 1 Planned Use of Questionnaires . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Table 6 1 ( a) Respondent Burden 2004 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Table 6 1 ( b) Respondent Burden 2005 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Table 6 1 ( c) Respondent Burden 2006 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Table 6 2 Respondent Universe, Total Burden and Costs . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Table 6 3 Agency Burden/ Cost for CAG, Public Meeting, Public Hearing, Listening Session, Small Group Discussion and FACA Meeting Effectiveness Surveys ( 6 surveys) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Table 6 4 Agency Burden/ Cost for CAG and FACA Process Effectiveness Surveys ( 2 surveys) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Table 6 5 Agency Burden/ Cost for Public Meeting, Public Hearing and Listening Session Follow up Surveys ( 3 surveys) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Table 6 6 Agency Burden/ Cost for CAG, FACA and Small Group Discussion Staff/ Contractor Assessments and CAG and FACA Participant Process Assessments ( 5 surveys) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Table 6 7 Agency Burden/ Cost for Negotiations Feedback Surveys ( 2 surveys) . . . . . . . . . 21 Table 6 8 Aggregate Agency Burden and Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Table 6 9 Aggregate Agency and Respondents Burden/ Cost . . . . . . . . . . . . . . . . . . . . . . . 24 Exhibits  EPA Survey Instruments and Associated Materials . . . . . . . . . . . . . . . . 25 1. Community Advisory Groups  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 33 c) Questionnaire for Feedback on the Effectiveness of a CAG Meeting ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 d) Questionnaire for Feedback on the Effectiveness of a CAG Meeting ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 e) Questionnaire for Feedback on the Overall Effectiveness of a CAG Process ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 f) Questionnaire for Feedback on the Overall Effectiveness of a CAG Process ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2. Public Meetings  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 46 c) Questionnaire for Feedback on the Effectiveness of a Public Meeting ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 d) Questionnaire for Feedback on the Effectiveness of a Public Meeting ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 e) Questionnaire for Feedback on the Effectiveness of Public Meeting Follow Up ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3. Public Hearings  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 60 c) Questionnaire for Feedback on the Effectiveness of a Public Hearing ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 iv d) Questionnaire for Feedback on the Effectiveness of Public Hearing ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 e) Questionnaire for Feedback on the Effectiveness of Public Hearing Follow up ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4. Listening Sessions  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 73 c) Questionnaire for Feedback on the Effectiveness of a Listening Session ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 d) Questionnaire for Feedback on the Effectiveness of a Listening Session ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 e) Questionnaire for Effectiveness of Listening Session Follow Up ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5. Small Discussion Group Session  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 a) Background Information for EPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 87 c) Questionnaire for Feedback on the Effectiveness of a Small Discussion Group ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 d) Questionnaire for Feedback on the Effectiveness of a Small Discussion Group ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 e) Questionnaire for Effectiveness of Small Discussion Group Follow Up ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 6. Stakeholder Negotiation  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . 101 c) Questionnaire for Feedback on Public Involvement in EPA's Stakeholder Negotiations ( stakeholder assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 d) Questionnaire for Feedback on Public Involvement in EPA's Stakeholder Negotiations ( public's assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 7. Federal Advisory Committee Act Committees  User's Guide . . . . . . . . . . . . . . . . 108 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . 112 c) Questionnaire for Feedback on the Effectiveness of a FACA Meeting ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 d) Questionnaire for Feedback on the Effectiveness of a FACA Meeting ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 e) Questionnaire for Feedback on the Overall Effectiveness of a FACA Process ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 f) Questionnaire for Feedback on the Overall Effectiveness of a FACA Process ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 g) Questionnaire for Feedback on the Overall Effectiveness of a FACA Committee ( DFO assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 1 U. S. ENVIRONMENTAL PROTECTION AGENCY VOLUNTARY FEEDBACK SURVEYS TO SUPPORT IMPROVEMENT OF EPA's PUBLIC INVOLVEMENT ACTIVITIES 1. Identification of the Information Collection 1( a) Title of the Information Collection: Obtaining Feedback on Public Involvement Activities and Processes 1( b) Short Characterization/ Abstract On June 6, 2003, EPA issued its new Public Involvement Policy. Evaluation is one of the key elements in the Agency's " Framework for Implementing EPA's Public Involvement Policy." Critical to overall evaluation is feedback from participants involved in the public involvement activities. A well designed evaluation plan will make it possible for EPA to better understand: ( 1) if the Agency is taking the necessary steps to gather and consider public input; ( 2) the quality of the Agency's public involvement processes; ( 3) how to consistently and systematically learn and improve, and ( 4) how the Agency can be more accountable to the public.

The Environmental Protection Agency is seeking from the Office of Management and Budget ( OMB) a generic clearance for public involvement activities surveys for a period of three years. The clearance will be used to conduct surveys about public involvement activities and processes. The surveys will help to identify participants' perceptions of their treatment during the activity, as well as the quality of pre activity information, the activity and follow up. All surveys will solicit opinions from participants in EPA public involvement activities on a voluntary basis, and will not involve " fact finding" for the purposes of regulatory development or enforcement.

By seeking a generic clearance for a set of questions and questionnaires for each type of activity, EPA will have the flexibility to gather the views of participants and better determine the extent to which our public involvement activities meet their needs or need to be improved. EPA will not deviate from the approved questions and will report annually on the actual use of the series of surveys. The generic clearance will enable regional and program offices' staff to regularly use approved questionnaires and apply the resulting data to improve the way the Agency conducts its public involvement processes.

EPA's sponsoring organizations that use any of the various questionnaires will report their use of the survey instruments to the public involvement staff in the Office of Policy, Economics and Innovation within the Office of the Administrator. That staff has developed a Lotus Notes spreadsheet program that enables the survey users to enter data from the returned questionnaires and quickly process the results, report the findings and take appropriate action to improve their next similar sessions.

EPA estimates that users of the public involvement questionnaires ( public meetings, public hearings, community advisory groups, Federal Advisory Committee Act groups, listening 2 sessions, small discussion group sessions, stakeholder negotiations ) will request voluntary overall.

2. Need for and Use of the Collection 2a. Need/ Authority for the Collection EPA's Public Involvement Policy states that the Assistant, Regional and Associate Administrators " Provide leadership and direction for their managers and staff by," among other responsibilities, " Evaluating the effectiveness of public involvement processes and taking action to improve these processes." To fulfill their responsibilities, the Agency's leadership and staff need a framework and tools to evaluate and measure how well EPA is implementing the policy's seven basic steps for conducting effective public involvement in both regulatory and non regulatory processes: 1. Plan and budget for public involvement activities 2. Identify the interested and affected public 3. Consider providing technical or financial assistance to the public to facilitate involvement 4. Provide information and outreach to the public 5. Conduct public consultation and involvement activities 6. Review and use input, and provide feedback to the public 7. Evaluate public involvement activities This ICR will be the cornerstone of a coordinated effort to track and improve the quality, effectiveness and consistency of EPA's public involvement activities and processes. Though individual programs and projects occasionally have conducted evaluations of their public involvement activities directly or through contractors, EPA has not encouraged a sustained agency wide effort to determine the extent and quality of public involvement processes and activities and to improve their effectiveness for both participants and EPA. One of the findings of a 2003 survey of staff from a cross section of programs and regions from 39 projects was that " absent a conscientious evaluation effort, project staff are unlikely to learn from their past mistakes."

The ICR will be part of a suite of tools being developed by public involvement staff and practitioners across the Agency that staff ( and other partners) can use to evaluate and measure public involvement activities on a consistent basis. That suite of tools will include:

Commonly used performance measures to help staff and managers better identify whether public involvement practices/ activities are performed and tools used appropriately, and whether they achieve the intended effects

Training in the use of the tools, measures and techniques ° Survey/ feedback templates that EPA staff can use to obtain input from the participating public for evaluating processes, activities and events ° A database/ spreadsheet program enabling staff to enter data from returned questionnaires, easily determine results and determine what actions to take to improve events, activities or processes 3 ° A central collection of evaluation instruments that EPA programs and regions have used to review their public involvement efforts, and

An OMB approved Information Collection Request ( ICR) to support gathering public involvement feedback based on a series of questionnaires that EPA staff can use to obtain input from the participating public for assessing processes, activities and events The Government Performance and Results Act of 1993 requires that agencies gather and use feedback. President Bush's Management Agenda underscores the need for citizencentered service delivery, increased satisfaction with government services and the ability to prove government is doing a better job through measuring outcomes. These questionnaires and the related tools will enable EPA to improve its public involvement processes, activities and events, enhance public involvement opportunities in ways the public will value and ultimately will improve EPA decision making by more consistently taking into account the values and views of the citizens we serve.

The key to building trust in a public involvement process is to pro actively gauge participants' perceptions and aggressively act on what they say went well and what needed correction. The techniques for obtaining such feedback do not have to be complicated or difficult to administer; however, they do have to be timely and the changes made because of public input must be visible and effective.

Some of the feedback needs in public involvement parallel those in customer satisfaction. Therefore, staff developing this Public Involvement ICR used OMB's Resource Manual for Customer Surveys ( dated October 1993), which provides guidance on obtaining quality survey results. They also used EPA's " Hearing the Voice of the Customer: Guidelines for Customer Feedback and Customer Satisfaction Measurement " ( November 1998).

EPA developed and used the " Guidelines" document to ensure proper design of EPA customer feedback and customer satisfaction measurement activities, increase the use and application of customer feedback, and build internal capacity to carry out those activities. The Guidelines use a five stage model for feedback: Plan, Construct, Conduct, Analyze and Act. A long series of detailed questions supplement the Guidelines to further assist those planning on obtaining feedback. Portions of the Guidelines ( available on the Internet @ http:// www. epa. gov customerservice/ guide. htm) will be included in public involvement feedback training.

The successful operation of the generic clearance for customer satisfaction surveys and the applications for 1997, 1999 and 2003 generic information collection requests served as the model for developing this application. Public involvement staff in the National Center for Environmental Innovation ( NCEI) will develop, distribute and post on the http:// www. epa. gov / publicinvolvement web site a fact sheet reflecting OMB's " Terms of Clearance" for the ICR, strongly stating restrictions on the use of this clearance. Burden will be defined on the surveys. The agency will display the OMB control number and clearance expiration date on every survey and will inform respondents of its legal significance ( see 5 CFR 1320.5( b)). In addition, every questionnaire will have introductory information for staff clarifying its purpose and any use restrictions. The spreadsheet/ database will also have instructions for staff use and the collection of results. 4 2( b) Practical Utility/ Uses of the Data Information gathered from these surveys will continuously help EPA to build, validate and improve public involvement measurement systems. Survey results may be used to identify: 1) Needs and expectations of public involvement participants 2) Strengths and weaknesses of EPA involvement activities 3) Ideas or suggestions for improvement of EPA involvement activities from people who participate in them 4) Barriers to achieving improved public involvement processes and activities 5) Changes in what and how we measure the success of public involvement activities and processes 6) Common performance measures By using these surveys, EPA personnel will be able to better define desired outcomes of public involvement processes and activities and determine whether the goals are met. For example, if achieving broad, inclusive involvement is a goal, there will be standards and questions to measure whether those involved believed that most ( if not all) potential interests participated or had the opportunity to do so.

While the information will not be used for regulatory development, the results of public involvement surveys could lead to changes in those activities and in certain Agency processes and policies, and to development of additional guidance related to public involvement. Ultimately, these changes could result in citizen driven improvements in the public involvement activities the Agency provides and the methods EPA uses to gather and consider public input.

In researching current literature for measures of success used to evaluate decision making processes and public involvement activities, EPA found many goals against which to measure attributes of processes, collaboration, decision quality, social and environmental outcomes. EPA recognizes that only retrospectively, after making a decision, can the Agency ask participants to evaluate the quality of the decision, the fairness of the process overall, or the degree to which their ideas are reflected in that decision. Measuring outcomes in the environment is much longer term and not in any way part of the work to be done under this ICR. The Agency is determined to focus on gathering data that will enable us to improve activities that are component parts of larger decision making processes. If we can use the information gathered to improve components of public involvement processes, people who participate in those events will have more meaningful and productive opportunities to provide input to the Agency.

Most of EPA's public involvement activities have similar components: finding the appropriate people to involve, ensuring they receive adequate notice of the opportunity to participate, providing information for their review, conducting the activity and doing follow up after the event. In most cases asking those who participated how well EPA did [ insert action ] 5 on a scale from 1 to 6 ( with 7, don't know, as the last alternate) will provide good information about what the Agency needs to improve before holding the next similar event.

The questionnaires are structured primarily to obtain quantitative data. Most questions are close ended and use the Lichert Scale. EPA will also solicit some qualitative data using a small number of open ended questions. Most of the qualitative questions ask for suggestions on how to improve an activity or process.

Achieving a performance standard of 75% positive responses ( 5 and above on a 6 point Lichert Scale) will be the first goal EPA uses as a measure of our success at engaging the public in Agency decision making processes and related involvement activities. This goal will provide the necessary framework for management to use survey results.

Process questions can address the clarity of the stated goals and roles of the public and EPA, and focus on the overall process transparency. Questions address attributes of the actions leading to the event ( publicity, information provision, access, etc.), the logistics ( site convenience and comfort, time of day, equipment, provision for people with disabilities, etc) conduct of the event ( courtesy, respect, equality, etc.) and ask for suggestions on how to improve the particular event or process. More areas covered in some of the of questionnaires follow.

Process

clear goals

clear process

clear roles for public and agency

process transparency to participants Appropriate people

diversity of views

balance ( on advisory groups)

Information

accessibility of information and technical information

clarity ( understandable)

quality

timely provision of

responsiveness to requests for Conduct of event

courtesy

respect

equal treatment of all

Logistics

convenience

timing

length 6

provision for people with disabilities Effectiveness

each event

overall process

of staff or other support ( FACA group, meeting facilitation)

of technical assistance provided ( FACA, CAG)

capacity building among participants

new alternatives from participants

of personal participation

otherwise unavailable information/ innovative ideas/ holistic views provided from participants Completed process or decision

fair

based on good science/ good technical information

economically and/ or socially viable

environmentally beneficial outcomes

learning extended beyond participants to the general public ( community capacity building)

resolved conflict

built trust

considered public values 3. Non duplication, Consultations, and Other Collection Criteria 3( a) Non duplication EPA staff who conduct public involvement activities will use the approved questions to learn how participants in their specific activities view various aspects of those activities. Therefore, the information collected will not overlap with other programmatic or satisfaction surveys. Every effort will be made to channel all public involvement related surveys through this ICR and to prevent misuse of this ICR for other purposes.

3( b) Public Notice Required Prior to ICR Submission to OMB EPA will conform to the requirement for public notice by publishing a preliminary and final Federal Register Notice concerning our intent under this ICR and requesting comment.

3( c) Consultations EPA's National Center for Environmental Innovation ( NCEI) public involvement staff requested input to this ICR application from practitioners across the Agency. This ICR was prepared in consultation with the Public Involvement Evaluation and Accountability Task Group of the Public Involvement Improvement Council and representatives from all organizations that 7 responded to the opportunity to outline their possible uses of the public involvement questionnaires through the year 2006. This feedback was used to develop the estimates of staff and respondent burden. EPA contacted professional practitioners through their networks and organizations to request their comments during the period of review for the draft ICR.

3( d) Effects of Less Frequent Collection This information collection could not be conducted less frequently. EPA will learn from participant reactions to and perceptions of public involvement activities that the Agency now provides in order to improve them. Programs will not survey all participants in every activity provided. There will be sufficient time between surveys to allow the actions taken in response to customer comments to show results. There are no technical or legal obstacles to reducing the burden.

3( e) General Guidelines This ICR complies with OMB's general guidelines for the collection of information.

3( f) Confidentiality Not applicable 3( g) Sensitive Questions Feedback questionnaires in this ICR application will not be used to collect any sensitive data.

4. The Respondents and the Information Requested 4( a) Respondents/ SIC Codes The EPA, by the very nature of its mandate, serves very large and diverse groups that participate in our decision making processes or could in some way be affected by EPA decisions. Participants in public involvement activities range from well known scientific experts to representatives of various interest groups, to individual business or property owners, local officials, interested young students and residents of environmental justice neighborhoods and tribal members. The term " the public" is used in the Public Involvement Policy in the broadest sense to include anyone, including both individuals and organizations, who may have an interest in an Agency decision. Appendix 2 of the Policy provides further definition:

" In addition to private individuals, " the public" includes, but is not limited to, representatives of consumer, environmental and other advocacy groups; environmental justice groups; indigenous peoples; minority and ethnic groups; business and industrial interests, including small businesses; elected and appointed public officials; the news media; trade, industrial, agricultural and labor organizations; public health, scientific and professional representatives and societies; civic and community associations; faith based organizations; and research, university, education and governmental 8 organizations and associations."

4( b) Information Requested ( I) Data items, including record keeping requirements The Agency will maintain records of the surveys' use. Offices sponsoring the surveys will retain files of the surveys, responses and analysis. Since these surveys will seek to measure public opinions on Agency public involvement activities, the surveys have not and will not involve respondents in extensive searching of existing sources, or reformatting information to submit to the Agency. The Agency does not anticipate any record keeping activities on the part of the public under this ICR.

( II) Respondent Activities EPA public involvement surveys will focus on the quality of the pre event work, the conduct of the public participation opportunity provided and the speed and completeness of follow up requested. All questionnaires will solicit qualitative feedback. Most questionnaires will be completed immediately following the public involvement opportunity before participants leave the event location. Several will be mailed or e mailed to participants as follow up to a single event or upon completion of a decision making process. All will involve reading instructions and completing questionnaires; some will also require mailing questionnaires. The questionnaires will require ten minutes or less to complete.

5. The Information Collected Agency Activities, Collection Methodology, and Information Management 5( a) Agency Activities.

Agency activities associated with the collection of information include:

National Center for Environmental Innovation staff:

Internal EPA review and approval of questionnaires ( completed in 2003)

Pretesting questionnaires submitted in this ICR application both internally and with less than 9 non federal persons ( completed 2003)

Headquarters Office/ Region staff:

Selecting public involvement event type and the appropriate pre approved questionnaire

Assembling data sources ( mailing lists, etc.) if necessary

Providing copies of the questionnaire at events, or disseminating questionnaire to potential respondents after an event or process

Gathering information from respondents

Answering respondent questions, follow up

Reviewing data

Recording submissions and analyzing results

Preparing findings 9

Storing and maintaining results

Reporting overall results to NCEI to share internally ( and possible post on the http:// www. epa. gov/ publicinvolvement Internet page)

Taking actions to improve activities, events and processes 5( b) Collection Methodology and Information Management The following process will be established:

EPA's sponsoring organizations report to PI staff in the NCEI their planned use of the questionnaires; report changes in those plans.

The database and spreadsheet program record the sponsoring office, point of contact, date, location, type of event and number of respondents.

In all cases, the purpose of the surveys is to obtain feedback from participants in EPA public involvement activities and processes so EPA can consider and use their ideas to improve pubic involvement.

Costs and burden to the Agency are estimated based on approximate time necessary for data entry and consideration of results for each questionnaire type

Costs and burden to the respondents will be tracked using the database program to count uses of and respondents to each questionnaire.

Complex statistical methods will not be used for the collections covered under this generic clearance. The spreadsheet/ database will perform simple calculations for those who enter respondent data.

Sponsoring organizations within EPA will maintain records of all the occasions they use the questionnaires. The database program and spreadsheet will streamline this record keeping burden. In general, survey results will be maintained for three years or until after follow up activities have been completed.

EPA organizations will notify the public involvement staff in EPA's NCEI of their use of questionnaires, enter response data in the database program and periodically report follow up actions taken based on survey results.

The public involvement staff will share results, lessons learned and success stories with other offices and provide feedback on overall survey results. This base of experiences/ lessons learned will be useful to state and federal partners that carry out public involvement activities and are seeking models for tracking and measuring success.

5 ( c) Small Entity Flexibility. Not applicable.

5 ( d) Collection Schedule.

The collection schedule needs to be flexible. Public involvement event schedules often slip and can drastically change if a decision process becomes more complex or controversial than anticipated. Such changes are not predictable, so estimating with precision is difficult. Table 5 1 is a draft plan for the use of questionnaires for 2004/ 5  2006/ 7 ( depending on approval date).

This application is based solely on good faith estimates that follow. The burden estimate is not precise since EPA has never before attempted to coordinate obtaining feedback from participants in involvement events across all programs and regions. The 10 number of surveys and the actual burden could be quite small during the first year and build as offices and regions see the usefulness of the data collected.

Community Advisory Groups ( CAGs) are not managed by EPA; however, the Superfund community involvement coordinators work to ensure that such groups are effective. Many of the groups would welcome help in improving their meetings and may make use of the meeting feedback form. There are approximately 70 active CAGs nationwide, with an average of 15 people.

Some of the CAGs meet as often as monthly; others meet only once a quarter. Were about 30% of the groups to use the forms every other meeting ( 6 or 2 times annually), there could be up to 21 groups using the surveys ( 30% of 70 = 21). Of the 21 groups, eleven may use the survey 6 times; ten may use it two times equaling 86 uses ( 11 x 6 + 10 x 2 = 86 ) of the meeting survey. Were 30% of the groups to use the process effectiveness survey once a year, there could be 21 uses annually. Based on those projected uses, respondent estimates would equal 1605 respondents ( 86 x15 ( 1290) + 21 X 15 ( 315)). We project slight increases in use as CAG managers make meeting and process improvements and more groups choose to use the questionnaires.

EPA issues approximately 150 rules each year. Publication of a Federal Register Notice requesting public comment is required; sometimes public hearings are required; sometimes they are requested ( RCRA). For the majority of rules and regulations, public comment is the only involvement activity. If one hearing were held for 30 of the rules ( 20%), and the feedback surveys were distributed at 50% of the events held, there would be approximately 15 uses of the public hearings survey the first year. There would be many fewer uses of the follow up survey, perhaps as few as 5. Annually we project a steady, small increase in use for both instruments.

EPA's Center for Conflict Resolution and Prevention ( CCRP) has a contract vehicle for agency wide use that enables organizations to request services for convening and facilitating public involvement activities. The activities can be meetings, hearings, small group workshops/ discussions, listening sessions, FACA sessions, negotiations, etc. The mix of activities changes from year to year.

Based on rough estimates, and the expectation that contractors will be required ( by the terms of the contract) to do formative evaluation, and to consider using the approved forms to obtain ongoing participant feedback to help gauge their success, we estimate contractors will distribute the various surveys to participants as many as 275 times. This estimate is based on the past history of the three five year contracts. During each contract cycle there is a steady build up to approximately 100 active task orders in year three, and then a slow decline in the number of active task orders until the end of year five. The contract is about to be renewed.

About 40% of the 100 task orders support contractors who do mediation and other alternative dispute resolution techniques; the remaining 60% are for public involvement activities. Of the 60 task orders, approximately 50 focus on a single project; the remaining ten average five projects each ( 50 + 10 X 5 = 100 projects). Of the 100 projects, approximately 50 hold only one public involvement event; the remaining 50 have three to six events. If all contractors are asked to distribute the surveys following all events, up to 275 uses are projected ( 50 + 50 X 4.5 ( events)). 1 11 EPA does not believe that it is realistic to expect that every contractor will distribute the surveys after every event to assess its effectiveness however, and will not require that they do so. We project about 70  90 uses per year of the various event surveys and we believe that contractors will administer fewer of the post event follow up surveys under the 100 active task orders of the CCRP.

group is dissolved or finishes a long term project, particularly if a contractor has been involved throughout a multi year process. Estimates include a few such assessments. Without contractor support, offices and regions sponsor the same types of events under community based, watershed, voluntary, environmental justice, tribal and other types of projects and programs. Without an Agency wide requirement to use the forms for a specific percent of all public involvement activities, EPA organizations will not widely adopt and use the surveys. However, if those who do so show continuous improvement in event outcomes and participant satisfaction, staff's voluntary use of the surveys should increase. Estimates reflect that expectation. We estimate that EPA/ contractor use of the surveys will equal approximately 90 to 110 uses per EPA has 26 FACA advisory groups, each with a designated federal official ( DFO). Most FACAs are long term standing groups that provide advice on a wide variety of topics to programs and the Agency. Based on the number of meetings held annually in fiscal years 2000 through 2003 ( reported in GSA database www. fido. gov/ faca database), the annual average is 150 meetings. Were 60 percent of the events followed by use of the meeting questionnaire, we estimate approximately 90 uses of the FACA meeting effectiveness survey in the second year, slightly fewer ( 86) in 2004; slightly more ( 100) in 2006.

Until a FACA ends or a specific task of FACA input ends, there is no use for the process effectiveness questionnaire. Over the past four years 308 FACA reports with recommendations were issued. Since the number has been steadily decreasing ( from 118 to 48), rather than use an average, we project a continued decline to not more than 40 reports in 2004. Annually, we expect not more than 15 uses of the process effectiveness questionnaire.

Annually, 26 DFOs will receive a request to assess the usefulness of their FACA's advice. ( This survey does not require OMB approval since it is a federal official responding.)

Table 5  1 Planned Use of Questionnaires 12 Questionnaires Number of Uses 2004 2005 2006 Process effectiveness ( participant) 15 15 15 Process effectiveness ( EPA/ contractor ) 2 2 2 DFO annual assessment** 26 26 26 ** EPA may also use the DFO FACA assessment that OMB approved in GSA's 2002 application for all federal agencies with FACA groups.

6. Estimating the Burden and the Cost of the Collection 6( a) Estimating Respondent Burden. 13 Staff who developed the questionnaires did so knowing that respondent burden should be as low as possible in keeping with the Paperwork Reduction Act. The questionnaire designs are simple, convenient, easy to respond to and clear in content and purpose. At the end of a public event people are ready to start home and will not remain and respond if the questionnaires are long or complex. Therefore, questionnaires have limited scope and require a short time to complete.

Responding to the meeting and process effectiveness questionnaires ( FACA, CAG, Meetings, Hearings, Listening Sessions, Small Group Discussions) averages about ten minutes. The questionnaires that focus on follow up activities after those events take not more than 5 minutes to complete. The first stakeholder negotiation questionnaire takes 15 minutes to complete while the second requires 10 minutes. The estimate of respondent burden are shown below in Table 6 1 ( a c). The estimate of respondent burden was based on obtaining responses for questionnaire use as summarized in Table 5 1.

Table 6 1 ( a) Respondent Burden 2004 Minutes/ response ( a) Uses ( b) Responses/ use ( c) Total responses ( d= b* c) Burden hours ( e= d* a/ 60)

Community 14 15 Table 6 1 ( c) Respondent Burden 2006 Minutes/ response ( a) Uses ( b) Responses/ use ( c) Total responses ( d= b* c) Burden hours ( e= d* a/ 60) 2 16 The EPA estimates respondent burden as follows: FY 2005: The three year total burden hours or per 6( b) Estimating Respondent Costs I Labor Costs. Since the respondents represent previously applicable instructions or requirements," " training personnel to be able to respond to a collection of information," " searching data sources," nor a need for the respondents to keep records. Burden activities include only a few steps: reviewing instructions, responding and submitting ( handing back or dropping into a collection box on site) or e mailing/ mailing responses when the surveys are not performed in person.

Table 6 2 displays the total burden estimates for respondents for each type of survey and total estimated respondent cost 17 Public Contributors 9 0.09 3 18 STAKEHOLDER NEGOTIATION TOTAL 18 0.198 Federal Advisory Committee Act Groups Meeting effectiveness ( participant) 276 4.14 Meeting effectiveness ( EPA/ contractor) 90 1.8 Process effectiveness ( participant) 45 0.9 Process effectiveness ( EPA/ contractor ) 6 0.12 DFO annual assessment**

FACA TOTAL 417 6.96 QUESTIONNAIRE TOTALS 1,819 54.571 Because some of the activities are regularly held meetings ( CAGs and FACAs), some respondents will complete the same questionnaire several times within the ICR period. This is necessary and appropriate since only they will be able to experience and comment on the effectiveness of the changes made in response to their input.

II Capital and Operations and Maintenance Costs Not applicable.

III Capital/ Start up vs. Operating and Maintenance ( O & M) Costs Not applicable.

IV Annualizing Capital Costs Not applicable.

6 ( c) Estimating Agency Burden and Cost.

Tables 6 3 through 6 7 provide the estimates for agency burden associated with each aggregated survey type for disseminating public involvement surveys and analyzing the results. Wage estimates were divided into three categories of labor: Management ( GS 15), Technical ( GS 13), and Clerical ( GS 7). Wage estimates were based on OPM's General Table 6 3. Agency Burden/ Cost for Meeting Effectiveness Surveys* 19 Activities for each Survey Use Gathering information Inputting data into database Reviewing data; follow ups Analyzing results Storing and maintaining results Preparing survey findings Acting on results Total burden by position ( manager, technical, clerical)

Total agency burden and cost per survey use * These include meeting effectiveness surveys for CAGs, Public Meetings, Public Hearing, Listening Session, Small Group Discussion and FACA advisory groups ( six surveys in total).

Table 6 4. Agency Burden/ Cost for Process Effectiveness Surveys*

Activities for each Survey Use Gathering information Inputting data into database Reviewing data Analyzing results Storing and maintaining results Preparing survey findings Acting on results Total burden by position ( manager, technical, clerical)

Total agency burden and cost per survey use * These include process effectiveness surveys for CAGs and FACA advisory groups ( two surveys in total).

Table 6 5. Agency Burden/ Cost for Follow up Surveys* 20 Activities for each Survey Use Gathering information Inputting data into database Reviewing data Analyzing results Storing and maintaining results Preparing survey findings Acting on results Total burden by position ( manager, technical, clerical)

Total agency burden and cost per survey use * These include follow up surveys for Public Meetings, Public Hearings, Listening Sessions, and Small Discussion Group Session surveys ( four surveys in total).

Table 6 6. Agency Burden/ Cost for EPA/ Contractor Meeting and Process Effectiveness Surveys*

Activities for each Survey Use Gathering information Inputting data into database Reviewing data Analyzing results Storing and maintaining results Preparing survey findings Acting on results Total burden by position ( manager, technical, clerical)

Total agency burden and cost per survey use 2.1 * These include meeting and process effectiveness surveys for CAGs, FACA advisory groups, Public Meetings, Public Hearings, Listening Sessions, and Small Group Discussion Session Surveys ( eight surveys in total; six for meeting effectiveness and two for process effectiveness.

Table 6 7. Agency Burden/ Cost for Stakeholder Negotiation Surveys* 21 Activities for each Survey Use Gathering information Inputting data into database Reviewing data Analyzing results Storing and maintaining results Preparing survey findings Acting on results Total burden by position ( manager, technical, clerical)

Total agency burden and cost per survey use * These include stakeholder negotiation surveys for Stakeholder Participants and Public Contributors ( two surveys in total).

A cross Agency staff group developed all questionnaires, so there are no additional costs of development and management review shown for using the surveys in any of the cost tables. Instructional materials accompany the surveys so their administration will take little staff time. Data entry and analysis are also streamlined when staff use the database spreadsheet package provided. Overall costs to the Agency for use of the feedback surveys is low and can be kept to a minimum because of the use of standardized surveys and data entry/ analysis.

Survey development costs are estimated at approximately $ 21,780 in EPA staff and intern salaries during 2002 and 2003. Costs are based on an average of 30 hours per survey surveys X $ 33.00 an hour. This total includes the time of members of the cross Agency Public Involvement Evaluation and Accountability Task Group, interns and practitioners across EPA.

Through a survey and personal interviews the Task Group and supporting staff asked EPA practitioners to state the most commonly used public involvement activities, whether they might do more evaluation were it easier to do and the kinds of participant feedback that would be useful to them. Responses helped to define the types of activities for which to develop surveys and the types of questions to ask.

In addition, the Task Group used related input from EPA's 2001 two week Internetbased " Dialogue on Public Involvement in EPA Decisions." Dialogue participants noted problems with the timeliness of pre event information, logistical problems at meetings, the quality of information provided and the treatment received before, at and after events. They 4 22 stated that small improvements to logistics, personal treatment and information dissemination aspects of public involvement activities made because of their input could make large differences in participants' satisfaction with events, their belief in the process and their trust in the decisions.

Various Task Group members drafted the individual questionnaires which the full group reviewed several times. A sample fact sheet for staff on how to use the surveys was the basis for a series of Guides on how to administer and use the six different sets of feedback templates. Several of the questionnaires and guides as well as the data entry system were pre tested in the field ( with fewer than 9 non federal persons) and discussed in conference presentations. Suggested changes were incorporated.

A Task group member developed the initial three models for the database spreadsheet and instructions for its use. Using the models, an EPA intern completed the remaining Guides, and will complete the database spreadsheets and instructional texts. These should be in place by the Summer of 2004.

Based on past use of the Customer Satisfaction ICR over the years, the Agency is not likely to administer each questionnaire as many times as now anticipated. Programs may not be ready to test whether the changes they make to a process or activity issue as quickly as they now predict, and it may take longer that they expect to institute changes.

6( d) Estimating the Respondent Universe and Total Burden and Costs Burden Table 6 1( a c) provides information on each survey by instrument type, number of respondents expected, burden per response, number of uses ( if more than one) and burden hours requested per survey. Table 6 2 summarizes the total burden and costs for respondents by type of survey. Activities have been grouped to reflect the various types of surveys and the total respondents expected for each instrument type. In all cases, the activities 23 CAG and FACA Process Effectiveness Surveys Public Meeting, Public Hearing and Listening Session Follow up Surveys EPA/ Contractor Meeting Effectiveness and Overall Process Effectiveness Surveys Negotiations Feedback Surveys Total III Variations in the Annual Bottom Line EPA burden hour projections are: FY 2004: 2,343 hours respondent burden from 14,819 individuals; FY 2005: 2,923 hours respondent burden from 18,382 individuals; FY 2006: 3,350 hours respondent burden from 21,370 individuals. Development costs which occurred prior to the use of any questionnaires do not factor into Agency survey administration costs, since all surveys will be pre approved.

IV Reasons for Change in Burden EPA expects to see a steady growth in the use of the surveys as programs and regions learn how to use them and the database supporting them, and to better apply the information they obtain to improve public involvement processes and events. During the three years of this ICR, there should be small increases in the costs for administering surveys, analyzing data and acting on the findings due to more usage and salary increases.

Table 6 9 Aggregate Agency and Respondents Burden/ Cost* 24 2004 2005 2006 Aggregate Respondents Agency * In addition to the cost the agency expects to incur over the three year period to administer the questionnaires, EPA incurred a cost of $ 21,780 to develop them ( see page 21).

V Burden Statement: The annual public reporting and recordkeeping burden for this collection of information is estimated to average 0.158 hours per response. Burden means the total time, effort, or financial resources expended by persons to generate, maintain, retain, or disclose or provide information to or for a Federal agency. This includes the time needed to review instructions; develop, acquire, install, and utilize technology and systems for the purposes of collecting, validating, and verifying information, processing and maintaining information, and disclosing and providing information; adjust the existing ways to comply with any previously applicable instructions and requirements; train personnel to be able to respond to a collection of information; search data sources; complete and review the collection of information; and transmit or otherwise disclose the information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control numbers for EPA's regulations are listed in 40 CFR part 9 and 48 CFR chapter 15.

To comment on the Agency's need for this information, the accuracy of the provided burden estimates, and any suggested methods for minimizing respondent burden, including the use of automated collection techniques, EPA has established a public docket for this ICR under Docket ID No. OA 2003 0009, which is available for public viewing at the Office of Environmental Information ( OEI) Docket in the EPA Docket Center ( EPA/ DC), EPA West, Room B102, 1301 Constitution Ave., NW, Washington, DC. The EPA Docket Center Public Reading Room is open from 8: 30 a. m. to 4: 30 p. m., Monday through Friday, excluding legal holidays. The telephone number for the Reading Room is ( 202) 566 1744, and the telephone number for the OEI Docket is ( 202) 566 1752. An electronic version of the public docket is available through EPA Dockets ( EDOCKET) at http:// www. epa. gov/ edocket. Use EDOCKET to submit or view public comments, access the index listing of the contents of the public docket, and to access those documents in the public docket that are available electronically. Once in the system, select " search," then key in the docket ID number identified above. Also, you can send comments to the Office of Information and Regulatory Affairs, Office of Management and Budget, 725 17th Street, NW, Washington, DC 20503, Attention: Desk Office for EPA. Please include the EPA Docket ID No. OA 2003 0009 in any correspondence. 25 EXHIBITS Questionnaires for OMB Review All future surveys under this ICR are to include, on or near the first page of the survey, a burden statement specific to that survey explaining the number of hours/ minutes per year per respondent and what that burden entails ( e. g.: Respondent burden for this survey is estimated to be minutes for reading and responding to the questions.) as well as the following paragraphs.

V Burden Statement: The annual public reporting and recordkeeping burden for this collection of information is estimated to hours per response. Burden means the total time, effort, or financial resources expended by persons to generate, maintain, retain, or disclose or provide information to or for a Federal agency. This includes the time needed to review instructions; develop, acquire, install, and utilize technology and systems for the purposes of collecting, validating, and verifying information, processing and maintaining information, and disclosing and providing information; adjust the existing ways to comply with any previously applicable instructions and requirements; train personnel to be able to respond to a collection of information; search data sources; complete and review the collection of information; and transmit or otherwise disclose the information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control numbers for EPA's regulations are listed in 40 CFR part 9 and 48 CFR chapter 15.

To comment on the Agency's need for this information, the accuracy of the provided burden estimates, and any suggested methods for minimizing respondent burden, including the use of automated collection techniques, EPA has established a public docket for this ICR under Docket ID No. OA 2003 0009, which is available for public viewing at the Office of Environmental Information ( OEI) Docket in the EPA Docket Center ( EPA/ DC), EPA West, Room B102, 1301 Constitution Ave., NW, Washington, DC. The EPA Docket Center Public Reading Room is open from 8: 30 a. m. to 4: 30 p. m., Monday through Friday, excluding legal holidays. The telephone number for the Reading Room is ( 202) 566 1744, and the telephone number for the OEI Docket is ( 202) 566 1752. An electronic version of the public docket is available through EPA Dockets ( EDOCKET) at http:// www. epa. gov/ edocket. Use EDOCKET to submit or view public comments, access the index listing of the contents of the public docket, and to access those documents in the public docket that are available electronically. Once in the system, select " search," then key in the docket ID number identified above. Also, you can send comments to the Office of Information and Regulatory Affairs, Office of Management and Budget, 725 17th Street, NW, Washington, DC 20503, Attention: Desk Office for EPA. Please include the EPA Docket ID No. OA 2003 0009 and OMB control number ( 2010 XXXX) [ insert the appropriate OMB control number] in any correspondence. 26 1. Community Advisory Groups  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 33 c) Questionnaire for Feedback on the Effectiveness of a CAG Meeting ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 d) Questionnaire for Feedback on the Effectiveness of a CAG Meeting ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 e) Questionnaire for Feedback on the Overall Effectiveness of a CAG Process ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 f) Questionnaire for Feedback on the Overall Effectiveness of a CAG Process ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2. Public Meetings  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 46 c) Questionnaire for Feedback on the Effectiveness of a Public Meeting ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 d) Questionnaire for Feedback on the Effectiveness of a Public Meeting ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 e) Questionnaire for Feedback on the Effectiveness of Public Meeting Follow Up ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3. Public Hearings  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 60 c) Questionnaire for Feedback on the Effectiveness of a Public Hearing ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 d) Questionnaire for Feedback on the Effectiveness of Public Hearing ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 e) Questionnaire for Feedback on the Effectiveness of Public Hearing Follow up ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4. Listening Sessions  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 73 c) Questionnaire for Feedback on the Effectiveness of a Listening Session ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 d) Questionnaire for Feedback on the Effectiveness of a Listening Session ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 e) Questionnaire for Effectiveness of Listening Session Follow Up 27 ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5. Small Discussion Group Session  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 a) Background Information for EPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . . 87 c) Questionnaire for Feedback on the Effectiveness of a Small Discussion Group ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 d) Questionnaire for Feedback on the Effectiveness of a Small Discussion Group ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 e) Questionnaire for Effectiveness of Small Discussion Group Follow Up ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 6. Stakeholder Negotiation  User's Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . 101 c) Questionnaire for Feedback on Public Involvement in EPA's Stakeholder Negotiations ( stakeholder assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 d) Questionnaire for Feedback on Public Involvement in EPA's Stakeholder Negotiations ( public's assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 7. Federal Advisory Committee Act Committees  User's Guide . . . . . . . . . . . . . . . . 108 a) Background Information for EPA Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 b) Checklist for Administering Questionnaires and Using Results . . . . . . . . . . . . . . . . . 112 c) Questionnaire for Feedback on the Effectiveness of a FACA Meeting ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 d) Questionnaire for Feedback on the Effectiveness of a FACA Meeting ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 e) Questionnaire for Feedback on the Overall Effectiveness of a FACA Process ( participant assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 f) Questionnaire for Feedback on the Overall Effectiveness of a FACA Process ( EPA/ contractor assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 g) Questionnaire for Feedback on the Overall Effectiveness of a FACA Committee ( DFO assessment) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 28 This page intentionally left blank 29 Assessing EPA supported Community Advisory Groups A User's Guide with Questionnaires Enclosed are information and tools for EPA staff to use in generating feedback on the effectiveness of Community Advisory Groups ( CAGs). Specifically, this user guide contains the following:

Background information for EPA Staff

Checklist for Administering Questionnaires and Using Results

Questionnaire for Feedback on the Effectiveness of a CAG Meeting ( participant assessment)

Questionnaire for Feedback on the Effectiveness of a CAG Meeting ( EPA/ contractor assessment)

Questionnaire for Feedback on the Overall Effectiveness of a CAG Process ( participant assessment)

Questionnaire for Feedback on the Overall Effectiveness of a CAG Process ( EPA/ contractor assessment)

For questions contact: Patricia Bonner ( bonner. patricia@ epa. gov)/ 202 566 2204 DRAFT: Last Updated 6/ 28/ 04 30 Assessing EPA's Community Advisory Groups Background Information for EPA Staff Introduction Enclosed are four questionnaires for EPA staff to use in generating feedback on the effectiveness of EPA supported ( and other) community advisory groups ( CAGs). The questionnaires are a component of the evaluation section in the Framework for Implementing EPA's Public Involvement Policy. The questionnaires are for people who regularly attend CAG meetings and for EPA staff responsible for coordinating the work of a CAG. The first and second questionnaires ( one for CAG participants and one for EPA/ contractor staff) focus on the effectiveness of specific CAG meetings. The third questionnaire, which should be administered once or twice a year as needed, deals with CAG participant perspectives on the CAG's success. The fourth questionnaire, which should be administered once or twice a year as needed, deals with Agency/ contractor perspectives on the CAG's success.

What is a community advisory group? A CAG is made up of diverse representatives from a community. Its purpose is to provide a public forum for discussing the interests and concerns the community has about some project or set of actions. CAG members meet on a regular basis to learn about project plans and activities and to offer ideas and suggestions to the project sponsor. A CAG may be selffacilitated or use an outside professional. It is not EPA sponsored and it is not subject to the Federal Advisory Committee Act procedures.

How can the CAG questionnaires be useful to you?

Use of the questionnaires should make the process of gathering feedback for your public involvement activity easier for you to implement directly, without spending contract dollars. These questionnaires provide an easy way to get useful feedback from participants about their CAG experience. This feedback should help you to better understand whether a particular CAG is working out well, whether you are missing opportunities for improvement, or whether you need to address certain problems. Once you have summarized the data, Agency staff can assess whether they met goals, make modifications, and compare progress over time.

When should I use the questionnaires? You can use the first and second questionnaires after every meeting or every couple of meetings. Use the third and fourth questionnaires once or twice a year, depending on the level of activity of the CAG. The questionnaires may also be helpful in situations where a particular problem hinders the successful working of the group. It is ultimately up to you, however, to decide when to use them. The key is to use the questionnaires to the extent that they provide you with valuable information and improve your ability to design and implement more effective listening sessions in the future.

Are the questionnaires in compliance with the Paperwork Reduction Act?

Not yet. The questionnaires will be cleared with the Office of Management and Budget ( OMB) 31 under ICR # XXXXX, OMB Control No: 2010 XXXX. This clearance will allow EPA to collect information ( via the questionnaires) from more than nine non federal entities.

Who designed these questionnaires? The Evaluation Task Group of EPA's Public Involvement Workgroup designed these questionnaires after consulting with EPA staff who are regularly involved in working with Community Advisory Groups.

What should Ido when preparing to administer a questionnaire? See " Checklist for Administering the Questionnaire."

What do I do with the questionnaire data once the forms are returned?

You can collect, collate and analyze the data from the forms using a pre formatted Lotus 1 2 3 spreadsheet program. In addition, the spreadsheet program allows you to summarize data over a period of time, making it easy to generate annual summary reports for your management. More information on how to use the program is available in the " Checklist for Administering the Questionnaire and Using Results" and within the Lotus 1 2 3 spreadsheet program. Once completed, Evaluation Task Group members encourage you to use your findings and send your completed data sheet files as e mail attachments to: bonner. patricia@ epa. gov. If you do so, we will be able to track the use of the questionnaires, and learn from you what is and is not working well, and gauge whether and how the questionnaires are useful to your efforts to improve public involvement activities. We will not use the information you send to judge the strength of individual public involvement efforts.

How can I use the results? The completed questionnaires should provide you with information that can help you design improvements to activities and to your overall involvement process. You can learn what worked well, what participants feel is important and what should change to better accommodate their needs. It will be up to you to decide how much data you need to prompt changes. While it is prudent to be cautious about making changes based upon a small amount of feedback data, you may find a number of the respondents' ideas useful. If you can act on their suggestions, even in a small way, participants will appreciate that you listened.

How should I communicate the results?

Use the performance and summary reports from the Lotus 1 2 3 spreadsheet program to talk with managers, staff, and CAG members about the effectiveness of CAG meetings and CAGs overall and where changes may be necessary. Communicate overall results to CAG participants in the CAG performance report or other follow up materials distributed to CAG participants. A few brief sentences describing overall what participants thought of the CAG, their suggestions for improvement, and what might be considered to improve the CAG process would let participants know that you valued their input.

What is EPA's Public Involvement Policy and where can I get more information? EPA's 2003 Public Involvement Policy underscores EPA's commitment to improving its practices and is based on seven steps for effective public involvement: 32 1. Plan and budget for public involvement activities 2. Identify the interested and affected public 3. Consider providing technical or financial assistance to the public to facilitate involvement 4. Provide information and outreach to the public 5. Conduct public consultation and involvement activities 6. Review and use input, and provide feedback to the public 7. Evaluate public involvement activities The Framework for Implementing EPA's Public Involvement Policy urges development of tools to improve public involvement information sharing, training and evaluation at EPA. This questionnaire and the database program are examples of such tools. Cross agency work groups developed the Policy and Framework using public comments and ideas gathered through a two week, Internet based Dialogue on Public Involvement in EPA Decisions. Copies of the Policy and Framework are available @ http:// www. epa. gov/ publicinvolvement/ public/ index. htm. 5Note: The instructions above are designed primarily for questionnaires that seek the perspective of participants about the effectiveness of a single event. Depending upon the type of questionnaire you wish to distribute and the type of public involvement activity you wish to acquire feedback for, you will need to tailor the steps of this checklist accordingly.

33 Assessing EPA's Public Involvement Activities Checklist for Administering the Questionnaire and Using Results5 Review questionnaires.

Determine whether a questionnaire should be administered.

Determine what questionnaire( s) should be administered.

Identify one person to be responsible for coordinating the overall dissemination and collection of the questionnaire and to input the data into the Lotus 1 2 3 data input, analysis, and reporting program once the questionnaires are collected.

Make plans to include the questionnaire feedback process in the agenda of your public involvement event. Specifically, plan to set aside a few minutes before the conclusion of your public involvement event and have participants fill out their questionnaires. You should also plan to make an announcement at the beginning of your public involvement event explaining the questionnaire feedback process and how feedback is needed from all participants in order to develop a more comprehensive understanding of the overall effectiveness of the event and how best to make improvements. Feedback is needed from participants who leave early and those who stay to the end. If you don't build the questionnaire feedback process directly into your event, and instead wait to the very end, chances are few participants will take the time to fill out their questionnaires. Participants may suggest that they would prefer to fill out their questionnaires at home or work. While this option is appealing for participants, chances are only a few will take the time to actually do this. The best chance for good feedback is when participants are actually at the event. Take the time to plan for this!

Estimate the number of participants who will be attending the public involvement event for which questionnaires will be distributed.

Make appropriate number of copies of the questionnaire.

Bring questionnaire copies and extra pens or pencils to the public involvement event.

Identify official who will be responsible for physically collecting the questionnaires.

Clearly identify areas where participants should return their questionnaires.

Hand out questionnaires with other public involvement activity materials either at the beginning of the event, or at a designated time during the event. 6Note: Separate Lotus 1 2 3 programs have been developed that match most types of questionnaires available ( e. g., one for single event effectiveness for participants, one for overall effectiveness for participants, etc.). Be sure to use the appropriate Lotus 1 2 3 program to input your data. The only type of questionnaire for which a spreadsheet program has not been developed is for the questionnaire dealing with post event follow up. However, if you need a spreadsheet program for this questionnaire, contact Pat Bonner ( bonner. patricia@ epa. gov).

34 Make it very clear to participants to whom, or to where, they should return their questionnaire. Urge participants to fill out their questionnaire even if they need to leave early.

Identify the total number of participants attending ( e. g., the total number of persons who could possibly fill out questionnaires).

Collect questionnaires.

Input data into the Lotus 1 2 3 data input, analysis, and reporting program within one month of session and share results.

Detach the Lotus 1 2 3 program onto your computer. 6 Open the Lotus 1 2 3 program.

Click on the " Getting Started" tab and follow the instructions.

Review performance report and identify whether certain changes should be considered in preparation for the next public involvement event.

Use the Lotus 1 2 3 performance reports to talk with managers and staff about the effectiveness of your public involvement activity and where changes may be necessary. Communicate overall results and what changes you intend to make to participants who attended your last public involvement event in the public involvement event summary report or other follow up materials distributed to participants. A few brief sentences describing overall what participants thought of the event, their suggestions for improvement, and what you will do to improve the next event would let participants know that you valued their input.

Sharing data... EPA's Public Involvement Policy Implementation Staff strongly encourages you to send your completed Lotus 1 2 3 program files as e mail attachments to: bonner. patricia@ epa. gov. If you do so, Implementation Staff will be able to track the use of the questionnaires, and learn from you what is and is not working well, and gauge whether and how the questionnaire was useful to your efforts to improve public involvement activities. We will not use the information you send to judge the strength of individual public involvement efforts.

When you conduct another public involvement event, repeat the same steps listed above. However, after doing this, be sure to go to the same Lotus 1 2 3 spreadsheet and follow the steps required to link the data from your first public involvement event to your second in order to summarize and compare your data over a period of time. 35 Directions Rate the following statements on a scale from 1 7 and write your own opinion to questions in the space provided. If you have no opinion, circle the number representing " Don't know." Please return your questionnaire to an EPA official before you leave. OMB CONTROL NO:

EXPIRATION DATE:

Questionnaire for Feedback on the Effectiveness of a Community Advisory Group Meeting Participant Assessment Background This questionnaire is designed to help Agency staff better understand what worked well and what improvements to consider implementing before holding future Community Advisory Group ( CAG) meetings. Once the data are summarized, Agency staff can assess whether goals for success were met, make modifications if necessary, and compare progress over time. Thanks in advance for taking time to fill out the questionnaire. We value your input! 36 Str on gly dis agr ee Dis agr ee So me wh at dis agr ee So me wh at agr ee Agr ee St ro ng ly ag re e Don' t kno w Meeting objectives were a) clear 1 2 3 4 5 6 7 b) achieved 1 2 3 4 5 6 7 Meeting was run effectively. 1 2 3 4 5 6 7 Sufficient technical information was available. 1 2 3 4 5 6 7 Participants were able to influence meeting outcome. 1 2 3 4 5 6 7 Participation in meeting was worthwhile. 1 2 3 4 5 6 7 EPA participation was useful. 1 2 3 4 5 6 7 Meeting time was convenient. 1 2 3 4 5 6 7 Meeting location was comfortable. 1 2 3 4 5 6 7 Meeting length was appropriate. 1 2 3 4 5 6 7 Do you have any comments or suggestions for improving the CAG meetings?

Please indicate which group you represent:

neighborhood group local non profit organization ( not including local neighborhood non profit groups) 37 national environmental organization business or industry or trade association state or tribal or local government federal government other ( please specify)

no organizational affiliation ( e. g., interested local resident)

Please return your completed questionnaire to an EPA official before you leave. Thank you for helping EPA improve its public involvement practices. 38 Directions Rate the following statements on a scale from 1 7 and write your own opinion to questions in the space provided. If you have no opinion, circle the number representing " Don't know." Please return your questionnaire to an EPA official before you leave. OMB CONTROL NO:

EXPIRATION DATE:

Questionnaire for Feedback on the Effectiveness of a Community Advisory Group Meeting EPA/ Contractor Assessment Background This questionnaire is designed to obtain an EPA/ contractor perspective on the effectiveness of a Community Advisory Group ( CAG) meeting or set of meetings. Agency staff will use this information to better understand what is working well and what improvements could make future CAG meetings more successful. Thanks in advance for taking time to fill out the questionnaire. We value your input!

Str on gly dis agr ee Dis agr ee So me wh at dis agr ee So me wh at agr ee Agr ee St ro ng ly ag re e Don' t kno w Meeting objectives were a) clear 1 2 3 4 5 6 7 b) achieved 1 2 3 4 5 6 7 Meeting was run effectively. 1 2 3 4 5 6 7 Sufficient technical information was available. 1 2 3 4 5 6 7 Participants were able to influence meeting outcome. 1 2 3 4 5 6 7 EPA participation was useful. 1 2 3 4 5 6 7 Meeting time was convenient. 1 2 3 4 5 6 7 Meeting location was comfortable. 1 2 3 4 5 6 7 Meeting length was appropriate. 1 2 3 4 5 6 7 Please explain your ranking of the previous statement about success: 39 Please provide any comments or suggestions for improving the CAG meetings:

Please return your completed questionnaire to an EPA official before you leave. Thank you for helping EPA improve its public involvement practices. 39 Directions Rate the following statements on a scale from 1 7 and write your own opinion to questions in the space provided. If you have no opinion, circle the number representing " Don't know." Please return your questionnaire to an EPA official before you leave. OMB CONTROL NO:

EXPIRATION DATE:

Questionnaire for Feedback on the Overall Effectiveness of a Community Advisory Group Process Participant Assessment Background This questionnaire is designed to help Agency staff better understand the overall effectiveness of a Community Advisory Group ( CAG) process. Agency staff will use this information to better understand what is working well and what improvements could be make this CAG process more successful. Thanks in advance for taking time to fill out the questionnaire. We value your input!

Str on gly dis agr ee Dis agr ee So me wh at dis agr ee So me wh at agr ee Agr ee St ro ng ly ag re e Don' t kno w Committee membership is inclusive. 1 2 3 4 5 6 7 All committee members are contributing to and sharing in the work. 1 2 3 4 5 6 7 Committee is adequately connected to the community. 1 2 3 4 5 6 7 Meeting frequency is appropriate. 1 2 3 4 5 6 7 Committee has a good process for reaching closure on issues. 1 2 3 4 5 6 7 Committee is fulfilling its chartered mission. 1 2 3 4 5 6 7 Committee is producing useful action agenda and recommendations. 1 2 3 4 5 6 7 Committee is influencing EPA decisions. 1 2 3 4 5 6 7 Personal participation is worthwhile. 1 2 3 4 5 6 7 Please provide any comments or suggestions for improving this CAG process: 40 Please indicate which group you represent:

neighborhood group local non profit organization ( not including local neighborhood non profit groups)

national environmental organization business or industry or trade association state or tribal or local government federal government other ( please specify)

no organizational affiliation ( e. g., interested local resident)

Please return your completed questionnaire to an EPA official before you leave. Thank you for helping EPA improve its public involvement practices. Draft Draft Draft 41 Directions Rate the following statements on a scale from 1 7 and write your own opinion to questions in the space provided. If you have no opinion, circle the number representing " Don't know." Please return your questionnaire to an EPA official before you leave. OMB CONTROL NO:

EXPIRATION DATE:

Questionnaire for Feedback on the Overall Effectiveness of a Community Advisory Group Process EPA/ Contractor Assessment Background This questionnaire is designed to obtain an EPA/ contractor perspective on the overall effectiveness of a CAG process. Agency staff will use this information to better understand what is working well and what improvements could be make this CAG process more successful. Thanks in advance for taking time to fill out the questionnaire. We value your input!

Str on gly dis agr ee Dis agr ee So me wh at dis agr ee So me wh at agr ee Agr ee St ro ng ly ag re e Don' t kno w CAG is inclusive. 1 2 3 4 5 6 7 CAG is well facilitated. 1 2 3 4 5 6 7 CAG has two way communication with community. 1 2 3 4 5 6 7 CAG is producing useful advice. 1 2 3 4 5 6 7 EPA is providing adequate technical information and support. 1 2 3 4 5 6 7 EPA is providing appropriate financial and other assistance. 1 2 3 4 5 6 7 Please explain your ranking of the previous statement about success:

Please provide any comments or suggestions for improving this CAG process:

Please return your completed questionnaire to an EPA official before you leave. Thank you for helping EPA improve its public involvement practices. Draft Draft Draft 42 Assessing EPA's Public Meetings A User's Guide with Questionnaires Enclosed are information and tools for EPA staff to use in generating feedback on the effectiveness of public meetings and the effectiveness of follow up after them. Specifically, this user guide contains the following:

Background information for EPA Staff

Checklist for Administering Questionnaires and Using Results

Questionnaire for Feedback on the Effectiveness of a Public Meeting ( participant assessment)

Questionnaire for Feedback on the Effectiveness of a Public Meeting ( EPA/ contractor assessment)

Questionnaire for Feedback on the Effectiveness of Public Meeting Follow Up ( participant assessment)

For questions contact: Patricia Bonner ( bonner. patricia@ epa. gov)/ 202 566 2204 DRAFT: Last Updated 6/ 28/ 04 Draft Draft Draft 43 Assessing EPA's Public Meetings Background Information for EPA Staff Introduction Enclosed are three questionnaires for EPA staff to use in generating feedback on the effectiveness of public meetings. The questionnaires are a component of the evaluation section in the Framework for Implementing EPA's Public Involvement Policy. The questionnaires are for people who attend a public meeting, for EPA staff/ contractors responsible for coordinating the work of a public meeting, and for public meeting participants who receive follow up information following the public meeting. The first questionnaire focuses on the effectiveness of a public meeting. The second questionnaire focuses on EPA/ contractor perspectives about the success of the same public meeting. The third questionnaire focuses on participant perspectives of the quality of the follow up information received after the same public meeting.

What is a public meeting? A public meeting is a forum that allows communities and the affected public to address EPA officials directly with their concerns. Public meetings promote two way communications and a means for all interested parties to ask questions and raise issues in an informal setting. The purpose of a public meeting is to share information and discuss issues, not to make decisions. Public meetings should: enable the public and the Agency to share data, ideas, advice and concerns; allow EPA to hear from a wide range of interested an affected parties; and compile a knowledge base of the public's various interests, ideas and needs, helping the Agency to better understand and consider the issues related to a particular decision.

How can the public meeting questionnaires be useful to you?

Use of the questionnaires should make the process of gathering feedback for your public involvement activity easier for you to implement directly, without spending contract dollars. The questionnaires provide an easy way to get useful feedback from participants and EPA about the public meetings they participated in. This feedback should help you to better understand whether a particular public meeting worked well, whether there are opportunities for improvement, or whether certain problems need to be addressed. Once you have summarized the data, Agency staff can assess whether they met goals, make modifications, and compare progress over time.

When should I use the questionnaires? You can use the first questionnaire ( for public meeting participants) towards the end of a public meeting. You can use the second questionnaire ( for EPA staff or their contractors who assisted in planning or leading a public meeting) after the same public meeting. You can use the third questionnaire ( for public meeting participants) when sending out follow up information to all public meeting participants. It is ultimately up to you, however, to decide when to use them. The key is to use the questionnaires to the extent that they provide you with valuable information and improve your ability to design and implement more effective public meetings in the future. Draft Draft Draft 44 Are the questionnaires in compliance with the Paperwork Reduction Act? Not yet. The questionnaires will be cleared with the Office of Management and Budget ( OMB) under ICR # XXXXX, OMB Control No: 2010 XXXX. This clearance will allow EPA to collect information ( via the questionnaires) from more than nine non federal entities. Currently, these questionnaires are undergoing a pre test.

Who designed these questionnaires?

The Evaluation Task Group of EPA's Public Involvement Workgroup designed these questionnaires after consulting with EPA staff who are regularly involved in the designing and holding of public meetings.

What should I do when preparing to administer a questionnaire?

See " Checklist for Administering the Questionnaire."

What do I do with the questionnaire data once the forms are returned? You can collect, collate and analyze the data from the forms using a pre formatted Lotus 1 2 3 spreadsheet program. In addition, the spreadsheet program allows you to summarize data over a period of time, making it easy to generate annual summary reports for your management. More information on how to use the program is available in the " Checklist for Administering the Questionnaire and Using Results" and within the Lotus 1 2 3 spreadsheet program. Once completed, Evaluation Task Group members encourage you to use your findings and send your completed data sheet files as e mail attachments to: bonner. patricia@ epa. gov. If you do so, we will be able to track the use of the questionnaires, and learn from you what is and is not working well, and gauge whether and how the questionnaires are useful to your efforts to improve public involvement activities. We will not use the information you send to judge the strength of individual public involvement efforts.

How can I use the results?

The completed questionnaires should provide you with information that can help you design improvements to activities and to your overall involvement process. You can learn what worked well, what participants feel is important and what should change to better accommodate their needs. It will be up to you to decide how much data you need to prompt changes. While it is prudent to be cautious about making changes based upon a small amount of feedback data, you may find a number of the respondents' ideas useful. If you can act on their suggestions, even in a small way, participants will appreciate that you listened.

How should I communicate the results? Use the performance and summary reports from the Lotus 1 2 3 spreadsheet program to talk with managers and staff about the effectiveness of the Public Meetings and where changes may be necessary. Communicate overall results and what changes you intend to make to public meeting participants in the public meeting summary report or other follow up materials distributed to session participants. A few brief sentences describing overall what participants thought of the session, their suggestions for improvement, and what you will do to improve the next public meeting would let participants know that you valued their input. Draft Draft Draft 45 What is EPA's Public Involvement Policy and where can I get more information?

EPA's 2003 Public Involvement Policy underscores EPA's commitment to improving its practices and is based on seven steps for effective public involvement: 1. Plan and budget for public involvement activities 2. Identify the interested and affected public 3. Consider providing technical or financial assistance to the public to facilitate involvement 4. Provide information and outreach to the public 5. Conduct public consultation and involvement activities 6. Review and use input, and provide feedback to the public 7. Evaluate public involvement activities The Framework for Implementing EPA's Public Involvement Policy urges development of tools to improve public involvement information sharing, training and evaluation at EPA. This questionnaire and the database program are examples of such tools. Cross agency work groups developed the Policy and Framework using public comments and ideas gathered through a two week, Internet based Dialogue on Public Involvement in EPA Decisions. Copies of the Policy and Framework are available @ http:// www. epa. gov/ publicinvolvement/ public/ index. htm. Draft Draft Draft 7Note: The instructions above are designed primarily for questionnaires that seek the perspective of participants about the effectiveness of a single event. Depending upon the type of questionnaire you wish to distribute and the type of public involvement activity you wish to acquire feedback for, you will need to tailor the steps of this checklist accordingly.

46 Assessing EPA's Public Involvement Activities Checklist for Administering the Questionnaire and Using Results7 Review questionnaire.

Determine whether questionnaire should be administered.

Determine what questionnaire( s) should be administered.

Identify one person to be responsible for coordinating the overall dissemination and collection of the questionnaire and to input the data into the Lotus 1 2 3 data input, analysis, and reporting program once the questionnaires are collected.

Make plans to include the questionnaire feedback process in the agenda of your public involvement event. Specifically, plan to set aside a few minutes before the conclusion of your public involvement event and have participants fill out their questionnaires. You should also plan to make an announcement at the beginning of your public involvement event explaining the questionnaire feedback process and how feedback is needed from all participants in order to develop a more comprehensive understanding of the overall effectiveness of the event and how best to make improvements. Feedback is needed from participants who leave early and those who stay to the end. If you don't build the questionnaire feedback process directly into your event, and instead wait to the very end, chances are few participants will take the time to fill out their questionnaires. Participants may suggest that they would prefer to fill out their questionnaires at home or work. While this option is appealing for participants, chances are only a few will take the time to actually do this. The best chance for good feedback is when participants are actually at the event. Take the time to plan for this!

Estimate the number of participants who will be attending the public involvement event for which questionnaires will be distributed.

Make appropriate number of copies of the questionnaire.

Bring questionnaire copies and extra pens or pencils to the public involvement event.

Identify official who will be responsible for physically collecting the questionnaires.

Clearly identify areas where participants should return their questionnaires.

Hand out questionnaires with other public involvement activity materials either at the beginning of the event, or at a designated time during the event. Draft Draft Draft 8Note: Separate Lotus 1 2 3 programs have been developed that match most types of questionnaires available ( e. g., one for single event effectiveness for participants, one for overall effectiveness for participants, etc.). Be sure to use the appropriate Lotus 1 2 3 program to input your data. The only type of questionnaire for which a spreadsheet program has not been developed is for the questionnaire dealing with post event follow up. However, if you need a spreadsheet program for this questionnaire, contact Pat Bonner ( bonner. patricia@ epa. gov).

47 Make it very clear to participants to whom, or to where, they should return their questionnaire. Urge participants to fill out their questionnaire even if they need to leave early.

Identify the total number of participants attending ( e. g., the total number of persons who could possibly fill out questionnaires).

Collect questionnaires.

Input data into the Lotus 1 2 3 data input, analysis, and reporting program within one month of public involvement event and share results.

Detach the Lotus 1 2 3 program onto your computer. 8 Open the Lotus 1 2 3 program.

Click on the " Getting Started" tab and follow the instructions.

Review performance report and identify whether certain changes should be considered in preparation for the next public involvement event.

Use the Lotus 1 2 3 performance reports to talk with managers and staff about the effectiveness of your public involvement activity and where changes may be necessary. Communicate overall results and what changes you intend to make to participants who attended your last public involvement event in the public involvement event summary report or other follow up materials distributed to participants. A few brief sentences describing overall what participants thought of the event, their suggestions for improvement, and what you will do to improve the next event would let participants know that you valued their input.

Sharing data... EPA's Public Involvement Policy Implementation Staff strongly encourages you to send your completed Lotus 1 2 3 program files as e mail attachments to: bonner. patricia@ epa. gov. If you do so, Implementation Staff will be able to track the use of the questionnaires, and learn from you what is and is not working well, and gauge whether and how the questionnaire was useful to your efforts to improve public involvement activities. We will not use the information you send to judge the strength of individual public involvement efforts.

When you conduct another public involvement event, repeat the same steps listed above. However, after doing this, be sure to go to the same Lotus 1 2 3 spreadsheet and follow the steps required to link the data from your first public involvement event to your second in order to summarize and compare your data over a period of time. Draft Draft Draft 48 OMB CONTROL NO:

EXPIRATION DATE:

Questionnaire for Feedback on the Effectiveness of a Public meeting Participant Assessment Background This questionnaire is designed to help Agency staff better understand what worked well and what improvements to consider implementing before holding future Public Meetings. Once the data are summarized, Agency staff can assess whether goals for success were met, make modifications if necessary, and compare progress over time. Thanks in advance for taking time to fill out the questionnaire. We value your input!

Directions Rate the following statements on a scale from 1 7 and write your own opinion to questions in the space provided. If you have no opinion, circle the number representing " Don't know." Please return your questionnaire to an EPA official before you leave.

1a. Pre meeting publicity availability Please indicate how you heard about the public meeting ( check all that apply): radio announcements television announcements newspapers ads flyers in store windows/ bulletin boards website information Other ( please specify)

Was the information about the public meeting made available in locally used languages? Yes No__

If you received a pre meeting notice via regular or electronic mail, how much in advance did the notice arrive? ___ a few days one week 2 weeks 2 4 weeks more than 1 month___ other ( please specify)

Please suggest any additional ways you think EPA should publicize future public meetings:

Please explain what you learned about the public meeting or issue that made you decide to Draft Draft Draft 49 attend:_____________________________________________________________________

1b. Pre meeting background information Was anything missing from the pre meeting background information? Yes___ No___ If yes, please specify:

Did you call to ask for specific information? Yes No If yes, what did you ask for?

2. Meeting logistics Str on gly dis agr ee Dis agr ee So me wh at dis agr ee So me wh at agr ee Agr ee Str on gly agr ee Do n't kno w Pre meeting invitation arrived early enough for me to plan attending. 1 2 3 4 5 6 7 Location was convenient. 1 2 3 4 5 6 7 Public transportation was close by. 1 2 3 4 5 6 7 Facility was comfortable for participants. 1 2 3 4 5 6 7 Time during which meeting was held was convenient. 1 2 3 4 5 6 7 The length of the meeting was about right. 1 2 3 4 5 6 7 Translators were available ( if needed). 1 2 3 4 5 6 7 Signers for hearing impaired were available ( if needed). 1 2 3 4 5 6 7 Provisions were made for handicap access. 1 2 3 4 5 6 7 Facility was well equipped for all planned activities ( tables, work space, break outs, supplies, etc). 1 2 3 4 5 6 7 Refreshments are important for a successful public meeting. 1 2 3 4 5 6 7 3. Conduct of meeting Draft Draft Draft 50 Str on gly dis agr ee Dis agr ee So me wh at dis agr ee So me wh at agr ee Agr ee Str on gly agr ee Do n't kno w Greeters made participants feel welcome. 1 2 3 4 5 6 7 The registration process was efficient. 1 2 3 4 5 6 7 I understood the purpose of the meeting. 1 2 3 4 5 6 7 I understood how the meeting would be conducted. 1 2 3 4 5 6 7 Meeting planners focused on the right 1 2 3 4 5 6 7 I was comfortable with the meeting format. 1 2 3 4 5 6 7 The format gave all ample opportunity to be 1 2 3 4 5 6 7 My ideas were heard. 1 2 3 4 5 6 7 There was good interaction among participants. 1 2 3 4 5 6 7 It was easy to sign up for follow up information. 1 2 3 4 5 6 7 There was a good mix of viewpoints. 1 2 3 4 5 6 7 I understand the " next steps" in the process. 1 2 3 4 5 6 7 Action items were documented ( flip charts). 1 2 3 4 5 6 7 Time frames and accountable persons were listed 1 2 3 4 5 6 7 Participants' input will make a difference. 1 2 3 4 5 6 7 Please suggest how EPA can improve the next public meeting:

Please indicate which group you represent:

neighborhood group local non profit organization ( not including local neighborhood non profit groups) Draft Draft Draft 51 national environmental organization business or industry or trade association state or tribal or local government federal government other ( please specify)

no organizational affiliation ( e. g., interested local resident)

Please return your completed questionnaire to an EPA official before you leave. Thank you for helping EPA improve its public involvement practices. Draft Draft Draft 52 Directions Rate the following statements on a scale from 1 7 and write your own opinion to questions in the space provided. If you have no opinion, circle the number representing " Don't know." Please return your questionnaire to an EPA official before you leave. OMB CONTROL NO:

EXPIRATION DATE:

Questionnaire for Feedback on the Effectiveness of a Public Meeting EPA/ Contractor Assessment Background This questionnaire is designed to obtain an EPA/ contractor perspective on the effectiveness of a Public Meeting. Agency staff will use this information to better understand what is working well and what improvements could make future Public Meetings more successful. Thanks in advance for taking time to fill out the questionnaire. We value your input!

Str on gly dis agr ee Dis agr ee So me wh at dis agr ee So me wh at agr ee Agr ee Str on gly agr ee Do n't kno w The registration process was efficient. 1 2 3 4 5 6 7 Participants understood the purpose of the meeting. 1 2 3 4 5 6 7 Participants understood how the meeting would be conducted. 1 2 3 4 5 6 7 The right topics were addressed during the meeting. 1 2 3 4 5 6 7 The format gave all ample opportunity to be 1 2 3 4 5 6 7 There was good interaction among participants. 1 2 3 4 5 6 7 The number of people that participated was appropriate. 1 2 3 4 5 6 7 There was a good mix of viewpoints. 1 2 3 4 5 6 7 EPA gained new, or clarifying, information, that will enable it to more effectively develop, revise, or implement rules or programs. 1 2 3 4 5 6 7 This public meeting was a success. 1 2 3 4 5 6 7 Please explain your ranking of the previous statement about success: Draft Draft Draft 53 Please suggest how EPA can improve its next public meeting:

Please return your completed questionnaire to an EPA official before you leave. Thank you for helping EPA improve its public involvement practices. Draft Draft Draft 54 Directions Rate the following statements on a scale from 1 7 and write your own opinion to questions in the space provided. If you have no opinion, circle the number representing " Don't know." Please return your questionnaire to an EPA official before you leave. OMB CONTROL NO:

EXPIRATION DATE:

Questionnaire for Feedback on the Effectiveness of Public Meeting Follow Up Participants' Assessment Background This questionnaire is designed to help Agency staff better understand what worked well and what improvements to consider before implementing future Public Meetings. Thanks in advance for taking time to fill out the questionnaire. We value your input!

Str on gly dis agr ee Dis agr ee So me wh at dis agr ee So me wh at agr ee Agr ee Str on gly agr ee Do n't kno w I received meeting notes. 1 2 3 4 5 6 7 Notes were clearly written. 1 2 3 4 5 6 7 Action items I remember were documented. 1 2 3 4 5 6 7 Actions, due dates and accountable persons were noted. 1 2 3 4 5 6 7 Issues discussed and decisions made were accurately recorded. 1 2 3 4 5 6 7 I received the information I requested in a timely manner ( in _______ days). 1 2 3 4 5 6 7 The packet I received was complete. 1 2 3 4 5 6 7 Now that some time has passed since the public meeting you may have thought about what could have made it a better experience. Please suggest how you think EPA can improve public meetings and follow up actions.

Please indicate which group you represent: Draft Draft Draft 55 neighborhood group local non profit organization ( not including local neighborhood non profit groups)

national environmental organization business or industry or trade association state or tribal or local government federal government other ( please specify)

no organizational affiliation ( e. g., interested local resident)

Please return the completed questionnaire in the pre addressed, stamped envelope provided. Thank you for helping EPA improve its public involvement practices. Draft Draft Draft 56 Assessing EPA's Public Hearings A User's Guide with Questionnaires Enclosed are information and tools for EPA staff to use in generating feedback on the effectiveness of public hearings and the effectiveness of follow up after them. Specifically, this user guide contains the following:

Background information for EPA Staff

Checklist for Administering Questionnaires and Using Results

Questionnaire for Feedback on the Effectiveness of a Public Hearing ( participant assessment)

Questionnaire for Feedback on the Effectiveness of a Public Hearing ( EPA/ contractor assessment)

Questionnaire for Feedback on the Effectiveness of Public Hearing Follow Up ( participant assessment)

For questions contact: Patricia Bonner ( bonner. patricia@ epa. gov)/ 202 566 2204 DRAFT: Last Updated 6/ 28/ 04 Draft Draft Draft 57 Assessing EPA's Public Hearings Background Information for EPA Staff Introduction Enclosed are three questionnaires for EPA staff to use in generating feedback on the effectiveness of public hearings. The questionnaires are a component of the evaluation section in the Framework for Implementing EPA's Public Involvement Policy. The questionnaires are for people who attend a public hearing, for EPA/ contractor staff responsible for coordinating the work of a public hearing, and for public hearing participants who receive follow up information following the public hearing. The first questionnaire focuses on participant perspectives of the effectiveness of a public hearing. The second questionnaire focuses on EPA/ contractor perspectives about the success of the same public hearing. The third questionnaire focuses on participant perspectives of the quality of the follow up information received after the same public hearing.

What is a public hearing? A public hearing is a forum to enable stakeholders and the public to directly address EPA officials and formally state their concerns for an official record. EPA does not respond to public comments at the hearing. Public hearings are held: when requested by a member of the public during a public comment period; during a public comment period following the issuance of a draft permit, major permit modification, or at the selection of a proposed corrective measure; and when the level of community concern warrants a formal record of communication. Typically, a public hearing will have a well defined procedure for presenting comments, formal time limits on statements, a hearing official or officials to hear the comments and ask clarifying questions, and the requirement to make all records available to the public. The purpose of a public hearing is to enable EPA to obtain input from a wide range of interested and affected parties so the Agency can better understand and consider the issues relevant to a particular decision.

How can the public hearing questionnaires be useful to you?

Use of the questionnaires should make the process of gathering feedback for your public involvement activity easier for you to implement directly, without spending contract dollars. The questionnaires provide an easy way to get useful feedback from participants and EPA about the public hearings they participated in. This feedback should help you to better understand whether a particular public hearing worked well, whether there are opportunities for improvement, or whether certain problems need to be addressed. Once you have summarized the data, Agency staff can assess whether they met goals, make modifications, and compare progress over time.

When should I use the questionnaires? You can use the first questionnaire ( for public hearing participants) towards the end of a public hearing. You can use the second questionnaire ( for EPA staff or their contractors who assisted in planning or leading a public hearing) after the same public hearing. You can use the third questionnaire ( for public hearing participants) when sending out follow up information to all public hearing participants. It is ultimately up to you, however, to decide Draft Draft Draft 58 when to use them. The key is to use the questionnaires to the extent that they provide you with valuable information and improve your ability to design and implement more effective public hearings in the future.

Are the questionnaires in compliance with the Paperwork Reduction Act? Not yet. The questionnaires will be cleared with the Office of Management and Budget ( OMB) under ICR # XXXXX, OMB Control No: 2010 XXXX. This clearance will allow EPA to collect information ( via the questionnaires) from more than nine non federal entities. Currently, these questionnaires are undergoing a pre test.

Who designed these questionnaires?

The Evaluation Task Group of EPA's Public Involvement Workgroup designed these questionnaires after consulting with EPA staff who are regularly involved in the designing and holding of public hearings.

What should I do when preparing to administer a questionnaire?

See " Checklist for Administering the Questionnaire."

What do I do with the questionnaire data once the forms are returned? You can collect, collate and analyze the data from the forms using a pre formatted Lotus 1 2 3 spreadsheet program. In addition, the spreadsheet program allows you to summarize data over a period of time, making it easy to generate annual summary reports for your management. More information on how to use the program is available in the " Checklist for Administering the Questionnaire and Using Results" and within the Lotus 1 2 3 spreadsheet program. Once completed, Evaluation Task Group members encourage you to use your findings and send your completed data sheet files as e mail attachments to: bonner. patricia@ epa. gov. If you do so, we will be able to track the use of the templates, and learn from you what is and is not working well, and gauge whether and how the questionnaires are useful to your efforts to improve public involvement activities. We will not use the information you send to judge the strength of individual public participation efforts.

How can I use the results?

The completed questionnaires should provide you with information that can help you design improvements to activities and to your overall involvement process. You can learn what worked well, what participants feel is important and what should change to better accommodate their needs. It will be up to you to decide how much data you need to prompt changes. While it is prudent to be cautious about making changes based upon a small amount of feedback data, you may find a number of the respondents' ideas useful. If you can act on their suggestions, even in a small way, participants will appreciate that you listened.

How should I communicate the results? Use the performance and summary reports from the Lotus 1 2 3 spreadsheet program to talk with managers and staff about the effectiveness of the Public Hearings and where changes may be necessary. Communicate overall results and what changes you intend to make to Draft Draft Draft 59 public hearing participants in the public hearing summary report or other follow up materials distributed to session participants. A few brief sentences describing overall what participants thought of the session, their suggestions for improvement, and what you will do to improve the next public hearing would let participants know that you valued their input.

What is EPA's Public Involvement Policy and where can I get more information? EPA's 2003 Public Involvement Policy underscores EPA's commitment to improving its practices and is based on seven steps for effective public involvement: 1. Plan and budget for public involvement activities 2. Identify the interested and affected public 3. Consider providing technical or financial assistance to the public to facilitate involvement 4. Provide information and outreach to the public 5. Conduct public consultation and involvement activities 6. Review and use input, and provide feedback to the public 7. Evaluate public involvement activities The Framework for Implementing EPA's Public Involvement Policy urges development of tools to improve public involvement information sharing, training and evaluation at EPA. This questionnaire and the database program are examples of such tools. Cross agency work groups developed the Policy and Framework using public comments and ideas gathered through a two week, Internet based Dialogue on Public Involvement in EPA Decisions. Copies of the Policy and Framework are available @ http:// www. epa. gov/ publicinvolvement/ public/ index. htm. Draft Draft Draft 1Note: The instructions above are designed primarily for questionnaires that seek the perspective of participants about the effectiveness of a single event. Depending upon the type of questionnaire you wish to distribute and the type of public involvement activity you wish to acquire feedback for, you will need to tailor the steps of this checklist accordingly. 60 Assessing EPA's Public Involvement Activities Checklist for Administering the Questionnaire and Using Results1 Review questionnaire.

Determine whether questionnaire should be administered.

Determine what questionnaire( s) should be administered.

Identify one person to be responsible for coordinating the overall dissemination and collection of the questionnaire and to input the data into the Lotus 1 2 3 data input, analysis, and reporting program once the questionnaires are collected.

Make plans to include the questionnaire feedback process in the agenda of your public involvement event. Specifically, plan to set aside a few minutes before the conclusion of your public involvement event and have participants fill out their questionnaires. You should also plan to make an announcement at the beginning of your public involvement event explaining the questionnaire feedback process and how feedback is needed from all participants in order to develop a more comprehensive understanding of the overall effectiveness of the event and how best to make improvements. Feedback is needed from participants who leave early and those who stay to the end. If you don't build the questionnaire feedback process directly into your event, and instead wait to the very end, chances are few participants will take the time to fill out their questionnaires. Participants may suggest that they would prefer to fill out their questionnaires at home or work. While this option is appealing for participants, chances are only a few will take the time to actually do this. The best chance for good feedback is when participants are actually at the event. Take the time to plan for this!

Estimate the number of participants who will be attending the public involvement event for which questionnaires will be distributed.

Make appropriate number of copies of the questionnaire.

Bring questionnaire copies and extra pens or pencils to the public involvement event.

Identify official who will be responsible for physically collecting the questionnaires.

Clearly identify areas where participants should return their questionnaires.

Hand out questionnaires with other public involvement activity materials either at the beginning of the event, or at a designated time during the event.

Make it very clear to participants to whom, or to where, they should return their questionnaire. Urge participants to fill out their questionnaire even if they need to leave early. Draft Draft Draft 1Note: Separate Lotus 1 2 3 programs have been developed that match most types of questionnaires available ( e. g., one for single event effectiveness for participants, one for overall effectiveness for participants, etc.). Be sure to use the appropriate Lotus 1 2 3 program to input your data. The only type of questionnaire for which a spreadsheet program has not been developed is for the questionnaire dealing with post event follow up. However, if you need a spreadsheet program for this questionnaire, contact Pat Bonner ( bonner. patricia@ epa. gov). 61 Identify the total number of participants attending ( e. g., the total number of persons who could possibly fill out questionnaires).

Collect questionnaires.

Input data into the Lotus 1 2 3 data input, analysis, and reporting program within one month of public involvement event and share results.

Detach the Lotus 1 2 3 program onto your computer. 1 Open the Lotus 1 2 3 program.

Click on the " Getting Started" tab and follow the instructions.

Review performance report and identify whether certain changes should be considered in preparation for the next public involvement event.

Use the Lotus 1 2 3 performance reports to talk with managers and staff about the effectiveness of your public involvement activity and where changes may be necessary. Communicate overall results and what changes you intend to make to participants who attended your last public involvement event in the public involvement event summary report or other follow up materials distributed to participants. A few brief sentences describing overall what participants thought of the event, their suggestions for improvement, and what you will do to improve the next event would let participants know that you valued their input.

Sharing data... EPA's Public Involvement Policy Implementation Staff strongly encourages you to send your completed Lotus 1 2 3 program files as e mail attachments to: bonner. patricia@ epa. gov. If you do so, Implementation Staff will be able to track the use of the questionnaires, and learn from you what is and is not working well, and gauge whether and how the questionnaire was useful to your efforts to improve public involvement activities. We will not use the information you send to judge the strength of individual public involvement efforts.

When you conduct another public involvement event, repeat the same steps listed above. However, after doing this, be sure to go to the same Lotus 1 2 3 spreadsheet and follow the steps required to link the data from your first public involvement event to your second in order to summarize and compare your data over a period of time. Draft Draft Draft 62 OMB CONTROL NO:

EXPIRATION DATE:

Questionnaire for Feedback on the Effectiveness of a Public Hearing Participant Assessment Background This questionnaire is designed to help Agency staff better understand what worked well and what improvements to consider implementing before holding future Public Hearings. Once the data are summarized, Agency staff can assess whether goals for success were met, make modifications if necessary, and compare progress over time. Thanks in advance for taking time to fill out the questionnaire. We value your input!

Directions Rate the following statements on a scale from 1 7 and write your own opinion to questions in the space provided. If you have no opinion, circle the number representing " Don't know." Please return your questionnaire to an EPA official before you leave.

1a. Pre hearing publicity availability Please indicate how you heard about the public hearing ( check all that apply): radio announcements television announcements newspapers ads flyers in store windows/ bulletin boards website information Other ( please specify)

Was the information about the public hearing made available in locally used languages? Yes No__

If you received a pre hearing notice via regular or electronic mail, how much in advance did the notice arrive? a few days one week 2 weeks 2 4 weeks more than 1 month__ __ other ( please specify)_________________________________________________

Please suggest any additional ways you think EPA should publicize future public hearings:

Please explain what you learned about the public hearing or issue that made you decide to attend: Draft Draft Draft 63 1b. Pre hearing background information Was anything missing from the pre hearing background information? Yes___ No___ If yes, please specify:_____________________________________________

Did you call to ask for specific information? Yes No If yes, what did you ask for?

2. Hearing logistics Str on gly dis agr ee Dis agr ee So me wh at dis agr ee So me wh at agr ee Agr ee Str on gly agr ee Do n't kno w I became aware of pre hearing notice early enough to plan attending. 1 2 3 4 5 6 7 Location was convenient. 1 2 3 4 5 6 7 Public transportation was close by. 1 2 3 4 5 6 7 Facility was comfortable for participants. 1 2 3 4 5 6 7 Time during which hearing was held was convenient. 1 2 3 4 5 6 7 The length of the hearing was about right. 1 2 3 4 5 6 7 Translators were available ( if needed). 1 2 3 4 5 6 7 Signers for hearing impaired were available ( if needed). 1 2 3 4 5 6 7 Provisions were made for handicap access. 1 2 3 4 5 6 7 3. Conduct of hearing Greeters made participants feel welcome. 1 2 3 4 5 6 7 The registration process was efficient. 1 2 3 4 5 6 7 I understood the purpose of the hearing. 1 2 3 4 5 6 7 I understood how the hearing would be conducted. 1 2 3 4 5 6 7 Hearing planners focused on the right questions. 1 2 3 4 5 6 7 I was comfortable with the hearing format. 1 2 3 4 5 6 7 Draft Draft Draft 64 3. Conduct of hearing ( continued)