Nationally, there is movement towards achieving high learning standards and promoting accountability in education. Increasingly, funders are reviewing a program's results to determine future funding. As adult education programs progress into the new customer service environment established by the Workforce Investment Act, programs must be able to document results to attract new customers and to become part of the local workforce investment board. Programs must achieve the state performance standards, operate in compliance with federal and state regulations, and create an organizational atmosphere of continuous improvement.

Summative and Formative Evaluation

Before discussing specific state performance indicators, it is important to have an understanding of the two kinds of program evaluation systems: summative and formative. Summative evaluations are a summary of the results of the program over a specified period. They study the success of a program in meeting its outcomes and objectives. The results of these evaluations are typically sent to funders, regulators, and other interested stakeholders. The National Reporting System for Adult Education is an example of a summative evaluation. For adult education programs, summative evaluations generally measure program success in having learners achieve the following:

• development of literacy skills/educational gain
• attainment of educational completion (GED or HS diploma)
• placement in a job or a further educational program
• job retention over a specific period of time.

Formative evaluations, on the other hand, are for internal use to study program effectiveness on an on-going basis. Program staff use formative evaluations to determine the effectiveness of specific strategies. Those strategies that are effective are continued and expanded, while those that are not are discontinued or modified. Formative evaluations are designed as early warning systems for areas where the program is not doing well, so corrective actions can be taken quickly to get the program back on track. Formative evaluations might study:

• educational needs the program is meeting most effectively
• educational strategies most effective in a particular community
• recruitment and retention strategies resulting in the best student mix
• the greatest barriers to learning.

Comprehensive Performance Accountability System

Title II of the Workforce Investment Act (WIA) requires all states to establish a comprehensive performance accountability system. Such a system assesses the effectiveness of eligible agencies in achieving continuous improvement of adult education programs. According to the US Department of Education's Office of Vocational and Adult Education:

At the core of the adult education performance system are the number of adult learners who complete one or more educational functioning levels during the program years. ...The remaining core measures are follow-up measures collected after the adult leaves the program. These measures include:

• high school completion
• entered postsecondary education or training
• entered employment and retained employment.

The core outcome measures are summarized below:

Each state negotiated levels of performance measures. Local agencies receiving grants from the New York State Education Department for the State performance levels.

National Reporting System for Adult Education

To support the documentation of the performance indicators and the management of the adult education delivery system, the USDOE worked with Pelavin Associates to develop the National Reporting System for Adult Education. The National Reporting System for Adult Education (NRS) is an annual report that state education agencies use to submit data on funded adult education programs. The report includes participant demographics, learner gains, and learner outcomes. In New York State, this report was amended to include information on program and service offerings.

The NRS and the performance indicators necessitate the use of an individual student record system (ISR). An ISR is a computerized database that allows interested stakeholders, notably adult education providers and the state education agency, to conduct analysis to support continuous improvement.

There is a wide variety of computer software that will support an ISR system. Some software applications may be used right "off-the-shelf" and others require local programs to design a custom application. Program managers should consult with managers from other agencies to identify software applications that are currently used and determine pros and cons before purchasing and implementing a system.

Continuous Improvement

While much of the drive of the performance indicator movement is towards accountability, an equally important aspect is to use the data collected to improve the delivery of instructional and support services. Continuous improvement may be approached in two ways. The first is to review the summative data of the program. This data is submitted to funders. Analysis of this data will enable program managers to review the results of the overall adult education program and make modifications in the future. The drawback of using a summative analysis of the adult education program and services is that the program is already completed and no action can be taken to improve program results.

To make adjustments to program and service delivery during, instead of after, the program year, managers and staff can use a formative evaluation. It may review recruitment and marketing, retention, and learner gains. Whether evaluations are formal or informal, it is important to obtain sufficient and reliable data to make informed decisions. An example of a formative evaluation is the comparison of pre-and post-test data on learner gains. Did the learner achieve the desired gain? If yes, what worked well and can be replicated for other learners in other classes? If the learner did not achieve the desired gain, why not? What other instructional strategies or curriculum could be used? Strategies that are working well should be expanded and replicated. Areas that are weak need to have new strategies developed to provide stronger outcomes.

Changes should be implemented incrementally. Slow, steady changes that are carefully assessed are often more effective in the long run than massive, radical changes implemented quickly. It is important to continually monitor and measure changes in the program to be sure that the new design is resulting in the desired outcome.

Compliance

Compliance is the process by which a program manager ensures that the adult education program is operating within the rules and regulations of the funder. This includes:

• learner eligibility in programs and services
• student data that must be collected at in-take as well as upon exiting of the program
• documentation of student attendance
• documentation of revenue and expenditures
• the number of learners that may be enrolled in a class.

It is important for program mangers to understand the slight differences among funding sources. Knowing these differences may enable the adult education program to work with the local workforce investment board and provide programs to a population not previously served. Not knowing the differences may result in an audit exception or closing of the program.