|
Evaluation
SECTION 6 OUTLINE
EVALUATION
To evaluate means, “to ascertain the worth of”. The evaluation process gives you information about the worth of what you are doing, and how to do the job better. Evaluation allows you to ask the questions you want to ask about the project, collect the appropriate information, and then use it to reshape, re-frame, and redirect activities, or keep them on track, depending on what the data tell you.
In this Section we elaborate on the reasons for conducting evaluations, describe the types of evaluation that you can engage in, and suggest how to go about conducting an evaluation. There is also information on what indicators can be used to show you’ve achieved project objectives. A project-planning framework (LogFrame) is provided as an example of a tool that can help people see the logic in a project. This tool also offers a clear process for developing a monitoring and evaluation system.
Benefits of Evaluation
When evaluations are done right, the people collecting the data know why they are doing so, and the data collected are used to improve the services offered, and to enhance the short-term and long-term benefits of the project. Briefly, evaluation allows you to accomplish the following:
• Determine if what you are doing is worthwhile.
• Identify unintended consequences.
• Determine if the program is actually being implemented as designed.
• Improve project management.
• Respond to the needs of stakeholders.
• Identify key variables.
• Assess the viability of expanding small-scale projects.
• Build institutional capacity.
• Be accountable.
• Build a case for increased support for young children and their families.
Types of Evaluation
There are several different kinds of evaluation, each of which contributes a somewhat different perspective on the project. The seven types we will describe here are diagnostic, monitoring, process, effectiveness, impact, relevance, and sustainability evaluation. With each definition we include a set of questions that are addressed by that type of evaluation.
• Diagnostic evaluations are carried out during the design and planning of the project.
• Monitoring is carried out once the project is under way.
• Process evaluation is designed to improve the activities that are being implemented within the project.
• Effectiveness evaluation is carried out after a project has been under way for some time, but this type of evaluation is still directed principally toward improving the project activities and design.
• Impact evaluation is used to determine if the project has had the desired effect on participants.
• Relevance evaluation has to do with whether or not the project is continuing to meet a need.
• Sustainability evaluation looks at what is likely to remain once initial funding comes to an end.
Developing an Evaluation Plan
The evaluation should be designed at the same time as the project is designed. Evaluation systems need to be in place as the project begins and data should be gathered along the way. All too often the project is well under way–and may, in fact be close to the end of a funding cycle– before evaluation is discussed for the first time.
The stakeholders–from parents to policy-makers–should be a part of defining the desired outcomes and be a part of collecting the data. It is important to work with those involved in developing and implementing the program to identify what they need to know in order to do their jobs better and develop ways to gather that information.
·SIDE TRIP LINK: Evaluation Design, UNICEF/United Nations Development Programme, Socialist Republic of Vietnam.
Who Needs the Information?
What Is To Be Evaluated?
How Often Should Data Be Collected, and When, So That Sound Decisions
Can Be Made?
• When the project begins.
• At a mid-point in the project.
• Toward the end of the project.
• Longitudinally.
Who Will Conduct and Be Involved in the Evaluation?
IDENTIFYING THE SKILLS REQUIRED
INTERNAL VS. EXTERNAL EVALUATION
How Will Data Be Collected? What Instruments Will Be Used?
INFORMATION THAT IS ALREADY AVAILABLE
CHILDREN'S PARTICIPATION
SURVEYS
QUESTIONNAIRES
INTERVIEWS
OBSERVATIONS
What Resources Are Required to Support the Monitoring and Evaluation Systems?
How Should the Data Be Presented?
• Provide a balance between numbers and description (i.e., quantitative and qualitative data).
• Present information on the challenges as well as the accomplishments.
• Provide information on what you will do next.
Identification of Appropriate Indicators
One of the keys to an effective evaluation is the identification of what you are going to look at to determine whether or not the project is on track and whether or not it is having the desired effect. For this reason, it is critical to spend time during the project planning to identify the indicators you will look at to make an assessment of project effectiveness, and how these indicators will be measured.
The development of appropriate indicators is one of the major challenges in the ECCD field. Should indicators be interpreted against a fixed and absolute standard or should the interpretation be relative and focus on improvements?
Interpreting Indicators
·SIDE TRIP LINK: Indicators of the Current Nutritional Condition of Women and Children.
Multiple Goals and Purposes
Early Childhood Psycho-Social Indicators
·ECCD BRIEF LINK: Psycho-social Skills Categories.
·SIDE TRIP LINK: Standardized Indicators of Program Impact Agency-Wide:and Example from the Christian Children's Fund, CCF.
·SIDE TRIP LINK: P. Greenfield. 1997. You Can’t Take it With You: Why Ability Assessments Don’t Cross Cultures.
·LIBRARY LINK: C. Landers and C. Kagitçibasi. 1990. Measuring the Psychosocial Development of Young Children: Innocenti Technical Workshop, Summary Report.
Cross-Cultural Research: A Discussion of the Issues
There is considerable debate regarding the extent to which cross-cultural measures can be developed. One position advocated by cross-cultural psychologists is that, “ability tests are intrinsically transportable from one culture to another”. An alternative position is taken by cultural psychologists.
The Local Definition of Intelligence
Cultural Representation in Development of Assessment Procedure
·ECCD BRIEF LINK: IEA Preprimary Project: An International Study of Early Childhood Care and Education.
·ECCD BRIEF LINK: IEA PREPRIMARY PROJECT: An Observational Study of Early Childhood Settings.
>
Maintain Validity
Accurately Assess Children’s Knowledge
Establish Tester-Child Relationships
Child Status Profile
Components of a Child Status Profile (CSP)
Measuring the Components Within the Child Status Profile
School Status Profile
Availability of and Access to Schooling
Quality
Teacher Expectations
Responsiveness to Local Needs and Circumstances
Measuring the Components of School Readiness
Conceptual Framework for an Evaluation
A variety of techniques have been developed to help program planners and managers develop an evaluation design consistent with project goals and objectives and program design. One of these is known as a Logical Framework Analysis (LogFrame). This is a tool that provides a structure for specifying the components of a project and the logical links between them.
·SIDE TRIP LINK: Project Appraisal Document on a Proposed International Development Association Credit to Uganda for a Nutrition and Early Childhood Development Project. M. Garcia.
Conclusion
There are several issues that make it difficult to conduct good evaluations. Some of these have been defined by Stern (1990). They include:
• Evaluation often needs to be responsive to the tensions between organizational accountability on the one hand and organizational learning on the other.
• A lack of clarity about objectives often makes it difficult to design an evaluation.
• When external evaluators are involved, the independence of the external evaluator is often threatened and difficult to maintain.
• Key groups are often excluded or given little involvement in evaluations.
• The timescale for evaluations is often problematic–they are often either initiated too late too early for valid conclusions to be drawn.
·ECCD BRIEF LINK: Calculating Cost Savings: The High/Scope Perry Pre-school Project.
|
