You Call This Exemplary? Lessons from an Unsung International Evaluation
This paper reflects on the role of academic discipline and epistemic community in judging what is an exemplary evaluation. It examines the case of an evaluation that was considered ‘exemplary’ by a panel of program evaluators, but methodologically flawed by evaluators from a different evaluation tr...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Evaluation Center at Western Michigan University
2017-05-01
|
| Series: | Journal of MultiDisciplinary Evaluation |
| Subjects: | |
| Online Access: | https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/469 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841561319253213184 |
|---|---|
| author | Douglas Horton |
| author_facet | Douglas Horton |
| author_sort | Douglas Horton |
| collection | DOAJ |
| description |
This paper reflects on the role of academic discipline and epistemic community in judging what is an exemplary evaluation. It examines the case of an evaluation that was considered ‘exemplary’ by a panel of program evaluators, but methodologically flawed by evaluators from a different evaluation tradition. The evaluation in question was carried out within an international agricultural research network (known as the CGIAR), with a rich tradition of economic impact assessment. The evaluation was carried out by a team of experienced program evaluators, who attempted to apply accepted good practices in the program evaluation community. The evaluation employed mixed methods and multiple data sources with heavy reliance on triangulated perceptual data. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. However, within the CGIAR, both the evaluation and the meta-evaluation study were rejected, as methodologically flawed. The paper closes with four propositions related to what is considered an “exemplary evaluation.”
Background: Program evaluators have reached broad agreement on principles for planning and conducting evaluations and on standards for judging their quality. However, many evaluation stakeholders, including key intended users, may judge evaluations on criteria that differ sharply from the professional standards and the criteria we commonly employ in meta-evaluations.
Purpose: This paper highlights the role of academic discipline and epistemic community in judging what is an “exemplary” evaluation, by examining the case of an evaluation that was considered exemplary by professional program evaluators, but methodologically flawed by professionals from different disciplinary traditions.
Setting: The evaluation in question was carried out within the Consultative Group on International Agricultural Research (CGIAR), a research network with a rich tradition of economic impact assessment.
Intervention: NA
Research Design: This is a case study that combines participatory action research and historical analysis.
Data Collection and Analysis: The study is based on the author’s personal involvement in the evaluation and on a review of publications and unpublished documents related to the case.
Findings: A team of experienced evaluators applied what are generally considered to be good practices in the program evaluation community. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. In contrast, within the CGIAR, both the evaluation and the meta-evaluation study were considered to be methodologically flawed and biased. Three lessons related to exemplary evaluation are formulated and elaborated upon:
Lesson 1. Being exemplary is in the eyes of the beholder.
Lesson 2. Epistemic communities are hard nuts to crack.
Lesson 3. You can’t win them all.
While the early efforts with program evaluation analyzed here were experienced as failures, a number of subsequent developments have led to greater understanding of diverse evaluation approaches, and some movement toward agreement on what constitutes exemplary evaluation in the CGIAR. Nevertheless, there is still a considerable way to go.
|
| format | Article |
| id | doaj-art-2e8d0df501f448578d2a852bec9bea1c |
| institution | Kabale University |
| issn | 1556-8180 |
| language | English |
| publishDate | 2017-05-01 |
| publisher | The Evaluation Center at Western Michigan University |
| record_format | Article |
| series | Journal of MultiDisciplinary Evaluation |
| spelling | doaj-art-2e8d0df501f448578d2a852bec9bea1c2025-01-03T01:46:32ZengThe Evaluation Center at Western Michigan UniversityJournal of MultiDisciplinary Evaluation1556-81802017-05-01132810.56645/jmde.v13i28.469You Call This Exemplary? Lessons from an Unsung International EvaluationDouglas Horton0Independent evaluator, University Park, FL This paper reflects on the role of academic discipline and epistemic community in judging what is an exemplary evaluation. It examines the case of an evaluation that was considered ‘exemplary’ by a panel of program evaluators, but methodologically flawed by evaluators from a different evaluation tradition. The evaluation in question was carried out within an international agricultural research network (known as the CGIAR), with a rich tradition of economic impact assessment. The evaluation was carried out by a team of experienced program evaluators, who attempted to apply accepted good practices in the program evaluation community. The evaluation employed mixed methods and multiple data sources with heavy reliance on triangulated perceptual data. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. However, within the CGIAR, both the evaluation and the meta-evaluation study were rejected, as methodologically flawed. The paper closes with four propositions related to what is considered an “exemplary evaluation.” Background: Program evaluators have reached broad agreement on principles for planning and conducting evaluations and on standards for judging their quality. However, many evaluation stakeholders, including key intended users, may judge evaluations on criteria that differ sharply from the professional standards and the criteria we commonly employ in meta-evaluations. Purpose: This paper highlights the role of academic discipline and epistemic community in judging what is an “exemplary” evaluation, by examining the case of an evaluation that was considered exemplary by professional program evaluators, but methodologically flawed by professionals from different disciplinary traditions. Setting: The evaluation in question was carried out within the Consultative Group on International Agricultural Research (CGIAR), a research network with a rich tradition of economic impact assessment. Intervention: NA Research Design: This is a case study that combines participatory action research and historical analysis. Data Collection and Analysis: The study is based on the author’s personal involvement in the evaluation and on a review of publications and unpublished documents related to the case. Findings: A team of experienced evaluators applied what are generally considered to be good practices in the program evaluation community. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. In contrast, within the CGIAR, both the evaluation and the meta-evaluation study were considered to be methodologically flawed and biased. Three lessons related to exemplary evaluation are formulated and elaborated upon: Lesson 1. Being exemplary is in the eyes of the beholder. Lesson 2. Epistemic communities are hard nuts to crack. Lesson 3. You can’t win them all. While the early efforts with program evaluation analyzed here were experienced as failures, a number of subsequent developments have led to greater understanding of diverse evaluation approaches, and some movement toward agreement on what constitutes exemplary evaluation in the CGIAR. Nevertheless, there is still a considerable way to go. https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/469exemplary evaluationevaluation principles and standardsdisciplinary perspectivesepistemic community |
| spellingShingle | Douglas Horton You Call This Exemplary? Lessons from an Unsung International Evaluation Journal of MultiDisciplinary Evaluation exemplary evaluation evaluation principles and standards disciplinary perspectives epistemic community |
| title | You Call This Exemplary? Lessons from an Unsung International Evaluation |
| title_full | You Call This Exemplary? Lessons from an Unsung International Evaluation |
| title_fullStr | You Call This Exemplary? Lessons from an Unsung International Evaluation |
| title_full_unstemmed | You Call This Exemplary? Lessons from an Unsung International Evaluation |
| title_short | You Call This Exemplary? Lessons from an Unsung International Evaluation |
| title_sort | you call this exemplary lessons from an unsung international evaluation |
| topic | exemplary evaluation evaluation principles and standards disciplinary perspectives epistemic community |
| url | https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/469 |
| work_keys_str_mv | AT douglashorton youcallthisexemplarylessonsfromanunsunginternationalevaluation |