THE ART OF EVALUATION
A HANDBOOK FOR EDUCATORS AND TRAINERS
Tara J. Fenwick and Jim Parsons
University of Alberta
THOMPSON EDUCATIONAL PUBLISHING, INC.
Toronto
Table of Contents
- Introduction, 11
CHAPTER 1
The Purposes of Evaluation, 13
- The Importance of Authentic Evaluation, 13
- Nine Purposes of Evaluation, 14
- Principles of Authentic Evaluation, 16
- Four Traps in Evaluation, 19
- A Final Thought, 20
CHAPTER 2
Evaluating Your Philosophy, 21
- Examining Your Own Beliefs, 22
- Reflecting on Your Own Learning and Teaching Biography, 23
- The Dilemma of Evaluating Other Adults, 25
- The Evaluative Context, 26
- A Final Thought, 26
CHAPTER 3
Planning for Evaluation, 27
- Why Should the Evaluation Take Place?, 27
- What Should Be Evaluated?, 28
- What Do You Want Learners To Know?, 28
- What Does the Institution Want To Know?, 29
- What Approaches Should Be Used?, 30
- What about Validity and Reliability?, 31
- How Much Time and Other Resources Do You Have?, 32
- Alleviating the Anxiety, 33
- A Final Thought, 35
CHAPTER 4
Developing Criteria for Evaluation, 39
- Criterion, Normative, and Self-Referenced Evaluation, 39
- Choosing a Frame of Reference, 40
- The Source for Criteria in Criterion-Referenced Evaluation, 40
- Indicators of Criteria, 44
- Involving Learners in Developing Indicators, 46
- A Final Thought, 47
CHAPTER 5
Choosing A Strategy, 49
- Multiple Methods, 49
- Choosing Assessment Methods, 49
- Questions to Consider before Using an Evaluation Strategy, 51
- Questions to Consider after Using an Evaluation Strategy, 51
- Involving the Learner in Assessment, 51
- Questions to Consider before Evaluating Learnersâ Progress, 53
- Management and Record-Keeping, 53
- A Final Thought, 53
CHAPTER 6
Evaluating Technical Skills, 57
- Two Kinds of Knowledge in Technical Skills, 57
- Operations Involved in Learning a New Skill, 58
- Context and Transfer in Skill Learning, 59
- Coaching and Evaluating Skills, 59
- A Final Thought, 61
CHAPTER 7
Evaluating Performance, 63
- Defining Objectives for Competency, 63
- Pre-Test/Post-Test Approaches to Determine Skill, 64
- Choosing Criteria for Evaluating Performance Objectives: Begin with the Learners, 68
- Evaluating ãEntry-Levelä Skills, 68
- A Final Thought, 69
CHAPTER 8
Evaluating Growth in Conceptual Knowledge, 73
- New Knowledge Is Actively Constructed, Not Passively Absorbed, 73
- Everyone Creates a Personal Knowledge, 73
- Learners Construct New Knowledge in Different Ways, 75
- Some Assessment Measures Can Strangle Conceptual Growth, 75
- Putting It All Together, 75
- A Final Thought, 77
CHAPTER 9
Evaluating ãLevelsä of Conceptual Growth, 79
- Bloomâs Taxonomy, 79
- Using Bloomâs Taxonomy, 79
- Formats That Assess Levels of Understanding, 81
- Limitations of Bloomâs Taxonomy, 82
- A Final Thought, 83
CHAPTER 10
Evaluating ãRelationalä Skills, 85
- How To Talk about Relational Skills, 85
- Relational Wisdom, 87
- What Can and Canât Be Measured, 89
- A Final Thought, 89
CHAPTER 11
Towards Dynamic Assessment, 91
- The Problem with Conventional Assessment, 91
- Dynamic Assessment÷An Empowering Alternative, 91
- Approaches to Using Dynamic Assessment, 93
- How To Incorporate Dynamic Assessment in Classrooms, 96
- Constructing Tasks for Dynamic Assessment in Classroom Situations, 101
- A Final Thought, 101
CHAPTER 12
Helping Learners Evaluate Themselves, 103
- Why Self-Evaluation Is a Critical Goal, 103
- Using Self-Assessment, 104
- Beginning Self-Assessment, 105
- Methods That Prompt Ongoing, Informal Self-Evaluation, 107
- Methods for Formal Self-Assessment, 109
- A Final Thought, 111
CHAPTER 13
Integrating Ongoing Evaluation into the Learning Process, 113
- Focusing the Learners on Assessment, 113
- Opening Evaluation Activities, 115
- Mid-Process Evaluation Activities To Be Used during Instruction, 115
- Follow-Up Activities for Ongoing Evaluation, 122
- A Final Thought, 122
CHAPTER 14
Evaluating for Grades, 123
- The Meaning of a Grade, 123
- What Does a Grade Mean to You, the Instructor?, 124
- What Grading Means in an Educational Institution, 124
- Grade Curves, 124
- Deriving the Grade: Issues to Consider, 126
- Alternative Methods of Deriving Grades, 128
- Maximize Your Minimal Time, 128
- Reporting Grades to Learners, 130
- Trouble-Shooting, 131
- A Final Thought, 137
CHAPTER 15
Evaluating Your Practice, 139
- Expanding Self-Evaluation and Growth, 139
- Returning to Your Beliefs, 141
- Evaluating Learner Satisfaction, 141
- Learner Reports to Instructors, 142
- Questioning Your Own Practice as an Evaluator, 143
- A Final Thought, 146
TOOLBOXES
TOOLBOX 1
Assessing Portfolios, 149
TOOLBOX 2
Assessing Learner Journals, 155
TOOLBOX 3
Assessing Written Assignments, 163
TOOLBOX 4
Assessing with Video, 175
TOOLBOX 5
Assessing through Performance Observation, 179
TOOLBOX 6
Assessing with Rating Scales, 189
TOOLBOX 7
Using Participant Course Evaluations, 193
TOOLBOX 8
Using Learner Contracts, 201
TOOLBOX 9
Using Case Studies for Evaluation, 209
TOOLBOX 10
Assessment in On-Line Courses, 215
TOOLBOX 11
Peer Assessment, 223
TOOLBOX 12
Using Objective Tests to Evaluate, 231
- Bibliography, 239
- Index, 241
List of Sample Rating Scales and Evaluation Activities
INSTRUCTORâS MATERIALS: These rating scales and evaluation activities are available in electronic form to instructors using this book as required text in a course of study. In these files, the background is clear and the page may be readily photocopied or made into overhead transparencies for classroom use. The files may be viewed and printed using the Adobe Acrobat Reader, which is free from www.adobe.com. To obtain the Adobe Acrobat (PDF) files for these pages simply, instructors should email the publisher with course details: publisher@thompsonbooks.com.
- Learner-Developed Criteria: Read-Around Groups, 46
- Evaluation Methods, 50
- Determining Learner Evaluation Preferences, 52
- Techniques for Evaluating Individual Student Learning, 54
- Developing Indicators To Rate Levels of Technical Skill Performance, 66
- Instructor Self-Evaluation: Teaching Technical Skills, 67
- Chart for Assessing Observable Outcomes, 70
- Sample Activities To Assess Learnersâ Growth in Conceptual Knowledge, 74
- Ongoing Self-Evaluation of Thinking Skill Development, 76
- Changes in Evaluation Approaches, 78
- Bloomâs Taxonomy, 80
- Developing Relational-Skills Criteria, 88
- Activities for Dynamic Assessment Incorporating Critical Reflection, 98
- Moving towards Dynamic Assessment, 100
- Moments of Awareness (MA) Exercise, 104
- Quick-Memo Ideas, 105
- Getting on the Same Track, 107
- ãLadder of Inferenceä Left-Hand Column Exercise, 108
- Self-Assessment, 110
- ãPAPAä: Participant Action Plan and Analysis, 110
- Reflection Notes for a Workshop, 111
- Self-Assessment: Work Experience, 112
- Ongoing Evaluation of Coaching Skills: Case Example, 114
- Listening Skills Inventory: ãCoaching Skills for Supervisorsä Workshop, 116
- Personal Inventory Opening: Change-Management Workshop, 118
- Personal Inventory: ãPowerful Presentationsä Workshop, 119
- Peer Feedback Rating Scale: Evaluating Teacher-Parent Interviews, 120
- Action Plan, 121
- Checklists for Monthly Self-Assessment on the Job, 122
- An Example of Two Grading Systems, 126
- Assignment List: History Course, 130
- Questions to Ask When Planning the Weight of an Assessment, 132
- Three Grading Alternatives for Class Participation, 135
- Four Small-Group Grading Approaches, 136
- Examining Seven Elements of Learning, 142
- Six Ideas for Speedy Memos, 144
- Three Methods for Evaluating Learner Satisfaction, 144
- Survival Advice Memo, 146
- Self-Reflective Teacher Questionnaire, 147
- Mirror Image Exercise, 148
- Sample Portfolio Assessment, 152
- Journal Starters, 158
- Journal Self-Evaluation, 159
- Journal Assessment: Holistic Scale, 160
- Rating Scale for Evaluating Journals, 162
- Sample Preliminary Writing Plan, 166
- Peer Responses in One-On-One Writing Conference, 168
- Peer Feedback to Individual Presentation, 169
- Scoring Guide for Opinion Paper, 171
- Scoring Guide for Argumentative or Research Essay, 172
- Holistic Guide for Evaluating Writing, 174
- Suggested Protocol for Group Analyses of Performance Video, 179
- Anecdotal Description, 185
- Group Checklist: Informal Skill Observation, 187
- Formative Feedback for Practice Session, 189
- Scoring Scale for Formal Writing, 193
- Participant Response to Course Experiences, 197
- Sample Participant Course Evaluation Form, 200
- Learning Contract, 204
- Independent Study Proposal, 206
- Rating Scale: Learning Contract for Focus-Group Research, 208
- Evaluation of Learning Project: Process and Product, 210
- Five Steps to Case Study Analysis, 212
- Two Sample Rating Scales: Case Studies, 213
What This Book Is All About
Evaluation is an integral part of learning. In this age of accountability, assessment of learning outcomes has become a critical, albeit daunting, responsibility for many educators. We are supposed to develop clear criteria for learner capacities that are frequently emergent, unclear, and constantly changing. We are supposed to report comprehensive and realistic measurements of highly complex and often invisible human understandings and abilities. Above all, presumably, we hope to honour learners and their individual processes and keep their growth and their own understanding of their development at the centre of our evaluation purposes.
This book is intended as an introduction to learner evaluation in various contexts of adult education. We have attempted to address issues and provide examples that would fit the interests of students of adult education, new college and university teachers, trainers in business and government and consultants whose work involves learner evaluation. Every educator works within a unique context of learners, purposes, structures, and cultures. Every reader, therefore, needs to be critical and selective when choosing among the materials offered in this book. Some of it will suit you, some may not fit your philosophy or your situation, and some will need adaptation to be useful in your work.
Primarily, the book is intended for use as a practical manual. This is why we have included many activities and a Toolbox section offering specific strategies ÷ the nuts and bolts of evaluation methods. Sample forms are provided, such as checklists and rating scales, which we encourage you to modify to adapt effectively to your own contexts and preferences. (These forms are available and can be downloaded for your use.) We hope that the book becomes a useful reference for beginning educators and trainers, and perhaps even for experienced instructors looking for some new strategies and perspectives.
Our focus here is the learner and instructor, rather than the program. Of course, all evaluation of learning must be considered within the total context of the program and institution. However, program evaluation seeks different purposes, resolving different questions than learner evaluation. For this reason we have touched only briefly upon issues of program evaluation, and only insofar as these affect learner assessment.
We start with the basics. In Chapters 1 through 5 we encourage readers to examine carefully fundamental issues in evaluation purposes, planning, developing criteria, and choosing strategies for evaluation. In Chapters 6 through 10 we explore different evaluation approaches for assessing conceptual growth, technical skill mastery, and relational skills. For instructors working within an institution, we have included a chapter about the meaning of grades and grading practices. You will see our bias towards integration of authentic assessment into all parts of learning and life in Chapters 11, 12, and 13. These are crucial chapters for us, offering arguments and suggestions for making assessment ãdynamic,ä and for focusing most on learnersâ self-assessment and continuous assessment as an ongoing integral part of education.
And finally, the book ends the way it begins: with a chapter encouraging readers to reflect deeply upon their own practice and philosophy in evaluating learners. We believe that the richest growth opportunities for us as educators are embedded in our own dilemmas of everyday practice and the selves which emerge in them. When we are able to confront these dilemmas knowledgeably, lovingly, critically, and courageously, we can begin to work through them. Only then do we begin doing what this book suggests that we help learners to do÷integrate reflective self-evaluation into the fabric of our lives.