In the user-experience profession, we preach iteration and evaluation. There are two types of evaluation, formative and summative, and where you are in the design process determines what type of evaluation you should conduct.
Formative evaluations focus on determining which aspects of the design work well or not, and why. These evaluations occur throughout a redesign and provide information to incrementally improve the interface.
Let’s say we’re designing the onboarding experience for a new, completely redesigned version of our mobile app. In the design process, we prototype a solution and then test it with (usually a few) users to see how usable it is. The study identifies several issues with our prototype, which are then fixed by a new design. This test is an example of formative evaluation — it helps designers identify what needs to be changed to improve the interface.
Formative evaluations of interfaces involve testing and changing the product, usually multiple times, and therefore are well-suited for the redesign process or while creating a new product.
In both cases, you iterate through the prototyping and testing steps until you are as ready for production as you’ll get (even more iterations would form an even better design, but you have to ship at some point). Thus, formative evaluations are meant to steer the design on the right path.
Summative evaluations describe how well a design performs, often compared to a benchmark such as a prior version of the design or a competitor. Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product. Summative evaluations occur less frequently than formative evaluations, usually right before or right after a redesign.
Let’s go back to our mobile-app example. Now that we’ve shipped the new mobile app, it is time to run a study and see how our app stands in comparison to the previous version of the app. We can gather the time on task and the success rates for the core app functionalities. Then we can compare these metrics against those obtained with the previous version of the app to see if there was any improvement. We will also save the results of this study to evaluate subsequent major versions of the app. This type of study is a summative evaluation since it assesses the shipped product with the goal of tracking performance over time and ultimately calculating our return on investment. However, during this study, we might uncover some usability issues. We should make note of those issues and address them during our next design iteration.
Alternatively, another type of summative evaluations could compare our results with those obtained with one or more competitor apps or with known industry-wide data.
All summative evaluations paint an overview picture of the usability of a system. They are intended to serve as reference points so that you can determine whether you’re improving your own designs over time or beating out a competitor.
The ultimate summative evaluation is the go/no-go decision of whether to release a product. After all is said and done, is your design good enough to be inflicted on the public, or do we think that it will harm our brand so badly that it should never see the light of day? It’s actually rare for companies to have a formal process to kill off bad design, which may be why we encounter many releases that do more harm than good for a brand. If you truly embrace our proposition that brand is experience in the digital age, then consider a final summative evaluation before release.
Origin of the Terms
The terms ‘formative’ and ‘summative’ evaluation were coined by Michael Scriven in 1967. These terms were presented in the context of instructional design and education theory, but are just as valuable for any sort of evaluation-based industry.
In the educational context, formative evaluations are ongoing and occur throughout the development of the course, while summative evaluations occur less frequently and are used to determine whether the program met its intended goals. The formative evaluations are used to steer the teaching, by testing whether content was understood or needs to be revisited, while summative evaluations assess the student’s mastery of the material.
When Each Type of Evaluation Is Used
Recall that formative and summative evaluations align with your place in the design process. Formative evaluations go with prototype and testing iterations throughout a redesign project, while summative evaluations are best for right before or right after a major redesign.
Great researchers begin their study by determining what question they’re trying to answer. Essentially, your research question is the same as the type of evaluation. Below is a list of possible research questions you might have and the corresponding evaluation. For that reason, this table is descriptive, not prescriptive.
Questions you might ask | Type of evaluation |
How is our interface performing compared to our competitors? | Summative |
What usability issues exist in our interface? | Formative, Summative |
How does our interface compare to the industry benchmark? | Summative |
Do users understand our navigation? | Formative |
How has our overall experience changed over time? | Summative |
Does our interface comply with recognized usability principles? | Formative |
Is this product good enough to launch? (Go/no-go decision) | Summative |
Research Methods for Formative vs. Summative Evaluations
After it is clear which type of evaluation you will conduct, you have to determine which research method you should use. There is a common misconception that summative equals quantitative and formative equals qualitative — this is not the case.
Summative evaluations can be either qualitative or quantitative. The same is true for formative evaluations.
Although summative evaluations are often quantitative, they can be qualitative studies, too. For example, you might like to know where your product stands compared with your competition. You could hire a UX expert to do an expert review of your interface and a competitor’s. The expert review would use the 10 usability heuristics as well as the reviewer’s knowledge of UI and human behavior to produce a list of strength and weaknesses for both your interface and your competitor’s. The study is summative because the overall interface is being evaluated with the goal of understanding whether the UX of your product stands up to the competition and whether a major redesign is warranted.
Additionally, formative evaluations aren’t always qualitative, although that is often the case. (Since it’s recommended to run an extended series of formative evaluations, it makes financial sense to use a cheaper qualitative study for each of them.) But sometimes big companies with large UX budgets and high level of UX maturitymight use quantitative studies for formative purposes in order to ensure that a change to one of their essential features will perform satisfactorily. For instance, before launching a new homepage design, a large company may want to run a quantitative test on the prototype to make sure that the number of people who will scroll below the fold is high enough.
Conclusion
Formative and summative evaluations correspond to different research goals. Formative evaluations are meant to steer the design on the correct path so that the final product has satisfactory user experience. They are a natural part of any iterative user-centered design process. Summative evaluations assess the overall usability of a product and are instrumental in tracking its usability over time and in comparing it with competitors.
For more on formative and summative evaluations, check out our courses like Usability Testing or Measuring UX and ROI.
References
Greenstein, Laura.What Teachers Really Need to Know About Formative Assessment. ASCD, 2010.
FAQs
For the modern learner, or for any learner, summative assessment is not ideal. Formative assessment fits much better with student needs, and also with the teaching and learning outcomes schools have in place.
Is summative evaluation is possible without formative evaluation? ›
Summative assessment happens at the end of a course, training, or program while formative assessment happens as learning takes place. This means that while there may only be 1 summative assessment for an instructional unit, formative evaluation happens many times within the learning process.
Can assessment be both formative and summative? ›
Generally, instructors seek to incorporate both formative and summative assessments in their evaluation as they can be combined in effective ways.
How do summative evaluations and formative evaluations differ? ›
What Is the Difference Between Formative and Summative Assessment? If formative assessment measures how a student is learning during a course of study, summative assessment is designed to measure “how much” a student has learned after a unit or course has reached its completion.
Which assessment is best? ›
Formative Assessment is the most powerful type of assessment for improving student understanding and performance.
- Examples: a very interactive class discussion; a warm-up, closure, or exit slip; a on-the-spot performance; a quiz.
- Examples: Chapter test; extended essay; a project scored with a rubric.
They are useful for evaluating learning objectives that require high levels of critical thinking, creativity, and coordination. Projects are good opportunities to provide summative feedback because they often build on prior formative assessments and feedback.
What is the link between formative and summative assessment? ›
Formative assessments have low stakes and usually carry no grade, which in some instances may discourage the students from doing the task or fully engaging with it. The goal of summative assessment is to evaluate student learning at the end of an instructional unit by comparing it against some standard or benchmark.
What are the disadvantages of formative evaluation? ›
When it comes to formative assessment, there are a few drawbacks to consider.
- Time-consuming and resource-intensive. ...
- Experts experienced with assessments. ...
- Creates complexity challenges. ...
- Evaluators must maintain objectivity.
The resources in this section compare the two, complementary functions of evaluation. Formative evaluation is typically conducted during the development or improvement of a program or course. Summative evaluation involves making judgments about the efficacy of a program or course at its conclusion.
Is it necessary for teachers to use both formative and summative assessment? ›
As a teacher, you'll likely need to employ both summative and formative assessments in your curriculum. An effective balance of these assessments will help you understand your students' needs while meeting your standards.
Record keeping is an essential aspect of grading and formative assessment. Each student's entries should identify the child's current proficiency level with each target and its criteria. A consistent record keeping system is essential for grading reliability.
How does formative and summative assessment support learning? ›
Formative assessments differ from summative assessments in that formatives are used to better understand how the learning experience is progressing while summatives are used to track the learner's progress at the end of a lesson.
What is the difference between formative and summative evaluation through example? ›
Formative assessment includes little content areas. For example: 3 formative evaluations of 1 chapter. Summative assessment includes complete chapters or content areas. For example: just 1 evaluation at the end of a chapter.
Can formative assessments be graded? ›
The short answer to this question is no. For several decades now, research has been quite emphatic that grades and scores can interfere with a student's willingness to keep learning.
Why is formative assessment more suitable for meaningful assessment? ›
Formative assessment builds students' “learning to learn” skills by emphasising the process of teaching and learning, and involving students as partners in that process. It also builds students' skills at peer-assessment and self- assessment, and helps them develop a range of effective learning strategies.
What is the best way to assess student learning? ›
How to Assess Students' Learning and Performance
- Creating assignments.
- Creating exams.
- Using classroom assessment techniques.
- Using concept maps.
- Using concept tests.
- Assessing group work.
- Creating and using rubrics.
Diagnostic assessment
Diagnostic assessment is the most preferred type of assessment of learning to check a learner's current knowledge base. Most of the time, it involves a series of questions given at the start of a class or training session to identify a learner's strengths and weaknesses prior to learning.
What makes a high quality assessment? ›
High quality assessment takes the massive quantities of performance data and translates that into meaningful, actionable reports that pinpoint current student progress, predict future achievement, and inform instruction.
What is the strength of summative assessment? ›
You can use summative assessments to evaluate the comprehensive performance of the classroom to gain more insight. In a way, summative assessments can help you in two ways: Evaluate what your students have learned during the course. Understand how prepared your students are for the next academic year.
Why do teachers use summative assessments? ›
The goal of summative assessment is to evaluate student learning at the end of an instructional unit by comparing it against some standard or benchmark. Summative assessments are often high stakes, which means that they have a high point value.
Summative assessments provide cumulative snapshots to evaluate and report on student learning. Summative data can help illuminate areas of strength and gaps in curriculum and instruction, and especially for student subgroups. Reporting summative results provides information to families and the general public.
What are the advantages and disadvantages of formative assessment? ›
Best of all, formative assessments are effective tools in personalized learning. The disadvantage of formative assessments is that they can take time, more time than teachers might perceive that they have. To repeatedly check students' learning takes more time than to administer one test at the end of a lesson or unit.
What are the major barriers to wider use of formative assessment? ›
Different standards. No common tagging scheme for content and assessment. No agreement on competency. Inadequate tools.
What are the 4 types of formative assessment? ›
Types of Formative Assessment
- Observations during in-class activities; of students non-verbal feedback during lecture.
- Homework exercises as review for exams and class discussions)
- Reflections journals that are reviewed periodically during the semester.
Disadvantages: assessments can change the way teaching is focused. This is away from enjoyment and inspiration towards succeeding purely in an assessment. It also puts perhaps an unnecessary pressure on students either by themselves, their parents or teachers .
How can summative evaluations impact student learning? ›
Summative evaluations are utilized to ensure the educator's accountability, demonstrate achievement, and judge the quality of a program in its entirety (Sewall & Santaga, 1986). Simply, formative methods are an assessment for learning whereas summative ones are an assessment of learning.
Do formative assessments count? ›
Formative assessment:
an assignment or exam that doesn't count towards your final mark. you get constructive feedback which helps you to work out your progress.
When should a teacher use formative assessment? ›
Using formative assessment at the end of a previous day's lesson or at the start of a new lesson will allow you to see what your students already know about the standard you have targeted to teach. Consider using an exit or entrance ticket, for example.
Why is it important to include both formative and summative assessments in a learning process? ›
The two together can be used to reach a common goal. By using 'summative' tests during the learning process, the information gained can be used formatively to highlight areas where improvement in learning and understanding is required.
What will happen if the teacher uses the same assessment strategy every day? ›
When a teacher uses the same strategy over and over again, it becomes boring to students. This is a great way to cause students to lose focus and lose interest in learning. When a teacher varies activities, changes them up, and uses a wide range of instructional strategies students stay engaged.
Continuous assessment has the potential to support student learning through feedback and to increase students' motivation for learning [7]. This could play a vital role in diagnosing and remediating areas of learners' weakness if properly anchored in what occurs in classroom [3].
Why is it that formative assessments are just recorded but not graded? ›
Formative assessment is about measuring where students are in their learning and giving them feedback, they say, and then working to fill in the gaps. Grading can shut down that process.
How often should formative assessments be performed? ›
Formative assessments are frequent, daily or weekly, learning checks that guide both the teacher and the student on two areas: where the student has achieved mastery and where the student needs additional support.
What is the effect if there were no formative assessment? ›
If students are not doing authentic work on formative assessments, this may lead to poor summative assessments. A potential problem is that students may not learn the material and move on to the next concept.
What are the advantages of formative evaluation over summative evaluation? ›
Unlike summative assessment that waits till the end to assess students, formative assessment provides instant feedback on students' progress by evaluating them as they learn. Another advantage of formative assessment is it allows for changes and modifications to the teaching method as learning progresses.
Who benefits from summative assessment? ›
1) Teachers and academic managers can use the results of summative assessments to guide their decisions for future curriculum development. 2) Summative assessments can help indicate overall progress and show if moving forward is practical.
What is the major difference between formative and summative assessments? ›
Formative assessment is an ongoing activity. The assessment takes place during the learning process, not just once but several times. A summative assessment is a one-time activity. The assessment takes place not during the learning process but at the end.
What is the difference between formative and summative feedback? ›
Examples of formative feedback techniques include many interactive classroom activities, homework and surveys. By way of contrast, summative feedback typically falls at the end of a topic or semester. It provides an evaluation of how much a student and the class has learned and is often connected to a grade.
How do we determine if the assessment is valid and reliable? ›
How to be sure that a formal assessment tool is reliable. Check in the user manual for evidence of the reliability coefficient. These are measured between zero and 1. A coefficient of 0.9 or more indicates a high degree of reliability.
Should summative assessments be graded? ›
Summative assessments are almost always formally graded and often heavily weighted (though they do not need to be). Summative assessment can be used to great effect in conjunction and alignment with formative assessment, and instructors can consider a variety of ways to combine these approaches.
Summative-assessment results are often recorded as scores or grades that are then factored into a student's permanent academic record, whether they end up as letter grades on a report card or test scores used in the college-admissions process.
What are formative grades worth? ›
Second, a category system (for example, Formative Assessments are worth 25% of final grade and Summative Assessments are worth 75%) allows for flexibility in number and types of assignments.
Which type of assessment would be the most reliable? ›
Formative Assessment is the most powerful type of assessment for improving student understanding and performance.
How does formative assessment improve student learning? ›
Formative assessments help teachers identify concepts that students are struggling to understand, skills they are having difficulty acquiring, or learning standards they have not yet achieved so that adjustments can be made to lessons, instructional techniques, and academic support.
Why formative assessment activities process did not go well? ›
Formative assessment is too time-consuming
This thinking can be something of a trap. Avoiding formative assessments because they take up too much instructional time decreases potential learning. The fact is formative assessment doesn't need to take much time at all.
Why is formative assessment more suitable for meaningful assessment? ›
Formative assessment builds students' “learning to learn” skills by emphasising the process of teaching and learning, and involving students as partners in that process. It also builds students' skills at peer-assessment and self- assessment, and helps them develop a range of effective learning strategies.
Why summative assessment is important? ›
Summative assessments provide cumulative snapshots to evaluate and report on student learning. Summative data can help illuminate areas of strength and gaps in curriculum and instruction, and especially for student subgroups. Reporting summative results provides information to families and the general public.
What is the purpose of formative and summative evaluation? ›
Formative evaluation is typically conducted during the development or improvement of a program or course. Summative evaluation involves making judgments about the efficacy of a program or course at its conclusion.
Why is formative assessment important? ›
The goal of formative assessment is to monitor student learning to provide ongoing feedback that can be used by instructors to improve their teaching and by students to improve their learning. More specifically, formative assessments: help students identify their strengths and weaknesses and target areas that need work.
Which type of assessment is most appropriate for assessing learning difficulties? ›
Diagnostic assessment
Diagnostic assessment is the most preferred type of assessment of learning to check a learner's current knowledge base. Most of the time, it involves a series of questions given at the start of a class or training session to identify a learner's strengths and weaknesses prior to learning.
How to Assess Students' Learning and Performance
- Creating assignments.
- Creating exams.
- Using classroom assessment techniques.
- Using concept maps.
- Using concept tests.
- Assessing group work.
- Creating and using rubrics.
When it comes to formative assessment, there are a few drawbacks to consider.
- Time-consuming and resource-intensive. ...
- Experts experienced with assessments. ...
- Creates complexity challenges. ...
- Evaluators must maintain objectivity.
Summative assessments help educators make adjustments for future school years. By analyzing how well a class has retained information, a teacher can decide whether to make changes when they teach the course again. Ideally, this process allows teachers to improve their lessons every year.
What are the disadvantages of summative evaluation? ›
Disadvantages of Summative Assessment
It does not provide an accurate reflection of the student's knowledge or learning. Students can develop anxiety as they prepare for the single year that can make or mar their academic progress. Anxiety, fear, and nervousness affect the student's performance.
Who benefits from summative assessment? ›
1) Teachers and academic managers can use the results of summative assessments to guide their decisions for future curriculum development. 2) Summative assessments can help indicate overall progress and show if moving forward is practical.
Can formative assessments be graded? ›
The short answer to this question is no. For several decades now, research has been quite emphatic that grades and scores can interfere with a student's willingness to keep learning.
Should teacher record results of formative assessment for grading purposes? ›
Record keeping is an essential aspect of grading and formative assessment. Each student's entries should identify the child's current proficiency level with each target and its criteria. A consistent record keeping system is essential for grading reliability.
What is the difference between formative and summative evaluation through example? ›
Formative assessment includes little content areas. For example: 3 formative evaluations of 1 chapter. Summative assessment includes complete chapters or content areas. For example: just 1 evaluation at the end of a chapter.
How can formative assessment improve learning? ›
By collecting information while learning is happening, formative assessments help teachers see where students are in the learning process, quickly identify if students are struggling with a particular subject and adjust their teaching approaches to help students stay on track.
What can teacher learn from formative assessment? ›
Formative assessments help teachers identify concepts that students are struggling to understand, skills they are having difficulty acquiring, or learning standards they have not yet achieved so that adjustments can be made to lessons, instructional techniques, and academic support.
Having students set their own goals and evaluate their progress toward achieving them is an effective part of the formative assessment process. Goal setting has a positive effect on student motivation and learning when the goals are specific and performance based, relatively short-term, and moderately difficult.
