EDUA 6380: Reflections on Data and Digital Assessments
Summary of
EDUA 6380:
Topics in Educational Technology
(Canva Images)
Introduction
As teachers and educators, it
is part of our jobs to better understand our students and how they learn.
As a teacher of a career technology education course, I find grading a daunting
task. For the subjects I teach, I'm more concerned that the student learn how
to perform an overall task or job, than whether or not they can pass a quiz or
test. To me, watching my journalism students in the field videoing and
photographing events, interviewing students and teachers, selling a local
business an advertisement, or creating social media posts shows me those lessons
that I have taught them. However, in education As and Bs are still the
standard when it comes to grading.
Assessments and the amount of data that given through them have become an essential part of education. It has also shown the importance of creating a data-driven learning environment in order to get everyone, from administrators to teachers, involved in helping teachers provide the data and results. This is why it’s important for schools to have plans in place on data analysis, action and creating a data-driven culture. Computer-based testing and digital assessments have now become a part of our world. I don’t know of any major test or assessment that isn’t online.
This special topic course has shown how digital assessments are being used to collect various amounts of data, which makes for better instruction and students excelling. Paul Brambrick-Santoyo emphasizes that data-driven schools are those that use data to turn school focus from “what was taught” to “what was learned” (2010).
As a learner and teacher, I agree with the statement in the Powell-Assessment Reading: “Students learn differently and should have a variety of opportunities to demonstrate what they know and can do” (Powell, 2012). Technology offers tremendous opportunities to the field of educational assessment in terms of convenience, efficiency, and features. As Brambrick-Santoyo states “assessments are not the end of the teaching and learning process; they’re the starting point” (2010).
(Ozona High School Lion Media Students covering the Shattered Dreams event in April 2024)
Embedded and Authentic Assessments
To me, as a learner and teacher, embedded assessments and authentic
assessments are the best ways to see what students have learned. Embedded
assessments feel like a natural part of the instruction, so you are not aware
you are being assessed (Koch, 2012). I have students give oral reports at
times. We do what is called “Show and Tell” on our yearbook pages. When a
student has a rough draft of the page done, they “show” it to the whole class
and then the class must “tell” the presenting student what is positive about
the page and what might need correcting or adjusting. Negative comments are not
allowed. This is constructive criticism to help make the page and the yearbook
better. The “Show and Tell” that my yearbook students do is an embedded
assessment. They know it’s part of their design process.
Authentic assessments, students performing a task relating to what they
have learned to some real-world problem or example, is the main assessment I
like to use as a teacher and as a learner (Koch, 2012). To me this really helps
you retrieve and rethink that knowledge from the classroom and put it to use
for all to see. Students must publicly demonstrate their understanding by
reshaping, expanding on, extrapolating from, and applying what they already
know (Koch, 2012). To me, students gain even more skills and knowledge from
these types of assessments than from just simply filling in blanks.
Rubrics give clear expectations for the assignments, allow teachers to
differentiate consistently among performance levels, provide guidelines for
student improvement, and make grading more transparent and consistent, which is
important with my assessments (Powell, 2012). I use a lot of analytic rubrics
because I have a lot of specific tasks, characteristics, and performance levels
that I look for from students (Powell, 2012). Rubrics also help me with
students who have learning disabilities, special needs, or need differentiated
instruction.
I know standardized testing is not going to go away, but I think teachers need to embrace other forms of assessments besides using a pen and pencil. We all learn differently, even those students who make straight As.
Standards-Based Grading and Competency-Based Education
Standards-based
grading involves measuring students' proficiency on well-defined course
objectives (Scriffny, 2008). While many schools adopt both traditional and
standards-based grading, there is an argument that standards-based grading can
and should replace traditional grades.
Seven
reasons that standards-based grading should replace point-based grades:
1.
Grades should have
meaning
2.
We need to challenge the
status quo
3.
We can control grading
practices
4.
Standards-based grading
reduces meaningless paperwork
5.
It helps teachers adjust
instruction
6.
It teaches what quality
looks like
7.
It’s a launchpad to
other reforms
(Scriffiny,
2008)
Competency-based
education has increased in popularity as the downfalls of traditional grading
systems have become more apparent, such as an over focus on
grades. Competency-based models have attempted to address these concerns
and improve the educational experience of students by expanding the focus from
grades to student growth and skill obtainment (Boss and McKendree, 2022).
One area of education
that competency-based education has become useful is the increase in Career
Technical Education (CTE). Research in broader post-secondary education
has also explored other models of competency-based education, such as mastery-oriented
assessment. A key foundation of mastery assessment is evaluating students’
learning relative to previously established learning goals, instead of other
student performance (Boss and McKendree, 2022). In most CTE classes, students
are learning skills that will lead them toward a job or a career in that
field.
The students' mission is
no longer merely to beat other students in the achievement race. At least part
of their goal must be to become competent. Teachers must believe that all
students can achieve a certain level of academic success, must bring all
of their students to believe this of themselves, must accommodate the fact that
students learn at different rates by making use of differentiated
instruction, and must guide all students toward the attainment
of standards (Stiggins, 2005).
Competency-based
education, whether in CTE or other content areas, represents a positive shift
in education, emphasizing understanding over traditional grading metrics
(Jensen, 2024). Recommendations for practice include suggesting CTE
teachers and secondary schools that have not yet implemented a competency-based
model consider implementing one – but with the understanding there will be
challenges. Teachers must adjust to the possibility of struggles initially, but
with administrative support and support of other teachers it can be an
effective model for student learning and engagement (Boss and McKendree,
2022).
One of the potential
challenges for teachers is the shift in what is expected of them before and
during instruction. However, meeting this shift in expectations may result
in more effective instructional experiences with students (Boss and McKendree,
2022).
One thing I love about CTE teaching is seeing the student, who may not perform the best in those core classes, shine in my classroom when they master a new skill. They may not be the top of their class, but in my classroom they are the stars.
(Canva Images)
Computer-Based Assessment
Computer-Based Assessment (CBA)
refers to tests administered to students by computer. The purpose of using CBAs
is to increase the efficiency of test administration and scoring and ensure the
standardization of testing procedures. CBAs have also been able to offer
innovations in test administration that cannot be available for paper and
pencil tests, such as embedded accommodations and modifications, as well as the
model of Computer Adaptive Testing (Dembitzer, et. al, 2017).
With
digital assessments, teachers can see student’s responses and the process a
student used to get to their answer, or the process of which they used to write
their essay (UNESCO, 2018). If a student doesn’t perform well, the teacher can
look at the process the student used to help them better understand the
information or lesson.
Teachers can also see demographics on how their students are doing, including minorities, English Language Learners, and students with disabilities. Schools that are high achieving may see that they are not serving these populations of students as well as they thought by using digital assessments (UNESCO, 2018). Tools like adjustable font sizes, text-to-speech, and highlighters are now available to all students, not just those with accommodations (NAEP, 2024).
There is also a greater emphasis on digital literacy. As standardized tests move to computer-based formats, teachers are increasingly focusing on developing students' digital literacy and keyboarding skills as part of their regular curriculum (Typingagent, 2023).
.jpeg)
Creating a Data-Driven Environment
Paul Brambrick-Santoyo’s core
idea is that “Standards are meaningless until you define how you will assess
them” (2010). Another way to put it is “why are we teaching this? What lesson
as a teacher do I want my students to learn?” As Brambrick-Santoyo states “no
one would start building a house without creating the blueprints, or training
for the Olympics without identifying what benchmarks define success” (2010).
Data-driven instruction is the philosophy that schools should
constantly focus on one simple question: are students learning
(Brambrick-Santoyo, 2010)?
Four key principles to data-driven instruction are:
1.
Assessment:
Create rigorous interim assessments that provide meaningful data.
2.
Analysis:
Examine the results of assessments to identify the causes of both strengths and
shortcomings.
3.
Action:
Teach effectively what students most need to learn.
4.
Culture:
Create an environment in which data-driven instruction can survive and thrive.
The benefits of data-driven instruction go deeper than just
test scores. Teachers learn that, across content areas, what concepts students
struggle with and which ones they excel at, allowing teachers to work to
increase students’ abilities to infer, see causal patterns, and universalize
themes, skills, and principles (Crites, 2016). These are the kind of critical
thinking skills that teachers want student to have when they head to college
and beyond.
One thing I like about Brambrick-Santoyo’s method is the that
he emphasizes that everyone from the teachers to administrators be involved in
creating the assessments how they can be an invaluable resource in building a
quality interim assessment program (2010).
.jpg)
(Canva Images)
Big Data
The prospect of “big data”
at once evokes optimistic views of an information-rich future and concerns
about surveillance that adversely impacts our personal and private lives (Cope
and Kalantzis, 2016). Big Data in Education is transforming ways to
understand student performance and learning patterns by analyzing vast amounts
of educational data. With this, educational Institutions can tailor learning
experiences to individual needs (Roberts, 2023).
Big Data
in Education involves collecting, processing, and analyzing vast datasets
generated by educational institutions. These include student demographics,
academic performance, attendance, online interactions, and assessment results.
Key characteristics of Big Data in Education are volume (large amounts of
data), velocity (rapid data creation), and variety (diverse data types)
(Roberts, 2023). Big data also focuses on advanced analytics and data mining to
uncover patterns and trends, which help educators identify at-risk students,
tailor teaching methods, and improve learning experiences.
To set
the stage with a definition, “big data” in education is:
1. the
purposeful or incidental recording of activity and interactions in
digitally mediated, network-interconnected learning environments—the
volume of which is unprecedented in large part because the data
points are smaller and the recording is continuous;
2. the
varied types of data that are recordable and analyzable;
3. the accessibility and durability
of these data, with potential to be (a) immediately available for
formative assessment or adaptive instructional recalibration and (b)
persistent for the purposes of developing learner profiles and longitudinal
analyses; and
4. data analytics, or
syntheses and presentations based on the particular characteristics of
these data for learner and teacher feedback, institutional
accountability, educational software design, learning resource
development, and educational research.
(Cope and
Kalantzis, 2016)
Issues with Big Data include student privacy, test-driven teaching, lack of digital literacy skills for educators and administrators, potential for bias, and ethical questions in fairness, transparency, and accountability decision-making.
Key benefits of Big Data are:
1.
Personalized
Learning
2.
Improved
Student Performance
3.
Enhanced
Decision-Making
4.
Predictive
Analytics
5.
Enhanced
Teaching Method
Overall, Big Data has brought tremendous potential to
transforming learning and is becoming a crucial for successful
implementation.
.jpg)
(Adobe Images)
Digital Grading
Automated assessment applications
have achieved better than human reliability and other methods of assisting
assessment have opened up additional venues for utilization in the
classroom and beyond. However, a lack of understanding of the differences between
the different types of applications and their limitations has made selecting
the appropriate application a difficult task (Aken, 2017). Computer-based
text analysis (CBTA) has been referred to by many names in the literature
including Automated Essay Scoring (AES), which is the most prevalent name
particularly within education literature.
CBTA falls into two
categories: automated assessment and machine-assisted analysis.
Automated writing assessment requires no human intervention (subsequent to
the initial configuration of the prompt) while machine-assisted analysis
is dependent upon human interaction to provide an analysis of the text
being analyzed (Aken, 2017).
Most CBTA programs are
developed to assess written text to provide a summary score
(summative assessment). Some applications, however, are primarily as a
learning environment designed to assist students in learning how to write
(formative assessment) as well as possibly providing
summative assessment. Although most formative assessment applications
also provide summative scores, the applications classified as summative
assessment do not include formative assessment capabilities (Aken,
2017).
After Ohio started using American Institutes for
Research in 2015 to provide and score state tests, Artificial Intelligence (AI)
programs have increasingly taken over grading. Computers are now scoring
the entire test for about 75 percent of Ohio students, State Superintendent
Paolo DeMaria and state testing official Brian Roget told the state school
board recently. The other 25 percent are scored by people to help verify the
computer's work (O'Donnell, 2018). Multiple other testing organizations
- like Pearson (which handled the old PARCC tests), McGraw Hill and
Educational Testing Service (which produces graduate school admissions tests) -
have developed automated scoring systems that can quickly compare student
essays to model answers humans provide.
Just this past spring, the Texas Education
Agency (TEA) used computers and artificial intelligence technology to
grade students' open-ended questions on the State of Texas Assessment of
Academic Readiness for Science, Social Studies, Reading, and Language
Arts. According to a scoring report by the TEA, students' responses will
be graded first by a computer. A hired human scorer will grade roughly 25
percent of those responses. If the computer has a "low confidence"
score, it will also be re-scored by a human. Some tests are also up for review
if the computer's programming catches unrecognizable responses like slang
words, phrases, or languages other than English (Sessions, 2024). Students
and parents who disagree with the computer and human scores can request a
rescore for $50. The fee can be waived if the computer's or human's score is
wrong. TEA guidelines state that the STAAR test measures how much a student has
learned about a subject.
While online grading saves time and money, I
question its accuracy. Some argue that these algorithms may struggle to
grasp nuances of language and could misinterpret students' work (Aken, 2017).
Furthermore, the assessment criteria of algorithms are limited, potentially
overlooking certain areas such as organizational skills or grammar usage.
.jpg)
(Canva Image)
Creating a Digital Assessment
During the course, we were given
the opportunity to create our own Digital Assessment using the Canvas learning
management system (LMS) platform. I have a passion for teaching students’
digital citizenship and media literacy. I created a quiz from the Digital
Citizenship curriculum I plan to use this school year from the International Society
for Technology in Education (ISTE). I had a chance to link learning objectives
to a quiz and promptly access the results upon classmates completing the quiz
as a student. Canvas LMS provides the choice to examine outcomes based on
question types, along with individual student analyses, revealing both areas of
struggle and success among students. Below are my questions and outcomes.
Grade Level Subject: 9th Grade Digital Citizenship Quiz
Conclusion
Data is continuously going to be part of schools. A judicious, targeted use of data by educators coupled with careful coaching for them and qualitative assessments of students can yield significant improvements, especially when the definition of data expands beyond testing (Berwick, 2018).
As a Career Technical
Education Teacher, I am interested in seeing how data plays in CTE programs
like mine where students don’t necessarily take paper and pen assessments but
show their skills in real world situations. Relationships between school
leaders, teachers, and students are essential in making a school environment
successful.
Oral PresentationProvided below is an oral video presentation of this blog.
References
(2024, February 21). Digitally Based Assessments. National Assessment of Educational Progress. Retrieved July 15, 2024, from https://nces.ed.gov/nationsreportcard/dba/
(2023, October 20). The Digital Shift in Education: Preparing Students for Computer-Based Standardized Tests. Typingagent. Retrieved July 15, 2024, from https://blog.typingagent.com/the-digital-shift-in-education-preparing-students-for-computer-based-standardized-tests/
Aken, Andrew. An Evaluation of Assessment-Oriented Computer-Based Text Analysis Paradigms. Higher Education Research. Vol. 2, No. 4, 2017, pp. 111-116. doi: 10.11648/j.her.20170204.12
Boss, C., & McKendree, R. B. (2022). Career and Technical Education Teachers’ Perspectives of Evidence-Based Grading. Journal of Research in Technical Careers, 6(2). https://doi.org/10.9741/2578-2118.1110
Brambrick-Santoyo, P. (2010). Driven by Data: A
Practical Guide to Improve Instruction (p. IntroductionandChapter1).
Jossey-Bass.
Cope, B., & Kalantzis, M. (2016). Big Data Comes to
School. AERA Open, 2(2), 233285841664190. https://doi.org/10.1177/2332858416641907
Crites, E. (2016, October 4). 7 Steps to Becoming a
Data-Driven School. Edutopia. Retrieved July 23, 2024, from https://www.edutopia.org/blog/7-steps-becoming-data-driven-school-eric-crites
Dembitzer, L., Zelikpvitz, S., & Kettler, R.
(2017). Designing computer-based assessments: multidisciplinary findings and
student perspectives. International Journal of Educational Technology, 4(3),
20-31.
Jensen, K. (2024, February 15). What all content areas can learn from CTE’s Competency-Based Grading. Atlas. Retrieved July 19, 2024, from https://www.onatlas.com/blog/what-core-and-arts-departments-can-learn-from-ctes-competency-based-grading#:~:text=Students%20are%20also%20evaluated%20on,understanding%20over%20traditional%20grading%20metrics.
Koch, J. (2012). TEACH (pp. 84-89). Wadsworth, Cengage Learning. https://lccn.loc.gov/2010935304
O'Donnell, P. (2018, March 19). Computers are now grading essays on Ohio's state tests. The Plain Dealer. https://www.cleveland.com/metro/index.ssf/2018/03/computers_are_now_grading_essays_on_ohios_state_tests_your_ch.html
Powell, S. D. (2012). Your introduction to education: Explorations in teaching (2nd ed., p. Chapter5). Pearson Education, Inc. https://lccn.loc.gov/2010047286
Roberts, S. (2023, October 16). Big Data in Education: Applications, Limitations, and Future Scope. Theknowledgeacademy. Retrieved August 6, 2024, from https://www.theknowledgeacademy.com/blog/big-data-in-education/
Scriffiny, P. L. (2008). Seven Reasons for Standards-Based Grading. Educational Leadership Expecting Excellence, 66(2), 70-74. https://ascd.org/el/articles/seven-reasons-for-standards-based-grading
Sessions, K. (2024, April 12). Details Emerge on Automated Grading of Texas' STAAR Tests. Government Technology. Retrieved August 6, 2024, from https://www.govtech.com/education/k-12/details-emerge-on-automated-grading-of-texas-staar-tests
Stiggins, R. (2005). From Formative Assessment to Assessment for Learning: A Path to Success in Standards-Based Schools. The Phi Delta Kappan, 87(4), 324–328. http://www.jstor.org/stable/20441998
[UNESCO]. (2018, March 5). Computer-based assessments: What you need to know - UNESCO-IAEA webinar [Video]. YouTube. https://youtu.be/n0ij0Sq4kbk?si=9tRigFAw-X84XhJr
Comments
Post a Comment