|
Statistical Testing Results for Recall/Comprehension
Testing
Statistic #1
We observed that participants were, on average (for both
stories in multimedia and text), 5 percent more likely to correctly
answer recall questions about story information if they read a text version
of the story.
This observation is marginally significant.
Two-tailed t-test results:
t-test(877)=1.727, p-value=0.0844
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
439 |
0.626424 |
0.02269 |
0.58189 |
0.67095 |
| Text |
440 |
0.681818 |
0.02266 |
0.63734 |
0.72630 |
Statistic #2
Participants were asked "Who is Michelle Sankowsky?" Sankowsky
is a nurse who works for Tyler Pipe, the company that is profiled in
the story. Twenty percent of the participants who saw the multimedia
version answered this question correctly, while 62.5 percent of the participants
who read the text version responded correctly.
This observation is statistically significant.
Two-tailed t-test results:
t-test(42)=3.062, p-value=0.0038
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20 |
0.200000 |
0.10250 |
-0.0069 |
0.40685 |
| Text |
24 |
0.625000 |
0.09357 |
0.43617 |
0.81383 |
Statistic #3
Recall of information that was part of a dramatic or shocking text
description is high.
Seventy percent who received this type of information in multimedia got the
question right. Ninety six percent who received the information in
text got the answer correct.
This observation is statistically significant.
Two-tailed t-test results:
t-test(42)=2.435, p-value=0.0192
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20
|
0.700000 |
0.07836 |
0.54186 |
0.8581 |
| Text |
24 |
0.958333 |
0.07154 |
0.81397 |
1.1027 |
Statistic #4
When participants carefully reviewed a moving illustration
of a process that defined the word "cupola," they
were better able to recall its meaning. Eight percent of participants
who received this information in text were correctly able
to define a cupola in a recall test. Sixty-five percent of
multimedia viewers got this answer correct.
This observation is statistically significant.
Two-tailed t-test results:
t-test(42)=4.801, p-value=.0001
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20
|
0.650000 |
0.08717 |
0.47408 |
0.82592 |
| Text |
24 |
0.083333 |
0.07958 |
-0.0773 |
0.24393 |
Statistic #5
When asked to recall how pipes were formed, the delivery
method of the information did not seem to affect recall. Seventy-one
percent of text readers answered this question correctly.
Eight percent of multimedia viewers answered it correctly.
This observation is not statistically significant.
Two-tailed t-test results:
t-test(42)=0.687, p-value=0.4959
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20
|
0.800000 |
0.09855 |
0.60112 |
0.99888 |
| Text |
24 |
0.708333 |
0.08996 |
0.52678 |
0.88989 |
Statistic #6
The
majority of questions with statistically significant differences
in performance between media types revealed that text versions
performed best.
|
Question
|
Question
type
|
Media
type
|
Performance
Delta *
|
MM
|
Text
|
t-test(42)
|
p-value
|
|
What
is the main point of this story?
|
Conceptual
|
Text
|
53%
|
60.00%
|
91.67%
|
2.632
|
0.0118
|
|
Who
is Michelle Sankowsky?
|
Factual
|
Text
|
213%
|
20.00%
|
62.50%
|
3.062
|
0.0038
|
|
Which
statement most accurately depicts what happened to Lopez
after he received an X-ray?
|
Factual
|
Text
|
37%
|
70.00%
|
95.83%
|
2.435
|
0.0192
|
|
What
is a cupola?
|
Factual
|
MM
|
680%
|
65.00%
|
8.33%
|
4.801
|
0.0001
|
|
What
is the average air temperature inside the Tyler pipe
foundry?
|
Factual
|
Text
|
46%
|
60.00%
|
87.50%
|
2.16
|
0.0365
|
|
Which
group of individuals most considered it an honor to
be drawn by Hirschfeld?
|
Factual
|
Text
|
153%
|
37.50%
|
95.00%
|
4.8
|
0.0001
|
|
What
hours did Hirschfeld usually work?
|
Factual
|
MM
|
53%
|
91.67%
|
60.00%
|
2.632
|
0.0118
|
*
The performance delta is calculated by taking the percentage of the
better performing media, subtracting from it the performance of the
worse performing media, and then dividing it by the performance percentage
of the worse performing media. For example, in a case where correct
responses for each are MM=60.0% and Text=91.7%, the following applies:
(91.7-60.0)/60.0 = 52.8%. Here, text performed 52.8% better than
multimedia. In another example where the correct responses for each
are MM=65.0% and Text=8.3%, the following applies: (65.0-8.3)/8.3
= 683%.
Statistics for individual questions in the chart above
Question:
What is the main point of this story?
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20
|
0.600000 |
0.08886 |
0.42067 |
0.7793 |
| Text |
24 |
0.916667 |
0.08112 |
0.75296 |
1.0804 |
Question:
Who is Michelle Sankowsky?
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20 |
0.200000 |
0.10250 |
-0.0069 |
0.40685 |
| Text |
24 |
0.625000 |
0.09357 |
0.43617 |
0.81383 |
Question:
Which statement most accurately
depicts what happened to Lopez after he received an X-ray?
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20
|
0.700000 |
0.07836 |
0.54186 |
0.8581 |
| Text |
24 |
0.958333 |
0.07154 |
0.81397 |
1.1027 |
Question:
What is a cupola?
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20
|
0.650000 |
0.08717 |
0.47408 |
0.82592 |
| Text |
24 |
0.083333 |
0.07958 |
-0.0773 |
0.24393 |
Question:
What is the average air temperature
inside the Tyler pipe foundry?
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20
|
0.600000 |
0.09402 |
0.41027 |
0.7897 |
| Text |
24 |
0.875000 |
0.08583 |
0.70180 |
1.0482 |
Question:
Which group of individuals most
considered it an honor to be drawn by Hirschfeld?
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20
|
0.375000 |
0.08076 |
0.21201 |
0.5380 |
| Text |
24 |
0.950000 |
0.08847 |
0.77146 |
1.1285 |
Question:
What hours did Hirschfeld usually
work?
| Level |
Number |
Mean |
Standard
Error |
Lower
95 percent |
Upper
95 percent |
| Multimedia |
20
|
0.916667 |
0.08112 |
0.75296 |
1.0804 |
| Text |
24 |
0.600000 |
0.08886 |
0.42067 |
0.7793 |
Statistic #7
When we analyzed our data for gender, we saw no significant
difference in men's and women's abilities to correctly recall
story information presented in text vs. multimedia.
|
Gender
|
Correct
answers when story was received in multimedia
|
Correct
answers when story was received in text
|
|
Male
|
62
percent
|
68
percent
|
|
Female
|
63
percent
|
68
percent
|
|
|
t-test(437)=0.234,
p-value=0.8154, no difference
|
t-test(438)=0.0,
p-value=1.0, no difference
|
Statistical breakdown for multimedia
|
Level
|
Number
|
Mean
|
Standard
Error
|
Lower
95 percent
|
Upper
95 percent
|
|
Female
|
220
|
0.631818
|
0.03269
|
0.56757
|
0.69606
|
|
Male
|
219
|
0.621005
|
0.03276
|
0.55661
|
0.68539
|
Statistical breakdown for text
|
Level
|
Number
|
Mean
|
Standard
Error
|
Lower
95 percent
|
Upper
95 percent
|
|
Female
|
220
|
0.681818
|
0.03147
|
0.61996
|
0.74368
|
|
Male
|
220
|
0.681818
|
0.03147
|
0.61996
|
0.74368
|
|
