Teacher Comments

Research Study

Project Title: Development and testing of LD/ELL-friendly computer tutorials in diverse subjects, classrooms, and formats.

U.S. Dept. of Education PR/Award Number: H133S030008

This material is based upon work supported by the U.S. Department of Education under funding agreement number H133S030008. Any opinions, findings, conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views or policies of the Department of Education.

Background

Goals of the study
In 2003, Visilearn began Phase II of its NIDRR-funded research study, to determine if tutorials based on its illustration-based Phase I prototype helped students with learning disabilities (LD) and English Language Learners (ELL) learn computer subjects better, faster, and easier.

The study also sought to determine if these tutorials were suitable for mainstream students, and if they were effective when used in electronic format.

Significance
Current computer tutorials are text-heavy and complicated; they make it difficult for LD and ELL students to learn computer subjects.

What's more, most LD and ELL students are taught computer subjects in mainstream classrooms, where it's impractical and awkward to provide them with special tutorials.

Also, current computer tutorials are expensive, and increasingly difficult for schools to afford.

If Visilearn could develop a series of LD/ELL-friendly computer tutorials suitable for use by mainstream students, it could:

• Improve computer instruction for LD and ELL students.
• Reach LD and ELL students learning computer subjects in mainstream classrooms.
• Show that using LD/ELL-friendly tutorials in mainstream classes benefits all students.

If the Tutorials were effective in electronic format, that could enable schools to buy computer tutorials for one-tenth the cost of current texts.

Phase I
In 2002, Visilearn conducted Phase I of the study. It set out to develop a prototype textbook that made it easier for LD and ELL students to learn computer subjects.

Description of Research
Visilearn first interviewed 15 classroom teachers teaching computer subjects to LD and ELL students, asking general questions about what methods worked best for them. The teachers' answers were then used to create a survey with more specific questions. The survey was posted online and administered to 117 teachers.

Visilearn developed a draft of the prototype based on teachers' requirements, and printed 1000 copies. The Tutorials were distributed to interviewed/surveyed teachers for use in their classrooms. After several weeks, the teachers were asked what changes/enhancements would improve the first draft.

The teachers' suggestions were incorporated into the second and final draft, and copies were printed and sent back to them for use in their classrooms.

Findings and Results
None of the teachers interviewed or surveyed were satisfied with current computer tutorials. They found them too wordy and complicated.

However, all of the teachers interviewed and surveyed praised the prototype textbook. Every teacher who responded to Visilearn' final query about the prototype said that its illustration-based approach was superior to that of other tutorials they'd used, and that it allowed them to teach computer subjects better, faster, and easier.

The teachers said that these instructional techniques and methods work best when teaching computer subjects to LD and ELL students:

• Hands-on labs
• Simple, step-by-step instructions
• Lots of large graphics
• Information presented in small chunks
• Real-world exercises

The teachers said that class materials should be organized and presented this way:

• Tasks: Computer tutorials should be organized by tasks: the things most students want to do with the program.
• Steps: Then for each task, list concise, step-by-step instructions.
• Illustration: Illustrate each step with a computer screen shot that shows exactly what to do. The bigger the better.
• Reinforcement: At the end of each important task, include stand-alone exercises so students can practice the tasks. Include multiple exercises so students who work quickly remain occupied.

The teachers said that this sort of textbook layout and illustration was optimal:

According to them, the optimal page layout has clear page numbering and a spot color. Not, however, full color: that can confuse students.

They said binding should be spiral, so the book lays flat on the desk. Also, optimal instruction required supplemental materials, but not multimedia demonstrations.

Conducting the Study

Creating tutorials
For Phase II of the study, the goal was to develop 20 different illustration-based Phase I-type tutorials. That goal was far exceeded--33 tutorials on the most common computer subjects were produced.

Test subjects
A total of 497 students in 14 classrooms participated in testing the Tutorials. The students completed a total of 1346 pre- and post-tests, and a mean of 2.7 Tutorials each.

The students were from grade levels 3-12:

Grade	Percentage of total
3	5
4	38
5	6
Elementary	49
6	6
7	7
8	16
9	8
Middle/Jr. High	29
10	9
11	3
12	2
High School	22

Their schools were located in ten different states:

Testing
Identical pre-and post-tests were administered for each book.

Because objective tests typically do a poor job of measuring proficiency in computer subjects, subjective tests—hands-on projects for students to complete—were administered.

Scoring
All tests were scored by the teachers who administered them.

Because some competency tests were based on a 10-point scale and others on a 100-point scale, all pre- and post-test scores were converted to percentages of accurate responses.

Quantitative Data

Increase in scores
Across all tests and students, pre-test scores averaged 39.2% and post-test scores 66.6%, for an increase of 27.4 percentage points. Using a paired t-test of differences for means, this was a significant increase (t=44.1, p<.0001).

Scores across titles
The table below shows mean pre- and post-test scores across titles (collapsing scores for 2002 and 2003 versions of titles).

The scores for the Dreamweaver Tutorials showed the lowest gain, but those students were seemingly already proficient with the material, given that the pretest scores averaged 94.5.

The Microsoft Access Tutorials showed the lowest mean pretest score as well as the highest gain score from pre- to post-test (53.0 percentage points).

Mean Scores Across Titles
	Pretest	Post-test	Gain
Access	11.0	64.0	53.0
Dreamweaver	94.5	98.2	4.2
Excel	35.4	69.6	34.2
FrontPage	43.2	83.2	40.0
Photoshop	75.8	87.4	11.6
PowerPoint	32.2	58.6	26.4
Publisher	43.4	63.6	20.2
Word	47.1	64.2	17.1

Scores across students
Of the 1346 total pairs of pre- and post-tests, 286 (21.2%) were completed by LD students (n=150), ELL students (n=133) or both LD and ELL (n=3). Although we were not able to target these students, we were gratified that they were well-represented in the final sample.

The table below shows the results of an analysis of variance (ANOVA) for mean gain scores for LD/ELL students versus non-LD/ELL students across all titles. Because the samples were not randomized, this method of analysis was selected over the analysis of covariance (ANCOVA) for this and subsequent analyses.

As this table shows, pre-test scores for LD and ELL students averaged nearly 30 percentage points below students who were not LD or ELL. However, LD/ELL students had a significantly higher increase in scores from pre- to post-test (F=7.1, p<.01).

Although LD and ELL scores were below other students at both pre- and post-test, they were able to "narrow the gap" to a significant degree between pre- and post-test.

Mean Scores Across Students
	Pretest	Post-test	Gain
LD/ELL	15.0	45.6	30.6
Non-LD/ELL	45.7	72.3	26.6

Scores across formats
Visilearn tutorials were available to the participating schools in either bound or electronic (i.e., Portable Document Format, or PDF) format.

The following titles were requested by school personnel in both bound and electronic formats, and only these titles were included in subsequent analyses:

• Excel
• PowerPoint
• Publisher
• Word

Scores were based on 895 tests in which the bound format was used, and 290 tests in which the electronic format were used. The table below presents the findings from the ANOVA. As the table shows, there was a significantly higher increase in scores for the texts in electronic format (F=397.9, p<.0001).

Although this suggests that students learned the material better from the electronic format, it should be tempered by the disparity between the sample sizes for the two formats.

Mean Scores Across Formats
	Pretest	Post-test	Gain
Bound	41.4	61.1	19.7
Electronic	33.0	77.7	44.7

Qualitative Data

13 teachers involved in the study completed a 12-question survey. The tables below contain their responses and comments.

Compared to the tutorials you were using before, did the illustration-based Visilearn tutorials you used in class help your LD students learn...
Significantly more	70%
More	15
Same amount	15
Much faster	15
Faster	70
Same speed	15
Much easier	46
Easier	54
Same effort

Compared to the tutorials you were using before, did the illustration-based Visilearn tutorials you used in class help your ELL students* learn...
Significantly more	17%
More	50
Same amount	33
Much faster	17
Faster	33
Same speed	50
Much easier	33
Easier	67
Same effort

*7 out of 13 teachers surveyed had no ELL students.

Did your mainstream (non-LD/ELL) students* find the illustration-based Tutorials suitable and useful?
Yes, definitely	67%
Yes	33
No
Definitely not
Why?
"The Tutorials let them review as needed, and they also used them as a reference to look up something they may have forgotten."
"They are easy to follow and understand."
"I had only LD students with various degrees of difficulty. All of my students benefited from the pictures in the Tutorials, and for some of them the Tutorials gave them more independence as learners. I think that mainstream students would find the Tutorials an easy way to learn quickly, and see what the screen is supposed to look like so they can self-correct their mistakes."
"My students thought having the illustration to look back on helpful, when they got behind during my instruction."
"Yes, mainstream students found the Tutorials useful because it was easy to read and quick to work through. Most high-end students were able to work through the Tutorials very quickly and this left more time for them to actually practice using the programs."
"I believe that the visual element of the Tutorials is a great help."
"The students found the Tutorials both useful and suitable. The students were able to work at their own pace and focus on the things they didn't know already."
"Yes useful--sometimes examples are not exactly student friendly (e.g. fax cover sheet; kids don't care), whereas 'frog farm' example-kids like the idea better."
"The mainstream students could work at their own pace to complete the task, while I would work with either the LD or ELL students to better explain terms and pictures. The mainstream students appeared to like working this way."
"They loved the pictures."
"Yes, my students greatly enjoyed using the Tutorials. They even took time before school to come to the computer lab to practice with them."

*1 out of 13 teachers surveyed had no non-LD/ELL students.

If you used the Tutorials in electronic format, how effective were they, compared to conventional bound texts?
More effective	33%
Just as effective	67
Why?
"I liked them because I could give assignments to do at home using Blackboard, and students had no trouble accessing them online. I have heard from parents who were impressed with their child's computer skills."
"The e-format was easier for the students, and portable."

*3 teachers surveyed used the Tutorials in electronic format.

Comments and Observations

"Very appropriate for my students."

"The Tutorials were easy to follow, but I had to go back and explain to the class what they were doing in each step. They would follow the steps in the Tutorials, but didn't understand what they were doing. There was no carry over."

"I enjoyed the Tutorials and found them helpful. I did have to adjust the assignments to fit the needs and wants of my students. I would also, once I taught something using the book, then create an assignment to practice the skills the students had just learned. My students could then use the book as their reference guide to complete the assignment independently. Many of my students really enjoyed the independence they gained by being able to use the Tutorials on their own."

"My only complaint is with the illustrated format, students only relied on the pictures. This format is good but it also makes students "lazy" workers. They don't bother to read ANY of the written text."

"These Tutorials are easy to use for all students. Even the lowest learning-disabled student in my class was able to understand these Tutorials and learn the software program."

"I feel the tutorials provided ALL students the opportunity to build their technology skills in a personal manner."

"I used the Tutorials for Windows XP. I felt the student would have learned and retained more if there were a completion test and also if I counted it as more than an extra credit grade."

"For the most part, kids enjoyed being able to go off on their own and complete work at their own pace."

"In the beginning, all of my students had trouble because of vocabulary (not knowing what some of the words meant). As we continued to work with the texts the issue of vocabulary became less and less the issue. The circles around where to click made it quite easy for my LD and ELL [students] as we continued to work with the texts. I'm hoping that a lot of the information will be retained and we can move forward. In PowerPoint and Publisher we got about halfway thru the Tutorials because of test timing. I'm hoping with some groups to review from this year and go ahead to the other half next year. I am really impressed with them all-they really like working independently and sharing what they got done. I'm looking forward to next year!"

"Overall, my students really enjoyed the simplicity of the Tutorials as well as the illustrative pictures."

"These Tutorials made it very easy to teach the students how to work with the programs, and they were able to follow along."

Conclusions

Illustration-based texts help LD/ELL students.
Illustration-based computer class tutorials help LD and ELL students learn computer subjects better, faster, and easier:

• In quantitative testing of the Tutorials, LD/ELL students had a high and significant increase in scores from pre- to post-test. Their gains were higher than those of non-LD/ELL students.
• In the qualitative survey, 90% of teacher responses were that the tutorials helped LD/ELL students learn more/faster/easier than the Tutorials they were using before.

They work best for beginners.
Tutorials on commonly-used programs such as Microsoft Word, or programs with which students had previous experience, such as Adobe Dreamweaver, produced a notable gain in scores.

However, the biggest gains were produced by Tutorials on complicated, technical programs such as Access and Excel in which the students' initial knowledge was lower.

They work for all subjects.
Test gains were observed across all titles. While gains were comparatively low with the Tutorials on Dreamweaver-a program that students had experience using-gains were high with the Tutorials on FrontPage, a similar program new to most students. Thus, the illustration-based tutorials work regardless of computer subject.

They work well for mainstream students.
In the quantitative testing, non-LD/ELL students showed pre/post-test gains almost as high as those of LD/ELL students.

In the qualitative survey, all of the teachers reported that their mainstream students found the Tutorials suitable and useful. Their comments were uniformly positive.

They are effective when used in electronic format.
Although some teachers resisted using the Tutorials in electronic format, those who did saw their students realize test score gains twice as high as those using standard bound texts.

In the qualitative survey, the teachers using the electronic-format texts reported that they were just as effective or more effective than standard bound texts.