"The tragedy of science -- the slaying
of a beautiful hypothesis by an ugly fact."
~ Thomas Henry Huxley ~
I frequently receive e-mails and
phone calls from people asking what reading programs are "scientifically
research-based." Unfortunately, that's not an easy question to answer.
Before I can answer the question, I have to ask two other questions --
"To do what?" and "For which children?"
To do what?
Let's consider the first question
first -- When you are conducting research, you must pay attention to both
the design of the study and the outcome measures you wish to examine.
The design and methodology of the study depend on what questions you want
to answer.
For example, let's say I wanted
to test a diet drug. The first question I might want to know is,
does taking this drug result in weight loss? That's fairly easy to
study -- I randomly assign a sizeable sample of people to treatment conditions,
give some of them the drug, give others a placebo, and monitor their weight
for a few months while they are taking the drug.
However, I may also want to answer
some other questions -- are there side-effects from taking this drug?
Could this drug cause long-term damage? Does the drug continue to
be effective over long periods of time? Those questions are also
easy to study, but the first study does not answer those questions.
This was illustrated by studies of the mix of diet drugs popularly known
as Fen-Phen. Studies showed that Fen-Phen did help people lose weight.
However OTHER studies showed that people who took Fen-Phen were more likely
to have heart valve disease.
Was Fen-Phen effective? Yes.
Was it research-based? Yes. Was it deadly? Apparently
for some people, yes.
Getting back to reading, there are
some reading programs which seem to offer short-term gains, but which do
not seem to offer much in the long-term. One intervention may involve
pulling a struggling reader out of his or her regular classroom for 20
minute tutorials by a highly-trained professional. And there may
be research studies that show that students make great gains while they
are in the program. However, other research may indicate that after
the students leave the program, the good habits and skills they learned
in the program are not reinforced in the regular classroom, and the student
begins to lag behind again. Thus, the program is effective in the
short-term, but does not seem to have lasting effects.
Also, it is important to remember
that reading is a skill which is built upon two subskills. The ability
to read with understanding is the product of the ability to fluently decode
text, and the ability to comprehend language (see S
is for Simple View of Reading). Research related to a reading
program may focus on that program's ability to help children learn, for
example, to decode with fluency. There may be no evidence that the
program helps children to develop comprehension skills or reading skills
in general -- only decoding skills.
For which children?
This leads to the second question
-- For which children? If you pick a classroom at random, some children
in that classroom will already have proficiency in all three of those skills
(reading comprehension, listening comprehension, and decoding fluency).
Some children will need help learning to decode text. Some children
will have some difficulty with comprehension. Some children will
have difficulties in both areas. One reading program may be highly
appropriate for some children in that classroom, but inappropriate for
other children. There are excellent programs available for helping
children develop phoneme awareness, and use that awareness to develop fluent
decoding skills. However, those programs are wasted on children who
are already adept at reading. They may also not be optimal for children
who primarily have comprehension difficulties. They are only effective
for certain children who need instruction in the areas that program addresses.
Thus, it does not matter if there
is a great deal of research showing a reading program is effective for
enhancing, for example, decoding skills if none of your students need instruction
in decoding. A program can be "research-based," and still be inappropriate
for your students.
How good is good enough?
There is another point I'd like
to raise about research -- a caveat emptor, if you will. Not all
gains are equal. Simply showing that a reading program "improves"
reading performance is not adequate. Some studies of reading programs
show that they raise scores "on average," but as a consumer, you should
be asking if the gains reported are satisfactory -- are they they gains
you want for your students? Let's say I have a reading program that
studies have shown moderately enhances reading scores for 25% of the students.
It is "research-based" -- specifically, the research shows that average
scores are raised, but not all that much for any one student. As
the consumer, you have to decide if that is adequate. As a consumer,
you have to decide if another reading program might be more effective and
bring you more "bang for your buck." It is not enough to ask if a
program will raise scores; as consumers, we must also ask, how much gain
can be expected from this program?
Quality
All of that said, there is another
concern that I have that I'd like to air here. What I have said up
to now in this essay assumes that the research in question is of high quality.
However, most of the research that I have read related to reading programs
has been of extremely poor quality -- poorly designed, poorly executed,
and poorly reported. The connection between the design of the study
and the conclusions drawn from the study must be clear and appropriate,
and all too often in reading research, that connection is tenuous at best.
Poor-quality studies should not be allowed to see the light of day, but
unfortunately, they are often published, and widely referenced.
In high-quality research, there
is a peer-review process that must be free from bias, politics, and polemics.
If I conduct a study of a reading program, before I am allowed to publish
a description of that study in any reputable journal, there should be a
panel of experts in the field who review my report to make sure the study
was designed well, conducted well, reported well, and that the conclusions
that I draw are clearly connected to the findings of the study. If
I make generalizations in my report, the peer-review should check to make
sure the generalizations are appropriate. The peer-review panel should
be skeptical and critical, and yet objective. Peer-review can seem
cruel, but it really isn't. Good researchers are used to it, and
they understand that it is a process that effectively elevates the quality
of the research that is published.
Another filter that helps to improve
the quality of research is the concept of "convergent evidence."
If I conduct one study that suggests that smoking does not cause cancer,
you should be skeptical in light of all of the other studies that have
been conducted that suggest a causal link between smoking and several types
of cancer. No single study is all that convincing, but a collection
of studies that are all in agreement about a conclusion is very convincing.
For now, the research in reading
and education programs is scant, sketchy, and of poor quality. For
35 years, there has been excellent basic research in the psychology of
reading -- we know a great deal about how children learn to read in general
-- but the body of research in reading instruction and reading programs
is quite poor. That is changing, and there are signs that the change
is here for good. Just as there was a revolution in medicine and
psychology research 50 years ago, so too is a revolution afoot in education
research. People are adopting more of a healthy skepticism about
programs -- people are asking vendors to show them the research supporting
the program. That's good. For now, the vendors are trying to
use poorly designed studies to support the effectiveness of their programs,
and for now, consumers are still buying it. But the consumer is getting
smarter, and in the future, the publishers and vendors will have to respond
with high-quality research evidence that shows their products and services
are effective and worth an investment of time and money.
