Methodological deficiencies notwithstanding, I'm not sure why anyone is surprised that the the interim Reading First Study seems to be showing null results.
Reading First was a product of political comprise. Instead of limiting grants to research validated programs reading curricula, the Reading First was watered down to permit reading programs "based on" scientifically based reading research. In reality, all this meant was that publishers needed to provide curricula appearing to have "explicit and systematic instruction in phonemic awareness, phonics, vocabulary development, reading fluency, and reading comprehension strategies.
Reading Recovery is the antithesis of this approach to teaching reading. Yet look at what those clowns managed to show (pdf).
Designing effective reading instruction targeted to at-risk kids requires an orchestration of minute details and variables.
Throwing a few disjointed phonics exercises in your previous whole language program is not effective instruction, though you'd likely be able to make a case that it falls within the statutory language of "explicit systematic instruction in ... phonics" since that undefined term is all but meaningless. Oh sure, such instruction is going to be successful with many "advantaged" kids, but a broken clock is correct twice a day as well yet we don't say that this clock tells good time.
Let's be honest, many "real" phonics programs only perform marginally better than phony phonics programs. Phonics is not a magic wand that can be waved over a reading program and make it effective instruction. Phonics is a tool. A tool that can be wielded many ways, only some of which are effective with "at-risk" children. And phonics is only one smallish part of an effective reading program.
Prior to Reading First the major education publishers were not exactly cranking out high quality instructional programs--nor were they known for their ability to design effective instruction. Then an opportunity came along to grab a larger share of the reading curricula market by putting out a product that would be selected by all those schools with Reading First grants. All that needed to be done was to redesign your program so that it appeared to comply with the undefined statutory language of Reading First. Not exactly a Herculean task.
There are an infinite number of ways to design a reading program that complies with the Reading First statute. The probability of any of the major publishers stumbling upon one of the few effective combinations is pretty slim, especially considering their previous track record. And, the probability of a school selecting one of these newly-cranked-out reading programs from one of the major publishers and seeing improved results is similarly slim.
Then there are other problems.
Even if a publisher did stumble upon an effective program, the chances that a school would actually implement it with fidelity is slim as well. There's a reason why the few reading programs that have been validated by research tend to be scripted: without the scripts, schools would screw them up.
And selecting a generic reading comprehension test as your measure of achievement is going to be mostly testing student IQ/SES and the amount of background knowledge they've acquired which arguably has little to do with reading ability.
Add all this up and the only conclusion you'd expect from evaluating Reading First schools as a whole is going to be a null set. That's what the interim study appears to have found. Lest you forget, that's what Project Follow Through found as well. Most of the Follow Through schools failed to achieve positive results. In fact, almost all of the Follow Through schools showed negative results. Reading First is only somewhat less of a failure than Project Follow Through. But that's only if you look at the programs as a whole.
It's a fair bet that when you look at individual reading programs, some of the Reading First programs will show significant positive results. One of the Project Follow Through programs showed positive results.
In education we expect a preponderance of losers. Education is not yet a mature profession. It's not even a profession. We're not going to see improvement until we make a concerted effort to separate the winners from the losers, scrap the losers, fund the winners, and find effective means of identifying and developing new winners. The Federal Government has already attempted to go down this route twice and failed both times. The Democrats tried it with Project Follow Through and the Republicans tried it with Reading First. In both cases political forces overwhelmed and weakened the attempts, returning us back to the status quo.
I expect the same outcome with Reading First. Some winner might be identified in the final report, but that outcome will be overwhelmed by the overall failure of the program as a whole. History will no doubt repeat itself again the next time we spend lots of money on a fancy grant program. You can count on that.
You're kidding yourself if you think there will be a governmental/political solution to our education woes. That's not the way the world works. It works the other way.