In his Comments, Daniel has asked a lot of questions, and I thought that I’d take a minute to answer two of them. First, after I reviewed a book (Elyn Saks, The Center Cannot Hold, Hyperion:New York, 2007) in which a schizophrenic individual provided her personal descriptions of her life with this illness, he asked how I (a scientist) could know that a first-person account like hers was truthful.
The short answer is that I (a scientist) cannot know. Subjective evidence provided by any individual introspective reporter has a limited value in science. On the other hand, I (scientists) know a LOT about the disease processes that alter behavioral function in schizophrenia, and scientists have conducted thousands of studies variously measuring the performance abilities of populations of schizophrenic patients, using experimental strategies that are designed to eliminate the possibilities of false reporting or deception. Scientists appropriately subject this accumulated information to statistical analyses that informs us about measures of certainty/uncertainty, and on the basis of this wider knowledge, and having read a number of other consistent first-person accounts, Gwyn Saks personal accounts ring true. I applaud the bravery and the effort that this apparently-strong, intelligent woman has made, to provide us with a highly informative and apparently unvarnished account of her life with this devastating illness. However, as a scientist, her individual reporting necessarily falls into a non-fact (subjective) category.
In the same questioning vein, Daniel asks how you can know that students who show powerful advances from using a brain plasticity-based training program (see May 9th blog) aren’t improving just because they perceive these new activities as being exciting, and rewarding. Daniel, the reason that randomly-assigned intent-to-treat controlled trials with psychologically-balanced “control” therapies are the Gold Standard of outcomes assessment is to eliminate just this kind of uncertainty. These training programs have long since passed that test: if you match the hoopla for the “control” class, Fast ForWord training still accounted for much larger gains than were recorded in an equally exciting/fun/intense/serious control limb of the trial. Moreover, the magnitude of those gains were directly correlated with the gains in performance recorded in specific FFW exercises. That argues strongly that those exercise gains translate to kid-performance gains. Or put another way, they can ONLY come from the specific training provided by FFW. Finally, other studies have shown that this training remodels the brains of trained children (they’re “higher-performing” = “more normal” after than before training). It’s hard to fool Mother Nature in such studies, Daniel!
If you look across the landscape of claims for the thousands of things that have been created and sold to make children or adults smarter or more capable, Daniel, you WOULD discover that the vast majority of claims come from non-controlled trials. Exactly the kinds of misgivings that you raise may apply for them. NOT SO, for the FFW kid-training programs, or the Posit Science Brain Fitness or Insight training programs. Controlled outcomes trials are costly, and complicated to organize and conduct. Alas, there is just no substitute for them, if the goal is to determine whether or not your programs really work.
