I was traveling with my kids this weekend across two states and the majestic views of the plains and the west. If measured correctly at the end of the drive, or even the end of the year, I am sure that there is no measurable impact on their standardized test achievement. Does that means they should stay home? Or conversly that we shouldn't teach 21st century skills.
The same question can be asked about any field trip, shadowing, or performance students are exposed to. Does that mean we should not do them? If the answer is no, we should continue doing these things as a part of a whole child education What then is our verdict about the quality of most of our educational research?
If we focus on short-term simple effects right after an intervention of any kind, we may well be missing two things: the long-term impact (or lack thereof- what Calfee called the poop-out effect), and exploring the impact of hard to measure meaningful activities.
How would we measure the impact of Day of Code, Read Across America, or Project based Learning?
I would argue that we need new paradigms, new instruments, and a vivid imagination exploring what outcomes of note can be. The relationship between researchers, teachers, students, and community members should change. The goal of research should change and become a cooperative endeavor that requires different structures than we have now. For example, a school can have a resident researcher who teaches and conducts a design experiment that serves school goals, as well as increasing our research knowledge. This is especially true of digital and other 21st century skills one's we know very little about. Still thinking about it...