So I have a pretty ambitious schedule at NARST this year. NARST is the annual science education research conference. (It used to stand for something, but doesn’t anymore.) This year, I am involved in not one or two, but six presentations at the conference. And yes, that is a lot. Two of them are ones where I am first author on the paper and am presenting at the conference and the other four I am one of the co-authors (which is associated with a varying amount of responsibility depending on the paper). And these six papers fall into three very different areas, which makes the whole thing even more onerous.
* I am presenting a subset of the final results of the simulation meta-analysis that I’ve been working on for the last year and a half (a subset focused on science, obviously).
* I am presenting some findings relating to a large efficacy study of the PBIS curriculum (my part is focused on analysis of the weekly online implementation logs, but I’ve also been working on analysis of classroom video observations and teacher professional development). This work is largely concerned with teachers’ implementation of the new science Framework and NGSS-related ideas (mostly the integration of scientific practices with content). These papers are part of a related paper set (but there’s also another one that is in its own session).
* I am helping a colleague put together a presentation on analysis of afterschool science materials that we have been working on.
Simulation Meta-Analysis
Last year we completed a meta-analysis of simulations for K-12 STEM learning. The study, along with a companion games for learning meta-analysis by another group (led by my PhD advisor), was funded by the Bill & Melinda Gates Foundation. It is part of a larger effort that is focused on studying the use of educational games for assessment and learning. The SimCityEDU game that is being developed by GlassLab is also part of this larger effort. While GlassLab is doing a lot of development work, researchers at SRI Education’s Center for Technology in Learning is working on the research side of things.
More information about the study and GlassLab-Research can be found on the project website: Research on Assessment in Games (GlassLab-Research). This is also where you can download the full reports, executive summaries, and short briefs for both the simulation and the games meta-analyses. (These are officially being released on Monday 3/31 so check back then for sure.)
It should not be a total surprise that we found an overall effect in favor of simulations. We looked at two main groups of research studies: those that compared an intervention involving simulations to a non-simulation (more traditional) learning situation and those that compared two versions of a simulation to each other (with some type) of enhancement or modification to the treatment condition. There was a significant effect size for both of these groups of studies.
One surprising thing was that although we searched for articles on all of the STEM fields we found mostly science simulations. There were very few mathematics simulations, even when we extended our search criteria for some additional keywords (like dynamic or linked representations). Another surprising thing was that there was relatively little information in most articles about the nature of the simulation and/or the intervention that was being studied. So that was disappointing and made it difficult to run some of the moderator variable analyses that we initially wanted to.
Again, if you’re interested in the study, please download the reports and read about all of the interesting findings. I’d love to hear what you think.
Implementing Scientific Practices in the Classroom
This work is part of a larger efficacy study that is looking at the implementation of the Project-Based Inquiry Science (PBIS) curriculum. The study is designed as a randomized control trial within a large urban district in the southeastern United States. There are about 100 teachers (and their students) involved in the study, randomized by school, where about half of them are implementing this new curriculum and the other half are continuing with their typical curriculum. We are collecting all kinds of data, including video observations of a subset of the classrooms, weekly online log data of instruction, end of unit assessments, teacher-assigned student work, teacher background surveys, and information about their professional development.
While some teachers are using the new currirculum, all of the teachers in the district are being encouraged to learn about and begin changing their teaching practice to embrace the vision of the new K-12 Science Framework and the Next Generation Science Standards (NGSS) (even though this state has not yet adopted NGSS). Teachers had PD on the Framework and NGSS and we are focusing a lot of research around their implementation of the integration of the core ideas and scientific practices (especially developing and using models and constructing explanations).
At NARST this year, our research team has a related paper set that looks at some of the different sources of data from this project and how we are beginning to make sense of studying the implementation of the Framework using a curriculum like PBIS which is only partially aligned to the goals of the Framework. Since the Framework has only been out for a couple years, there aren’t yet any well-designed and researched materials available for teachers to use and so, for a while at least, teachers will need to make adjustments to existing curricula in order to address all of the Framework (and NGSS) goals. PBIS is in a good place to help teachers do this, but it is not perfect and does not go far enough (which is to be expected since it was designed to be aligned with prior standards and goals in mind).
I helped out with the papers on our video observations of teacher talk moves during classroom discussions and on the teacher professional development. But I will be presenting on my work of analyzing the weekly online logs that teachers fill out to help give us a sense of what they and their students are doing in the classroom each week. I have been looking at how frequently teachers report engaging their students in the eight scientific practices, how much this frequency varies week to week both within a single teacher and across the teachers (and research conditions), and ways to visualize and account for this variation. The findings so far suggest that many teachers may not have a very nuanced understanding of what the Framework is calling for w/r/t the practices and that it may require a lot of time, PD, and explicit curricular supports to help them achieve this vision.
More on the findings and implications related to the weekly logs can be found in my paper.
Afterschool Science Materials
This paper/presentation is about an analysis we did that focused on categorizing the types of materials used in afterschool science programs and then coding the materials for support features that help instructors teach and students learn. This substudy is part of a larger study – the Afterschool Science Study – that has been looking at the afterschool program that participate in California’s ASES network. This network in California has more afterschool programs than the rest of the country combined, so it was a great opportunity to look at a wide range of afterschool programs.
Through our study of the materials used, we found that most afterschool sites used what we called enrichment materials (those that are not specificaly designed for instructional settings and are typically stand-alone materials). Few sites used curricular materials (those that are designed to be used in instructional settings and typically involve sequenced lessons) and even those that did used them in a more stand-alone fashion. Enrichment materials are easy to find on the internet or through activity books (think 101 Science Activities kinds of things) but they typically lacked most of the support features that facilitators said they needed/wanted in a lesson and that the recent NRC report on informal learning said were desirable. Afterschool sites deal with a large number of constraints so it is not surprising that they tend to use enrichment materials on an ad-hoc basis, planning lessons one at a time, but we feel that afterschool science instruction could be more engaging to students and result in more learning (both content and practices) if they use materials with more support features included (which is typically curricular materials).
Cynthia D'Angelo 
One thought on “NARST 2014 Presentations”