research – Page 3 – Cynthia D'Angelo

NARST and AERA 2013

My “spring conference series” just ended. NARST (the conference formerly known as the National Association of Research on Science Teaching) was in early April (in Puerto Rico!) and AERA (American Educational Research Association) was about a week ago (here in San Francisco). Here are my notes and thoughts from the two conferences.

NARST

The big topic of the conference was, of course, the Next Generation Science Standards (NGSS) which officially were released at the tail end of the conference. Most people were referring to it during their presentations, even though we didn’t know exactly what it was yet. (Some people were conflating the new Framework with the new NGSS, but that’s a different story.)

There were a few presentations about one of the large studies that I am working on, an efficacy study of a middle school science curriculum. These presentations on some of our preliminary findings went well and I am really looking forward to next year’s conference when we will have even more results to report on and some awesome graphs to show.

Continue reading “NARST and AERA 2013”

six months in

Today marks six months at my new job. If someone had told me a year ago, when I was deep in the academic job hunt madness, that I would have found a job as intellectually fulfilling, interesting, challenging, and enjoyable as the one I currently have I’m pretty sure I wouldn’t have believed it. Part of that is probably because I didn’t know this kind of job existed. As it turned out for me, everything worked out: the job hunt didn’t actually destroy me, I learned a lot about myself and the type of work I really want to do and the manner in which I want to do it, and I found this awesome job at an awesome place in California (and not half way around the world).

I have had the great pleasure to work on a number of really interesting and diverse* projects just in these first six months, many of which would never had been available to me in a more traditional academic setting. Moreover, the people that I have been working with are great in so many ways. There is a true sense of collegiality and cooperation, which I had experienced before, but never thought possible at the scale at which we have it here at SRI. The teamwork and support structure makes our large and complicated projects possible.

———

*Diverse meaning that I am working on one project that reaches across the whole U.S. at multiple grades, another where I am helping evaluate a middle school science curriculum, another where I am part of a small team that is looking into the different use cases of a specific technology in science classrooms, and a big one where I’m leading a team doing a meta-analysis. Every day is a new adventure. :)

exporting Mendeley tags to csv or xls

I recently was struggling with a data problem, one that I hadn’t anticipated being so … annoying. I eventually found a solution, and thought I would share it so others wouldn’t have to go through the same thing that I did. [Update: I have a link to the files that I made at the bottom of the post.]

One of the projects that I am working on is a huge meta-analysis of research studies surrounding simulations used for learning STEM content, inquiry, and related skills. I am leading a team of researchers in this task (some of whom are at a different institution), and initially tried to figure out the best way to collaborate in reading and coding thousands of article abstracts. I decided to go with Mendeley, mostly because I knew it would be easy to get the citations into it, it’s free, it works really well, and I could set up collaborative groups to facilitate our coding process. What I hadn’t anticipated would be how difficult it would be to get the citations out of Mendeley and into a spreadsheet (which I was using for analysis of the abstract data and the next steps of our overall project). Specifically, I needed to get the tags in Mendeley out, since we were using the tag field for our coding.

[Sidenote: Using tags to code the abstracts in Mendeley actually worked pretty well in my opinion.]

Mendeley, while awesome in many respects, does not have many options for exporting data. You can do BibTeX (.bib), RIS (.ris), or EndNote XML (.xml). Through some sleuthing, I managed to figure out that BibTeX was probably my best bet as the tags were definitely part of that export (they didn’t seem to be in the xml file and I didn’t really want to deal with RIS if I didn’t have to). Now, of course, the new problem is that BibTex isn’t a spreadsheet and it’s only readable by other citation management software (and there’s a good reason why I don’t use EndNote anymore). So I still needed to get it into a csv type format. Luckily, through some interwebz searching, I found the program JabRef, which is an open source citation manager and is able to export to csv. Hooray!

However, my problems still weren’t quite over, as the default csv export filter didn’t include the tags that I wanted. Although free, open source programs are great in a lot of ways, they are usually not so great in the documentation of certain operations. And this was one of those situations.

So here’s how to make a custom csv export filter for JabRef that includes the tags that Mendeley uses. [Update: you can skip this step and just download the files I made – links at the bottom of the post.]

First, download the src files. This includes the the standard export filter layouts that you can then edit and tweak to your heart’s content. They can be found here: http://sourceforge.net/projects/jabref/files/jabref/ Open up the file openoffice-csv.layout (in your text editor of choice) and decide which fields you want to be included in your export to csv. “\mendeley-tags” is the correct way to refer to the Mendeley tags in the BibTeX file. My suggestion is to definitely delete the “\annote” field (especially, if like me, you had a bunch of random crap in the Note field in Mendeley, some of which took up a lot of space). You can of course change the order of these and delete other fields as well. After going through the rest of this process, I would have taken out more fields due to the regular expressions weirdness that came out the other side when there were strange symbols or characters (which happened a lot in the author and abstract fields – more on that later). If you don’t need the address, I would suggest deleting that since there were a bunch of formatting issues with it in the csv version.

You can also edit the openoffice-csv.begin.layout file to change the headings for the csv export – and obviously, it’s easier later on if you have ensured that the data and the headings match up. :) Make sure to rename your edited files (I called mine custom-csv.layout and custom-csv.begin.layout) and then save them somewhere convenient.

Then go back to JabRef, select Menu Options – Manage custom exports – Add new. Fill in the fields: export name – call it whatever you want; main layout file: navigate to your saved .layout file; file extension: .csv. Click Ok and Close. Then go to Menu – File – Export and your newly defined export filter should be listed there.

Now, this solution is not perfect by any means. There were definitely lots of formatting and regexp hiccups in the transformation and export processes. I’m not sure where these occurred and I’m not skilled enough at regexp to fix them. So, I had to go through my csv file and correct a lot of the issues (the concatenate function in Excel was exceedingly helpful in this regard). But, in the end I got all 2392 of my citations into an xls format and can now manipulate my data with ease.

[Update: Here are the files that I created. This is the .layout file and this is the begin.layout file.]

Advice for Grant Writers

I have written or helped write a lot of grants (like >10) in the last two years of my academic career. I have also recently gone to NSF and reviewed proposals. And people keep asking me if I have any advice for them when they are writing grants. So I thought I would put together something that might be helpful. Some of this advice is just general writing advice, but I think it is especially important or relevant for grant writing.

[caveats: I am putting this list together from the perspective of someone (me) who is relatively new to grant writing and still very early in her career and who therefore is approaching this as a new form of writing. I mostly write grants to go to NSF in the area of STEM education research, so this advice is based on my experience with funding through that organization.]

Read the call/request for proposals very carefully

This may seem like an easy one, but it’s really important. The tone and details of the request/call for proposals can tell you a lot about not just what kinds of projects the program or agency is interested in, but also how you should frame your argument for funding your project. Also, sometimes there are very specific requirements, such as asking you to identity the type and/or size of the proposed project in the first sentence or the title. These types of seemingly minor mistakes sometimes make it difficult for reviewers to properly situate your proposal and can lead people to think you won’t be able to pay attention to details.

What’s the big idea?

This is perhaps the most important part of this process. What is the big idea you want to address? What is the big problem you are trying to solve (and how will you solve it)? Once you have figured this out, you need to create a story or an argument for (a) why this is a big idea/problem, (b) why this is the best way to solve/fix the problem, and (c) why you are the team to do it. These three points all need to be addressed early and often in the proposal. You need to convince the reviewers that this is important and innovative work that is achievable.

Talk to a program officer

Two important things to remember: (1) program officers (at least at NSF), as federal employees, are required by law to respond to your emails and phone calls, and (2) program officers want you to submit the best proposal you can because it’s better for everyone if they are funding innovative and successful projects. This means that you should not hesitate to contact them about your initial ideas to make sure that your proposal fits within the program boundaries. Program officers are usually good about giving you advice about which program would be a good fit for your project and whether or not it is missing important pieces. Of course, keep in mind that there are limits to the type and amount of guidance they can give you at the proposal stage.

With a little help from your friends

Having a third party read your proposal is essential as well, especially if the proposal is about a topic that you have been immersed in for a while. You have probably developed some shortcuts, slang, or ways of talking about this topic that are not immediately apparent to someone who hasn’t been inside your brain recently. A third party, even someone who is familiar with your work, can help bring an outside view or perspective (like someone who might be reviewing your proposal) that should clarify unclear concepts or phrases, help you structure your argument effectively, and ensure that you have properly explained what you’re planning on doing, how you will do it, and how your team is well-suited to perform the work.

Make it easy for your reviewers

Things that reviewers are looking for specifically: overall big idea, prior work/results related to this project, intellectual merit, broader impacts, prior NSF support (or the equivalent), team, and workplan. If you write your proposal in such a way that it is difficult for a typical reviewer to find any one of these pieces, it is not going to help your case. A good proposal is clear, as concise as possible, and complete. Note that the prior NSF support section should include not just that you were part of a funded project but also any major findings of that project. Additionally, there is a fun catch-22 with funding. In order to get funding, you (usually) need to have shown that you have already been successfully funded. This means that if you’re an early career researcher and applying for your first grants, it is extremely important to find more senior researchers to team up with who have a history with that funding agency. A well-known advisory board also helps in this regard.

All of the pieces

There are a LOT of documents and details to keep track of during the proposal writing process. It is extremely important that you stay organized throughout, otherwise the few days before the deadline are going to be hellish (I’m not kidding). Make sure you start the process soon enough to ask for appropriate letters of support from your advisory board and collaborators. Make sure that all of the details, language, and formatting match across the narrative, summary, references, budget, budget justification, letters, subaward information (if relevant), current & pending, biosketches, data management plan, and mentoring plan. Make sure you don’t forget any of those pieces. Make sure that you review the final pdf before it gets submitted – especially if you have a lot of figures or diagrams (sometimes certain programs don’t render these correctly when converting to pdf).

Sign up to review

A lot of young researchers seem unsure about this. But basically, if you have finished your Ph.D., you can review for NSF. They have some advice for how to do this buried on their website somewhere, but the easiest thing to do is to create a short version of your CV (about 2 pages) that highlights the kind of work that you do and your areas of expertise and send it to a program officer in a relevant area, letting him or her know that you are interested in volunteering to be on a review panel. If that specific program officer doesn’t need someone with your expertise for their panels, they usually pass along your information to their colleagues to see if someone else does. Being on a review panel was one of the most interesting and valuable experiences I have had in terms of learning how to write grant proposals and discuss research projects. You get to see things from the other side, learn about lots of cutting edge research and ideas, meet people in related fields, and most importantly, see what parts of the proposal are most important during those funding conversations.

Keep trying

In general, the world of academia and research includes a lot of rejection: rejected conference proposals, rejected journal articles, and unfunded research proposals. Research proposals are notoriously hard to get funded. The acceptance rates at NSF are in the single digits in some programs (overall, NSF’s acceptance rate is around 28%). All of this rejection requires a thick skin and an optimistic outlook. Whenever you get reviews back from a proposal, try to use the feedback constructively to either make improvements to your funded project or to write an even better proposal next time.

The Next Chapter

I am pleased to announce that I have accepted a research job with the Center for Technology and Learning at SRI International! I will be starting in mid-July and can’t wait to start working on some of the projects there and continue building my research program. I think that SRI will be a great place for me and allow me to take advantage of my unique background and training in order to do really innovative work in the intersection of science education, technology, and measurement. SRI supports research in all sorts of STEM areas, not just education, and there are a ton of resources available.

SRI is located in Menlo Park, a few blocks away from Stanford (the “S” in SRI used to refer to Stanford – although they haven’t been affiliated with them since 1970). I am excited to move back to California (but I think my family is even more excited) and it should be interesting living up in the Bay Area instead of SoCal like I’m used to.

I would also like to say that I was lucky enough to have to make a decision about where I wanted to work next, and although my other option was very exciting in a lot of ways (due to its location – in Singapore – and research opportunities), I feel like SRI will be a really good fit for the next chapter of my career.

some thoughts on AERA 2012

I just got back from AERA 2012, which was in Vancouver, British Columbia during the past week. It was my fourth AERA, and I hadn’t attended last year because I had basically sworn off the conference and decided I wasn’t going to go very frequenly anymore. Luckily, I was invited to a session at NCME (the sister conference to AERA that was co-occurring) and decided to go (it also didn’t hurt that the conference was in Vancouver). It turned out to be a great conference all around, and I went to a few good sessions and met with lots of old and new friends and colleagues.

The best thing about AERA usually is that everyone goes to AERA. I heard that this year there were over 13,000 people registered. That’s a bit crazy. This huge zoo is people is usually one of the reasons that I don’t like AERA, but for some reason it didn’t seem so bad this time around. Perhaps it was the way the conference ~~center~~ centre and hotels were located/configured, which forced people to walk around and run into people more than in other places. Or maybe it was that many of us chose not to get international cell/data packages and were reliant on congregating in certain places having wifi (and then consequently spreading around the passwords for the wifi). Whatever it was, I ended up running into lots of people that I wanted to talk to and had lots of interesting discussions.

I think also that one difference this time around was that I felt that for the first time I was able to help start connecting other people together, instead of relying on my advisors and professors to connect me to people they knew. That type of thing still happened, of course, and is an essential part of academic networking, but I was taking on that role for other, younger researchers now. And that felt kind of nice.

One particularly interesting set of conversations centered around using games in classrooms, how students might be viewing educational games within the context of a classroom setting, and how that context might (a) change how the student views the game/simulation and/or (b) change the type and amount of learning that is possible in that game (versus an out-of-school, informal, or student-directed session). This could possibly be highly dependent on what educational value the student might attach to the game outside of the classroom in the first place. For example, a commercial game being used in the classroom might have a larger “learning differential” than a game specifically designed to be used for learning (either informally or in a classroom). Definitely something to think some more about.

Vancouver was also a great city to visit. We had wonderful weather almost the entire time, which just highlighted how beautiful the city is. There was lots of great food to be eaten, especially seafood (yum!).

On my last afternoon, I (along with some others) went to the Vancouver Aquarium. It was a really nice facility and I would totally recommend going there if you are visiting and you like awesome things. Here are a couple pictures of the aquarium and Vancouver. More pictures will be posted soon (somewhere).

Part of the Vancouver skyline, from Stanley Park.

Awesome jellyfish at the Vancouver Aquarium.

Part of the sea lion feeding/training non-show.

Baby clownfish. Awww.

Totem pole exhibit in Stanley Park.

Watching sea planes land from the Convention Centre.

[side note: all of these pictures were taken with my iPhone 4 because (a) it’s what I had with me the entire time and (b) my better camera just broke.]

Don’t believe everything you hear

Yesterday I saw this BBC video about the McGurk effect. Basically, it’s a phenomenon where your brain interprets sounds differently based on what you are seeing. It’s a really amazing and robust effect, especially since it is not dependent on you not knowing about it.

This reminds me a lot of when I was teaching undergraduate physics labs and we would do the famous ball drop experiment where you would release two balls (same size but different mass) from the same height and see which one hits the ground first. The majority of these intro students would usually predict that the more massive ball would hit the ground first (a common intuition to have). The interesting thing was that a few of these students would still think this after we had done the experiment.

I remember asking some of them why they still thought that the more massive ball hit the ground first. And inevitably, some of them would say, “I heard it hit first.” At the time, this seemed kind of crazy to me because I had been there and heard the two balls hit at the same time and other people in their group had also heard them hit together. But they insisted that they heard a separation.

Later when I started studying science education I learned the name for this phenomenon: theory-laden observations. The students thought that the more massive ball would hit first and this influenced how they took in information and therefore their observation confirmed this idea. However, it seems possible that it might be even more complicated than that. Although the McGurk effect seems primarily to be focused on hearing different phonemes and sounds based on lip movements, it’s possible that the parts of the brain being confused in that area are the same as the ball-drop phenomenon. Students *think* they see the massive ball hit first and therefore *hear* it hit first.

(via kottke)