research – Cynthia D'Angelo

Trying out some ungrading

Even though we are still in the midst of a raging global pandemic and there’s tons of bad stuff happening, I decided nonetheless to make a substantial overhaul to my fall classes this semester. I guess we will see in time if this was a good decision or not.

I am trying out a variation on ungrading. My version will include additional reflection prompts and discussions with the students, to help them be more cognizant of their individual learning goals and how they are progressing toward them during the semester. We will check in at multiple times to see how they are progressing and see what kinds of additional or alternative supports they might need from me or fellow students. At the end of the semester, each student will make a written case to me about what grade they think they deserve and why and if it diverges greatly from my assessment of their learning, we will discuss it. If they are underselling their achievements, I will give them my higher grade; if they have an overblown idea of what they accomplished, we will have a chat (I don’t expect this to be a big issue, at least not with these students). Most students in these two classes usually get an A or A-, so I’m not overly concerned with this screwing things up too badly. The hope is that it will do two things: 1) reduce student anxiety and 2) have everyone focus more on how and what they are learning.

I got rid of all points associated with assignments in Canvas. This was the most nerve-wracking part for me. It finally felt real, even though I had been talking about it and thinking about it and planning around it for months. No points. The grade tab would be meaningless. 😳

I am teaching two graduate courses this semester, both of which I’ve taught multiple times. If I was teaching undergrads or a brand new course, I doubt I would be trying this out. Too many new things at once could be hard, and I feel like grad students are going to be easier to convince that this approach will work than undergrads (also, the undergrads I typically teach are in a teacher licensure program and they have a lot of specific requirements). This approach will probably end up taking up more time than what I used to do, but I think it will be a lot better. Grading always made me uncomfortable, and I feel like this approach is going to help me give better feedback to students and support them in better, more tailored ways.

I just finished teaching this first week and introducing the students to the concept of ungrading and walking them through the general plan. Some students seemed a bit puzzled when I talked about it and others were visibly excited (those folks seem to have recognized the term and had some familiarity with it). I think a lot of them are cautiously optimistic. For next week, they will each write up a summary of what their learning goals are for the course.

If you’re interested here’s the language I put in my syllabi around this:

I am trying out a non-traditional grading approach this semester, a variation on “Ungrading” (see Blum, 2020 for more on this). This is partly to relieve some anxiety around points and grades and also to help focus our attention on the process of learning rather than a particular letter grade. This might be in flux a little bit and your thoughts during the process would be helpful for tweaking it and making it successful for everyone. It will involve likely more effort on your part, mostly in the form of being more self-reflective about your learning throughout the semester. I will provide feedback on my observations of your learning as well as providing structure and opportunities for this reflection. Sometimes this feedback will be part of whole class discussions and sometimes it will be individualized. At any point during the semester, I would be happy to sit down and talk with you about how you feel you are progressing with the course content.

You are the person most responsible for your learning. My job as an instructor is to create and structure an environment that will facilitate this learning. I can’t do the learning for you and I also am not the person best suited to knowing whether or not you have really learned the material in a way that is consistent with your learning goals. Each of you are in this class for a different reason. The more that you can reflect on your progress toward your particular learning goals and communicate that to me, the more successful this class will be.

Learning will happen through your engagement in the class content, working on the assignments, and discussing the topics with your fellow classmates. Being fully present in class is an essential part of this process. The more you can be honest with yourself and your classmates about what you don’t understand or are struggling with conceptually, the better we will all be in terms of addressing those concerns together. Of course, not everyone will feel comfortable at first sharing with others in the class, but one of my goals is to create a learning environment in which you feel safe in openly talking about what and how you are learning and what you still need to understand.

What this will involve:

Reflection activities about your learning goals and progressing towards those (and perhaps editing them) throughout the semester
Self-evaluation of your major assignments and final project
At the end of the semester you will make a written case to me about what grade you think you deserve and why and I will compare that to my assessment of your learning. If there’s a big disagreement between our assessments, we will meet and discuss.

Finally, I’d like to thank the members of our little ungrading book club in the spring of 2022 for sharing your thoughts and concerns and ideas around this. The few of you who had already tried this gave me hope for trying this out.

Another Year of Pandemic Teaching

Well, we are doing this again. This past June, when we had a few weeks of almost-normalcy (at least for us vaccinated folks), I really didn’t think we’d be back here again facing another semester of masks and testing and uncertainty and worry. But here we are.

I am lucky because my university has been taking this pandemic seriously from the very beginning. I want to mention that specifically because I know that many other colleges and universities have not done this or have really leaned into ignoring the real and serious harms that can occur when this is not taken seriously. We have had on-demand saliva-based PCR testing available to us as often as we’ve wanted it for the past year. I go to campus, spit into a tube, and a few hours later get a result pushed to an app (and email). That alone was a huge help to me during the last year before I got vaccinated in order to give me some relief to my anxiety about covid.

For this fall semester, they have required everyone to be vaccinated, for which I am very grateful. We still have testing available and I will still go get tested every week (and will encourage my students to do the same), especially now that we’ll be back on campus a bit.

I have chosen to teach in person this semester (and yes, it was a choice, and again, I am lucky that I had this as a choice and that it was not forced on me like at other institutions). This was due to the current set of circumstances, which could change throughout the semester. First, I am teaching two graduate-level courses that are relatively small (10-15 students typically). I have taught them both before, both in-person and remote, and I feel strongly that the class is better when it is in person. It is also, frankly, easier for me to teach in person. Because I am vaccinated and everyone else should be vaccinated and we will all have to wear masks the whole time and I am bringing my portable HEPA filter with me, I feel like it is worth trying to doing this in person. (If I had small children at home who are unvaccinated or if it was a larger class of undergrads – like I’ll have in the spring – I might be making a different choice.) But I am also, in the back of my mind, making alternate plans in case things get worse in a few weeks or a month or two and we’ll have to transition back to remote. I think that’s definitely possible and it won’t be awesome and hopefully we won’t have to do that (for multiple reasons), but at least I know I can manage that and it will be fine.

It’s very strange seeing people back on campus. The calendar in my office is still on March 2020. The vibe seems to be hopeful but uncertain. At least, that’s how I feel. My dog is upset that I am not working from home every day anymore. But, like us, she will adapt.

Also, last fall I added a pandemic and coronavirus statement to my syllabi. I’m including it again this year and have copied it here in case you might find it helpful to adapt to your syllabus:

Pandemic and Coronavirus statement
We are attempting to have as normal of a semester as possible during a global pandemic. I think it is important to remember that this is happening in the background of our learning this semester. The pandemic will affect us all in different ways and at different points in time throughout the semester. It is crucial that we are kind and empathetic towards ourselves and each other during this time. We need to be flexible and adaptable and we need to center caring for each other.
I want to be very clear: your health and the health of your family, classmates, and your community is the most important thing. This includes both your physical and mental health. Please keep me updated with how you are doing and if you need extensions on due dates or other support. I don’t need any specifics about what is going on, I just need you to tell me what you need.
Try not to compare yourself to others; you don’t know what others are or are not going through (this is good advice even when we’re not trying to get through a pandemic).

Toward a more open science practice with R

Recently I did a webinar with my colleague Joshua Rosenberg, hosted by the Center for Open Science, on Analyzing Educational Data with Open Science Best Practices, R, and OSF. You can find a recording of the webinar here and our slides and an example R Notebook are in an OSF repository here. I thought I would do this blog post to summarize some of the main things I talked about there and highlight some of the more important aspects.

This webinar was ostensibly about open science for educational data. I think most of us want to engage in more open science practices (which could include open data, open materials, preregistration of studies, replication, posting preprints, and reporting null results) but don’t know necessarily where to begin or what tools to use. I think we tried to make the argument that workflows, procedures, practices, and behaviors that are good practice for you by yourself, future you, and your internal team can also be good for open science. And that using R and its many packages and tools is a good way of achieving those goals.

I’ve written many times before about how much I love using R and how I want others to incorporate it more into their practice. I’ve now collected the series of blog posts as well as other related blog posts (like this one!) and slides onto one page for easy access. You can go to cynthiadangelo.com/r/ to see all of the R related stuff that I have worked on linked in one place.

In general, I’ve been thinking a lot lately about my values and commitments as a researcher and how I approach my work in a very basic way. What is important to me, my collaborators, and my field? How could I be doing things differently or looking at my assumptions differently?

This thinking led me to this set of things to consider for a more open science approach:

There are a lot of technical tools and solutions to some of the open science problems. But there are also philosophical/ethical/moral issues to consider.
Humans are participants that helped produce your data. All humans deserve respect and so do their data.
There’s no easy answer for some of these situations you might face. That’s ok. Part of what open science asks is to consider your options and document your decision making.
Reflect early on in your process about what your goals are and how you want to achieve them. What are your values? How do these match up?

Some of the tips and guidelines that I talk about in the webinar: (1) Documentation is so important. It’s also really difficult. Making things clearer for you and your future self will also make them clearer for others who might eventually see your code. (2) A tidy data structure will make things easier for you and easier for others to understand. If you’re not already on board the tidyverse train, it’s never too late to start. (3) Make sure you have a data dictionary somewhere that explains all of your variables. This sounds obvious, but it doesn’t usually happen because in the moment you think you understand all of your variables. But future you will not remember all of those things. Write it down. Preferably in a R Notebook (more on that later). (4) Pick a consistent and clear file naming convention early on in your project (preferably before you begin data collection). Think about the date format you use and think about the unit of analysis you will care about later and try to incorporate it directly into your filename to help with filtering and analysis later on. (5) Of course I want you to visualize your data. Descriptive statistics can be misleading sometimes and visualization is an important step in your process and is not just an end product.

The thing that ties all of this together is using a R Notebook within RStudio. R Notebooks make use of RMarkdown, a flavor of Markdown, my favorite way to write. It is a plain text file, so it’s easy to version control and easy to share, both things that are hugely important when thinking about open science. I really like R Notebooks because you can easily incorporate explanatory text alongside your code and figures/graphs are persistent across the page so you can scroll and easily refer back to something above or below where you are working. This, in my opinion, is a much better way to use R than the older way with scripts and the console.

R Notebooks can produce an html file that you can send to your colleague or friend who doesn’t have R installed and they will be able to open it up in a browser and see all of your wonderful thoughts and figures. It’s really great. You can also execute code in Python or JavaScript or D3 (or a few other programming languages) in addition to R, so it’s very versatile. There are a lot of output formats as well, including pdf, Word, slide decks, dashboards, and books. And they are all customizable. Check out the RMarkdown website to see all of the options and more details on how they work. For me, they dramatically changed (in a good way) how I do my work.

Maybe a good question to leave you with is to try and answer “What is the best way for you to work toward open science?” It doesn’t have to be a big thing; it can be a bunch of small changes over time. This hopefully shouldn’t feel too overwhelming. There are lots of us here to help.

Teaching in the time of Coronavirus

Our spring semester ended a little bit ago and after having taught online for the second half of it and now planning for an online fall semester, I have some thoughts. First, a little bit about me and my position in all of this, so you can understand where my perspective is coming from. I am an assistant professor at a large public Midwestern university. I study learning and educational technologies. I am an old millennial and have seen first-hand how technology has dramatically changed during my schooling career (from early personal computers in elementary school to broadband internet in college to smart phones and social media now). I have a serious underlying medical condition (type 1 diabetes) that affects my everyday life in sometimes unpredictable ways and makes me much more vulnerable to adverse complications with Covid-19 but also has supplied me with coping mechanisms and resilience to deal with this physical distancing¹ and uncertainty that we all now face. Alright, here are my thoughts on teaching (and just being in the world) in this time of crisis and how we can adapt and get through it safely.

Guiding principles during an uncertain time

My guiding principles around this transition to online were flexibility and empathy. Being flexible with myself in how I adapted the course and being ok with making changes week to week as we were figuring it out, and being flexible with my students in understanding what their needs were and how they were changing as the pandemic continued. Empathy is equally important because it’s more important than ever to create a supportive and nurturing classroom culture. And that is built on empathy and trust.

Continue reading “Teaching in the time of Coronavirus”

Data Visualization

This is the fourth part in an ongoing series on how and why you should be using R, especially if you are a social science researcher or education researcher, like me. If you missed the earlier ones, you can check out part 1 (Intro to R), part 2 (R Basics), and part 3 (Data Cleaning and Manipulation). This post will go into some more specifics relating to data visualization.

There are many ways to visualize your data using R. By far the most popular (and I think robust and flexible) is using the ggplot2 package. This post will talk a bit about why and how to visualize your data and some tips and basics to using R’s ggplot2 package to help you achieve your visualization goals.

Why visualize?

There are lots of reasons why you might want to visualize your data (or rather, why you should visualize your data). It can be a useful tool at various stages of research, and depending on where you are in your analysis process, different aspects of visualization might be more or less important to focus on. The way I see it, there are three main purposes for data visualization: examining your data, showing your data/findings, and sharing your data/findings.

What question are you trying to answer with your data? How can a visualization help you answer that? Do you have a really complex data set that is too hard to easily capture with a few numbers? Are you interested in variation and distribution rather than just means and medians? Are you exploring different relationships between variables and want to see how they interact?

Data Cleaning and Manipulation/Organization

This is the third part in an ongoing series on how and why you should be using R. If you missed the earlier ones, you can check out part 1 (Intro to R) and part 2 (R Basics). This post will go into some more specifics relating to data cleaning, organization, and manipulation.

In my opinion, the dplyr package is a game changer for those trying to learn R. It is what motivated me from just recommending that people use R to basically demanding that my friends and co-workers switch to R. I remember the day that I finally got around to learning how to use the package’s functionality and all of the ways in which it lets you easily and clearly manipulate your data frames¹. I just kind of stared at my computer screen and imagined how much better my data-life was going to be with these tools. I realized that the hours and hours I used to spend in Excel trying to massage my data into the right form were over². Also, I wouldn’t have to decipher weird R base code anymore when trying to create new variables or filter datasets. The dplyr package and its friends make your code/scripts much easier to read which will help both you and future you in trying to decipher what is going on.

Continue reading “Data Cleaning and Manipulation/Organization”

Back to academia!

In August I moved across the country (again) for a new opportunity that I am so excited about. I have started a new position as a tenure-track assistant professor at the University of Illinois at Urbana-Champaign. Yay! I am part of a new campus-level initiative called Technology Innovation in Educational Research and Design (or TIER-ED). I have a split appointment within the College of Education; I have a 75% appointment in the Educational Psychology department and 25% in the Curriculum and Instruction department. The plan is to do a lot of interdisciplinary work across campus (e.g., with VR/AR researchers, speech researchers, and the physics department).

Continue reading “Back to academia!”

10,000 Tweets

I have been on twitter for almost ten years. Twitter has changed a lot in that time and my enthusiasm for the platform has waned a bit over the years, but I still find it to be a compelling communication platform. Initially I used it to share about the more mundane, personal parts of life and my stresses as I finished graduate school. Lately it’s become more professionally-focused (most of the time) and more reflective of the many things that are happening in the world (but with important dog pictures also). I have met lots of people through twitter as well as listened and learned from thousands of people who I would never have met in my day-to-day life. It has helped me gain a wider audience for my academic work and has allowed me to share pictures of my awesome dog with strangers and friends alike.

I just hit 10,000 tweets (if I did this correctly then the tweet linking to this post would be number 10,000). And I thought it would be a good opportunity for me to go back through my twitter archive and get a sense of what all of those tweets were about and how I tweeted. (The analysis that follows is actually only on my first 9,945 tweets because I had to request my tweets a couple weeks ago and do the actual analysis.) This was also a fun R exercise for me¹.

Continue reading “10,000 Tweets”

Multi-modal Learning Data Collection at (Small) Scale

subtitle: even the best-laid plans…

Last year (spring 2015) we collected a really nice set of data of students collaborating in groups of three. The data collection process wasn’t entirely smooth or perfect, but it generally went off without any major technical or logistical problems. We ended up with a really nice dataset of almost 150 students with high quality audio data (four channels per group), video recordings (one per group), and computer log files (ideally one per group, practically more than one). [NB: The annotated audio from this first phase of data collection will be made available soon to other researchers. You can read the paper about the data set (presented at Interspeech 2016) here.]

In the spring of 2016 we set off to do our second phase of data collection, in classrooms during a regular class period. So unlike the first phase where we had just two groups at a time with kids who had volunteered and were excited to try out some math problems (a.k.a. the best kids), we had up to 10 groups at once with varying levels of excitedness and/or willingness to follow directions. We mostly wanted to test out how well the audio capture worked with all of the background noise in a typical classroom environment and see if our speech models still held up.

Continue reading “Multi-modal Learning Data Collection at (Small) Scale”

R Basics

This is the second part in an ongoing series I’m doing about why I think R is awesome and why you should be using it. (Check out part one!)

So now that you have downloaded and installed RStudio and have some data you want to play with, what are the next steps? How do you get started really working with your data? In this post I’ll cover an overview of the basics of working with R. Future posts will have more details on some of these topics.

Project spaces and working directories

So RStudio has you create a “project” when you get started. You tell it where you want the project to be and then it creates a file with “.Rproj” at the end. The location where this project resides is also your working directory. This will be relevant when trying to load in data

You can have more than one project (in different places if you want) and I have found creating multiple projects is mostly helpful for keeping different R projects separate. For instance, I have a main R project called “R Stuff” and then also separate projects for a couple of the bigger research projects that I work on. Things not attached to one of those two bigger research projects go in R Stuff and then I sort them out later and move them if they grow into their own thing.

My suggestion is to create most of your code/scripts/whatever in an R script file (extension .R) instead of just using the console to type in commands when you need them. You can load one of these in the main RStudio panel and type and edit your code here. Once you have some code/commands you like, you don’t need to copy them down into the console, you can just hit command-return (on a Mac, probably control-return on Windows) (or use the “Run” command in the upper right corner of that main window.

This script will allow you to do a couple of things: first, you can see your whole data manipulation/analysis/graphing workflow all at once; second, you can make changes to one step (e.g., switching the size of your graphed data points) and then re-run the code easily; third, you can write comments.

Now, I am not always the best at writing comments. But I try. And it’s really important. Even if you don’t think anyone else is ever going to see your code, you might need to look at it later. And no matter how smart and clever you think you are (well, actually I think if you’re super cleve then this is going to be more important because on a future day you may not be having a super clever day), you will probably need to read your code again. You are always, at a minimum, collaborating with yourself. And you deserver to have well-commented and documented code. So do yourself a favor and write some sensible comments.

Loading and viewing your data

Ok, so you have a data file and you want to start working with it. You have a few options. Most likely, it’s a .csv file and I’m going to assume to start that it’s in your working directory so you can use the command

d1 <- read.csv("MyDataFile.csv")

This will create a new dataset called d1 that is made up of what was in your csv file. You can use the “Import Dataset” button in the Environment panel. If your data file is in another location, you will have to enter the correct file path.

For the rest of the examples here, I’m going to use one of the sample data sets that comes with some R packages. The mpg dataset is one of the typical datasets for examples, as it comes in the base package. It is a datatset of car models and gas mileage data. Play along at home with the following commands.

To start, load the dataset: data(mpg). This should create an entry in the Data section of the Environment panel on the right. It should tell you the name of the dataframe and that there are 234 observations of 11 variables. Alright, but if we want to look at the data? If you type head(mpg) the console will output the header of the dataframe: the column names and the first six rows of data.

I prefer using glimpse(mpg) which is actually a command from the dplyr package. (If you haven’t already downloaded the dplyr package, now is a good time. We will be using it a lot in later posts.) Glimpse gives you a more compact view of more of the dataset and also tells you how R is interpreting each variable. For instance, R thinks that manufacturer is a factor (true) and that year is an integer (also true). displ is a “double integer” which is a bit weird, but for now, let’s just go with that it’s a special class of numerical variable. None of the text-based variables showed up as strings, which is good for our purposes with this dataset.

This is fine if you have a relatively small dataset, but it begins to get unwieldy if you have a lot of variables. The summary(mpg) call will give you a different view of your data. For the text-based variables, it gives you a count of them (up to a point) and for the numerical variables, it spits out the minimum, quartiles, mean, and maximum values. Pretty handy for a quick check.

If you want to see the whole dataset (or at least, a lot more of it, depending on how big it is) in a format more closely resembling that which you’re used to in Excel or something, you can use View(mpg). This will pop up a “normal” looking dataset in the main window for you to peruse.

Alright, now that we have looked at our data, let’s talk about variables. To access a specific variable, you will use the dollar sign. So, if you want to look at (or refer to) the model variable in the dataframe, you will call it by mpg$model. This way R knows that you are looking in the dataframe mpg and you want the variable model. You can use this in combination with lots of other things. For instance, if you wanted to find the minimum year of car that is in the dataset, you could use min(mpg$year) and it should output 1999.

If you make some changes to your dataset (e.g., adding a variable, reshaping it, filtering it, etc. — all topics for a future post), you can also save your dataset in a recognizable format. So if your new dataframe is called mpg2 you can export a csv of that using write.csv(mpg2, file="mpg2"). This will put a new csv file in your working directory with the filename mpg2.csv.

Other things to think about with R

In order to maintain an up-to-date version of R within RStudio, there are three separate things you need to update: RStudio itself (the application), R (the base), and all of your packages.

Updating your packages is easy in RStudio. In the Packages tab in the lower right corner (using the default set-up), there is an “Update” button that will easily show you which packages have updates available and let’s you download and install them. Super easy. (Updating RStudio is easy too: look in the Help menu (at least on Macs).)

When you start up RStudio, the console will give you a readout of the current version of R that you are running. As of today, that is version 3.2.2 (“Fire Safety”), but if you have an earlier version of R — as long as it’s not too old — most things should run fine. Updating R is sometimes a pain because you can’t do it directly in RStudio (which I think is confusing to people because you can update your packages easily in RStudio). When you download a new version of R, RStudio will automatically detect that, so that’s not too bad. However, RStudio tries to be helpful and store your downloaded packages in the correct place, but a major version update to R actually creates a new location and you have to migrate all of your packages over to that new place. It’s a bit of a hassle, but there is an easy way around it.

update.packages(checkBuilt = T, ask = F, type = "binary")

RStudio also has support for version control. Woo! You can use either git or SVN. I have more experience with SVN, but I am in the midst of switching over to git so maybe I’ll post about that at a later date. I’m not going to go into all of the details for how to set up and use version control, but, I will say that it’s a good idea even if you don’t need it for collaboration or sharing purposes.

Next post: we’ll look at how to organize and manipulate your data using my favorite package dplyr! Check it out here.