Hi Andres, I can speak to the automatic pose inference part: it definitely makes the occasional mistake! However, the advantage of doing it automatically is the large and fine-grained nature of the video data that can be handled. So, on average, we anticipate that occasional errors will be offset by a volume of correct inferences in surrounding frames of the video -- though that remains to be seen, and we are also considering ways for a teacher/researcher to get involved in the process and evaluate or correct these scenarios.

With respect to context, we do have a bit of additional data regarding the type of activities (e.g., group work, student presentations) going on in the classroom at each point. From the qualitative perspective, this info is certainly relevant to interpretation, though it is a bit tougher to consider in computational analyses. This is an area we plan to explore, however!

I see, the pose inference occurs on a per-frame basis, so then you get an average or consensus inference over time?

Hi Andres -- I work with Nigel on doing automatic pose inference work on the project. We still need to explore how the overall poses change over time, but I think it will for sure be an important first step to see if our team's qualitative observations on classroom poses are corroborated by the fluctuations of pose inferences on the computational side! 

Great questions! We have some future plans to explore exactly how qualitative researchers can take advantage of the tool; for the scope of the current grant project, the tool is a bit too much of an early prototype. But eventually, we plan to explore the tool as a sort of filtering mechanism to help qualitative researchers find interesting and relevant parts of large video datasets, which might consist of visualizations of behaviors over time and key moments in time that could fit with a particular behavior. This may indeed somewhat constrain the types of behaviors qualitative researchers examine, but at the same time it might expand the types of behaviors they examine, if they were not already looking for these types. For new qualitative researchers, such as research students, it might be especially effective for pushing them in new (to them) directions.

Yes, absolutely. We are also looking at how the encouragement of student discussion in the absence of immediate teacher presence (as you had mentioned) relates to shifts in students' physical movement. Maybe the students become more relaxed, and more expressive with hand gestures or head movements? Would love to hear your thoughts based on you and your team's observations from analyzing math classroom discourse!

That is a great and important question regarding the bias in the visual algorithms, but one that I'm not sure I can sufficiently answer (perhaps another person more knowledgeable could jump in, if available!). I will say, though, within our application of OpenPose's pose detection methods, we are able to see some limitations of the algorithms. Characteristics present in the low camera angles of the classroom videos, such as the occlusion of certain students in the back of the class, and the density/close proximity of individuals' bodies, lead to inconsistent pose detections in the OpenPose output. It is difficult to confidently say whether this is due to biases present, or even tell in which situations (certain contrast, brightness, etc.) the inconsistencies tend to manifest, since there are instances when pose detection fails (for a few frames) even when it has been working up fine up to that point. While it is difficult to account and protect individuals against these situations, we hope to eventually explore more privacy options for individuals who do not want to be included in the detection output.

Thank you for linking that paper -- I have not seen it before! We are aware of Jacob Whitehill's other work and are inspired by it and other work in the area of developing automatic methods for analyzing classroom data. 

Hi Daniel, thanks for viewing our video and for asking about this - we're at a point in the project at which we'll soon be working to integrate output from the two methods (computational and human researcher-driven), but we haven't done this just yet. We do have some ideas about how we'll be doing this, some high-level and more at the conceptual stage for now, and some quite concrete/technical. 

At a high level, we're curious about whether the computational output can serve as a starting point for qualitative analysts, such as by identifying segments of the data at which certain kinds of activities of interest might be more likely to occur. Then a qualitative analyst could investigate those segments in an in-depth way.Christina and project colleagues presented a paper at the most recent AERA conference on this and some other ways we can integrate the data. 

At a concrete/technical level, we've discussed how selecting the time scale at which we'll join data (e.g., based on second or millisecond units) requires us to make certain decisions about what a meaningful unit of data is. But, we're very much working out these details now. I hope we can share more next year on what's working (and what's not) after we've explored this in greater depth.

thanks again for this!

Josh