NSF Awards: 1822865
Multicraft is a project on multimodal interfaces to empower individuals to play and create in the popular video game Minecraft. The platform includes speech, gaze, and tangible interfaces, and intends to advance the design and evaluation of naturalistic multimodal interfaces in the field. The combination of these various human-computer interaction modalities may also support learning for a broader diversity of learners and facilitate collaboration between young students in a more naturalistic and productive setting. Considering this, the video provides an overview of the motives of this platform, the changes made to Minecraft to achieve these goals, the experiences had with students using these technologies, and the explorations made in the context of improving collaborative learning and spatial reasoning skills in young learners.
Jeff Ginger
Neat project - has some similarities to what we've been up to! The command wrapper is something the game needs more generally (especially if it could capture commands that are so long they have to go into command blocks). Do you think it has advantages over other programming interfaces available for Minecraft, like the Python/Block editor in Education Edition or the Turtles of ComputerCraft?
Is most of your analysis of the spatial reasoning qualitative assessment? For a long time I've wanted to use block placement/alteration data to see if we can automate measures of engagement, collaboration and spatial reasoning (pulling from the mysql output of Core Protect, for instance).
H Chad Lane
Marcelo Worsley
Assistant Professor
Thanks for your questions, Jeff. At a high level, the Minecraft commands have some neat functionality that can be hard to replicate using blocks. Particularly, the in-game commands can live in the game, as opposed to being tied to individual players. Another factor is that the block based environment does not allow for undo actions after the code has been executed.
Relatedly, we find that many students seem more interested in doing coding with commands than with block coding or Python.
With regard to analyzing spatial reasoning, most of that has been qualitative, but we are in the process of doing some automated work with eye tracking data and also using log data. We have used some automated approaches of log data to locate important moments within Minecraft summer camps, but are just getting started with using automated approaches.
Pati Ruiz
Learning Sciences Researcher
+Jeff and +Marcelo this is a new idea for me, can you please say more about how teachers are/would be involved in the process of automating measures of engagement, collaboration and spatial reasoning? This is something that I did at the qualitative level as a classroom teacher, and wonder how automation might support(or hinder) this task if I were still in the classroom.
Marcelo Worsley
Assistant Professor
Good question Pati. I think one challenge with automation is that it can over simplify what students are doing. When we reduce the extensive building activity to a small set of numbers about how many blocks moved or destroyed we might over look a lot of the rich ways that students are thinking and engaging in the activity. In my work, I've mostly used the log information to see when students may be deviating from their typical build patterns. So, there might be moments where students are building much faster than they normally do, or much slower than normal. This might key me into important moments to look at qualitatively.
I tend to think about automation as something that can help us see different points of salience in our data, and less in terms of producing labels about students being "productive" or "disengaged".
Using something like eye tracking data, I might get a better window into what students are paying attention to during the building process. In our work, we see this in the form of attentional or visual anchors. As we get better at computationally analyzing this data in ways that don't oversimplify user behavior, we might start to see different ways that students are demonstrating changes in spatial reasoning practices, for example.
Dalila Dragnic-Cindric
Jeff Ginger
@Pati 100% agreed with Marcelo. I'd add that I think automation or AI-driven components should be tools to be leveraged by the teacher - a lot of the time the interest in them is to replace the teacher to cut costs and speed up exposure to material. If we went in the opposite direction, and saw automation as a way to help teachers find more venues for assessment and feedback to deepen project-based learning we'd be in better shape. The kind of R&D time required for it I've only witnessed at the private and university level, or in informal learning settings, unfortunately.
Pati Ruiz
Learning Sciences Researcher
Thank you for sharing this project - I explored Minecraft years ago but not in a collaborative way. Can you talk more about how you are seeing students collaborate in Minecraft through this work? There is a brief mention of this idea in the video and I would love to learn more about what this looks like in the Minecraft environment.
Marcelo Worsley
Assistant Professor
The collaboration piece has come into play on a few different levels. First, one goal in using tangible blocks as an input modality for Minecraft is to allow people to collaboratively build, and not have to debate who will control the keyboard and mouse. The tangible modality can also add a new element for engaging individuals who find the keyboard and mouse controls to be cumbersome. For example, in a recent K-2 camp that we ran, students and family members used tangible blocks as part of a building challenge. Parents could build something with wooden blocks, and the kids would try to recreate that structure in Minecraft. In that case, the tangible blocks served as a tool for parents and kids to collaborate in the Minecraft experience without everyone being glued to a computer screen. Their interactions could still take place in the physical world. We have also used the blocks in K-2 after school programs where kids build out their design plans in pairs, using the wooden blocks.
Another area for collaboration that we are currently testing is the utility of created shared named objects. For instance, let's say I created a neat tower. I can give that tower a name, and other people can recreate that tower by using Multicraft's clone command. We anticipate that this will support different types of collaboration within the game environment, but are still in the process of seeing if this hypothesis is true. Note: Minecraft has a built in command for cloning objects, but this requires students to determine the coordinates and size of the existing structure. This might be great for having students practice with cartesian coordinates, but can be a deterrent for easily sharing builds with one another.
Pati Ruiz
Dalila Dragnic-Cindric
Jeff Ginger
@Marcelo
>> Particularly, the in-game commands can live in the game, as opposed to being tied to individual players.
This is a great point. A lot more potential for altering the world or making an interface or experience for other players.
>> Another factor is that the block based environment does not allow for undo actions after the code has been executed.
This is one of the benefits of the Forge platform with ComputerCraft - you can add in other mods that can revert block states to undo alterations.
I'd also add that you can have a greater degree of parallel processing and interactive systems with command blocks or ComputerCraft turtles, compared to the lone player-based agent.
>> eye tracking data and also using log data
Cool! I look forward to hearing how that goes!
Pati Ruiz
Jeff Ginger
(also Marcelo I just noticed your advisor was Paulo Blikstein - I was the director of the Champaign-Urbana Community Fab Lab for half a decade - the world is a small place :)
Pati Ruiz
Marcia Linn
Hi David, It's great to learn about the exciting work you are doing. I love the interface. Your video is thought provoking. We share your interest in making science more accessible and understandable. See our work exploring the ways teachers use data while teaching for social justice in science--including helping students interpret scientific data. Enjoy,
Suzanne Otto
Teacher / Fellow
Thank you for sharing your work. I am completely new to Minecraft but am quickly learning that it may have many educational uses when purposefully applied. Your ideas for increasing access for all learners are important and I'd like to hear more about the specific modifications you plan.
Also - can you expand more on where you see this type of technology integrated into a classroom or school setting? Is it something that can be integrated into core classes to meet learning standards or is your vision to use it in enrichment classes or programs?
Marcelo Worsley
Assistant Professor
We are in the process of doing additional testing to see what types of features people would like to have. In parallel, we are exploring the use of gesture-based input (using a standard camera or a wearable device with an inertial measurement unit). The other major modification that is still in development in allowing participants to build using standard wooden blocks.
Broadly speaking we have seen Minecraft integrated into several different learning contexts. Within the current project we support middle school classrooms with lessons and activities related to coding, architecture, science (mainly sustainability), and pre-algebra. In some cases teachers are looking for connection points to specific learning standards, while in other cases it is used more as an enrichment. For example, we also partner with the local library and the YMCA to offer summer programming with Minecraft.
On the topic of learning standards, other practitioners and researchers have worked to identify some standards that are met within existing Minecraft Education Edition lessons. Seeing teacher creativity in coming up with ideas that align with different countries' standards has been particularly neat.
Dalila Dragnic-Cindric
Postdoctoral Researcher
I have truly enjoyed learning about your fascinating work—thank you for sharing it.
I have recently observed two Fortnite players in Creative mode teach each other how to build quickly. That situation made me think of the differences between how kids teach other kids and how adults teach kids. I wonder if you plan to investigate how players teach each other and what peer teaching and learning look like in Multicraft.
I look forward to continuing to follow your progress on this project.
Marcelo Worsley
Assistant Professor
Peers teaching each other is one aspect of what we look at across the various contexts that we work in. We have seen less of this during Covid-19 with students being remote, but it does certainly take place. We also see students using a number of web-based resources created by other kids to facilitate their learning of new approaches. Beyond this, our focus has largely been thinking about how youth engage in peer learning related to computational thinking. For example, when do students share code with one another versus explaining how to do something? When do students simply do something for someone else versus pointing them to a web-based resource? And when do students ask an adult to assist? Anecdotally, many of the younger students that we have worked with (K-2) seem more likely to want help from an adult than from a peer. For middle school students they have seemed more willing to accept help from a peer.
Pati Ruiz
Dalila Dragnic-Cindric
Postdoctoral Researcher
Thank you, Marcelo. Very interesting observations about the help-seeking dynamics. BTW, the Fortnite players I observed were also middle-schoolers. I study self-regulated learning and social regulation of learning, so it is interesting to think about the differences in help-seeking behaviors and the best ways to support learners of different ages in various learning contexts. Congratualtions to the whole team on this work!