Great questions Billy (and hello!). For the evaluation, we were primarily looking at efficacy of the workshop series and teachers intentions to integrate resources, content, etc. Many teachers (45%) indicated they signed up for the workshop because of interest in the content and/or to improve their own knowledge and skills to teach this content. Post-workshop surveys showed 75% of the teachers intended to integrate the resources, activities, etc into their instruction with another 20% unsure. This is right on par with the in-person PD responses showing these online PD workshops are holding up to the integrity of the in-person ones. 

There were a couple features of the online PD that stood out to teachers - 83% of teachers said the live virtual streaming session with the expert scientist was very to extremely useful. Teachers appreciated the opportunity to engage firsthand with a scientist and to ask their questions. This was a definite perk of the online format that was not typically experienced in the in-person PD. In addition, teachers valued the follow-up workshop opportunity to engage in collaborative discussions with other teachers on how to use the content and resources. The breakout rooms in the online follow up were a good tool for these discussions. 

So in terms of delivery method, the online approach is meeting if not surpassing some of the benefits of the in-person format. Now, teachers of course would like more time to do hands on activities with one another and to see/feel materials. That is difficult to accomplish with online. However these other assets like the live time with scientists would not otherwise have been accomplished in person. 

Hope this helps! 

Hi Joy! The comparison of evaluation results is so interesting, and sounds like it reveals some unexpected benefits that could help to inform professional development best practices post-COVID. 

Hi Billy! Adding on to the awesome feedback and results from the educator surveys, one of the aspects of this virtual model that we have heard educators mention over and over again is that they like the 3-part aspect of these new mini-series. Some have shared that they find that breaking up one topic into several parts makes the content easier for them to digest. Others like that the events are shorter and find them easier to attend than an all day, in person PD. Lastly, having a few weeks between the live NOAA hosted portion of the event and the follow-up events hosted by the Alliance Partner sites allowed for several teachers to try out the activities in their classroom and come prepared to discuss! As part of the NOAA Ocean Exploration education team leading the PD opportunities, I personally found that this made for lively discussions and debriefs during the follow-ups. What would be more of a theoretical conversation in a single day event about how teachers might adapt the lesson, become more real as educators shared what they did, what worked and what didn't, and how they would adapt it for future use! 

Hi Stephen - thanks for the question about translation to instruction in the classroom. That's the pinnacle of PD and yet difficult to capture. For this evaluation, the scope was limited to looking at the immediate PD experience and intentions of the teachers. Ideally, there would be a 3-6 month follow up to understand what teachers did in the classroom, how it worked, etc. We've asked such questions of the in-person PD experience but as this online format is new this year there has not yet been the time to pursue those questions. 

In general, teachers gravitate toward these NOAA PD offerings because of the hands-on nature of them and really bringing science to life in the classroom. We would love to see more student artifacts of how these activities played out with the students as another data collection point. Perhaps in the future! 

I wanted to add to Joy's comment- one other interesting part about this particular series of workshops is that we do have a cohort of teachers who want to sign up for as many of these particular workshops as they can. It would be so interesting (a dream! really) to see if this heavily-invested cohort of teachers translate these materials differently/ more/ or in what ways they translate materials into their classrooms because of their continued participation over time. 

Hi Stephen - great question about how the educational materials are being translated into the classroom. As one of the NOAA Ocean Exploration education team members leading the PD events, one of my favorite aspects of this new model is the 3-part aspect. Specifically, I have been impressed by the number of teachers that have been able to test out at least some of the materials presented in the NOAA hosted event (part 2) before participating in the follow-up sessions (part 3). Most of the follow-ups have been hosted 2-4 weeks after the initial event. This has allowed several teachers to come to the follow-up sessions ready to discuss the activity and speak to how they used it in their classrooms, what worked/what didn't, and how they would adapt for the future. This has been a great addition to the conversation between educators as compared to our more traditional one-day in person PDs, as it allows other participants to ask their peers from the same teaching region questions about the materials, especially how they were adapted. In several of the follow-ups I participated in, this seemed to energize teachers and give those that hadn't tried the materials yet the confidence to do so. I agree with Joy, that having a longer time period between parts 2 and 3 of the mini-series would only allow more educators to come ready to talk!

Thanks for your comment, Julie! I am happy to address any questions you may have.

Agreed, thanks Julie for your comments! It has been really rewarding to hear/see how well-received these programs have been, and that many participating educators will consider engaging in virtual PD programs even when in-person events can resume. The flexibility of virtual programs is key, in that they meet educators where they are, at that point in time, and can work with potential travel limitations, family obligations, or other contributing factors.

Meghan, thank you for your comment! Accessibility was front and center for this team and we were constantly checking and re-checking ourselves to help build bridges of accessibility for the audience and the subject matter.

One example throughout the entire series was to embed a conversational interview with a subject matter expert in each virtual professional development. This was intentional so that we were not "explaining" the science but letting the subject matter grow out of interpersonal conversation between the host and guest, as well as, interaction with the live audience.

Meghan-  HI!  Thank you for checking out this amazing work. The ability of people from all over to participate and be actively involved in the conversation was definitely amazing in this format.

Yes, a few keys to answering the great questions were: having some people "behind the scenes" monitoring the chat AND then also saving the transcript of the chat so we could address some of the questions in the follow-up Zooms (smaller groups).  I was so pleased with the results of that.

Nickolay, I think you are absolutely correct that we cannot replace those intensive experiences and retreats with work online. Our aim was not to REPLACE those, but to do the best we can in the current situation with essential restrictions that prevent such experiences. We don't aim to permanently replace in-person PD with online. However, in terms of exposure, accessibility, and providing exciting experiences to the most educators (and therefore students) possible, there are definite benefits to this model that are exciting and could be leveraged moving forward. Personally, I could see us (and many others) doing some sort of combination of the various models moving forward. We can provide higher level overview, engagement, and initial excitement for the content and program online and then the most passionate educators who have access and are able can follow that up with in-person experiences.... or the other way around! I don't believe it has to be one or the other.

And I would add that there is a range of PD formats and end goals. PD varies from one day workshops to intensive summer institutes, more like what you were highlighting Nickolay. You wouldn't expect a one day workshop to transform a teacher's instructional practice for the long term and completely shift their way of teaching. That would not be an appropriate expectation for that sort of intervention. But a one day workshop or series of online workshops like this are well suited to introduce teachers to resources and new content and help them gain familiarity and comfort with that content. Hope that helps add some context to any comparisons. 

(My dissertation research was on teacher PD so I am always happy to talk more about what "works" in PD!)

Good points Nikolay, Spent part of this last year working on creating spherical constructions (think 3-d interactive tours), spending time behind a set of Oculus Rift glasses, doing autocross simulations, and trying to understand more of the gamers worlds and intersections with the real world. Fascinating - certainly I train for my autocross race weekends in my home simulator - can't drive my car that fast on public roads and I have to be amped up and used to the speed before I hit the track. And there can be adrenaline in learning online. But we are not there yet in the clunkiness of the virtual tools relative to the real world outdoors (probably a good thing). So what will be interesting to me also, is to see what we carry on with us as we head back to the world we knew. I have a 2nd grade teacher that has just proposed creating a Minecraft version of our research institute for her Masters, link it to standards, then we will link it to fieldtrips. Might be useful or maybe not.  We definitely link deep sea experiences (necessarily virtual) with real world shallow water, backyard or classroom phenomena in our PD's. Overall, seems to me that reality is  a bit different these days for our students (I am an oldish guy). But cyberspace interactions do not physically kill you yet (nod to Willam Gibson) although tripping over things while gaming does make it possible  :) Can't wait to get the students and teachers back outside with us.

Tami, Joy and Mark, thank you for your thoughtful responses and additional information! Indeed, that mindset to expand and enhance, rather than simply replace, is a very helpful framework for the way forward and this is where what we benchmark has important design consequences for the outcomes.

In the interaction design space, a good rule of thumb to guide innovation is whether technology is leading that transformation or the user expectations for the need of it. The failure of 3D TV, recent retreat from “everything VR” and decades long iterations of sleep-inducing teleconference technologies, are good examples of what happens when this relationship is reversed. UI/UX designers have a hard time convincing us that this is not just pampering of the senses and hollow entertainment - the haptic, broader sensory and cognitive engagement fascinate the human mind and senses and are thus inextricably linked to understanding, learning and action.

In the educational psychology space, Papert’s Principle rings true as well - if after the hundreds of hours that we have spent on Zoom, Teams, Google Hangouts and Meets, StreamYard too, if we can’t point to a long list of what we as the users, designers, teachers, PD leaders and mentors want more of ... ... ...  there is a gap to fill!  15-inch screens, thumbs up and applause icons on the participant grid will not cut it. This is where some folks see opportunities for more. Much more! I encourage you to push further in that space, and in the meantime, if it comes across as clunky, because Mark, I agree, it still very much is, don’t give up on it - experiment, demand better design, integration and support. It is not a technology shortcoming, really, it is a design opportunity. I am excited to follow your work! In the integration of unique content development and educational design, I think you are really on to something! 

Thank you Nickolay! I am really enjoying (and learning from) this dialogue! Your statement, "It is not a technology shortcoming, really, it is a design opportunity" is spot on- with StreamYard, it's a great tool, but ultimately it's the collaborative process we have developed, media integration, and ability to have genuine conversations with scientific experts (as opposed to just moving through a slide deck) that has led to the success of these programs (I feel).  StreamYard is just another tool in our growing box of engagement tools. I, too, look forward to where this work takes us! (and I equally look forward to learning from others!)

Thanks Jean for your comments! I would love to know how you have used StreamYard in your work- is it for virtual interactions with associated media and video or for other purposes? You are correct, emerging digital technologies, especially those that enable work in cloud environments, are impressive- and I am always interested in learning how individuals are using these tools (usually in different ways!)

Thanks for the kind words Jean! These PD mini-series events were my first time working with the StreamYard platform and I am still blown away by the production potential of the platform. The ability to adjust who is on camera and include media as people are talking makes it really feel like a show, rather than just another webinar. What is your favorite aspect of the platform? We are always look for new ways to leverage the software to make our programs even better!

Troy, Thank you! It is a specialized and amazing community with a lot of passion for what we do! I would say that the best way to connect is professional organizations. The National Marine Educators Association (NMEA, www.marine-ed.org) and its regional chapters are all incredibly helpful in reaching formal and informal educators in a variety of settings across the US and around the world. I could also help you connect with some folks who specifically do ed research in marine science if you send me an email at tami.lunsford@gmail.com. NMEA has an ed research committee.

Hi Troy! To piggy back off of Tami's comment, and echoing NMEA as a great resource, a lot of people in the marine science field are usually pretty responsive and often willing to connect and share their knowledge with others (or if they are simply too busy, they will communicate that).  So I would suggest if there is an area of ocean science, or maybe a specific research project, or even a region your are interested in, find people doing the work and reach out.  Many scientists are also tasked with "broader impacts" associated with their work, and will need to communicate results to wide audiences, beyond that within their scientific community. So there is a distinct requirement to actually connect and communicate about marine science research results.  Happy to connect with you offline and speak further to this if you'd like (holly_morin@uri.edu).  Thanks!

Gracias Ivelisse. Muchos maestros que nosotros trabajamos con estan ensenando cursos cientificos que no tienen un emphasis en cosas marinas. Con nuestras activididades, los recursos estan alineando a los Next Generation Science Standards, y pueden conectar a fenomenos y conceptos de ciencias en general. Lo siento si mi espanol no es perfecto :)

Thank you, Lance! We would be interested in learning more about your ocean simulation partnership and are always curious to identify cross-collaborations. We can be reached at oceanexeducation@noaa.gov. 

I second this question...I was going to post the same inquiry.

As a teacher who uses the materials in my classroom and a facilitator of these workshops, I can say that parts of these workshops will DEFINITELY be used moving forward.  The videos explaining the tools, the interviews with researchers and technicians in the field, and some of the digital lessons will be invaluable moving forward.  Even when classes are full-time in person, some students benefit from videos explaining something that they can watch on their own. Plus, as schools are mostly now 1:1, even if they were not before, I personally plan to balance digital and hands-on lessons moving forward.  I am a huge fan of multiple learning and teaching styles to try to reach all kids, so having these pre-made videos and digital lessons are going to be great!

In addition, as was briefly discussed above, we can reach more teachers if we have a variety of PD options.  So having some asynchronous videos, webinars, AND in-person PD expands the reach of this amazing work!

Something related in terms of reuse of the materials created, I believe it is important to consider some of the creative process that has become part of this model. Specially, using cloud based tools to host these virtual PD events that are accessible and do not require extra software or high learning curves. One more example of possible reuse would be to intentionally incorporate conversational interviews with subject matter experts. This could be repeated in the classroom (in-person or hybrid) and create a collaborative learning environment where subject matter and the expert messengers behind the subject can "feel" more accessible to both students and teachers.

Yes, Alex, I completely agree- from the "behind the scenes" aspect of things, the ISC team has learned so much through this collaborative effort- it will only inform and enhance future efforts as we look to manage and produce more projects using cloud-based tools.  The distinct ease all presenters experienced, that "oh, that's it?" moment, when they realized the streamlined processes in place (thanks to the cloud-based tools in play), allowing them to  focus on telling their story, connecting with the audience, and conversing with the host was always a highlight for me.

Hi Perrin!  Thank you for your kind words and positive feedback. You too have been doing amazing work for years! So great to see ACRES thriving, and providing important strategies and resources to a new group of teachers who had to immediately pivot to virtual learning over this last year. I feel like your team was definitely ahead of the curve in providing meaningful and engaging online professional development programs and creating supportive, virtual networks for teachers (which was SO critical this past year!).

Sounds great, Josh. I agree, many potential synergies to explore! It would be really cool to connect and learn more about each other's work.

I will follow up through the Explorer at Large contact us page. 

Thanks for viewing our video and for your comments, Josh! I agree with Alex about learning more about the Explorer at Large program and potential parallels. Exploring and understanding potential synergies across different projects, platforms, and communities is something the ISC is always interested in.  Thanks again!

Thanks April for taking the time to view our video and provide such positive feedback! We will be offering the PD programs on a wider scale this summer- stay tuned, as there may be opportunities for your teachers to also connect with these virtual programs!