Project: When Data Science Goes Bad

Contribute email twitter instagram facebook

(Also available in Pyret)

Students investigate four types of threats to validity by pretending to be “bad data scientists” who fail to consider the impact of selection bias, bias in the study design, poor choice of summary data, and confounding variables. This project supports the learning goals of our lesson on Threats to Validity.

Lesson Goals

Students will be able to…

consider how threats to validity can emerge when research is conducted
gain awareness of how data can be misconstrued

Student-facing Lesson Goals

Let’s be "bad data scientists"!

Materials

Preparation

Teachers are encouraged to edit and adapt the student-facing rubric for their unique classroom context and distribute it to help students understand the scope of the project - and your expectations - at the outset.
Decide what sort of final product you would like students to create. A slide deck template is included in the lesson, but any format will work.
Students will work with a partner for this project. You may assign partners, or allow students to choose.
Sharing is an important piece of this project. Think about if you have time for each group of students to share their work, or if you would prefer to use a different format.

Key Points for the Facilitator

This project is designed to span approximately 2-3 class periods.

🔗Threats to Validity flexible

Overview

Students develop a faulty research plan on a question of their choosing to demonstrate their understanding of four unique threats to validity.

Launch

Congratulations! You’re a data scientist who has been granted funds to conduct research on a topic of your choosing! There’s one complication…

You’re a BAD data scientist!

For this project, you and a partner will develop a statistical question that you would be interested in exploring and then develop a research plan that will result in data that isn’t valid.

With your partner, come up with a statistical question that you want to pretend to study together.
Record your question on Q1 of Our Research Plan.
Then review the Major Threats to Validitiy to ensure that you both understand what each one entails.

As a class, invite students to share their questions and begin to consider how they might incorporate the threats to validity. You’ll want to ensure that all students have developed statistical questions before moving onto the next portion of the project.

Complete the rest of Our Research Plan with your partner, developing a brief research plan and describing in detail how you plan to misrepresent your data.

Investigate

It’s time to consider how the threats will impact your analysis!

Turn to Analysis of Research Plan Predicted Outcomes.
For each threat listed:
- Explain how the validity of your conclusions will be impacted.
- Identify what a good data scientist could do to minimize each of the threats.

Finally, students create a presentation highlighting the (very bad!) choices that they have made.

If you’d like, you may opt to have students conduct a peer review of the pages they have completed before proceeding to their final draft. The Threats to Validity Rubric is useful for this.

Create a presentation of your plan! You can use this Threats to Validity Project Template to get started - but feel free to personalize it and make it your own.
When you’re done, refer to the Threats to Validity Rubric to ensure you’ve met the project objectives and included all of the required components.

Synthesize

Did you notice any new threats to validity we should be on the lookout for?
Can you think of any other reasons why we might mistrust data?

A fun way to conclude this project is to invite each pair to present their flawed research plan to the rest of the class. If using the Threats to Validity Project Template, each group will present only their first three slides. Then, invite the rest of the class to discuss and critique the (hopefully very flawed) research plan.
Have fun with the sharing or this project - and encourage students to be silly as well! Maybe you want to encourage costumes! Maybe the class is a panel of very serious and critical scholars judging the work of each group.
If you can’t carve out time for every group to give a presentation but want to give students the opportunity to practice identifying threats, you could pair up groups. Each group would present their research plan, and then create the other group’s research plan.

These materials were developed partly through support of the National Science Foundation, (awards 1042210, 1535276, 1648684, 1738598, 2031479, and 1501927). Bootstrap by the Bootstrap Community is licensed under a Creative Commons 4.0 Unported License. This license does not grant permission to run training or professional development. Offering training or professional development with materials substantially derived from Bootstrap must be approved in writing by a Bootstrap Director. Permissions beyond the scope of this license, such as to run training, may be available by contacting contact@BootstrapWorld.org.