Research on Design League Badge Portfolios

Project overview

Design League Badge Portfolios was an effort to iteratively develop and research an innovative credentialing process that enables high-school-age youth who come from communities underrepresented in science, technology, engineering and math (STEM) fields to showcase their digital design talents. As part of this project, youth who participated in an after school program called Design League learn to use a digital platform to curate representations of their work in ways that are personally meaningful and that also address the expectations of higher education institutions using a badge system that was developed and endorsed by Parsons School of Design. Education Development Center conducted research on this initiative to build knowledge about how informal program developers and higher education faculty can create a system that establishes the value of out-of-school learning experiences in college and career pathways. This project is funded by the National Science Foundation’s Innovative Technology Experiences for Students and Teachers (ITEST) program (award #1614727).

Read more about this project at the following links:

• Programmatic Foundations for Badge Portfolios
• Endorsement Journey
• Mouse Create Technical Infrastructure
• Partnership Lessons Overview

What is Design League?

Mouse’s Design League is a design and technology program in which high school students create inventions to make a positive impact. Design League members develop creativity, problem solving, and collaboration skills, and build confidence in designing technology with purpose.

Over the course of the program, youth use human-centered design to brainstorm, prototype, and present assistive technologies designed to improve the lives of others. Design League enables high school students with diverse interests and backgrounds to gain applied design, technology, workplace, and leadership experience.

This program works with the “Design with Purpose” course on the Mouse Create learning platform. Through Design League, youth earn Digital Badges that document and validate the college-level skills they have developed. The program currently runs at three sites: Mouse’s Manhattan office, at DreamYard Project in the Bronx, NY, and Minneapolis, MN.

Project goals and objectives

The Design League Badge Portfolios project had three goals: (1) to create a digital platform that allows Design League participants to earn badges endorsed by Parsons School of Design and present them in portfolios of their design work; (2) to investigate whether assembling these portfolios of digital badges affects students’ motivation and persistence for pursuing digital design careers and (3) to standardize the credentialing method so that other institutional partnerships can use it to enable the use of badge portfolios in college admissions, thereby extending its utility and, possibly, establishing its validity.

Research and development design

We studied the research and development process and the impact of that process on youth across the three phases of work:

Phase 1: Program expansion, iterative development, formative testing, and documentation;

Phase 2: Implementation feasibility testing and revision; and

Phase 3: Pilot testing, revision, and dissemination.

We designed the research to answer the following questions.

1. Does the Design League Badge Portfolio process support participant engagement by increasing their STEM self-efficacy and interest in STEM careers?
2. What revisions have to be made to the existing systems to enhance participants’ capacity to create Badge Portfolios that are personally meaningful and present their digital design capabilities to higher education admissions officers?
3. What revisions have to be made to the existing Design League Badge Learning Portfolio systems to ensure the continuity of the process for recognizing and credentialing youth digital design ability across the partner organizations?

This research occurred at two sites in New York City. In Year 1, approximately 30 high school students participated in Design League at one site in Manhattan. In Year 2, approximately 40 students participated in the program across two sites (Manhattan and the Bronx). Finally, in Year 3, 50 students participated at the two sites.

Phase 1: Program expansion, iterative development, formative testing, and documentation

In Phase 1, the primary activities were: (1) to develop and test the technical infrastructure to enable the delivery of digital badges and construction of digital portfolios at multiple project sites and for multiple users, using assets from the existing Design League badge system; (2) to interview new and returning Design League youth to understand whether and how receiving badges and constructing portfolios affects their STEM identify (3) to interview Parsons admissions officers and educators at Mouse and DreamYard to understand their perceptions of the badges and e-portfolios as digital design credentials; and (4) to draft a Design League Portfolio resources to help DreamYard and future satellite organizations and higher education partners institute the badge and portfolio system.

In early fall 2016 Mouse began to design new portfolio-related functionality on its online platform, Mouse Create. The Create platform hosts Mouse’s online programming and enables educators and youth to implement activities, communicate with each other and comment on work across programs, and store youth projects.

Mouse’s design and web development team conducted over two dozen meetings and convened partners from Parsons and DreamYard Project. The team gathered feedback on design and functionality from those partners, as well as from current Design League participants via two focus groups responding to stakeholder feedback including information gleaned from student focus groups. The feedback led to new functional specifications and revised plans for a technical infrastructure that builds a more seamless user experience for students in sharing and curating their work and badges via digital portfolios. Several important outcomes (among others) of this work included:

●     An engaging, but simple, user experience to maximize communication between students and their peers, and students and educators across project work;

●     Robust export functionality to help high school students use their Design League in portfolios required by colleges and universities in applications

●     Clearer mapping between badge and portfolio functionality

Phase 2: Implementation feasibility testing and revision

In Phase 2, the primary activities were (1) to conduct a nine-month feasibility study of the badge portfolio system at two Design League sites with approximately 40 high school students in grades 9–12; (2) to continue to document collaboration activities among project partners and revise the Design League Badge Portfolio process; and (3) to recruit one additional higher education partner.

During this phase of the project, Mouse and Parsons began to revise programmatic elements of the endorsement agreement between the organizations to ensure that admissions partnership is more clearly built into the infrastructure of the partnership. Partners would be collaborating to have admissions train youth-facing staff in admissions procedures, and structured the university’s “portfolio mentor” role, filled by existing undergraduates, into a clear job description. The team also initiated meetings with Hostos Community College, based in the Bronx, with the intention to endorse Design League badges and to establish the endorsement as a step toward a deeper admissions partnership, with the potential for students to use badged portfolios toward college-level credits.

Phase 3: Pilot testing, revision, and dissemination

In Phase 3, the primary activities were to (1) investigate whether Design League Badge Portfolios show promise of increasing engagement in and capacity for STEM pursuits among underrepresented youth, (2) produce a toolkit and disseminate findings, (3) continue conversations with potential badge system partners. Some significant outcomes from Phase 3 included:

(1)  The partners authored and published a Design League Badge Portfolio Resource. Project partners leveraged existing relationships to identify a public digital host, Badge Wiki, “a knowledge repository for open badges.” The team authored five posts which can be found here: 1)Research on Design League Badge Portfolios 2) Partnership Lessons Overview, 3) Technical Infrastructure (for badge portfolios platform), 4) Foundations for Badge Portfolios Program 5) Endorsement Journey.

(2)  Hostos Community College agreed to commit to endorsing Design League badges and to establish the endorsement as a step toward deeper admissions partnership, with the potential for students to use badged portfolios toward college-level credit.

(3)  Parsons admissions and faculty expanded their support for the project. Parsons and the partners worked to draft a “portfolio review protocol” for the Parsons admissions office. Parsons also trained two undergraduate students to serve as portfolio mentors and to help Design League youth strengthen their admissions portfolios for Parsons in particular and other schools of design in general.

Analyses of surveys and interviews of Design League youth across the three years show many consistencies, but also some changes as the Badge Portfolio system evolved.

●     Across all years youth find their work in Design League to be personally meaningful and, for some, rewarding in ways that are different from experiences and achievements in school. Youth consistently discussed “collaboration,” “problem solving,” “being creative,” and “helping others” as important parts of their experiences in Design League. “Collaboration” and “acquiring technical skills” were the most frequently referenced reasons for how experiences in Design League might help youth in future pursuits.

●     Across all years Design League youth expressed interest in applying to college and indicated a wide range of possible academic interests, with most indicating that “Engineering,” “Computer Science,” “Design,” or “Architecture” were among the majors they were most likely to choose.

●     In the first year, Design League youth attached relatively little value to the digital badges they acquired. This was due in part to the fact that Mouse did not have a formal system for providing youth with feedback about the work associated with their badges. But most youth indicated that they did not ascribe much value to the badges because they are generally not recognized in higher education admissions (though two youth in Phase 1 shared their badge-related experiences and described their work in Design League in their college admissions applications). However, by Phase 3, some Design League youth describe the badges as an acknowledgment of mastery and a symbol of what they accomplished, while others felt that the work or experience that make up the badges is what’s important. Design League youth also view the badges as a currency that others (i.e., universities, organizations) can recognize and trust.

●     In Phase 1 Design League youth are unclear about what a badge portfolio is. By Phase 3 Design League youth understand what a portfolio is and what its purpose is, but still need additional guidance around curating portfolios that can be submitted for college applications.

●     In Phase 3, researchers surveyed Design League youth. Between pre- and post- surveys, youth reported an increased interest in pursuing STEM careers, with all seventeen at post- expressing interest in at least one career (versus fifteen at pre-), and nine in three STEM careers (3 at pre-). These results were statistically significant (.002) with a large effect size (.91). We also found that 100% of the participants surveyed planned on applying to college or had applied to college, and believed that Design League would help or did help them in their applications.

●     In Phase 3, researchers found that the badges provide Design League youth with a vocabulary to describe the work they did and the skills they gained throughout the process of their project. Design League youth spoke to the validation of their work that the badges represent, their personal growth, and the confidence to talk about the processes and their expertise.

●     In Phase 3, some Design League youth also were skeptical about the value of badges. Some participants in the program reported a lack of clarity on the practical uses of the digital badges as a credential outside of the Design League program.

Findings from the research were presented at numerous conferences and can be found in the article:

Digital Badges Forging Connections Between Informal and Higher Education, by Wendy Martin, Jaime Gutierrez and Maggie Muldoon in the fall 2020 issue of Afterschool Matters.

Informal Ed to Higher Ed Convening

An additional piece of work that was funded by NSF allowed for the project team to host a convening called Informal Ed to Higher Ed (IE2HE) at Carnegie Mellon University. The goal of the convening was to generate ideas and establish partnerships for projects that focus on making digital micro-credentials, particularly those issued through STEM experiences occurring outside of school, valued by higher education gatekeepers. We brought together stakeholders in the alternative credentialing enterprise who can actually make necessary changes happen. The convening included 22 participants representing four groups: faculty from two- and four-year institutions, providers of informal learning experiences for youth, micro-credential developers and issuers, and educators with experience implementing micro-credentials. Every participant was invited with the intention that they could generate ideas for moving the field forward by working together on innovative and interconnected projects. For more information, see the full IE2HE workshop report.