Research on Design League Badge Portfolios

From Badge Wiki
Jump to: navigation, search

Project overview

Design League Badge Portfolios: Investigating Digital Badges as Alternative Credentials to Broaden ICT and STEM Participation Among Underrepresented Youth, is a three-year ITEST Strategies research and development project. The main objective is to iteratively develop and study an innovative credentialing process called Design League Badge Portfolios. The process will provide underserved youth a technology-supported method for presenting their Interactive Communication Technology (ICT) achievements in an out-of-school program in ways that are personally meaningful and that address the expectations of higher education institutions. This provides underrepresented high school students with social and technological supports to turn their out-of-school learning experiences into viable and sustainable college and career pathways.  

Read more about this project at the following links:

What is Design League?

Mouse Design League is a design and technology program in which high school students create inventions to make a positive impact on the lives of others. Design League members develop creativity, problem solving, and collaboration skills, and build confidence in designing technology with purpose.

Over the course of the program, youth use human-centered design to brainstorm, prototype, and present assistive technology and projects designed to improves the lives of others. Design League comprises high school students with diverse interests and backgrounds, who gain applied design, technology, workplace, and leadership experience.

This program works with the “Design with Purpose” course on the Mouse Create learning platform. Through Design League, youth earn Digital Badges that document and validate the college-level skills they have developed. The program currently runs at three sites: Manhattan and the Bronx, NY, and Minneapolis, MN.

Project goals and objectives

The primary objective for this project is to iteratively test and refine the Design League Badge Portfolios process. There are four goals associated with this objective: (1) to identify the features that increase the badge system’s utility as a portfolio-making and credentialing process for underrepresented youth and higher education administrators; (2) to document differences in stakeholders’ perceptions of the meaning and value of badges and the badge portfolio; (3) to standardize the credentialing method so that other institutional partnerships can use it to enable the use of badges in college admissions, thereby extending its utility and, possibly, establishing its validity; and (4) to investigate whether and in what ways participation in assembling a portfolio of digital badges from informal learning experiences affects underrepresented students’ motivation and persistence for pursuing ICT careers, as understood through the lenses of self-efficacy and STEM identity (Britner & Pajares, 2006; Carlone & Johnson, 2007).

cover of a zine for recruitment of Design League youth

Research and development design

To ensure that the Design League Badge Portfolios project meets the needs of the youth, program developers, and higher education administrators it seeks to support—and to increase the likelihood that it can be implemented effectively and sustained over time—we have used a design-based implementation research (DBIR) approach (Penuel, Fishman, Cheng, & Sabelli, 2011). DBIR is an approach to organizing research and development that is collaborative, iterative, and grounded in systematic inquiry. We used this approach to study the implementation process and to assess the impact of that process on youth across the three phases of work, i.e., Phase 1: Program expansion, iterative development, formative testing, and documentation; Phase 2: Implementation feasibility testing and revision; and Phase 3: Pilot testing, revision, and dissemination.

We designed the research to answer the following questions.

RQ1: Does the Design League Badge Portfolio process support participant engagement by increasing their STEM and ICT self-efficacy and interest in STEM and ICT careers?

RQ2: What revisions have to be made to the existing systems to enhance participants’ capacity to create Badge Portfolios that are personally meaningful and present their ICT capabilities to higher education admissions officers?

RQ3: What revisions have to be made to the existing Design League Badge Learning Portfolio systems to ensure the continuity of the process for recognizing and credentialing youth ICT ability across the partner organizations?


To investigate RQ1, we are engaging in the following data collection activities:

  • Survey students using an instrument based on social cognitive career theory framework (Lent, Brown, & Hackett, 1994)—the STEM Career Interest Survey (STEM-CIS) (Kier, Blanchard, Osborne, & Albert, 2013)—at the beginning and end of Years 1 & 3.
  • Conduct separate interviews with 6–9 youth at each site (Mouse: Years 1–3, and DreamYard: Years 2 and 3) at three points during each project year; in Years 2 and 3, we will conduct the interviews about their ICT capabilities with youth using their badge portfolios as the focus of discussion.


To investigate Question 2, we are

  • conducting formative testing of the prototype badge portfolio system in Year 1;
  • integrating developmental evaluation findings from the project evaluator into formative research findings;
  • conducting separate interviews with 6–9 youth at each site, as described above;
  • conducting interviews with two Design League alumni about their higher education experiences; and
  • reviewing portfolios that participants create in Years 2 and 3, using the ICT ability criteria developed through the Year 1 iterative design process.


To investigate Question 3, we are

  • conducting separate interviews with key project stakeholders (program administrators, admissions officers, and educators) at three points during each project year to determine their understanding of the project goals and the meaningfulness of the badge portfolio process in their respective domains, their perception of challenges, and ideas for sustaining the process over time; and
  • integrating developmental evaluation findings from the project evaluator into research findings.


Sample and location. This research occurs at two sites in New York City. In Year 1, approximately 30 high school students participated in Design League at one site in Manhattan. In Year 2, approximately 40 students participated in the program across two sites (Manhattan and Bronx). Finally, in Year 3, 50 students are participating at the two sites.

Phase 1: Program expansion, iterative development, formative testing, and documentation

In Phase 1, the primary activities were: (1) to develop and test the technical infrastructure to enable the delivery of digital badges and construction of digital portfolios at multiple project sites and for multiple users, using assets from the existing Design League badge system; (2) to interview new and returning Design League youth to understand whether and how receiving badges and constructing portfolios affects their perceptions of themselves as “ICT or STEM people;” (3) to interview Parsons admissions officers and educators at Mouse and DreamYard to understand their perceptions of the badges and e-portfolios as ICT credentials; and (4) to draft a Design League Portfolio Toolkit to help DreamYard and future satellite organizations and higher education partners institute the badge and portfolio system.


There were a set of key findings from Phase 1. The findings are as follows:


In early fall 2016 Mouse began to design a new, portfolio-related functionality on its online platform, Mouse Create. The Create platform hosts Mouse’s online programming and enables educators and youth to implement activities, communicate with each other and comment on work across programs, and store youth projects.


Mouse’s design and web development team conducted over two dozen meetings and convened partners from Parsons and DreamYard Project. The team gathered feedback on design and functionality from those partners, as well as from current Design League participants via two focus groups (led by EDC) responding to stakeholder feedback including information gleaned from student focus groups. The feedback led to new functional specifications and revised plans for a technical infrastructure that builds a more seamless user experience for students in sharing and curating their work and badges via digital portfolios. Several important outcomes (among others) of this work include:

  • An engaging, but simple, user experience to maximize communication between students and their peers, and students and educators across project work;
  • Robust export functionality to help high school students use their work in Design League developed in the software required by schools and in their own portfolios as applicants
  • Clearer mapping between badge and portfolio functionality.


Based on interviews with Design League students over the course of Year 1, findings are as follows:

  • Surveys and interviews indicate that most youth find their work in Design League to be personally meaningful and, for some, rewarding in ways that are different from experience and achievements in school. Youth consistently discussed “collaboration,” “problem solving,” “being creative,” and “helping others” as important parts of their experiences in Design League. “Collaboration” and “acquiring technical skills” were the most frequently referenced reasons for how experiences in Design League might help youth in future pursuits.
  • All Design League youth expressed interest in applying to college and indicated a wide range of possible academic interests, with most indicating that “Engineering,” “Computer Science,” “Design,” or “Architecture” were among the majors they were most likely to choose.
  • Most Design League youth who we interviewed currently attach relatively little value to the digital badges they acquire. This is due in part to the fact that Mouse does not yet have a formal system for providing youth with feedback about the work associated with their badges (that is, feedback via the Mouse Create platform). But most youth in the two focus groups also indicated that they do not ascribe much value to the badges because they are generally not recognized in higher education admissions (though two youth have shared their badge-related experiences and described their work in Design League in their college admissions applications).
  • In the project’s first year, the response from higher education administrators to the possibility of partnering (or to the value of micro-credentials in general) has been mixed. Admissions personnel from Stamps, the art and design school at the University of Michigan, have expressed strong interest in the possibility of endorsing Design League badges. In contrast, program administrators from Stevens Institute of Technology are very interested in badges, but feel that central administrators will not see the need for accepting micro-credentials because the school is already successful in the areas of admissions (including among underrepresented youth), retention, and job placement.


a page from a zine depicting the work of Design League youth

Phase 2: Implementation feasibility testing and revision

In Phase 2, the primary activities were (1) to conduct a nine-month feasibility study of the badge portfolio system at two Design League sites with approximately 40 high school students in grades 9–12; (2) to continue to document collaboration activities among project partners and revise the Design League Badge Portfolio process; and (3) to recruit one additional higher education partner. To understand how youth understood and used the badge portfolios, we conducted a series of three interviews with 10 students at two sites, as well as pre- and post- surveys with 15 students at both sites. Surveys were adapted from a validated survey of youth STEM interest and self-efficacy (Sources of Science Self-Efficacy Scale (Zeldner, Britner, & Pajares, 2008).


There were some significant achievement during Year 2. They are as follows:


(1) Revisions to endorsement with Parsons: Mouse and Parsons began to revise programmatic elements of the endorsement agreement between the organizations to ensure that admissions partnership is more clearly built into the infrastructure of the partnership. Partners will be collaborating to have admissions train youth-facing staff in admissions procedures, and structured the university’s “portfolio mentor” role, filled by existing undergraduates, into a clear job description. (2) New endorsement with Hostos Community College: Bronx-based Hostos plans to commit to endorsing Design League badges and to establishing the endorsement as a step toward deeper admissions partnership, with the potential for students to use badged portfolios toward college-level credits.


There were also findings based on student interviews at both locations. They are as follows:


During the second of the three participant interviews, EDC asked a subset of 13 youth from both sites questions about the platform’s usability, and about their awareness of any portfolio functionality. Based on those interviews, EDC made the following recommendations to the development team:

  • Consider how Design League educators and the language of the Create platform can help youth to think about and explain the relationships between the skills they develop in DL and the specific ways in which those are applicable in different college majors, or careers. Or, to explain how a badge (or part of work submitted for a badge) is a valid indicator of a skill or achievement that is valued in a particular college major or job.
  • If Create is to be a part of a technical infrastructure that enables youth to bridge their experiences from DL to higher education institutions, then the partners should consider the following questions:
    • What aspects of portfolio development (e.g., reflection on work products and curation of badges or work related to badges), if any, should be supported by Create? What specific features (e.g., reflection prompts and the ability to use keywords to tag badges) will support portfolio development?
    • If Create will not support portfolio development, what other technical resources—if any—will the partners provide?


Phase 3: Pilot testing, revision, and dissemination

In Phase 3, EDC will investigate whether Design League Badge Portfolios show promise of increasing engagement in and capacity for ICT pursuits among underrepresented youth, and how the partners can ensure continuity of the process across the partner organizations. We will again conduct a series of three interviews with 12 youth participating in Design League, as well as pre- and post- surveys of all program participants.