Why a 5-Year Impact Evaluation?
The public interest technology space has seen an increase in Fellowship programs over the past five years, all focused on helping the public and civil society sectors better understand the impacts of technology on society, and build policies that put people first – in 2015, TechCongress launched to place technologists in the halls of Congress as advisors to policy makers; in 2016 the Ford Foundation started embedding technologists into their grant-making teams; and in 2017 the Media Democracy Fund paired graduate or PhD students with a tech background and DC-based public interest technology policy organizations. We’ve even piloted a few fellows models ourselves with our Secure Usable Design Fellows and our Underexposed Residency.
The Mozilla Fellowships and Awards program has been a leader in the tech and civil society sector since their first fellowships in 2014. We have partnered with Mozilla before, on projects such as Mozilla Rally and YouTube Regrets, and our Executive Director, Georgia Bullen, was a mentor to a 2016 Mozilla Fellow during her Open Technology Institute days. We know how important this program is to the space, and jumped at the chance to support them in reflecting on and maturing their funding arm. Partnering with funders to leverage human-centered design research practices to better understand socio-technical ecosystems is critical work to help further the internet health field. Previously, we’ve worked with funders like Luminate on the ‘On Trust and Transparency’ report, published in 2019, and various projects for the Open Technology Fund.
The evaluation provides an opportunity for the Mozilla Foundation to reflect on what works and what doesn’t in their own programs – and, in the spirit of openness and transparency, it poses useful learnings for the wider digital nonprofit funding ecosystem and internet freedom community.
Methodology
Our team, using human-centered design research methodology, conducted three surveys and held 47 interviews with program alumni, current fellows and awardees, funders, Mozilla Foundation and Corporation staff, and other key stakeholders. We hosted and participated in several workshops with staff and community members, designed to engage iteratively with early findings and challenges. Building on the foundation of internal Mozilla documentation and data, we qualitatively and quantitatively analyzed all of the data into the final evaluation – outlining five years of the programs’ impacts, strengths, and challenges. We also partnered with our friends at Objectively to create engaging visualizations of program data. (See the Evaluation Executive Summary and Impact Narrative sections for their great work.) The evaluation is supported by a program origin story, an impact narrative, and an interview with Mozilla Foundation management, written by Ayana Byrd and Kenrya Rankin. You can find the full evaluation both on Mozilla’s site and in PDF form.
Reflections for the Ecosystem
Check out the full evaluation for Insights and Findings around 9 key themes - ranging from External Perceptions to Program Structure to DEI – as well as 12 concluding opportunities for the Mozilla Foundation, many of which might be useful for the wider digital rights and internet freedom funding landscape.
The following ecosystem findings were echoed across the interviews and survey results. We identified these themes across our conversations with F&A program alumni, recipients, and staff, as well additional funders and broader community stakeholders. We collected these here as a summary of some reflections the report offers the public interest technology funding ecosystem at large.
1. Funding mechanisms – as exemplified by the success and growth of Mozilla’s Fellowships and Awards – are essential within the digital rights ecosystem.
- In the case of Mozilla, “the impacts of the fellowship program have rippled throughout the internet health movement and ecosystem through narrative shifts and knowledge production.”
- The Mozilla F&A programs are career catalysts for many technologists looking for an entrance into civil society, and for those in civil society looking to expand into digital rights.
- Funding like this opens opportunities for individuals and organizations outside of the for-profit “Big Tech” industry which dominates much of the space.
- Cohorts in fellows and awards create community camaraderie that lasts long beyond the funding period, creating lasting networks in the space.
- Still, there are few follow-on funding opportunities to support the experts who come out of these programs. For-profit companies pay and recruit more than civil society. There is a need to grow the nonprofit sector and prioritize funding so that experts continue working for the public interest – such as advocacy, open source, and digital equity work.
2. Measuring impact is hard.
- Every funder struggles to define impact – it’s the nature of the work. It is especially challenging in sectors that are relatively new with significant adversaries – such as the internet freedom community.
- What we can measure are individual stories and narrative shifts. This can be achieved through mapping and networking models, but these strategies must be built-in from the onset of the funding process. This means investing in responsible data collection, but also human connections – equitable opportunities to convene, share work, and network with each other.
- In the report, we propose that Mozilla “leverage the relationships it has built over the last five+ years of programming to develop a network-based model of impact measurement for the ecosystem.” This would not only benefit Mozilla’s modeling and understanding of its impact, but strengthen the growing field of practice with data and open processes.
- Mozilla is exemplifying impact-in-action by publishing the evaluation on a new Learning hub found on their site. Working on change in the open – as an iterative process –- allows everyone to learn and build together. This is a process-oriented approach to impact versus a traditional outcomes-approach - which is a powerful opportunity to build on the evaluation itself.
- Other impact outputs could include: “an in-depth maintained public-facing project database, public speaking and writing about the program design and outputs, funder advocacy around the program design strategies and what they should learn from them, and a “ripple map” that tracks the programs’ influence on individuals and the network” in a consentful and privacy conscious way.
3. Community is at the core.
- One of the chief findings in the evaluation is the importance of community for personal and professional growth. Fellows and awardees widely felt that the cohort experience, opportunities to attend and present at convenings such as Mozilla Festival (MozFest), networking, and supportive relationships with the Mozilla staff were among the most significant benefits of their time as funding recipients.
- On the other hand, this is also something that can be better fostered through the conscious stewardship and building of a network. We proposed that Mozilla “invest in developing, maintaining and supporting a network, inclusive of alumni and key partners/stakeholders, to help fuel the movement and support past participants.” Mozilla has already made big strides in this direction – resourcing new staff to build an alumni network as well as other community and movement building measures. (Check out their blog post here for more info!)
- Community networks are spaces for knowledge and resource sharing, which will further strengthen the impact of investments and build this field of practice. “Where previously the idea of an alumni network has had a specific focus on fellows, there is an opportunity for the network to be a broad group of stakeholders who can help inform programs, strategy, and overall priorities and give insights on the space as a whole … Mozilla can leverage all of these partnerships to run a more sustainably engaged, global program. This means the network can be more than just a place to share events and job opportunities, but can be the grounding framework for the movement as a whole.”
Funders looking to support public interest technology might find useful roadmaps in this in-depth evaluation of Mozilla Fellowships and Awards programs. Funding experts and promoting knowledge sharing community-wide, rather than simply uplifting individual profiles, expands and strengthens the movement at large. Investing in networks means more supported expertise in this sector – toward the goal building more compelling and sustainable avenues beyond Big Tech. The work isn’t just about the outputs we produce, but the communities and ecosystems and resources that we build. Reflections like this evaluation help the broader community design programs and impact measurement tools that support us all.