MENU
Measurement & Evaluation Manager
Green Workforce Academy Measurement & Evaluation Fellow
Director of 3E Research & Evaluation
Tomatoes. Photo credit: Roland Dahwen
Learn more about the four principles we adopted to better guide the design and execution of our evaluation.
The practice of evaluation at Ecotrust has evolved rapidly in the past six years. While Ecotrust engaged in occasional evaluation work since the organization’s start, we officially founded a Measurement and Evaluation (M&E) team in 2018. The following year, we developed our first Theory of Change, internal to the organization, and refined it in 2020.
At the same time, we began gathering data on key results from programs and projects throughout the organization, compiling and aggregating these data to publish Indicators Reports covering the years 2019 and 2020. These reports provided a snapshot of Ecotrust’s programmatic work, yet left a lot of the story missing. They are heavily focused on scale, such as the number of people served or supported, rather than the depth or significance of our impact. During this period (2018-2021), our organization went through many changes: a founder transition, the development and implementation of our Racial Equity Action Plans, the development of a 5-year Strategic Plan, and a deep focus on weaving and centering equity in all our work.
Evaluation is a discipline that is often entrenched in dominant culture, to the detriment of equity, justice and liberation. At times, evaluation efforts focus on outputs rather than outcomes1. This leads to perennial pressure to expand scale at the expense of program quality or meaningful impact.
At other times, evaluation focuses on measuring outcomes defined by the evaluator or funder, rather than by the developers or beneficiaries of the program, and measured using methods that are unaccountable to program participants. These approaches can lead to inequitable decisions, such as onerous data collection burdens, inappropriate programmatic changes, or withdrawal of funding. Traditional data collection and analysis methods, for example, have historically burdened people of color, with the resulting data often used in harmful, invasive, or extractive ways.
With this history in mind, our Measurement & Evaluation team, led by Jamese Kwele, Vice President of Organizational and Food Systems Equity, and Olivia Rebanal, Chief Impact Officer, came together to rethink evaluation at Ecotrust on the basis of equitable evaluation practices as defined by the Equitable Evaluation Initiative. In 2022, we sought out consultants, Amani Austin of Austin Advocates With and Katie Winters of Insight for Action, with the goal of creating guiding principles to inform our work as evaluators.
1 (An output refers to the deliverable created by a program; an outcome refers to a change in the world, such as an increase in resources, power, or skills, created by that program)
Based on the core principles of the Equitable Evaluation Initiative, the Culturally Responsive Evaluation field, and the American Evaluation Association, and with guidance from our consultants, we settled on principles that aligned with our work.
As we collaborate across our region, our program work and evaluation methods must reflect the needs, worldviews, and priorities of frontline communities and communities that are Black, Indigenous, and people of color (BIPOC). These principles are aimed to do just that: be a lens through which we approach the design, execution, and meaning making process. These principles also serve as reminders for us to recognize our individual biases, the historical and cultural contexts of our evaluands2, the power dynamics that exist between evaluators and participants, and most importantly, remind us to center equity in all our evaluative activities.
2“Evaluand” is a term from the evaluation field that refers to the program, project, or initiative that is being evaluated.
Ecotrust’s Equitable Evaluation Principles
Four principles adopted by Ecotrust in 2023 will guide the design and execution of our evaluation.
The practice of evaluation should be in service of and contribute to advancing equitable and just outcomes in the communities in which we partner. Our evaluations will be culturally responsive, be developed collaboratively, and prioritize the needs and goals of participants.
This principle is the core of what we set out to do in rethinking our evaluation efforts. We seek to create evaluations that support fair and just outcomes within communities through elevating program successes and harvesting relevant feedback. We aspire to address systemic barriers that contribute to inequities, and work towards solutions that promote justice and resilience through evaluation.
We recognize the power dynamics between evaluators and participants. We co-create our evaluations with our partners to empower program participants by reflecting their visions of success and desired outcomes.
Evaluative work can often be subject to a power imbalance between the evaluators and the participants in programs being evaluated. Traditionally, evaluators have been viewed as holding power due to the decisions that come from an evaluative activity. We recognize that many, especially people of color, have experienced negative repercussions from such power dynamics. As such, our team chose this principle to nudge us in the direction of being transparent and creating participatory processes with affected parties3 in a given project. Sharing space for input and feedback in areas that are conventionally held by the evaluator (such as data collection and meaning making processes) allows us to provide nuanced results that better serve affected parties and communities.
3We use the term “affected parties” in place of the more common term, “stakeholders”, which has colonial origins. See additional resources at these links: 1, 2, 3
We understand that the practice of evaluation must be informed by historical and cultural context; it must seek to understand the effect of a strategy both on differently situated groups and the underlying systemic drivers of inequity.
Ecotrust projects focus on underserved communities, such as Black and Indigenous farmers and food producers who have historically faced challenges like accessing land and funding; Alaska coastal communities that live in rural areas with limited infrastructure; and Indigenous communities practicing agroforestry methods rooted in indigenous traditional knowledge. These examples of our work also reflect the historical and cultural context within which some of our evaluations are set. We chose this principle recognizing that our programmatic work seeks to address systemic drivers of inequity. Keeping this in mind can assist us in making important decisions in evaluation planning and implementation, such as determining the appropriate methods for data collection and dissemination of measurement tools to participants. It also helps us understand the broader context in which interventions do or do not work with differently situated communities.
Our team works to recognize our individual biases. As evaluators, we acknowledge how our background and other life experiences may impact the conduct of an evaluation. We work to understand how aspects of identity, including race, gender, class, indigeneity, ethnicity, culture, ability, language, and positional power, can influence an evaluation.
Everyone on our team approaches this work with their own personal life experiences, backgrounds and biases, which could potentially influence the outlook of an evaluation. This principle keeps us aware of how identity, including race, gender, class, indigeneity, ethnicity, culture, ability, language, and positional power, can influence our day-to-day decisions, including actions to conduct an evaluation. Recognizing our biases allows us to ensure the integrity of our work, as we actively work to address them in our evaluation activities.
We use our principles as grounding material as we engage with teams working on diverse subject matters, from indigenous agroforestry to food systems equity. We have found that sharing our principles at the start of any evaluation engagement with partners helps us set the tone for the role that we play as evaluators: we work together to co-create an evaluation, from design to implementation, centering affected communities and their goals.
In addition to selecting and refining the above principles, our team engaged with consultants from Austin Advocates With and Insight for Action for several other activities, including workshops to identify white supremacy culture characteristics in evaluation and how to counter them, and applying our principles in an evaluation process, from design to meaning making. We then chose a pilot project to apply our newly established processes. This project was the Viviane Barnett fellowship (VBF), a cohort-based program designed for aspiring, emerging, and experienced BIPOC food system leaders. We held design sessions with alumni, program partners, and facilitators, who informed the evaluation design for the next cohort of VBF. We include more detail on our process in our brief, “Equitable Evaluation at Ecotrust.”
Watch the video below for a free webinar that we held on April 18, 2024 to discuss the process of how Ecotrust adopted and equitable evaluation approach. Our Equitable Evaluation Toolkit is also available online.
For additional information or questions, contact Denise Chin (dchin@ecotrust.org) or Noah Enelow (nenelow@ecotrust.org).
Project page
Six tools to help nonprofit evaluators create a more equitable evaluation process
Project page
Six tools to help nonprofit evaluators create a more equitable evaluation process
Report
This report presents Ecotrust’s second annual organization-wide Indicators, organized by six top-line categories: Acres, Audiences, Businesses, Inclusion/Equity, Jobs, and Partners/Trainees. The data were collected in 2021 and cover the calendar year 2020.
