By Lehn M. Benjamin & David C. Campbell
In 2003, the Greater Ithaca Activity Center (GIAC), located
in upstate New York, received a letter from a funder
that expressed concern about the outcome measures that
GIAC had presented. The funder suggested that GIAC—a multi-purpose community center that has served culturally
diverse young people and families since 1972—could achieve
greater impact by narrowing the scope of its programs and services.
About this interaction between GIAC and the funder, one observer
later said: “Programs took center stage; the relationship between
people was nowhere to be found.” Soon afterward, GIAC staff members
began working with a facilitator on a yearlong effort to articulate
the difference that their organization was making in its community.
Together, they and the facilitator concluded that the funder’s
preferred outcome measurement model failed to capture what was
most essential about the work that GIAC had been doing. To account
for that work adequately, GIAC would need to adopt an evaluation
framework that treats relationship-building not merely as instrumental
to programmatic outcomes but as valuable in its own right.1
Today, more than a decade later, nonprofit organizations still
struggle to represent their work in the context of prevailing outcome
measurement models. In a 2010 survey of 415 US-based nonprofits
conducted by Johns Hopkins University, 80 percent of nonprofit
leaders called for the development of better tools to measure qualitative
impact.2 Underlying the concern of these leaders is a sense
that existing evaluation models overlook something essential about
how nonprofits serve individuals and communities. “I think [outcome
measurement] is a worthwhile thing,” the executive director
of a mental-health counseling service told the authors of a 2009
study. “I am not complaining about the time it takes; it is a tool that
needs to be continuously refined and made better, but it needs to
get closer to the actual activity.”3
Our research on nonprofit organizations has sought to develop
a better understanding of this “actual activity.” This research has
led us to conclude that standard outcome measurement models are
limited by their assumption that programs, and only programs, are
what lead to participant outcomes. (In this article, we apply the
term “participant” to every type of person who potentially benefits
from interaction with nonprofit staff members. Depending on the
institutional setting, staff members might use other terms to refer
to a participant—“client,” “resident,” “constituent,” “patient,”
“beneficiary,” and so forth.)
Anyone who has ever set foot in a nonprofit organization understands
that the work done by staff members extends beyond the
simple task of program delivery. Nonprofit workers spend time getting
to know the people they serve. They take time to adapt services
to meet particular needs. They work with participants to identify
outcomes that will be relevant and meaningful to those participants.
They connect participants to resources that are available outside
their own organization. Not only is all of this work instrumental to
achieving program outcomes, but it can also lead to beneficial outcomes
that program-centric models are ill equipped to anticipate.
To understand nonprofit performance fully, we need to broaden
the lens through which we view the work that staff members do to
achieve outcomes for participants. Programs and program outcomes
matter a great deal, to be sure. But an outcome measurement model
that relies exclusively on “the program” as its unit of analysis will
miss a good portion of the work that staff members do. Not surprisingly,
many nonprofit staff members come to view outcome measurement
as a burdensome funding requirement, rather than as a
practical tool to help improve their work.
In making this argument, we do not wish to criticize outcome
evaluation in general. The field of evaluation is incredibly diverse,
and it includes many approaches that do recognize factors other
than programmatic intervention. Yet most nonprofits don’t have
the resources to employ their own evaluator, and so they depend on
popular outcome measurement guides or on whatever model their
funders happen to require. And these default models, in the main,
continue to focus pivotally on program activity.
The Legacy of Program Evaluation
Why did the standard outcome measurement framework for nonprofits
come to rely on the program as its primary unit of analysis?
Evaluation as an identifiable field of practice emerged during the
1960s as part of the US War on Poverty.4 After the federal government
launched a variety of social policy innovations, public officials
and other funders recruited social scientists to address the issue of
effectiveness: Were these policy interventions, in fact, producing the
intended outcomes? Initially, the main concern of social scientists
was to determine whether there was a causal relationship between
a given policy and a given measurable impact. Over time, however,
evaluators began to develop methods that would go beyond what was
essentially a “black box” approach. Their goal was to understand not
just whether an intervention produced a desired effect, but how: What
were the intervening causal mechanisms that resulted in that outcome?
Evaluators fastened on the programmatic intervention as a way
to organize their analysis of those causal mechanisms. They also
adopted the concept of “program theory,” which posited that every
intervention rests on a set of assumptions—sometimes explicit but
often implicit—about why a given program would lead to a given outcome.
By specifying a program theory for each intervention, evaluators
hoped both to identify causal mechanisms and to understand
why the intervention did or did not succeed. Also during this period,
evaluators drew on insight from the field of policy implementation
studies. A principal theme in that field involves the way that implementer
discretion can undermine the pursuit of policy objectives.
As a consequence, evaluators have tended to regard any deviation
from the design of a program as a threat to its effectiveness.
The programmatic legacy is evident in the outcome measurement
models that many funders currently require from their nonprofit
grantees. That legacy, in our view, has several problematic
consequences. First, these models portray staff members mainly
as program implementers, thereby overlooking the way that they
partner with their target population in pursuing change. Second,
these models do not account for staff member discretion in tailoring
programs to meet participant needs or to match the context of
participants’ lives. Third, these models treat participants as targets
of intervention and therefore ignore their agency in the change. And
fourth, these models neglect the role that outside organizations,
programs, and services play in achieving intended results.
A number of evaluation scholars have challenged the prevailing
focus on program implementation. Michael Patton argues that in
certain complex settings, the framing of desired outcomes needs to
emerge not from predetermined program goals, but from iterative
dialogue and deliberation.5Madine VanderPlaat points out that the
instrumental nature of program-centric evaluation fails to accommodate
empowerment-oriented initiatives that treat active engagement
and mutual support among participants as critical drivers of change.6
And Huey-Tsyh Chen suggests that a program theory should consider
not only the hypothetical causal mechanisms of a program, but also
the actions that staff members undertake to run the program and
the contributions that partner agencies and participants themselves
make to achieve outcomes. Together, these elements make up not just
a program theory, but “an action theory,” as Chen calls it.7
The next step for funders and nonprofit leaders, we believe,
is to develop a more complete understanding of frontline work—of the “actual activity” that staff members do, apart from program
activity. Research across a range of disciplines emphasizes the importance
of this work. The social policy scholars Lisbeth B. Schorr and Frank Farrow, for instance, argue that funders and nonprofit
leaders should seek out evidence that would enable them to learn
from “interventions that are complex, interactive, and relationship-based;
that can be adapted to a variety of cultures and populations
as well as new and changing contexts; and [that] require significant
frontline flexibility and sensitivity.”8 So far, however, frontline work
has received little attention in the debate over how best to measure
nonprofit effectiveness.
The time has come to enlarge the frame of reference that we use
in assessing nonprofit performance. By looking closely at frontline
work, we can create a more accurate picture of the efforts that enable
staff members and participants to achieve transformational change.
Varieties of Frontline Work
In our research, we have found that nonprofit staff members commonly
engage in four types of frontline work. (All of the examples
and quotations used below come from interviews that we have conducted
with nonprofit professionals as part of our ongoing research.)
Relational work | Staff members build relationships with participants
in order to understand their problems and aspirations.
Relationship-building can result in significant outcomes (self-confidence,
a sense of efficacy) that are independent of program activity.
Relational work can also help anchor and organize a program by
ensuring that it meets a real need and that it matches participants’
existing capacities or goals. Indeed, research in mental health, elder
care, youth development, and other fields shows that frontline relationships
are often more central to outcomes than are program
techniques. The same principle applies to nonprofits that engage
in grassroots organizing—an activity in which success, almost by
definition, pivots on the quality of relationships.
In interviews, frontline staff members refer frequently to the role
of relational work in their daily practice. In fact, this kind of work
often must occur before a staff member and a participant can do
any other work together. Relating to participants requires skill and
discernment, and in many cases it requires a staff member to cross
certain formal boundaries. “There is a lot of discussion in the legal
field: … ‘How close do you allow your clients to become?’ Some lawyers
never talk about their personal lives with clients, ever, period.
I don’t take that tack,” said one immigration attorney.
To build enduring relationships, frontline professionals listen
carefully and avoid falling back on pat responses. In a nonprofit that
provides services to refugees and immigrants, one staff member developed
a technique in which she would ask clients to write a list of
their goals; at the same time, she would write down what she understood
to be their goals, and then they would compare what each of
them had written. When clients saw that her list aligned with their
own, they felt that she had truly listened to them. That practice
helped to build trust, and it laid a foundation for other types of work.
Adjustment work | Staff members often adjust their approach in
response to a participant’s goals, needs, and immediate circumstances.
“When you work with trauma, you have to be able to tolerate
ambivalence,” said a frontline professional who helps people who
suffer from PTSD. This professional described a fairly typical client
interaction: “This guy has come back and left, he has come back
and left, he wants to explore it [his issue] and he does not want to
explore
it.” The best solution in that case, according to this frontline
worker, involves “letting him [the client] go at it at his own pace.”
Adjustment work also reflects an understanding that change is
not linear. What might seem like a step backward can be, in fact, a
critical step in a long-term transformation process. Frontline workers
must be able to judge whether a particular “step backward” will
promote a desired transformation or whether it will undermine a
participant’s growing self-confidence. In interviews, many frontline
workers shared vignettes in which adjustment follows a trial-and-error
pattern. One frontline staff member, who helps residents in a
homeless shelter in their job search efforts, explained how he debriefs
clients after they go on job interviews: “I say to them, ‘You know
what? A ‘no’ to you is more power. Don’t take it as a rejection. Just
get confident. You didn’t lose anything; they’re losing you, because
you’re a good worker.’ So you have to build the self-esteem, and
get them back to another interview.” Through this kind of debriefing,
the staff member and the client can experiment with different
strategies for matching the client’s aptitudes with employer needs.
Adjustment work is not a matter of “anything goes.” Nor is it
always effective. But it’s real, and it’s important, and it requires
experience and judgment. The frequency with which the frontline
professionals whom we have interviewed mention adjustment work
is striking, and it calls into question the fixed nature of program-centric
outcome measurement models.
Codetermination work | Staff members work with participants to
sort through various possible courses of action. Codetermination
work starts with the conviction that participants have the right to
claim ownership over their path toward transformation. “When all
is said and done, it’s really up to them,” a frontline worker in a faith-related
employment program said. “We can support them, but unless
they make a decision to change, their lives are not going to improve.”
Codetermination work can be highly demanding. It requires complicated
situational judgments about how to reinforce a participant’s
sense of agency. The impact of such work can be quite subtle—as
much a shift in perspective as a change in approach. Consider this
interchange between an employment counselor and his client:
“These things I’m asking you, they’re not for me. You don’t come
dressed up like this to see me, to please me. It’s for you. If you look
good, you feel better. How do you feel when you finish showering,
and you [dress] up, and you come to see me? How do you feel?”
“Well, everybody gives me a compliment.”
“That’s what I’m talking about. You do it for you.”
“Oh, I thought I was doing it for you.”
“Oh, no, you don’t do it for me.”
Frontline work of this kind involves using techniques to ensure
that participants embrace their ownership of the change process.
One immigration attorney, for example, described how she provides
clients with her supervisor’s contact details. Those clients reflexively
defer to her—she is a lawyer from the dominant racial group—but
she tells them that they are hiring her and so they can also fire her.
Linking work | Staff members connect participants with resources
from other parts of their own agency, from other nonprofit organizations,
and from local community groups. Individuals and families
often have multiple needs that go beyond what any one organization
or any one program can meet. In addition, their needs may change
over time. Frontline staff members therefore achieve success not
just by running their own programs, but also by serving as a portal
to other programs and services. When this kind of linking work is
absent, the impact of a given program is less likely to be sustainable.
Consider the example of employment programs, which often
require frontline workers to complement job training activities
with referrals to organizations that provide various products and
services: work clothing, dental work, tattoo removal, transitional
housing, and so forth. Success in this area comes from being able
to piece together disparate resources. A frontline staff member who
works with people who have ADHD, dyslexia, and other mental or
emotional disabilities told us that his job covers “whatever it is that
a client needs to get back into the workforce and become [a] productive
[member] of society.”
In interviews, staff members often refer to the amount of time
that they spend on linking activities—getting to know colleagues at
agencies that offer related programs, attending partnership meetings,
working out the terms of complex collaborative arrangements.
A common lament of these professionals is that they generally don’t
get credit for that time. “I do a lot of finding resources for clients: ‘You
need to go here. You might find food here.’ … I do a lot of case management.
I don’t always document it,” said a frontline professional
who works with people who suffer from depression, among other
problems. “Talking to them about what depressed them is not effective
while their basic needs are not met,” this staff member noted.
Nonetheless, a willingness to perform linking work can matter
a great deal for participant outcomes. One frontline professional,
for example, makes a point of joining her clients when they visit
a vocational counselor. “Sometimes it’s hard for members and clients
to articulate what they think their barriers are,” she said. “So
if I can articulate what I see, … it gives the vocational counselor a
better picture of how best to serve [each client].” In another case,
this staff member noted, a colleague didn’t engage in that kind of
linking work with a client. As a result, important information about
the client got lost in the hand-off to another agency. “This process
should’ve taken, at the most, two months. It’s probably going to
take a year,” the staff member said.
In sum, when we judge the effectiveness of an organization solely
by its programmatic outcomes, we risk misunderstanding the factors
that determine the success or failure of that organization. When we
ignore the relational work that staff members do, we neglect the way that high-quality relationships may compensate for a poorly designed
or under-resourced program. When we do not consider the adjustment
work that staff members do, we lose access to valuable information
about the potential mismatch between the logic of a program and
participants’ actual path to transformation. When we do not account
for codetermination work, we fail to see how programmatic “failure”
may reflect real progress for clients whose choices do not conform to
program logic. And when we do not pay attention to the linking work
that staff members do, we underestimate how dependent client
outcomes
are on resources that lie outside any one organization.
Principles of Outcome Measurement
To overcome the limits of models that focus solely on program
performance, we need to develop models that encompass all of the
work that takes place between frontline staff members and participants.
Our research has led us to formulate four principles of a more
comprehensive outcome measurement framework—principles that
reflect the various forms of frontline work.
Honor relationships | An outcome measurement framework should
take into account the pivotal role that relational work plays in the
transformation process. Healthy and mutually supportive relationships
can be valuable outcomes in their own right, especially when
participants have suffered from a lack of meaningful relationships
in their lives. Equally important, achieving programmatic outcomes
often depends on an organization’s ability to enable high-quality relationships.
Such relationships include those that connect frontline staff
members with participants, those that connect participants with one
another, and those that connect participants with members of their
social network. A critical component of these relationship-building
efforts is the attitude that staff members bring to their work. It has
long been known that in many human service activities—teaching,
therapy, and social work, for example—frontline workers’ attitudes
toward participants are a significant driver of change.9
Allow variation | The ebb and flow of participants’ lives, the necessarily
improvisational nature of frontline adjustment work, and the
desire to facilitate creative problem-solving all point to a need for
outcome measurement models that give space for variation. Take the
example of substance-abuse treatment. Although abstinence might
be the desired long-term outcome for people with a substance-abuse
problem, the short-term outcomes that signal progress toward that
goal can vary. Outcome measurement models that allow for this kind
of variability enable treatment professionals to discover patterns of
individual change and to adjust their services accordingly. In some
cases, long-term outcomes will vary as well. In drug rehabilitation,
for instance, “harm reduction” can serve as an alternative goal for
participants who are unable or unwilling to practice abstinence.
Respect agency | Too often, standard outcome measurement
models treat clients as if they were merely inputs to programs. Instead
of treating participants as objects of intervention, however,
an outcome evaluation framework should acknowledge the codetermination
work that they pursue in collaboration with staff
members. As we have noted, supporting participants in their progress
toward greater self-determination is a critical yet often invisible
aspect of frontline work. In some nonprofit organizations, participants
actually exercise significant agency not only in the design and
delivery of the services that they receive, but also in organizational governance. Some existing outcome measurement models do give
scope to participants’ “cooperation” with, or “acceptance” of, an
intervention. But outcome measurement models should account
for the self-determination of participants as well.
Support collaboration | In a traditional outcome measurement model,
nonprofit leaders generally have no way to assess the value of the linking
work that staff members perform in order to connect participants
with resources that exist outside the boundaries of a given program.
Nonprofit workers, as we have seen, frequently must collaborate with
colleagues both inside and outside their organization. What’s more,
the linking work conducted by frontline staff members mirrors the
work that participants themselves must do as they piece together various
services to address a particular problem. An outcome measurement
framework should focus on how a program fits into the life of a
participant—not how the participant fits into the program. By recognizing
that a program is one factor among many, we can better understand
how services work (or fail to work) within a larger ecosystem.
If we fail to situate programmatic activity within the context of
frontline work, we cannot reliably measure nonprofit effectiveness. We
have identified four reasons why this is so. First, non-programmatic
work is instrumental in achieving program outcomes. By motivating
participants or adjusting each program to particular needs, staff members
help ensure that a program will actually work for those who take
part in it. Second, non-programmatic work involves important outcomes
that are separate from program outcomes. For participants,
these outcomes might include a decrease in feelings of isolation or
an increase in feelings of efficacy or self-confidence. Third, non-programmatic
work is a necessary prerequisite to defining appropriate
outcomes. If frontline workers don’t listen carefully to the issues that
participants bring to them, and if they don’t systematically explore
how various resources might help to confront those issues, then they
cannot properly determine the most suitable outcomes for participants.
Fourth, non-programmatic work enables staff members to appreciate
the kinds of micro-transformations that the blunt instrument
of a program-based measurement model will often fail to capture. For
one client, a job secured might be just another step in a long journey;
for another, it might be a life-changing moment.
Emerging Alternatives
In our research on the evaluation of nonprofit effectiveness, we
have encountered several organizations that have undertaken innovative
efforts to capture the role that frontline work plays in
driving participant outcomes. Here are a few examples that point
the way toward an outcome measurement model that honors the
full breadth of the work that contributes to transformative change
in individuals and communities.
Measuring relational work | Roca, an organization based in Chelsea,
Mass., that helps young people move out of violence and poverty,
recognizes the significance of relational work. The organization
works with 17-to 24-year-old men who are involved in the criminal
justice system, who have no employment history, and who face a high
likelihood of re-incarceration. The leaders of Roca put a premium on
the transformational relationships than can emerge between young
people and frontline staff members, and for that reason they have
developed a performance measurement system that tracks important
relationship milestones. Such relationships, they have found, are what enable staff members to know when and how to challenge a young
client as a means of supporting behavior change. Staff members at
Roca also recognize that relapse and even “failure” are a part of the
transformational process, and they focus on addressing problems
of that kind in the context of a solid relationship.10
Accounting for adjustment work | The Full Frame Initiative (FFI),
based in Greenfield, Mass., partners with government agencies,
philanthropic organizations, and human service nonprofits that are
working to break both the cycle of poverty and the cycle of violence.
FFI has identified a set of principles and practices that it calls the
Full Frame Approach. This model, which FFI leaders believe is characteristic
of effective organizations that work with people who have
multiple challenges, raises several problems for traditional evaluation
methods. For one thing, it is highly tailored to the specific situation
and trajectory of each participant. For another, it allows program
activity to evolve over time and within a community context. For
still another, it avoids the tendency to see participants as “clients in
a program,” and instead treats each program as part of an ecosystem
of resources. FFI has created individual assessment tools, and
it is developing new methods that will enable other organizations
to track and evaluate their work. The core FFI method focuses on
tracking changes in five crucial aspects of life (FFI calls them the
Five Domains of Wellbeing): social connectedness, safety, stability,
mastery, and meaningful access to relevant mainstream resources.11
Recognizing codetermination work | Liberty Hill Foundation supports
nonprofit organizations in Los Angeles that promote grassroots
social change. Its grantees work simultaneously to build leadership
capacity within marginalized communities and to change the policies
and practices that contribute to that marginalization. In partnership
with its grantees (and with help from an outside consultant),
the foundation has developed an outcome measurement model that
accounts for the codetermination work that grantee organizations
pursue with their target populations. The goal is to overcome certain
limitations of the traditional outcome measurement framework.
First, that framework tends not to recognize the way that members
of a target population help define and carry out the work that
grantee organizations do. Second, the traditional focus on short-term
program outcomes encourages organizations to work mainly
with more-experienced members of a target population—a practice
that undermines a core mission of these organizations, which is to
cultivate new leaders.12 The Liberty Hill model, therefore, considers
two kinds of outcomes: one that focuses on external policy change,
and one that emphasizes leadership development.
Acknowledging linking work | The Women’s Funding Network, a
philanthropic collaborative based in San Francisco, supports initiatives
to meet the needs of women and girls. Over the past several
years, the organization has developed an evaluative framework called
Making the Case. The purpose of the framework is to help network
members and their grantees to design, implement, and evaluate social
change initiatives. Currently, 40 funders around the world are using
Making the Case with their grantees. The framework uses five indicators
of social change to measure organizational effectiveness: “a shift
in definition” (how people frame issues); “a shift in behavior” (how
people act within a community); “a shift in engagement” (how people
pursue change); “a shift in policy”; and “maintaining past gains.” A
notable feature of Making the Case is that it takes into account the
importance of what we call linking work. Within the “shift in engagement”
category, for example, Making the Case tracks alliance building.
Here the framework calls on grantees to show whether they have
engaged in activities such as forming coalitions, bringing public- and
private-sector organizations together, and encouraging institutions
to share information. The framework then asks grantees to describe
the outcomes for each activity.13
People at these four organizations are doing their part to transform
the practice of outcome measurement. But leaders and scholars in the
nonprofit world must devote more systematic attention to the challenge
of developing evaluation models that fully account for frontline
work. The point here is not to burden frontline staff members with
additional data-collection demands or to justify micro-management
of their efforts. Instead, the purpose is to recognize the full scope of
the work that they do. We need to understand the conditions that
affect this work, the kinds of skills that it requires, and the range of
outcomes that result from it. The new framework that we envision retains
the focus on performance that has long been central to nonprofit
evaluation: Did this activity help to improve the lives of its intended
beneficiaries? But it challenges the assumption that programs alone
enable improvement.14 Evaluators and evaluation researchers have
spent several decades refining techniques for measuring program
performance. Now it’s time to make a similar investment in understanding
how frontline work contributes to participant outcomes.