Download (PDF): 
Revisiting the Disciplinary Home of Evaluation- New Perspectives to Inform PR Evaluation Standards
Jim Macnamara
Professor
School of Communication
University of Technology Sydney
Fraser Likely
Likely Communication Strategies
University of Ottawa
Abstract
From historical analysis of the early development of public relations evaluation (early 1980s to the early 2000s), this paper shows that public relations scholarship and practice have drawn heavily on media and communication studies in developing models and methods of evaluation, but have not significantly engaged with the large related body of knowledge on program evaluation. While communication and media studies are logical and formative disciplinary homes for public relations (PR), this paper argues that PR is a transdisciplinary field and that program evaluation is a mostly overlooked source of influence and heritage in relation to evaluation. This analysis presents evidence that a disciplinary ‘home visit’ to program evaluation, which nestles within program theory and theory of change, offers much to overcome the long-standing stasis in PR evaluation and to inform the search for standards.
Introduction – why revisiting disciplinary homes is necessary
Evaluation methods for PR have been discussed for a century since Edward Bernays described PR as an applied social science that could be “precisely evaluated” (Watson, 2012, p. 391) and Arthur Page advocated use of opinion research (Likely & Watson, 2013). Likely and Watson (2013) noted that the search has received intensive focus over the past 40 years. Nevertheless, the search for PR evaluation models and methods has been likened to the search for the Holy Grail (L’Etang, 2008; Pavlik, 1987). Despite considerable efforts and some progress, Gregory and Watson (2008) lamented a “stasis” in PR evaluation and a number of studies since have confirmed a lack of implementation of evaluation, particularly at the level of achieving organisational objectives (Cacciatore, Meng, & Berger, 2016; Wright, Gaunt, Leggetter, Daniels, & Zerfass, 2009; Wright & Hinson, 2012; Zerfass, Verčič, Verhoeven, Moreno, & Tench, 2012) . Also, scholars and practitioners alike have lamented a lack of standards in PR evaluation (Michaelson & Stacks, 2011). As recently as 2015, Macnamara described PR evaluation as caught in a “deadlock” (Macnamara, 2014, 2015).
This paper presents historical analysis of the first 20 years of this intensive period of development of PR evaluation from the early 1980s to the early 2000s and compares and contrasts this with the development of program evaluation in other fields, particularly the application of theory of change, program theory, and program evaluation frameworks and tools. While this body of knowledge was developed around the same time, it has been drawn on only occasionally in PR and has been largely forgotten in recent initiatives to establish standards for PR evaluation. As Brown has argued, there are “dominant narratives” in PR that produce a “writing out” of other potential narratives that may be relevant and important (2006, p. 206). It is argued here that revisiting this disciplinary home of evaluation and rediscovering this lost heritage can provide theoretical frameworks and practical tools to overcome the long-standing stasis and deadlock in PR evaluation and inform the search for standards.
Methodology
This analysis is based on historical research examining the development of PR evaluation models and methods informed by literature review, interviews, and autoethnography, combined with critical comparative analysis of findings vis-à-vis documented ‘best practice’ program evaluation. While program evaluation is now conducted across a wide range of fields, as noted by Rossi, Lipsey and Freeman (2004, p. 6), it has been a particular focus of study and development in public administration and to some extent in and organizational development.
Relevant literature is reviewed across three fields: public relations evaluation; communication and media studies, which is a disciplinary home from which much PR theory is derived; and program evaluation. Interviews were conducted with a purposive sample of authors who were prominent in the field of PR evaluation during the period studied, including eminent figures such as Walter Lindenmann, Emeritus Professor Tom Watson, and authors of the landmark UK Institute of Public Relations ‘Toolkit’ and the International Public Relations Association Gold Paper on Evaluation (IPRA, 1994). Interviews were undertaken in an unstructured, open-ended format as is applicable to exploratory qualitative research and were conducted mostly informally using a mixture of telephone and e-mail communication.
Autoethnography was applied by the authors based on their own long-term participation in discussion and development of PR evaluation over more than 30 years. Autoethnography is described by Stanley Geertz (1973) as a qualitative research method conducted to learn and understand cultural phenomena that reflect the knowledge and system of meanings guiding the life of a cultural group. In particular, Geertz described ethnography as thick description, meaning such analysis is based on detailed observation and interpretation during an extended period of fieldwork. Barbara Tedlock notes that ethnographers ideally ‘live in’ the studied group or field for an extended period of time (2008, p. 151) and gain first-hand observation, or even participation. Geertz (1973) similarly identified the primary research methods used in ethnography as participant observation and sometimes participation by the researcher. Autoethnographic information is collected in field notes, recordings, and/or diaries, or reconstructed from data sources such as minutes of meetings, letters, reports, papers, speeches, and other records. The latter approach was mostly used in this analysis.
Interpretation of interviews and autoethnographic reflections followed the principles of narrative inquiry, a process that recognizes personal and social experiences as valid and important sources of knowledge (Clandinin & Connelly, 2000; Tedlock, 2008). That said, the authors were reflexive in relation to their own subjectivity and potential influence on the participants and/or research findings, and addressed these risks by applying Maréchal’s recommendation to connect observations to wider social, cultural, and political meanings and understandings. For example, observations and comments gained in interviews and discussions were compared with published literature and archival records to verify claims wherever possible. Thus, while statistical reliability is not applicable to this study as it is to ‘scientific’ quantitative research, this study produced findings that have credibility, dependability, confirmability, and overall trustworthiness as defined by Lincoln and Guba (1985), Silverman (2000), Shenton (2004), and other authors in describing the criteria for rigorous qualitative research.
The birth of PR evaluation
While the practice of public relations and related and largely synonymous fields such as corporate communication and communication management involve a range of activities including budgeting, planning, and reporting, communication and media theory are foundational given definitions of PR as “the management of communication between an organization and its publics” (Grunig & Hunt, 1984, p. 6). In particular, the development of PR as a contemporary practice in the early twentieth century coincided with the golden age of mass communication thinking, information processing theory based on early systems theory and psychology, and a resulting belief in strong media effects. Based on the propaganda and mass communication studies of Harold Lasswell (1927, 1948) and conceptualizations of communication such as the ‘sender, message, channel, receiver’ models of Shannon and Weaver (1949), Schramm (1954), and Berlo (1960), which contributed to ‘injection’, ‘hypodermic needle’ and ‘bullet’ notions of communication, PR evolved during the first half of the twentieth century mostly assuming effects. While a few pioneering practitioners such as Bernays and Page advocated audience research, the predominant disciplinary focus on mass communication, information processing, and lingering belief in direct effects meant that evaluation was mostly not seen as necessary. Apart from recognition of ‘noise’ as a potential disrupter of communication, early models of mediated communication mostly assumed a ‘domino’ effect from messages to awareness, attitude change, and behaviour (see Figure 1).
Figure 1. The Shannon and Weaver (1949) ‘mathematical model of communication’ that, along with other information transmission models, influenced early thinking about PR.
In the second half of the twentieth century things started to change. After referring to the importance of public opinion research in the first edition of their text Effective Public Relations (Cutlip and Center, 1952), Scott Cutlip and Allen Center added evaluation as the fourth step in the ‘PR process’ after “fact-finding, planning and communicating” in their second edition in 1958 (Hallahan, 1993, p. 198). But, while evaluation continued to be discussed in articles and PR textbooks, it was 1985 before Cutlip and Center joined by Glen Broom presented the first model of evaluation with their ‘Preparation, Implementation, Impact’ (PII) model (Cutlip, Center, & Broom, 1985). This was an important advance, as models summarize complex processes and provide a visual illustration that can serve as a tool for practitioners to use.
In their historical analysis of PR evaluation, Fraser Likely and Tom Watson identify a major focus on evaluation from the late 1970s (Likely & Watson; 2013; Watson, 2012; Watson and Noble, 2014). Likely and Watson (2013) cite a conference organized and chaired by Jim Grunig at the University of Maryland in 1977 as a “prime catalyst” for increased attention to evaluation in PR scholarship and practice, as well as a special issue of Public Relations Review on ‘Measuring the effectiveness of public relations’ published in the same year. A number of important publications followed in the 1970s and through the 1980s including the work of Walter Lindenmann from Ketchum (1979, 1980); David Dozier and Glen Broom (Broom & Dozier, 1983; Dozier, 1984, 1985); and Jim Grunig (e.g., Grunig, 1979, 1983). John Pavlik (1987) published one of the first books focussed on evaluation for PR titled Public Relations: What Research Tells Us, which was followed soon after by E.W. Brody and Gerald Stone’s (1989) Public Relations Research. Also, special issues of a number of academic journals including Public Relations Quarterly and Public Relations Review were published during the 1980s.
The 1990s opened with a third book focused on PR research, Using Research in Public Relations: Applications to Program Management by Glen Broom and David Dozier (1990), as well as another special issue of Public Relations Review (Lindenmann, 2005, p. 5). A raft of articles, papers, and book chapters followed including several chapters discussing evaluation in the first Excellence Study book (e.g., Dozier & Ehling, 1992; Ehling, 1992). However, several factors limited practitioner uptake of the now widely circulating advice on PR evaluation. First, with the exception of a number of articles by Lindenmann, most discussion was published in academic books and journals. Second, all except Cutlip, Center and Broom’s ‘PII model’ focussed on theories, with few applied tools such as models and practical guides. It was not until the end of the 1990s that specialized publications focussed on evaluation were produced for practitioners (e.g., Total Communication Measurement produced by Melcrum Publishing).
Based on historical records and interviews, this analysis posits a further limitation to the development of applied evaluation in PR – the focus on mass communication theories and models and information processing and a corresponding lack of attention to the growing discipline of program evaluation that emerged during the same period. These two approaches are compared and contrasted in the following analysis, with lessons emerging on ways to advance PR evaluation and achieve standards.
Information processing – a dominant paradigm in PR evaluation
While early transmissional notions of communication based on basic systems theory broadened with the growing influence of psychology and sociopsychology, as well as scholarship in sociology, phenomenology, semiotics, and cultural studies (Craig, 1999; Craig & Muller, 2007; Griffin, 2009; Littlejohn & Foss, 2008), information processing has remained a dominant mode of thinking in PR. For example, the influential work of social psychologist W. J. McGuire shaped thinking about the steps and stages of communication and these became identified as key elements to evaluate.
Figure 2. Six stages of communication (McGuire, 1985).
After initially identifying six key steps in information processing (see Figure 2), McGuire expanded his model in subsequent work to 13 steps, which he identified as:
Exposure;
Attention;
Liking;
Comprehension (understanding);
Cognition (thinking about the message);
Acquiring skills/knowledge to deal with the issue;
Attitude change;
Storing information (retention);
Retrieving information;
Deciding to act in accordance with information (intention);
Action / behaviour;
Cognitive integration of behaviour;
Encouraging others to behave similarly – i.e., advocacy (McGuire 2001).
The communication-persuasion matrix
The steps of communication proposed by McGuire identified many of the specific steps or elements of a communication program that can be evaluated, and many of these such as exposure, attention, liking, intention, and action are included in PR evaluation models today. McGuire’s work became the basis of the communication-persuasion matrix, described as a basic input-output model (Atkin & Rice, 2013, p. 4), that has been used extensively in advertising where it was condensed and simplified in the AIDA model (awareness, interest, desire, action) and, later, the ‘sales funnel’ used in marketing communication and sales – albeit these are over-simplified and the ‘hierarchy of effects’ implied in these models has been criticized (e.g., Barry, 1990).
A nascent ‘turn’ to public administration and management research
A different approach began to emerge in the early 1990s, although focus on mass communication and information processing continued in PR during the following decades. In the early 1990s, a ‘Macro model of PR evaluation’[1] was published in International Public Relations Review, the journal of the International Public Relations Association (IPRA), by Jim Macnamara (1992). This used a vertically arranged pyramid model to incorporate steps of communication drawn from Cutlip, Center and Broom (1985), but also added suggested measures and methods of evaluation in an attempt to make the model practical – a feature not previously included in PR evaluation models. Furthermore, Macnamara arranged the steps of communication in three stages, which he identified as inputs, outputs, and results. This was one of the first uses of these terms in PR evaluation – a significant shift that marked a turn towards the body of knowledge emerging in the disciplinary field of public administration and being increasingly taken up by management studies in a growing focus on performance management. However, there was not a smooth or universal shift – to the contrary, PR evaluation continued to be a fragmented field of study and practice.
Shortly after Macnamara’s ‘Macro model’ was published, Walter Lindenmann (1993) published his ‘Effectiveness Yardstick’ also as a vertically arranged model. Lindenmann called the stages (1) “basic”, which he described as outputs; (2) “intermediate”, which he described as outgrowths, thus introducing a new term to the PR evaluation lexicon, and (3) “advanced”, which he described as outcomes. Lindenmann’s model usefully identified a number of specific steps within the stages such as reception, awareness, comprehension, retention, opinion change, attitude change, and behavior change. This model became widely cited and adopted to some extent due to Lindenmann’s position as a practising researcher at Ketchum during most of this period following more than a decade with Hill & Knowlton’s research subsidiary, Group Attitudes Corporation, and his prolific writing on the subject of evaluation (e.g., 1979, 1980, 1990, 1997a, 1997b, 1997c, 1998, 2003). Lindenmann acknowledges that he was influenced by the work of McGuire and other communication researchers saying:
The theory behind the model came primarily from well-known communications scholars, especially William J. McGuire, Wilbur Schramm, Everett Rogers, Charles R. Wright, Elihu Katz … and Jim Grunig. I was especially taken with the excellent essay of McGuire, ‘Persuasion, resistance and attitude change’ which appeared in the 1973 Handbook of Communication. McGuire had a matrix that he used in his essay, which I relied on heavily to explain the Yardstick model. The matrix and the arguments that McGuire presented led me to create the stages in the model the way I did. (W. Lindenmann, personal communication, January 18, 2016)
In 1994 another model emerged that has been widely overlooked in bibliographies and histories of PR evaluation. Under the leadership of the then president of IPRA, Jim Pritchitt, an eight-member committee of practitioners and academics from Australia, the UK, and South Africa was commissioned to produce an IPRA Gold Paper on evaluation. The committee was made up of Australian PR practitioners Chris Hocking, Jane Jordan, Jim Macnamara, and Bill Sherman; Dr Gael Walker, an academic from the University of Technology Sydney (UTS); Sandra Macleod, UK CEO of pioneering media analysis firm CARMA International; and Anna Mari Honnibal, a practitioner from South Africa (IPRA, 1994, Preface, n. p.).[2] The largely Australian composition of the committee occurred partly because IPRA president, Jim Pritchitt, is an Australian. But also, several other factors indicate that the mostly Australian/UK initiative in producing the IPRA Gold Paper No. 11 (IPRA, 1994) was not simply a matter of geography. At the time, Australia had the third highest national IPRA membership in the world after the US and the UK (Macnamara & Watson, 2014). Among those members, Walker was advocating evaluation in PR research and teaching at UTS (Walker, 1992) and earlier that year Walker (1994) had replicated a study done by Lindenmann (1990) in the US. The IPRA model of evaluation cited and drew on the Cutlip, Center and Broom ‘PII model’, Walker’s research, and Macnamara’s ‘Macro model’. In the previous year Macnamara completed a Master of Arts degree by research with a thesis on evaluating the impact of PR on media and was in the process of founding CARMA (Asia Pacific) working closely with Sandra Macleod, and CARMA International sponsored the IPRA Gold Paper. So there was significant academic and practitioner research expertise and commitment to evaluation in this group.
The IPRA model was the first to use the stages of inputs, outputs, and outcomes, although these terms were also being talked about in the UK Institute of Public Relations (now CIPR). However, neither Macnamara’s ‘Macro model’ nor the IPRA model are mentioned in Bibliography of Public Relations Measurement published by the Institute for Public Relations (IPR) in the US (Carroll & Stacks, 2004).
Figure 3. The largely overlooked International Public Relations Association model of evaluation (IPRA, 1994).[3]
Tom Watson completed a PhD at Nottingham Trent University in the UK in 1995 with a thesis on PR evaluation and published his ‘Short term model of evaluation’ and his ‘Continuing model of evaluation’ in 1996. While usefully showing an iterative approach to evaluation with feedback loops informing strategy and tactics, these models illustrate only broad stages such as research, objective setting, strategy development, tactical choices, and effects identified through “analysis” with no details (Watson, 1996). In the same year, Anne Gregory (1996) published a chapter on evaluation in the first edition of her Planning and Managing a Public Relations Campaign in which she attempted to integrate evaluation into PR planning and management.
The following year in early versions of his ‘Guidelines and Standards’ published by the Institute for Public Relations and a number of industry journals, Walter Lindenmann (1997b, 1997c) evolved his thinking and renamed his three stages of PR programs as outputs, outcomes, and business/organizational outcomes. He still did not include inputs and appeared to have wavered in use of the term outgrowths. Around the same time on the other side of the Atlantic, Michael Fairchild (1997), an evaluation specialist working with the then IPR in the UK (now CIPR) and the International Committee of Public Relations Consultancies Associations (ICO, now ICCO), published a paper identifying the stages of PR programs as outputs, outtakes, and outcomes. In addition, as Watson (1999, p. 14) notes, Fairchild hinted at inputs as an important stage. Thus, Fairchild seems to be the first to use the term outtakes and the first to suggest four stages in PR programs as inputs, outputs, outtakes, and outcomes. In a revision of his Guidelines and Standards for Measuring and Evaluating PR Effectiveness (Lindenmann, 1997b), Lindenmann (2003) subsequently dropped the term outgrowths and adopted Fairchild’s concept of outtakes in Guidelines for Measuring the Effectiveness of PR Programs and Activities produced “with input and suggestions from Fraser Likely” (Lindenmann, 2003, preface).[4] Fairchild says that a number of the pioneers in PR evaluation communicated and shared ideas. Now in retirement, he reflected:
Walt [Lindenmann] and I had a discussion about ‘out-take’ that seemed to fill an obvious gap: the need to know how you got from output to outcome; what did the audience understand from the output, indeed whether they got the message at all? (M. Fairchild, personal communication, May 24, 2016)
Fairchild emphasizes that his focus was to integrate evaluation into “the process of planning, research and evaluation” and “to move beyond academic debate into pragmatic solutions” – although it must be noted that Lindenmann, Macnamara, and Watson all had practitioner backgrounds. He said “there was a need to show clients that we could evaluate the worth of what we did, but there was an equally pressing need to persuade PR practitioners that the whole approach to PR needed to be re-thought” (M. Fairchild, personal communication, May 24, 2016).
Throughout this period, Jim Grunig continued to advocate evaluation and in 1999 he and Linda Childers published a paper on evaluating relationships in which they identified the key stages of evaluation as outputs, outcomes, and relationships (Childers & Grunig, 1999). In the same year, Michael Fairchild and Nigel O’Connor (1999)[5] produced the first edition of the IPR ‘Toolkit’ for measurement and evaluation and this continued to promote the stages of outputs, outtakes, and outcomes within what they called the ‘PRE process’ (planning, research, and evaluation). Also in 1999, Paul Noble and Tom Watson presented a paper at the Transnational Communication in Europe: Practice and Research Congress in Berlin reviewing a number of models and introducing their ‘Unified model’. This identified the stages as input, output, impact, and effect (Noble & Watson, 1999, p. 20), consolidating the concept of four stages, but still using different terms. Nevertheless, Watson’s ‘Unified model’ illustrated a broadening focus in PR evaluation. He recalls:
My research into measurement and evaluation started from a practitioner perspective. I wondered why some campaigns worked well and gained desired results and other, similarly-designed campaigns didn’t. My reading of literature on evaluation methods identified work by Cutlip and Center, other PR texts, and Broom & Dozier’s excellent book on research methods in PR. I was also aware of Jim Macnamara’s early work. However, my testing of some models, particularly Cutlip and Center’s PII model, in practice was not satisfactory. I felt they could not be implemented in practice. So, I developed four case studies as a central part of my PhD research to investigate the realities of measurement and evaluation. From these case studies, I identified two practice-related models – the Short-Term model for tactical activity and the Continuous model that used iterative loops to undertake formative assessment during the length of a campaign.
The Unified Model was an attempt by Paul Noble and me to create a more integrated model based on both our work in the mid-1990s and to position PR as a measurable strategic communication activity. (T. Watson, personal communication, May 16, 2016)
As the twentieth century ended and the new millennium began, PR researchers working in the Deutsche Public Relations Gesellschaft (DPRG) and Gesellschaft Public Relations Agenturen (GPRA) in Germany produced the first of a series of ‘communication controlling’ models, which identified four stages of PR programs as input, output, outcome, and outflow, thus introducing yet another term to PR evaluation terminology (DPRG/GPRA, 2000).
These variations, which have since spawned dozens of models of PR evaluation that identify a range of stages and multiple steps in evaluation, beg the twin questions of why this fragmentation has occurred and how this might be overcome to achieve standardization. As Michaelson and Stacks note, standards are important as they allow “comparative evaluations” over time and they ensure appropriate methods are used (2011, p. 4). These questions are explored by turning to another disciplinary field that has produced a large body of knowledge specifically focussed on evaluation.
Program evaluation – The disciplinary home of evaluation
Around the same time as communication studies was emerging as a distinct and growing discipline drawing on systems theory, sociopsychology, cultural studies, and other fields, and PR evaluation was starting to receive intense focus, program theory and theory of change were being developed through the pioneering work of Edward Suchman (1967); Carol Weiss (1972); Joseph Wholey (1970, 1979, 1983, 1987); Claude Bennett (1976); Huey Chen and Peter Rossi (1983); Leonard Bickman (1987); Mark Lipsey (1993); and others. Program evaluation is a central focus of program theory and theory of change, which explore how programs can be designed and implemented to achieve their objectives.
Program evaluation has been advanced most notably in the disciplinary field of public administration, being developed first in relation to human service programs such as the delivery of social services and health promotion campaigns, but has spread to a wide range of fields from agricultural programs and construction projects to the testing of military hardware. Rossi, Lipsey, and Freeman say that program evaluation based on program theory and theory of change is “useful in virtually all spheres of activity in which issues are raised about the effectiveness of organized social action” and note its relevance for advertising, marketing, and other communication activities (2004, p. 6).
Theory of change
Theory of change, which emerged from research in environmental and organizational psychology, provides a broad overview of how a program is intended to work, identifying the basic stages that lead from planning to demonstration of effectiveness in achieving its objectives with particular emphasis on outcomes and impact. However, theory of change models usually provide little detail of activities undertaken or how these will be evaluated (e.g., see Figures 4 and 5 that, strictly speaking, are theory of change models).
Program theory
Program theory involves the conceptualization of how a specific program is intended to work and includes identification of a chain of activities that are expected to produce the intended impacts stated in the program objectives. This inclusion of details is what distinguishes program theory and resulting models based on that theory from the broad theoretical overview provided in theory of change. Rossi et al. identify three key inter-related components of a program theory: (1) the program impact theory; (2) the utilization plan; and (3) the program’s organizational plan. Thus, while this approach starts with theory, it moves quickly towards practical implementation. The program impact theory, in simple terms, is the theoretical projection of what a program will achieve – i.e., its desired effect and impact. This must be more than an aspirational statement. Program impact theory is a causal theory, designed to describe the cause and effect sequence that leads to the desired impact. Rossi et al., note that the utilization plan is “usefully depicted as a flow chart” that tracks the various stages and elements in a program (2004, p. 142), while the organizational plan describes the management actions necessary such as assigning the resources required and planning and implementing activities to achieve the desired effect.
Program logic models
The various stages and elements of a program theory are very commonly explicated in program logic models, a graphic illustration of the processes in a program from pre-program planning to its outcomes and impact. Use of the term ‘program logic model’ and its basic construction is most commonly attributed to Joseph Wholey’s (1979) text, Evaluation: Promise and Performance and is also informed by Claude Bennett’s (1976) The Seven Levels of Evidence. Program logic models were used by the US Agency for International Development (USAID) in the 1970s and have been extensively applied in public administration across a wide range of sectors since. Early program logic models developed for USAID and other organizations identified the causally-connected stages of programs as inputs, outputs, achievement of the project purpose, and achievement of the program goal (PCI, 1971, 1979).
However, the Kellogg Foundation, which has been a leader in the field of program evaluation and program logic models for several decades, advocates a widely-used model that identifies five stages in programs as inputs, activities, outputs, outcomes, and impact (see Figure 4). This serves as a planning model as well as an evaluation framework, facilitating identification and assessment of the adequacy of resources and other inputs as well as later stages. In communication and PR programs, inputs can include baseline data, formative research, and pre-testing, which indicate that this stage should not be overlooked as some PR evaluation models do.
Figure 4. The basic structure of a classic program logic model (Kellogg Foundation, 1998/2004)
The University of Wisconsin Extension program (UWEX), another leader in the field, describes the components of basic logic models in its guide as inputs, outputs, and outcomes (Taylor-Power & Henert, 2008, p. 20) – the stages reflected in the IPRA (1994) model and in the ‘Macro model of PR evaluation’ (Macnamara, 1992) that evolved into the ‘Pyramid model of PR research’ (Macnamara, 2000, 2002a, 2002b). In addition, more advanced versions of the UWEX model segregate outputs into activities and participation, making it quite similar to the Kellogg Foundation model, and split outcomes into short, medium, and long term – what are also called proximal and distal outcomes. When this done, long-term (distal) outcomes are synonymous with impact (see Figure 5).
Figure 5. A more advanced program logic model (Taylor-Power & Henert, 2008, p. 5).
As the UWEX Developing a Logic Model: Teaching and Training Guide notes: “many variations and types of logic models exist” (Taylor-Power & Henert, 2008, p. 2). The Kellogg Foundation similarly says “there is no one best logic model” (2004, p. 13). However, there are a number of common concepts and principles in program logic models. These are illustrated in Table 1, which compares some classic and widely used program logic models with PR evaluation models developed between the early 1980s and early 2000s. As well as illustrating the fragmentation that has occurred in PR evaluation thinking, this shows that, among 15 variants of models examined, the most commonly used stages are, in order of usage: outputs (14 times); outcomes (12 times), inputs (9 times), impact or closely related terms such as results (6 times), and outtakes (4 times). Arranged in ‘logical’ order, this suggests a model made up of at least four stages (inputs, outputs, outcomes, and impact) and possibly five stages (inputs, outputs, outtakes, outcomes, and impact). Activities is listed as a stage in classic program logic models, but not in any PR evaluation models. Outgrowths is used twice (Lindenmann, 1993; Likely, 2000), but to describe two different levels, and outflows is used only once. (See Table 1.)
Writing in the UWEX guide, Taylor-Power and Henert (2004, p. 6) note a limitation of program logic models is that many are uncomfortable with their apparent linearity. However, this is largely a misinterpretation of their use. Most program logic models include feedback loops or note in accompanying text the need for iterative planning that uses findings at each stage to adjust strategy and tactics if required. Some logic models are drawn as a cycle to highlight this iterative approach (e.g., Knowlton & Phillips, 2013, p. 39), although the horizontal or vertical arrangement of ‘boxes’ is the most common depiction.
Even with a program logic model in place, evaluation requires an evaluation plan that identifies the methods to be used to obtain data required. While a range of metrics can be automatically generated in today’s digital world such as counts of visitors to Web pages, views of content, likes, follows, and so on, most researchers agree that evaluation requires social science research methods, particularly at outcomes and impact levels (Atkin & Rice, 2013). However, program evaluation tools such as program logic models provide a ‘roadmap’ and a framework to bring consistency to evaluation as well as a number of other benefits.
Discussion and conclusions
Communication models based on systems theory and sociopsychology such as McGuire’s information processing have contributed details on the many steps in the process of communication and are valuable in populating an evaluation model – that is, providing the specific steps and elements that can and should be evaluated. However, PR evaluation models based on communication and media theories alone lack an overarching coherent framework. Without this, they present an array of processes, methods, and terms that is confusing to many practitioners and to management. It is curious that PR has mostly ignored the specialist discipline of program evaluation in its pursuit of evaluation models and methods and it is argued that, rather than continuing to create new terms and new models, PR evaluators should revisit the disciplinary home of program evaluation where a large body of academic and practice-orientated literature and tools exist. While some PR evaluation models have drawn on program theory and program evaluation, and some such as Sherry Devereaux Ferguson (1991) and Fraser Likely (1994-95) have argued that communication managers should work with specialist program evaluators, line managers, and internal audit units to undertake evaluation, the PR industry has suffered because of its narrow disciplinary focus and its frequent ‘go it alone’ approach and bifurcation that has characterized the search for the Holy Grail of evaluation.
It is proposed that program theory and program logic models can make at least three significant contributions to PR evaluation as follows.
Program theory and program logic models provide a framework to guide the process of evaluation throughout planning and implementation and allow identification of the appropriate evaluable elements at each stage. As Cutlip, Center and Broom note repeatedly in editions of their text from 1985 to the late-2000s, “the common error in program evaluation is substituting measures from one level for those at another level” (1985, p. 295; 1994, p. 414; Broom, 2009, p. 358), a warning echoed by Jim Grunig who said that many practitioners use “a metric gathered at one level of analysis to show an outcome at a higher level of analysis” (2008, p. 89). The UWEX guide to program evaluation says, for example, “people often struggle with the difference between outputs and outcomes” (Taylor-Power & Henert, 2008, p. 19). Program logic models help identify not only the overall stages, but the specific elements that are indicators or milestones of progress in each stage.
Program logic models provide a more holistic and comprehensive approach, such as identifying that impacts are not only those that concern the organization. The Kellogg Foundation Logic Model Development Guide says “impacts are organizational, community, and/or system level changes” (Kellogg Foundation, 2004, p. 8). This aligns with PR Excellence Theory, which advocates evaluation at (1) program level; (2) functional level (e.g., department or unit); (3) organizational level; and (4) societal level (L. Grunig, J. Grunig, & Dozier, 2002, pp. 91–92). Most contemporary PR evaluation models, including the latest evaluation frameworks developed by AMEC (2016) and the UK Government Communication Service (GCS, 2015) continue to focus solely or predominantly on organizational objectives, indicating a need for further development.
Program logic models, which have been developed in public administration and are the basis of most performance management systems used today, connect PR evaluation to recognized management processes. They can bring greater consistency to PR evaluation and approaches that are widely recognized, evidence-based, and while retaining flexibility, are based on standards, which PR scholars and practitioners are seeking.
References
AMEC (Association for Measurement and Evaluation of Communication). (2016). AMEC evaluation framework. London, UK: Author.
Atkin, C., & Rice, R. (2013). Theory and principles of public communication campaigns. In R. Rice & C. Atkin (Eds.). (2013). Public communication campaigns (4th ed., pp. 3–19). Thousand Oaks, CA: Sage.
Barry, T. (1990). A review and critique of the hierarchy of effects in advertising. International Journal of Advertising: The Review of Marketing Communications, 9(2), 121–135. doi: 10.1080/02650487.1990.11107138
Bennett, C. (1976). Analyzing impacts of extension programs, ESC-575.Washington, DC: US Department of Agriculture Extension Service. Department of Agriculture, Extension Service.
Berlo, K. (1960). The process of communication. New York: Holt, Rinehart & Winston.
Bickman, L. (1987). The functions of program theory. In New Directions for Program Evaluation, 33, 5–18. San Francisco, CA: Jossey-Bass.
Brody, E., & Stone, G. (1989). Public relations research. New York, NY: Praeger.
Broom, G. (2009). Cutlip & Center’s effective public relations (10th ed.). Upper Saddle River, NJ: Pearson.
Broom, G., & Dozier, D. (1983). An overview: Evaluation research in public relations. Public Relations Quarterly, 28(3), 5–9.
Broom, G., & Dozier, D. (1990). Using research in public relations: Applications to program management. Englewood Cliffs, NJ: Prentice-Hall.
Brown, R. (2010). Symmetry and its critics: antecedents, prospects, and implications for symmetry in a post-symmetry era. In R. Heath (Ed.), The SAGE handbook of public relations (pp. 277–292), Thousand Oaks, CA: Sage.
Cacciatore, M., Meng, J., & Berger, B. (2016, March). Measuring the value of PR? An international investigation of how practitioners view the challenge and solutions. Proceedings of the 19th International Public Relations Research Conference. Miami, FL.
Carroll, T., & Stacks, D. (2004). Bibliography of public relations measurement. Gainesville, FL: Institute for Public Relations. Retrieved from http://www.instituteforpr.org/wp-content/uploads/PRM_Bibliography.pdf
Chen, H., & Rossi, P. (1983). Evaluating with sense: The theory-driven approach. Evaluation Review, 7(3), 283–302. doi.org/10.1177/0193841X8300700301
Childers, L., & Grunig, J. (1999). Guidelines for measuring relationship in public relations. Gainesville, FL: Institute for Public Relations. Retrieved from http://www.instituteforpr.org/measuring-relationships
Clandinin, D., & Connelly, F. (2000). Narrative enquiry. Experience and story in qualitative Research. San Francisco, CA: Jossey-Bass.
Cutlip, S., & Center, A. (1952). Effective public relations. Englewood Cliffs, NJ: Prentice-Hall.
Cutlip, S., Center, A., & Broom, G. (1985). Effective public relations (6th ed.). Englewood Cliffs, NJ: Prentice Hall.
Cutlip, S., Center, A., & Broom, G. (1994). Effective public relations (7th ed). Upper Saddle River, NJ: Prentice Hall.
Devereaux Ferguson, S. (1999). Constructing a theoretical framework for evaluating public relations programs and activities. In M. Roloff (Ed.). Communication yearbook 21 (pp. 191–229). Thousand Oaks, CA: Sage.
Dozier, D. (1984). Program evaluation and the roles of practitioners. Public Relations Review, 10(2), 13–21. doi.org/10.1016/S0363-8111(84)80002-8
Dozier, D. (1985). Planning and evaluation in public relations practice. Public Relations Review, 11(2), 17–26. doi:10.1016/SO363-8111(82)80115-Xdoi:10.1016/S0363-8111(82)80115-X doi:10.1016/S0363-8111(82)80115-X doi:10.1016/S0363-8111(82)80115-X
Dozier, D., & Ehling, W. (1992). Evaluation of public relations programs: What the literature tells us about their effects. In J. Grunig (Ed.), Excellence in public relations and communication management (pp. 159 – 184). Hillsdale, NJ: Lawrence Erlbaum.
DPRG/GPRA (Deutsche Public Relations Gesellschaft and Gesellschaft Public Relations Agenturen). (2000). PR-evaluation: Messen, analysieren, bewerten – empfehlungen für die praxis. Booklet des evaluationausschusses von DPRG und GPRA, 2, Bonn, Germany.
Ehling, W. (1992). Estimating the value of public relations and communication to an organisation. In J. Grunig (Ed.), Excellence in public relations and communication management (pp. 617–638). Hillsdale, NJ: Lawrence Erlbaum.
Fairchild, M. (1997). How to get Real value from public relations. London, UK: ICO.
Fairchild, F., & O’Conner, N. (1999). The public relations research and evaluation toolkit: How to measure the effectiveness of PR. London, UK: Institute of Public Relations (now CIPR) and Public Relations Consultants Association (PRCA) with PR Week.
Fairchild, F., & O’Connor, N. (2001). The IPR Toolkit: Planning, research and evaluation for public relations success (2nd ed.). London, UK: Institute of Public Relations.
GCS (Government Communication Service). (2015). Evaluation framework. London, UK: Cabinet Office, HM Government. Retrieved from https://gcs.civilservice.gov.uk/guidance/evaluation/tools-and-resources
Geertz, C. (1973). Thick description: Toward an interpretive theory of culture. In The interpretation of cultures: Selected essays (pp. 3–30). New York, NY: Basic Books.
Gregory, A. (1996). Planning and managing a public relations campaign. London, UK: Kogan Page. (Second edition published 2000)
Gregory, A., & Watson, T. (2008). Defining the gap between research and practice in public relations programme evaluation: Towards a new research agenda. Journal of Marketing Communications, 24(5), 337–350. doi.org/10.1080/13527260701869098
Grunig, J. (1979). The status of public relations research. Public Relations Quarterly, 20(1), 5–8.
Grunig, J. (1983). Basic research provides knowledge that makes evaluation possible. Public Relations Quarterly, 28(3), 28–32.
Grunig, J. (2008). Conceptualizing quantitative research in public relations. In B. van Ruler, A. Tkalac Vercic, & D. Vercic (Eds.), Public relations metrics: research and evaluation (pp. 88–119). New York, NY: Routledge.
Grunig, J., & Hunt, T. (1984). Managing public relations. Orlando, FL: Holt, Rinehart & Winston.
Grunig, L., Grunig J., & Dozier, D. (2002). Excellent organizations and effective organizations: A study of communication management in three countries. Mahwah, NJ: Lawrence Erlbaum.
Hallahan, K. (1993). The paradigm struggle and public relations practice. Public Relations Review, 19(2), 197–205. doi.org/10.1016/0363-8111(93)90009-2
Henert, E., & Taylor-Power, E. (2008). Developing a logic model: Teaching and training guide. Retrieved from http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.pdf
IPRA (International Public Relations Association). (1994). Public relations evaluation: Professional accountability. Gold Paper No. 11. Geneva, Switzerland: Author.
Kellogg Foundation. (1998/2004). Logic model development guide. Battle Creek, MI: Author. Retrieved from http://www.epa.gov/evaluate/pdf/eval-guides/logic-model-development-guide.pdf
Knowlton, L. & Phillips, C. (2013). The logic model guidebook: Better strategies for great results (2nd ed.). Thousand Oaks, CA: Sage.
Lasswell, H. (1927). Propaganda techniques in the world war. New York, NY: Knopf.
Lasswell, H. (1937). Propaganda. In E. Seligman & A. Johnson (Eds.) Encyclopaedia of the social sciences (Vol. 12, pp. 521–552). New York, NY: Macmillan.
Lasswell, H. (1948). The structure and function of communication in society. In L. Bryson (Ed.), The communication of ideas (pp. 37–51). New York, NY: Harper.
L’Etang, J. (2008), Public Relations: Concepts, Practice and Critique, Sage, London and Thousand Oaks, CA.
Likely, F. (1994-95). The knock and the roles in public relations/communication. Journal of Corporate Public Relations, 2, 7–13.
Likely, F. (2000). Communication and PR: Made to measure. Strategic Communication Management, 4(1), 22–27.
Likely, F., & Watson, T. (2013). Measuring the edifice: Public relations measurement and evaluation practice over the course of 40 years. In J. Sriramesh, A. Zerfass, & J. Kim (Eds.), Public relations and communication management: Current trends and emerging topics (pp. 143–162). New York, NY: Routledge.
Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
Lindenmann, W. (1979). The missing link in public relations research. Public Relations
Review, 5(1), 26–36. doi.org/10.1016/S0363-8111(79)80005-3
Lindenmann, W. (1980). Hunches no longer suffice. Public Relations Journal, 36(6), 9–13.
Lindenmann, W. (1990). Research, evaluation and measurement: A national perspective. Public Relations Review, 16(2), 3–16. doi: 10.1016/SO363-8111(05)80001-3
Lindenman, W. (1993). An ‘effectiveness yardstick’ to measure public relations success. Public Relations Quarterly, 38(1), 7–9.
Lindenmann, W. (1997a). Using research to measure the effectiveness of PR programs. Quirk’s Marketing Research. Retrieved from http://www.quirks.com/articles/a1997/19970204.aspx
Lindenmann, W. (1997b). Guidelines and standards for measuring and evaluating PR effectiveness. Gainesville, FL: Institute for Public Relations.
Lindenmann, W. (1997c). Guidelines and standards for measuring and evaluating PR effectiveness. PR Pundit. Retrieved from http://prpundit.com/pdf/prTools/MeasuringandEvaluatingPREffectiveness.pdf
Lindenmann, W. (2003). Guidelines for measuring the effectiveness of PR programs and activities. Gainesville, FL: Institute for Public Relations. Retrieved from http://www.instituteforpr.org/wp-content/uploads/2002_MeasuringPrograms.pdf
Lindenmann, W. (1998). Only PR outcomes count: That is the real bottom line. Journal of Communication Management, 3(1), 66–73. doi.org/10.1108/eb023485
Lindenmann, W. (2005, February). Putting PR measurement and evaluation in historical perspective. Gainesville, FL: Institute for Public Relations. Retrieved from http://www.instituteforpr.org/wp-content/uploads/PR_History2005.pdf
Lipsey, M. (1993). Theory as method: Small theories of treatments. New Directions for Program Evaluation, 57, 5–38. doi.org/10.1002/ev.1637
Macnamara, J. (1992). Evaluation of public relations: The Achilles Heel of the public relations profession. International Public Relations Review, 15(2), 19–25.
Macnamara, J. (2000). Jim Macnamara’s public relations handbook. Melbourne, Vic: Information Australia.
Macnamara, J. (2002a, November). An international perspective on priority issues in public relations. Paper presented to the IPRA World Congress, Cairo, Egypt.
Macnamara, J. (2002b). Research and evaluation. In C. Tymson & P. Lazar, The new Australian and New Zealand public relations manual (21st Century ed., pp. 100–134). Sydney, NSW: Tymson Communications.
Macnamara, J. (2014, June). Breaking the PR measurement and evaluation deadlock: A new approach and model. Paper presented to the Association for Measurement and Evaluation of Communication International Summit on Measurement, Amsterdam, The Netherlands. Retrieved from http://amecorg.com/downloads/amsterdam2014/Breaking-the-PR-Measurement-Deadlock-A-New-Approach-and-Model-Jim-Macnamara.pdf
Macnamara, J. (2015). Overcoming the measurement and evaluation deadlock: A new approach and model. Journal of Communication Management, 19(4), 371–387. doi: 10.1108/JCOM-04-2014-0020
Macnamara, J., & Watson, T. (2014, July). The rise and fall of IPRA in Australia: 1959 to 2000. Asia Pacific Public Relations Journal, 15(1), 23 – 36. Retrieved from https://ojs.deakin.edu.au/index.php/apprj/index
Maréchal, G. (2010) Autoethnography. In A. Mills, G. Durepos, & E. Wiebe (Eds.), Encyclopedia of case study research (Vol 2, pp. 43–45). Thousand Oaks, CA: Sage.
Michaelson, D., & Stacks, D. (2011). Standardization in public relations measurement and evaluation. Public Relations Journal, 5(2), 1–22.
Moll, M. (2012). The quintessence of intercultural business communication. Berlin, Heidelberg, Germany: Springer.
Noble, P., & Watson, T. (1999). Applying a unified public relations evaluation model in a European context. Paper presented to Transnational Communication in Europe: Practice and Research International Congress, Berlin.
Pavlik, J. (1987), Public relations: What research tells us. Newbury Park, CA: Sage.
PCI (Practical Concepts, Inc.). (1971). The logical framework. Approach and training materials developed for U.S. Agency for International Development. Washington. D.C. [Unpublished manuscript].
PCI (Practical Concepts, Inc.). (1979). The logical framework. A manager’s guide to a scientific approach to design & evaluation. Washington, DC. Retrieved from http://pdf.usaid.gov/pdf_docs/pnabn963.pdf
Rossi, P., Lipsey, M., & Freeman, H. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.
Schramm, W. (1954). How communication works. In W. Schramm (Ed.), The process and effects of communication (pp. 3–26). Urbana, IL: University of Illinois Press.
Shenton, A. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22(2), 63–75.
Silverman, D. (2000). Doing qualitative research: A practical handbook. London, UK: Sage.
Suchman, E. (1967). Evaluative research: Principles and practice in public service and social action programs. New York, NY: Russell Sage Foundation.
Taylor-Power, E., & Henert, E. (2008). Developing a logic model: Teaching and training guide. Retrieved from http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.pdf
Tedlock, B. (2008). The observation of participation and the emergence of public ethnography. In N. Denzin & Y. Lincoln (Eds.) Strategies of qualitative inquiry (3rd ed., pp. 151–171). Thousand Oaks, CA: Sage.
Walker, G. (1992). Communicating public relations research. Unpublished paper presented to a research conference at the University of Technology Sydney, Australia.
Walker, G. (1994). Communicating public relations research, Journal of Public Relations Research, 6 (3), 145.
Watson, T. (1996). New models for evaluating public relations practice. In B. Baerns & J. Klewes (Eds.), Jahrbuch public relations (pp. 50–62). Dusseldorf, Germany: Econ Verlag.
Watson, T. (2012). The evolution of public relations measurement and evaluation. Public Relations Review, 38(3), 390–398.
Watson, T., & Noble, P. (2007). Evaluating public relations: A best practice guide to public relations planning, research and evaluation (2nd ed.). London, UK: Kogan Page.
Watson, T., & Noble, P. (2014). Evaluating Public Relations: A best practice guide to public relations planning, research and evaluation (3rd ed.). London, UK: Kogan Page.
Weiss, C. (1972). Evaluative research: Methods of assessing program effectiveness. Englewood Cliffs, NJ: Prentice Hall.
Wholey, J. (1970). Federal evaluation policy. Washington, DC: Urban Institute Press.
Wholey, J. (1979). Evaluation: Promise and performance. Washington, DC: Urban Institute Press.
Wholey, J. (1983). Evaluation and effective public management. Boston, MA: Little Brown & Co.
Wholey, J. (1987). Evaluability assessment: Developing program theory. New Directions for Program Evaluation, 33, 77–92. doi.org/10.1002/ev.1447
Wholey J., Hatry, H., & Newcomer, K. (Eds.). (2010). Handbook of practical program evaluation (3rd ed.). San Francisco, CA: Jossey-Bass.
Wright, D., Gaunt, R., Leggetter, B., Daniels, M., & Zerfass, A. (2009). Global survey of communications measurement 2009 – final report. London, UK: Association for Measurement and Evaluation of Communication. Retrieved from http://amecorg.com/wp-content/uploads/2011/08/Global-Survey-Communications_Measurement-20091.pdf
Wright, D. and Hinson, M. (2012), “Examining how social and emerging media have been used in public relations between 2006 and 2012: A longitudinal analysis”, Public Relations Journal, Vol. 6 No. 4, pp. 1–40.
Zerfass, A., Verčič, D., Verhoeven, P., Moreno, A., & Tench, R. (2012). European communication monitor 2012: Challenges and competencies for strategic communication. Brussels, Belgium: European Association of Communication Directors (EACD), European Public Relations Education and Research Association (EUPRERA), and Helios Media.
[1] The name ‘macro’ was taken from the definition of macro communication as communication between an organization and a range of external publics (Moll, 2012, p. 4) and also because Macnamara founded and headed the PR agency MACRO Communication in 1984 and MACRO Communication Research in 1986.
<a href="http://www.instituteforpr.org/#_ednref2" name