ARDEN HANDLER: We are going to wait about two or three more minutes because of the upstairs/downstairs arrangement here, with people eating breakfast upstairs. You can hear, okay. Mary, they can't hear me. Can you guys hear now? Kind of? Only if I go like this? Is that better? Okay. All right. I'll make sure. It is so far away, the mic, you know the speaker has to be like this. I'm so short, but...Better? Okay. We are going to give it another minute or so before we start because of the upstairs/downstairs arrangement of the hotel. Then we will have our welcomes and our first keynote speaker for today. I think we will get started. Can you hear me back there? Everybody? Okay.
I'm Arden Handler. I'm MCH professor at University of Illinois School of Public Health. I’m excited to welcome you today to our 17th annual MCH leadership conference making change happen. We have a lot in store for you today. We are very excited that you could join us. We are happy to say people have come from far away this time. We have many state and local health agencies represented. Better than last year. People are being allowed more travel maybe this year. As well as, individuals from as far away a Korea, who saw the speakers for our conference and were so excited to come and join us. The arrangements are more complicated in terms of the location. We picked the hotel because we think is it fun to be more in the heart of the magnificent mile. But there is an upstairs/downstairs. You have to pay attention to where you are in the schedule, if you are going to be on the first floor or going back upstairs for break out rooms, etcetera.
I wanted to say this is the third of our conference series translating research to practice. This is our last year. Our funding has run out. We applied for more. Hopefully we will be continuing this theme. We will be hearing about our renewal soon. Last year we decided on the role of evaluation in translating research to practice, so that is the theme of the entire conference. We are happy today that you will be hearing from Michael Quinn Patton who I'll introduce in a few minutes. I want to make sure I say a couple things today. We are being webcast live over the Internet for this session and four other sessions. I want to welcome all the webcast participants and let them know they can E-mail their questions at any time during the webcast. What will happen is that their questions will be handed to the speakers. It is very important for a speaker, when you receive a n E-mail question, to repeat the question. It is important if you have a question from the audience to please go to the mics, so that those who are listening through the webcast can hear you and even if the speaker can repeat that question again, just for clarity, for those observing through their computers.
I want to take the time too, to thank the planning committee for this conference for all of their hard work and dedication to planning this conference. I want to thank Maternal Child Health Bureau for helping to fund the conference. The CDC, who is paying for the webcast for this conference. For all the staff and volunteers who worked on the conference, Joan Chou, Anna, Peg Dublin, and others. I also want to let you know that we want a lot of input. We are assuming we will get funded. We would love a lot of input about the conference this year, and how you would like to see the conference next year. In your packet, everything that is salmon colored is something we want you to turn in. Most of the salmon-colored forms are in a section called evaluation forms. They are at the back of the packet. There is one salmon form that is in the front, close to the program. That is a more general form. We would like you to comment about the conference, the format, the structure, what you would like to see change, different, etcetera.
If you really want to have input into the conference and we would love to have your input, we are having a planning committee meeting tomorrow afternoon once the conference is finished. If you are not officially on a planning committee but would like to join us, go to the registration desk and tell the people who are sitting there that you would like to join the planning committee meeting tomorrow, and they will give you the time and location of the meeting.
I want to say that we stay in communication with everyone who comes to our conference as well as others through our maternal and child health list serve. It is a special conference list serve. We send you updates on a weekly basis on a variety of topic areas and areas of MCH. Similar to the list serve that comes nationally, we come up with salient articles that we think relate to the issue of translating research into practice. Let me see if there is anything else I need to tell you before we get started. I'll introduce our introducers. Just to say that we are going to be going from here after this whole session back up to the round tables. You will be in; there is breakout rooms, 40 people per breakout room. You will go to two poster sessions. The idea that we started doing this last year, where we have both a poster and a round table, so the poster presentation, and then a formal way or in formal way to discuss what is on the poster. We thought it worked well last year. We are replicating it for this year. For you standing near the door in back, there is lots of seats here and here. Come fill out these tables in front. We are happy to have you. I'd like to begin by inviting Dean Susan Scrimshaw, dean of school of public health to provide a warm welcome. Thank you very much. (Applause).
DEAN SUSAN SCRIMSHAW: Thank you very much, Arden. I'd like to say welcome. (Speaking in Spanish). And to say (inaudible) I'm pleased to be here and see familiar faces. It is wonderful that not only do we have new people coming into the group every year. But that we have people who come year after year and find that getting together and talking about the common issues that we have in maternal and child health, and then going back to our daily lives and using some of the thinking and being re fresh ed in the conversation, to find that is helpful is important to us. It means a great deal. We are proud of our MCH program. I think in particular, because it is a very community-focused program. There is this kind of outreach and this kind of networking.
I also, like today's -- I also like today's topic, the translating of research into practice, and the evaluation. I can't tell you how important it is. I spend now a lot of time, often in congress, or at health and human services, or at CDC and try there to convince the policy-makers and I have to say HSS, and CDC people are on board and it is educating congress to do the job better. But we talk about the fact that often people have this image that we can do all this great research, and we can discover new vaccine s and discover etiologies of diseases and sew forth and then we have done our -- and so forth and that we have done our job.
There is a push for funding for National Institutes of Health. That is important. We need the basic work. Some of us have pushed N I H to fund more translational work that we will be talking about today. But the basic work doesn't get us anywhere if you are not out there applying it. And applying it isn't helpful if we don’t evaluate it. If we don't understand it. One of the examples that I give of the gap between knowledge and application is measles. We have now had a measles vaccine for over 40 years. We still see measles in this country and low-income areas and of course we still see it around the world. We see less of it now. But as early as the ‘90s, we had children in Chicago who were dying of measles. I know Detroit is another area where there has been a big problem. So having the technology to solve a problem isn't enough. It really has to be your work and all of our collective work that gets the technology out to people and not just to a few people, but to everybody, and particularly to people in communities where they don't have very good access to healthcare and they may not have the funds to go get private healthcare. We need to get this to everyone.
I’m preaching to the choir here. But the issue of being able to put it in to practice is crucial and to come back to the issue of evaluation, I spent seven years, almost last seven years on the centers for disease control task force, which prepared guidelines for community preventive services. The guidelines are actually about to come out. This is a new task force that is producing a companion to the clinical preventive service s task force. Our job was to think of the community as our patient, as opposed to the clinical, and to develop recommendations based on evidence. In other words, evidence based recommendations of community public health practice interventions. Our first task, actually our first task was define things. There it was interesting, because there was a group that wanted to do it disease by disease. There were some of us who argued that communities don't function that way and that we need to do it by underlying causes of diseases. We needed to address food, diet. We needed to address exercise. We needed to address, yes, vaccination. We needed to address reproductive risk taking, drug abuse, and etcetera. The underlying causes, rather than heart disease. Because if you are looking at food and exercise, you are addressing heart disease. And you are addressing diabetes and other things. So we won that battle.
The next struggle though was to look at what are the methodologies that are acceptable in terms of good evidence, what is the good evidence on which we will base our recommendations? I have to say that Dr. Patton, I'm afraid that since I was almost the only one in that group who did qualitative as well as quantitative, that the method logical emphasis turned out to be heavily quantitative and they are now regretting that. And thinking that we need to go back to more qualitative. But in looking at well done interventions, we would take a pathway. What do we think is a link, say, in a teenager, deciding to take action so that she wouldn't become pregnant either too early or when she didn't want to be or not get an STD? What is the pathway to that? We work out a pathway. We would say, what evidence do we have of interventions in this pathway that work?
We might look at 3,000 articles. Out of the 3,000, there might be 300 that had, that look like they would be worth extracting. We go to be abstract and original articles and if we were lucky we would have five or six articles with good enough evidence on which to base a recommendation. Many of the recommendations in the task force where we know the pathways work are insufficient evidence. Not that it doesn’t work, but that we couldn't find a documentation. That is why what you are talking about today is so important and the work you will be doing is so important. Because if we do intervention, we have to think through the design as carefully as possible, and we have to evaluate them. So that the people who come a long behind us, and also us, in future years, can say, we have got five ways to do this but we know this one works and it is the most cause effective. -- cost -effective. This one, our guts tell us it would be good but the evidence doesn’t show that it works.
I have to tell you one more thing about the CDC task force, and that is that in all of the work we have done, and we have done probably by now close to 200 recommendations, we came out with a lot with insufficient evidence, and of course, some with evidence of what did work, including some very interesting things, like housing and community conditions working in terms of improving health. These are all published in MMWR, and American journal of preventive medicine. We came out with one thing where there was evidence where it didn't work, did not work. Anybody want to guess what it was? Think about, it hasn't been -- I mentioned this in other conferences and somebody said I went to the Website and it wasn't there.
There is a political reason it is not on the CDC Website. Abstinence only. There is evidence that it does not work. But it is difficult for the CDC right now to put some information on the Website. But that doesn't mean we shouldn't be doing evaluation. I want to say that it's particular pleasure for me to be here to welcome Dr. Patton. I've taught qualitative research methods. I hate to think about it. It is probably about 30 years now. His work is so important. It was so important in those earlier years, when in schools of public health, people thought qualitative was absolutely crazy and didn't get it. And to have the articulate, clear work that you have done to present to students and then have students be able to use, to be able to cite to convince other professors that maybe this was the way to do things, has been extremely helpful. I'm really very grateful for the, really your life's work in this area. And very pleased that you could be with us today. Thank you. (Applause).
ARDEN HANDLER: Thank you so much, Susan. I'm very pleased to now introduce Dr. Steve Saunders, to provide another welcome from our state of Illinois. He is the associate director at the Illinois Department of Human Services. (Applause).
STEVE SAUNDERS: Thank you, Arden. I too want to welcome everybody here to this 17th annual conference. The title is translating research into practice. I think all of us would agree in these times, that is a very important topic. We who work at the state level have to deal with this issue on a regular basis. I’ll give you a couple examples in a moment . We are aware of the issues of limited resources. And how to make the best decisions with the best data and best evaluation is a very important issue. Here in Illinois, this year we are facing about a 2 billion shortfall. Last year it was only maybe 2-1/2, 3 billion. I guess we are getting better. But we are facing tough decisions. This kind of conference, this kind of work is valuable for our work. Let me give you a couple examples.
One of the topics we will be hearing about at the conference is the issue of birth outcomes. One of the best enter -- what are the best interventions to improve birth outcomes. All of us have been struggling with this for years. If we had the answer, we would have solved it by now. But we don't. The question is, do we want to do more cessation, improve weight gain in mothers, do we want to reduce stress, do we want to reduce infections, have better health of pregnant women, more pre natal care, all of the above? Which is the most effective intervention, with limited resources? In Illinois, we had a Perinatal planning presses led by our Medicaid agency, our Department of Public Aid, we came up with a nice report. It documents these interventions and effectiveness of them and says they are all great. But now the question is where is the money? Which ones are we going to implement? These kind of research we will hear about today and we can think about will help us with that difficult problem.
The next issue we will hear about at this conference is racial disparities. We in Illinois and I think Michigan as well -- anybody here from Michigan? Is being offered the opportunity from the federal government for a closing the gap grant. They have chosen three or four states with high African-American infant mortality and are giving us resources to address the issue of black prematurity and SIDS. Once again, what is the right intervention? It is not as if we haven't been thinking about the issues for 20 or 30 years. What are we going to do that is new and creative and innovative? A couple sessions we will hear about today is about how to address the issue of black SIDS deaths. In the City of Chicago, almost 85 percent of all SIDS deaths are to black women. So clearly we have a problem here in this city. And I'm sure in other cities, Detroit and so on with this issue. But what is the intervention? The back to sleep campaign has been effective at reducing SIDS. Perhaps that is not the exact correct message in this situation. We have to look at that. What is the best approach to support parents?
We will hear about the issue of whether it is a nursing model of support from David Olds or whether healthy families of Illinois para professional model. This is a debate that has been going on for a few years. That will be an exciting session this afternoon. In Illinois for example, we have a time 1 million healthy families program which uses the para professional model. We put a lot of money into public health nursing. We have public health nurse s working with families throughout the state. What is the best use of this money? I don't know. Hopefully we will find out more about that today.
Finally, there is a session about teen pregnancy, sexuality. We have already heard from the dean about the issues with abstinence. But clearly, there is no one answer to this problem. Is it abstinence? Is it youth development? Is it health education or what? I'm not sure we will have the answer by this afternoon. But certainly it will provoke us to be thinking about that. Finally, this conference will talk about methodlogical issues. We will hear about that this morning. What kind of studies are the most effective? We love randomized control trials and think they are the gold standard. But we know many times that is not possible. It is not possible in the public health work we do. What can we do in lieu of that gold standard to come up with credible evaluation, with good evaluation that will work?
I'll end this discussion by saying, putting all that into perspective and all the research and data is terribly important. But we all know when it gets down to the bottom line, you have to be able to sell these programs politically. You have to be able to work through the political and legislative process. The data is important, the evaluation is critical, but there is that third component which is how do we trans late that data and evaluation into political will, which then can translate into resources, which we can utilize to use these good interventions. Hopefully we will have time to talk about how to use evaluation and data to convince folks that have authority over resources to spend it on this issue, versus that issue. Remember, members of the general assembly are approached daily, multiple times daily with many good causes, many good programs. Everybody says their program is going to save money. If they added them up, there would be no state shortfall because every program savings money if you listen to folks that talks to general assembly members. But we know though they may save money, they don't necessarily save money in the short term. We need to think, how can we translate this information into real work? I'll leave you with that challenge. I want to welcome everybody here today. We are pleased to have you come join us here in Chicago. Looks like it might not be a sunny day, but it will be a very product ive conference. So thank you very much. (Applause).
ARDEN HANDLER: Thank you so much, Steve. That was a great introduction. Now I'm going to introduce our keynote speaker, Michael Quinn Patton who is an organizational development and evaluation consultant. After receiving his doctorate in sociology from University of Wisconsin, he spent 18 years on the faculty of the University of Minnesota including five years as director of the Minnesota center for social research. He received the university's Morris a ward for outstanding teaching when he was there. He is the author of five books, many of you probably have one or two or three of those books on your shelves. I'm kicking myself because I didn't bring my most recent book. I wanted to have it autographed. But including a new edition of qualitative research and evaluation methods. His books have been used in over 300 universities. He also wrote a nonfiction book called the grand canyon celebration, a father/son journey of discovery. He also is a former president of the American evaluation association and the only recipient of both Alvin Myrtle a ward from the evaluation research society for out standing contributions to evaluation use in practice and the Paul Laserfield award for lifetime contributions to evaluation theory from American evaluation association. The society for applied sociology honored him with a -- honored him with a 2001 a ward for out standing contributions to applied sociology. For many of us, he is the guru of evaluation and today we are thrilled to have him speak to us on making evaluation useful. Welcome, Dr. Patton. (Applause).
MICHAEL PATTON: Thank you, Arden. Thank you all for the honor of being part of this important conference. I was especially delighted to be able to participate in a conference with an underlying leadership theme, because much of what I want to share with you involved thinking about evaluation as a leadership function, which is not often how it is thought about. But as we have come to learn how to make evaluation useful, issues of leadership become central as they do in other areas of effectiveness and translating research into practice turns out to have some leadership dimensions to it that I will want to review with you. Part of which begins with a common challenge in issues of leadership, which is, what is your vision for evaluation? The old vision thing.
A couple years ago, a coffee shop opened up near my office. The biker coffee shop, people who ride Harley Davidson motorcycles and big hogs. I got to know the owners, the husband and wife, young couple that started this up, who are bikers. I got to stopping off there every morning getting coffee. I was the only nonleather clad person in the group. As near as I could tell, probably the only nonsmoker in the group. But I enjoyed Connie and Matt and got to know them. I was leaving for a couple weeks work overseas, doing evaluation training, and my last morning there, Connie told me that she was about to go in and have the laser surgery that they are doing now, because she didn't like wearing glasses especially on the motorcycle. I got back from being in Australia and New Zealand and went by the coffee shop and found it closed. On the front door, it said, closed because of vision problem. I was really unhappy to imagine that Connie's laser surgery had gone bad. Two more weeks, three weeks went by. The coffee shop didn't open up. I was driving down the street on that main street where it was located, when I saw Matt coming the other way on his motorcycle. I honked and waved him down, did a U-turn, went up and said, what happened? Did Connie’s surgery go bad? I saw the shop is closed and you had a vision problem sign on. He said, no. Connie is fine. I just couldn't envision myself serving coffee the rest of my life. (Laughter). He had a vision problem. And got in touch with it.
One of the things we learned about evaluation is it suffers from a vision problem. People treat it as a compliance activity, as paperwork, as the pain that comes with funding, essentially one of those things that, like hearing the word about as much as they like hearing words like subpoena and biopsy. It doesn't turn out to be one of their favorite kinds of activities. What we have become convinced of through our research on evaluation use is that the barriers are not primarily methodological or measurement. They are primarily organizational culture and psychological. They have to do with the way in which people think about evaluation, position it in their organizations and programs, and approach the whole enterprise. What I want to reflect on you with is ways that we can change that practice, become more effective at translating research into practice, using evaluation knowledge and evaluation practices, as a part of that contribution. Despite some of the barriers to evaluation use, evaluation has been growing tremendously. I bear the aged honor of having been around at the creation of the professional field. When I started out, there wasn't a profession of evaluation formally.
We date the beginning of the profession to 1975 in the United States, and the publication of the evaluation research handbook. The evaluation research society and the evaluation network were form in ‘75 and merged into the American evaluation association in ‘9 4. The Canadian evaluation society also emerged at that time as did the Asian evaluation society. Evaluation went global. Until ten years ago there were only three major national associations. In the last ten years, evaluation has become a global activity. There are now over 40 national evaluation societies around the world, including new ones in Japan, 20 African countries, 15 Latin-American countries, all European countries. There is a European evaluation society. In Peru, this October, I’ll be there for the launching of the Latin-American evaluation society.
Last year, in South Africa for the launching of the South African society. One of my most memorable experiences was participating in Nairobi in the launching of the African evaluation association a couple years ago. This was especially meaningful to me in the relation to the topic of translating research into practice. My first experience of that challenge was as an agricultural extension Peace Corps worker in Africa in the 1960s. I went -- in the 1960s. I went to a French speaking country in the center of western Africa, worked among about a half million people doing agricultural extension. My wife worked at the local maternity in programs for pregnant women and lactating women. We worked together since food was a critical issue there.
The agricultural program and the maternity programs were combined as we in the field were trying to translate some of the research knowledge of the time into practice. The people spoke a Gurma language. They called themselves human being or person, gorma means human or person. What they called me translated into "white thing. “We got the status hierarchy clear very quickly. I had a wonderful experience there. When I had a chance to return to Africa for the launching of the African society, I was especially hoping that there would be representatives from Berthinea. In fact, there were three one of whom was from the land of the Gorema. I sought him out the first day. We had lunch, caught up on the news from Bertinea. I reminded him that when I had been there, I was called “white thing” and we were the first workers there. Now all the major international development organizations work there. The road has been changed and paved. It has become part of the international community. In that context, I was imagining that perhaps some manner of political correctness might have penetrated to Gorma. I asked him if I came back today would I still be called that. He said, no. We wouldn't call you that now. I said, what would I be called? He said (inaudible) he said "old white thing." So that is how I come to you today, with the miles that suggests and with the limitations of perspective that that implies, to share with you what we have learned about making evaluation useful as a part of the profession.
One of the beginning places of this as a leadership function and the vision challenge is the commitment to making evaluation use. The single biggest barrier that we have documented in a quarter century of research on the utilization of evaluation is that people doing it, people in organization s collecting, evaluating data, often, in fact usually under some mandate, the biggest barrier we find is that they don't expect it to be useful. There is no expectation that evaluation will be used. It is viewed entirely as a compliance function, something that comes with the territory, if you get money, you have to produce evaluation reports by somebody, to get sent in to somebody else. And nobody believes that anybody pays any attention. It is not surprising then that a great deal of what gets generated in evaluations are generated at the last minute, poorly done, sent off to somebody else, and not paid much attention to.
Over a quarter century, I've talked to people at all levels of government and philanthropy and programs of all kinds in the private sector and nonprofit sector. They can't point to many examples of useful evaluations. They don't remember getting feedback, people at local programs send evaluation reports off barely getting them confirmed that they were received, much less that anybody read them or paid attention to them or get any feedback about their quality. At the same time, the American Evaluation Association in collaboration with a number of other professional associations, including public health officials, and professionals, went through a process in the late 1970 s and early '80s, examining the question of what would constitute a high quality evaluation that ought to be used and created standards for evaluation.
You can find those standards on the American Evaluation Association Website, if you Google American Evaluation Association, or go to eval.org. You will find a menu for standards and principles. You will get the list of standards. They are listed in four major categories. There are 30 standards all together. They are in four major categories. The first category is that evaluations ought to be useful, a set of utility standards. The second is they are practical and feasible, done in a way that makes sense to
Practitioners. The third is they be ethical. The fourth is they be accurate. A high quality evaluation is one that is useful, practical, ethical and accurate. What that means is that no matter how method owe logically rigorous your publication is, if it is not used, it is a lousy evaluation. That means I would alter somewhat what Steve said about the gold standards. The gold standard from an evaluation perspective is not a methodological standard. The gold standard from an evaluation perspective is a utility standard. No matter how rigorous a n evaluation is by whatever criteria of rigor one may use, if it is not used, it is a bad evaluation. Not to mention a tragic waste of resources in as you noted a time of squares resources. -- scarce resources.
Utility takes different dimensions. One of the criteria of feasibility in the standards is that data are available in time for decisions. That they are available when decisions are going to get made. That means you have to do it quickly sometimes. Doing a more rigorous study that takes longer and misses the decision point isn't a better study. A lot of this is in some ways counter intuitive. The most common time evaluations are done around the world, indeed international standard for evaluations, is that they are done at the end of projects. Makes a great deal of sense to do an evaluation at the end of a project except for the fact that all decisions about the project will already have been made when it's over. There is no audience for an ends of projects evaluation. All future decisions about whether that project will continue, new design decisions, improvements in it, changes in it, would have been made before the evaluation report is submitted, in end of projects evaluations. So when we think about it from a feasibility point of view and from a utility point of view, it means doing standard diagrams, working backwards to when actual decisions are going to be made for funding, for program improvements, for changes, and being sure that you have the best possible evaluative data available at that time for it to be useful.
There are also challenges of people being afraid of evaluations. Evaluations have been used to abuse people, to control programs inappropriately. And there is the good old general human being fear of being evaluated that we all experience to some degree. This is a big enough problem that evaluators have methodized it in our own version of the creation story, evaluation being a n inherent and intrinsic human characteristic that has been with us from the beginning despite the recent professional formation of evaluation associations around the world.
We trace its roots back to the beginning. We have our own version of the genesis story I'd like to share with you. It goes like this. In the beginning, that is how you know it is a genesis story. You with me? In the beginning god created heaven and earth. God saw everything that he made. Behold, god said, it is very good. And the evening and the morning were the #th day. On the 7th day, god rested from his work. His arch angel came to him asking, god, how do you know that what you created is very good? What are your criteria? On what data do you base that judgment? Just what result are you hoping to achieve? And aren't you just a tad close to the situation to make a fair and unbiased evaluation? God thought about these questions all that day and his rest was greatly disturbed. On the 8th day, god said, Lucifer, go to hell. (Laughter) that was an evaluation born in a blaze of glory so to speak.
Those are central evaluation questions. The field has become complex. There are lots of evaluation approaches and models. There are lots of different evaluation methods. There are different frameworks for evaluation. But they boil down to those basic questions. How do you know that you are doing good? We hear a lot about evil in the world these days. But I find most folks are wanting to do good, anxious to do good, committed their time and lives and careers to doing good. Of course that is one of the challenges of asking evaluation questions. One of the difficulties at the field level is a lot of folks feel when evaluators come a long, asking evaluative questions, how do you know you are doing good, that we are questioning folks' motivation, questioning their sincerity, question whether or not they care. In working with people to adopt an evaluative perspective, to develop an evaluation vision, what I've come to understand in working with folks, that evaluation is only going to be used by people who care. If you don't care about making the difference, if you don't care about effectiveness, you don't care about spending your short time on this earth doing stuff that works, don't bother to evaluate. It is only the people who care. So we have to find ways of framing it as a part of a vision of leadership that makes it feel not like we are questioning people’s motivations but in fact supporting them. That means bringing them inside the logic of evaluation, bringing them inside the values of evaluation, to embrace it as ways to enhance what we are doing. Not become barriers to it. We have learned in our research that there are major tensions between account ability driven evaluation, and learning-driven evaluation.
One of the leadership characteristics of translating knowledge into practice is to deal with that tension. Account ability driven evaluations are done for someone else. Learning driven evaluations are done by people who want to do a better job at what they are doing, who care about doing the most effective work they can do. What we have learned is that accountability driven evaluations will trump learning driven evaluations every time unless there is leadership that keeps that balance in per spective. We have learned that the highest form of accountability is internal accountability. The account ability of your own determination not to waste your time and resources on things that don't work. We have learned that if people do a good job and embrace internal account ability, they can more than satisfy external account ability demands. But evaluation systems that are set up primarily to answer external account ability and compliance demands will die internally of overweight. They will become a burden. They will be hated. They will not be used. One of the functions of leadership is to establish program and organizational culture, to help people understand what is important, to send the messages about what should happen and what the leadership and the organization, whatever level you may be, is paying attention to. That is what leaders do, set the stage. They nurture it. They reinforce what is important. When the leadership and organizations treat evaluation as a compliance and paperwork activity, it is no surprise that other people in the organization treat it the same way.
I not long ago had lunch with the director of a nonprofit agency who receives both government funding and funding from the local united way. I was beginning evaluation process with him working with their staff, taking them through a process of building a positive culture of evaluation. We had lunch. She was saying all the right things. She wanted evaluation to be useful. She wanted to build a positive culture. She was committed to it. This workshop began with her introducing me. As she introduced me, she said, I've got good news and bad news today. Just before this workshop, I received word from the local United Way that they have increased our allocation 1 percent for next year, which is a big increase because a lot of programs are getting cut. So we have done well. We feel good about that. The bad news is they are going to make us do more evaluation. Now, I know it is a pain in the ass, but we will try to make it as pain less as possible. That is why we have Dr. Patton here today to try to tell us how to do this without it becoming too burden some. But the good news is we got the money. She didn't even realize the message she was sending. We just spent an hour and a half talking about building a positive evaluation culture. But when she went into where she lived, evaluation is a pain. It is a n add on. The strings come attached to the money.
Changing your thinking about the vision for evaluation is actually useful in setting up a culture of practice, involving the leaders in the organization, modeling evaluation use. It means spending staff meeting time, going over evaluations, dialoguing about them, thinking about them. Considering their consequences. Not just moving that thing off your desk as fast as possible. And delegating it down to lower levels in the organization. A lot of leaders go to the workshops that train them, the first thing they should do when there is an evaluation assignment is get that sucker off their desk as fast as possible and send it down some where else in the organization. I can tell in a lot of organizations who screwed up that quarter because they have that quarter's evaluation report assignment. That is how people know they screwed up. They get to write that quarter's assignment for evaluation reporting. The stories and images we build around this are important not only within our organization, as organizational culture but cross culturally. Internationally and domestically, a lot of groups have been beat up by evaluation. They feel like they have been studied to death. Some people come in and do things to them. They get promised results will come back and they never come back. One of the great sins we will spend lots of lives living off is promises from researchers they will get results back to people in a form they can understand. The evidence is it almost never happens.
When I was, had the opportunity to go to work in New Zealand, I was working among other groups with the Maori People. The Maori People worked hard to contain cultural traditions and are assertive about not allowing things to be imposed on them. As a result of bad evaluation experiences, they were resistant to engaging in evaluation. I had the opportunity to read some about their culture and their myths before going to New Zealand. As we started working together, I remind them of their own creation story. One of my hobbies is collecting creation myths around the world. Most creation myths actually take somewhat the form of the, some version of the genesis story that I recited a moment ago. That is, that gods and goddesses create the world, stand back and look at what they did and say, we did a bang up job. It is a little self-serving, self evaluation of some of the good ness but the Maori, have a wonderful creation tale of a formative evaluation, a learning oriented evaluation. Which I suggested to them once we talked about it, made them naturally evaluators out of their tradition. The story begins with Father Sky, and Mother Earth, locked in a fierce embrace. They bore their children between them in these fierce embrace. But because of that fierce embrace, no light ever penetrated between them. So their children were born into darkness.
Their children became disgruntled in this darkness and began to plot to push their parents apart, so that the light would pour in, so they would have more space and air. The most powerful of their offspring, who became the god of all living things, needed to join this conspiracy because he was the strongest. They would need his strength if they were going to succeed in pushing their parents apart. He was worried about what the effects might be. But eventually, his nagging brothers and sisters convinced him he added his strength to the others. They pushed and managed to separate the mother and father. They immediately saw that Father Sky was sobbing at the loss of his deep connection with Mother Earth. And those tears became the rain. They saw that mother earth was naked and they were embarrassed for having brought in the light and exposed her nakedness. So Tana set out to clothe his mother by planting trees but having been born in the darkness and had no experience of this, he didn't know how to plant trees.
The first time he planted them, he put the leaves in the ground and the roots in the air and they died. He observed them dying. He realized he must have done it wrong. So then he tried laying them flat on the earth. And again they withered and died. And he realized that he had done it wrong. So a third time, he planted the roots in the ground and the leaves in the air. This time the birds came and the animals came and he realized that he had done it right. The story of improvement oriented, learning oriented evaluation, which the Maori adopted as their model of evaluation which they call the Tana model of evaluation. Trying out things, examining them, seeing what works, discarding what doesn't work and building on the strength of what does work.
One of the functions of leaders is to find the stories, the examples that will help people connect with evaluation. We all engage in it all the time. There is lots of evidence about our not using knowledge, not simply in maternal health, but in our ordinary lives. I recently saw a study of how people buy cars or buy computers. Major purchases. The interesting thing about the study of cars is people's intentions before they bought the car was to look at consumer reports, was the look on the web and compare different cars, look at the safety records, look at the evidence, test drive different cars, compare models, but in the end, they went down to the same dealership they bought their last car from, and spent about ten minutes making this decision. Or talked to one friend or one family member who had strong opinions and followed that advice. Very few people walked the talk of a careful evaluation before making these major decisions. They intend to, just like people in programs intend to use evaluations but time gets away and other things take over. Which basically sends the message that this isn't a priority. To make it a priority, means to deal with the genuine barriers of perception, of vision, of fear, of politicalization.
Dean Scrimshaw referred to political activities at CDC. I was there myself doing training last year and find it outrageous that CDC Websites are being censored of evaluation studies that don’t support policy. Whether you are Republican or Democrat, you ought to be out raged by the censorship of information about what works and doesn't work. But evaluation is a political activity. Knowledge is power. Part of the coming of age of evaluation has been to deal with the power. Feasibility standards of evaluation association call for evaluators to be sophisticated about politics, for researchers to be sophisticated about politics, not to be shocked and pretend that evaluation findings are not politicalized but to create mechanisms that make transparency and balance so built into the system that the findings can't be suppressed. That is a utilization responsibility.
When I was President of the American Evaluation Association, one of my responsibilities as president is pick the theme of the meeting for that year, as you have the conference theme of translating research into public health practice. When I was president, during a president ale election year, they picked politics of the situation, dealing with politicalization. As part of our conference, we have a n essay contest in which people are invited to write a n essay on the topic, the essay question I posed to the membership was given that evaluation had come of age and realized that evaluation is inherently a political activity, without wanting to be tautological, my essay question was, when is evaluation not political? What aspects of evaluation are nonpolitical? The winning essay was short, succinct and insightful. Evaluation is not political under the following conditions, all of which must be met: No one knows about the program, no one cares about the program and no one making decisions about the program, involved in the program, has any power or resources at stake, and no one involved in the program making decisions about the program or otherwise engaged in the program in any way is sexually active. (Laughter). So if your program fits that description, you don't have to worry about the politics of evaluation. Otherwise, it is a political activity.
One of the strategies we have developed to increase the use of evaluation and keep it from being politicized is make sure they have advise recommittees with people from different per spectives, representing different stakeholders groups, that findings would be impossible to suppress. You create multiple and diffuse outlets for getting research findings into the hands of the people who need them. Creating a culture of learning then translating research into practice, using evaluation findings for making a difference begins with the expectation of using the findings for change. And creating an organizational and policy culture where people aren't punished when things don't work out, but is treated as what it is, learning. The only real failure is not to learn. We should expect things not to work.
The private sector understands -- the private sector understands this. We have private sector mantra stuff coming in, about how we should be modeled after the private sector. They understand businesses will fail in the first year. There are 80,000 products introduced in the American marketplace. Only 600 find a regular shelf space over five years. Why then would we expect every single program we would do would work? Doesn't happen with new businesses. Doesn't happen with new products. What they do is evaluate what works and move on when it doesn't work. They don't try to marshal resources to keep products in the marketplace that nobody is buying. But we do. We build stakes around our programs out of our ideology and out of our values.
Being evidence based, as Dean Scrimshaw was talking about, involves more than one kind of evidence. Indeed when we talk about lessons learned, I've become convinced, being at this a long time, that when we talk about translating research into practice, we have to also talk about translating practice into purchase. I come out of the cooperative extension service and the agricultural extension service s which is based upon a two way flow of information. Researchers valuing their knowledge of practitioners and practitioners valuing knowledge of researchers. It is a two way street. It is not a one-way translation, folks. We have been thinking about what constitutes high quality lessons learned. I would submit contrary to the prevailing wisdom, that high quality lessons learned are not simply research-based.
High quality triangulated findings from my perspective, think about the hotel ratings that you see, the restaurant ratings that you see, a five-star restaurant rating, five-star movie rating. What would be equivalent of a five-star finding to in form public health practice? Evidence-based research is one of the stars to be sure. But it is only one of them. Methodologies being inherently fallible, paradigm s within research being powerful, wasn't that long ago to where the best practice was hormone replacement therapy. There is a long list of those things that we used to think were best practices. Research is only one of those stars. Perhaps the most shining, though I think that is arguable, because of those fallibility of paradigm. I treat evaluation as different from research because standards are different. Standards of utility and fees ability are not part of most research.
Practical advertiser wisdom ought to be a star. Practitioners know a lot. Practical advertiser wisdom needs to be valued, needs to be collected, honored, sifted and synthesized. Participant wisdom. Participants know what works and doesn't work. For me, a five -star lesson learned is one that triangulates research findings. People dedicate their lives to studying and thinking about and looking at what works. A combination constitutes a lesson learned, that will connect theory and practice in a two-way flow of communication. How do we know then if research and triangulated knowledge and lessons learned have been turned into practice? We know it because practice has changed. One of the barriers we faced in extension programs which was born as a mechanism in universities for translating research into practice, one of the things that is regularly debated among research is whether they have responsibility for worrying about that translation. If their responsibility is simply to produce knowledge and let somebody else worry about translation.
Seems to me inherent in your theme is an understanding that we share this responsibility for the translation, for the interaction between practice and research, in dialogues that make sense in both ways, in which researchers have to abandon much of their traditional arrogance and jargon and talk to practitioners in ways they can understand what is being said, and practitioners have to struggle with methods and theory to be able to make sense of the research, that we need a collaborative efforts a long these lines. Collaboration s, they are much touted and demanded for almost everything but turn out not to be easy to do.
I was at a youth conference in California a couple years ago that was focused on the theme of collaboration across traditional boundaries and youth programming. One of the speakers compared collaboration to teenage sex. Collaboration is like teenage sex in that everybody talks about it all the time, everybody thinks everybody else is doing it, those who are doing it, which isn't very many, are not doing it very well, despite that afterwards they all talk about how great it was. (Laughter). We have a lot to learn about how to create these collaborative kind of partnerships. But a part of that is having a vision about what it would mean for high quality lessons learned to be turned into practice. It seems to me that what it means is people are doing something different. We can evaluate that.
There is a wonderful piece that I like, that I suspect many of you have seen but perhaps not in this context, that’s in a way an evaluation biography or auto biography of a lesson learned, of what it means to translate something into practice. It is nice and clean and I like to use it as part of a vision of the use of knowledge and learning and practice and research, because it is simple and clean and gets to the bottom line. It is in five chapters. Chapter one: I walk down the street. There is a deep hole in the sidewalk. I fall in. I'm lost. I'm help less. It isn’t my fault. It takes a long time to find my way out. But I eventually get out. Chapter 2: I walk down the same street. There is a deep hole in the sidewalk. I ignore it. I fall in again. I can't believe I'm in the same place. It doesn't feel like my fault. But at least it takes less time to get out. Chapter 3: I walk down the same street. There is a deep hole in the sidewalk. I see it is there this time. But I fall in. It is a habit. But now my eyes are wide open. I know where I am. I know it is my fault. I get out immediately. Chapter 4: I walk down the same street. There is a deep hole in the sidewalk. I walk around it. Chapter 5: I walk down a different street. (Laughter). A part of the vision of evaluation use, of making evaluation useful is a vision of changed practice. That means spending sufficient time with evaluation findings to interpret them and make sense out of them. One of the most difficult challenges in having people use evaluation research, practicing advertise, wisdom, -- practical advertiser wisdom, -- practical advertiser wisdom, that requires interpretation. There isn't a linear interpretation of most findings into actions. That requires interpretation, dialogue, trying things out, figuring out what the meanings are, in the context of what it is we want to accomplish, including dealing with both the outcome implications of those change s and the value implications.
When we are dealing with leadership in evaluation, and I’ve been working with a couple of leadership training programs with philanthropic funding that are actually taking directors of nonprofit organizations and senior managers and government through leadership training, I got into this because what I found, having done years and years of evaluation training, is that usually the people who have leadership positions won't come to evaluation training. They think that evaluation training is methodological, it will be about measurements, it will be dry. So they send their lower level folks and their technical people to those workshops, and then we talk about how to build an evaluation per spective, evaluative thinking into the organization on program culture, to build a foundation of use and they come up to me afterwards and say, I wish my director had been here because I can't do that from the place I'm located.
We learn from our evaluations that leaders wouldn't come to evaluation workshops. So we created leadership work shops in which the contents is all about evaluation, but we never use that word. (Laughter). I have to ask you to keep that secret for my future workshops in this regard. So what we started doing were workshops on leadership and the -- were workshops on leadership. The model we offered was reality testing, results oriented leadership. It turns out leadership is a big topic these days. Last three years there have been 400 non fiction books published on leadership. Any historical figure has a leadership book modeled after them. I happen to like the general Patton leadership book. Others have other choices. There is leadership a la Moses, ala Mohammad, whatever religious figure you like. There is leadership ala Genghis Khan. Leadership ala Henry Ford, Rockefeller. There is authentic leadership, servant leadership, feminist leadership and anti feminist leadership. Top down leadership. Bottoms up leadership. Middle to bottom leadership. Middle to middle leadership. Nowhere leadership. Everywhere leadership. We tell folks the only leadership training they need is results oriented reality testing leadership. Because that is what evaluation is about.
How do you know that what you think is going on is going on? That is the reality testing piece. You have all had lots of social science which is basically the study of how we as human beings distort reality. We have evolved as premier reality distorters. When individuals get out of touch with reality, we try to help them find mental health services. When organizations get out of touch with reality, we call in evaluators. We are the organizational therapists, to try to help organizations stay modestly in touch with reality Let me assure you, think about your organizes, organizations get more difficulties toured views of reality -- get more distorted views of reality than individuals do. How would you know if you were out of touch with what is going on in your organization? How would you know whether or not you were doing good? How would you know if you weren't doing good? What kind of feedback mechanisms do you have? What we know is most of those systems, reinforce people’s belief systems about what is going on without testing those beliefs. We have the challenge of helping people think about what they want to accomplish. And internalize ways of asking those questions. Building evaluation into the process . In so doing, the contribution of evaluation and this would be my final comment before we stop for some interaction, is that we have discovered that when we talk about making evaluation useful, we are only partly talking about using the findings of evaluation.
In the last few years, based upon studying use, what we have learned is that what is often most useful is not just the findings from a discrete evaluation. Those actually get outdated quickly. It is building into people’s ways of thinking in organizations, evaluative thinking. Going through the process of internalizing evaluations' values, creating a culture where people can have honest discussions about what is working and not working on an ongoing basis. Continuous progress improvement efforts, at looking at what is going on, creating an experimenting culture not just experimental designs. Where people are openly talking in staff meeting about how they know what is going on and sharing reality testing. And speaking knowledge to each other. Building in data that they pay attention to.
When I was in South Africa, we did some work around metaphor s as I'd done in New Zealand trying to find local ways to make a connection to evaluation that would make sense to them. As I laid out some of the general evaluation principles, people began coming up with wonderful metaphors to connect evaluative thinking to their own cultural moat eave s -- motifs, in their local environment. Let me share a couple of those with you for use in your own world, your own cultures, your own organizational cultures. In the big animal areas of Africa, in South Africa, there is a large bush where a small trio or a small tree, call the buffalo thorn. It is a succulent plant, one of the favorite of giraffes. If two or three giraffes descend upon a single tree and start eating it quickly, the plant produces tannin so quickly, it gets picked up in the air and gets carried to surrounding trees that immediately start producing tannin. To protect the orchard. To protect the group of trees. That became for my South African colleagues a metaphor for them of having early warning systems that the environment has changed. The clients have changed. The participants have changed. Something is going on out there that they ought to pay attention to and change what they are doing within their practice. They came to view that as natural in highly evolved plants.
Another example, I had the chance to visit South African vineyards, they are prying high quality South African wine in the cape town area, nestleed under the green mountains there, there are places you can stand and see that lovely greenery against the mountains. It is speckled with roses. Because around each of the Vineyards they planted white rose bushes. It is beautiful. I was getting a tour from the head of one of the vineyards and commented on the aesthetic beauty of the roses. He said, I'm glad you find them beautiful but that is not why we plant the rose. We plant the roses around the vineyards because anything that will attack the vines will attack the roses first. Every morning we look at the rose petals. If there is fungus or disease or insects on the roses that tells us what we need to do to protect the vines. That is that their early warning system to find out what is going on. That is their data system. We need ways of listening to participants, getting feedback from them, of hearing what happened in the staff, so that the research and practice connection is a two-way street. Researchers are hearing what is going on in practice and what practitioners are experiencing with participants and beneficiaries in the programs and those participants and beneficiaries need to be able to hear and understand research and evaluation findings and have a vision of dialogue in creating knowledge and lessons learned. Let me stop with that. And invite your questions and questions by E-mail from the webcast participants.
ARDEN HANDLER: If you have a question, please go to the mike. The closest mic. Is there a mic in the back? Come up here.
>> Thank you very much for telling us the Maori creation story. I'm an anthropologist and an evaluator. I worked in the Pacific also. One thing people in the Pacific said about the Maori, they are philosophers. They reflect themes that we have seen in other Pacific cultures. For me one of the important things about that story is it is about relationships. As an evaluator, one of the things I struggled with is relationships, because it intersects with politics. What people talk about as political problems are often relationship problems. In creating utilization focused evaluation, it seems to me we have to go from finding s that only reflect what exists, and what has been going on in the programs, to recommendations about what could exist. That is where sometimes I've run into difficulties because a lot of times people’s expectations about evaluation are that they want you to confirm that what they are doing is making a difference and what they are doing is the right thing to do. They want the evaluation to justify to the fun der they -- fund er they should keep getting funding. So how as a relationship problem can we help move people to thinking about what they could do differently rather than just thinking about how to justify keeping on doing more the same thing?
MICHAEL PATTON: That is a wonderful set of questions we could devote a lot of time to. I resonate to the relationship emphasis. Indeed, as you are building evaluation relationships with both researchers and practitioners and participating in the translation process, negotiating relationships and being explicit about them and attentive to them and evaluating them is a part of that interaction. One of the ethical principles that comes to us from anthropological field work is importance of reciprocity in relationships. I find that a n important guideline in working an evaluation relationships, is to look at what the different partners in evaluation are going to get out of it. What are the mutual benefits? We stretched the notion that people ought to participate in research for the greater good about as far as it can go. We need concrete reciprocity for people involved in participating in evaluations. Of course, there are some new models of evaluations that have emerged, many of them coming from developing countries, including the South Pacific.
You have got somebody featured in the conference, participate Tory firms of research that break down distinction between who is researcher and who is the research that makes them partnership s in collaborative research. It is usually harder for researchers to adapt to norms than participants, who can get excited about doing research. In the work that I do, as an evaluator, I describe myself as an evaluation facilitator rather than the evaluator, because I don't make judgments about whether the program is good or bad. I don’t make the recommendations because I'm not the primary stakeholder for the program. What I do is bring together the people who are the primary stake holders and facilitate an evaluative process for them to make those decisions. Because they have to own those judgments and they need to own the recommendations.
So for evaluators to learn to be facilitators and for participatory researchers to learn to be facilitators are not typically skills you learn in graduate school. Getting a PhD doesn't make you a good teacher, we have learned in graduate school, although universities regularly ignore that finding. And having a PhD in something or master s degree also doesn't make you a good facilitator. There are facilitation skills that people have to learn if they will value those reciprocal kind of relationships. A gate deal of evaluation training these days involves learning facilitation skills, learning people side of the business. Not just the technical side. When one is dealing in facilitating recommendations and success, two quickies around trying to change that view of things. Given the learning emphasis throughout my presentation this morning,
I work with folks to try to create a definition of success that includes and indeed centers on learning , not just confirming. Because there is catcha about being in a learning organization, if a program can position itself as a learning organization and the leadership positions at that way, that actually helps the funding process. To be a learning organization, it means what you have done is been able to say here is the things we are doing that we have been able to document works and we are strengthening those. And here are the things we found out don't work well and here is how we are changing those. If you don't have anything to change, you can't demonstrate that you have learned anything. So getting people's heads around that and positioning themselves in that way is a piece of that action.
The final piece I'd say about recommendations, very quickly, to give you, provoke additional thinking, is that I got interested in the problem of recommendations out of data. I think it is the hardest thing to teach, hardest thing to work with people on. I came to believe that a part of that challenge is that because evaluations at the moment they are produced are essentially historical documents. They are about what's happened in the past. In order to do a recommendation, you have to become a futurist. The remissions are about futuring. I got involved in futuring, the world futures society, future techniques. I'm not talking about the National Enquirer version of futuring. There is a scholarly version of futuring. They run scenarios, which means instead of having singular recommendations, we ought to do X, Y or Z, a scenario based approach to recommendations says if the world unfolds in this way, we ought to do X, Y or Z. If the world unfolds in this way, we ought to do A, B and C. If the world unfolds this way, we ought to do something else.
One of the results of the good evaluation done collaboratively is to hold before people the things they ought to be paying attention to, going forward. Not simply the programmatic actions they ought to take. It is a contingency form of evaluation, a set of if/then statements, rather than you ought to do this come hell or high water. It sensitizes and builds into it a learning framework. The if/then framework says we got to pay attention to what is going on. Evaluation in that framework is not just correcting weak nesses, it is also adapting programs to new conditions. Clientele change rapidly. The world changes rapidly. Even effective programs, evidence-based practice programs, have to pay attention to what is going on because what we know from 30 years of evaluation is effectiveness deteriorates over time if programs don't continue to improve. Models don't stay stagnant. Participants don’t stay stagnant.
ARDEN HANDLER: I have a question too. We will take another one. You might as well line up. We only have ten more minutes. There is only one mic. I was intrigued by your comments about how businesses, 90 percent of businesses collapse. We expect that in the private sector. We don't expect that in the public sector. The thing that is difficult for me and I’m thinking about the healthy start example nationally is the whole notion of programs that we deal with, usually the most vulnerable populations. It is hard when you have your health accountable -- you are held account ability for reducing infant mortality which we know is multi faceted in terms of what causes it. When the Feds say, you didn't reduce infant mortality by 50 percent, we will shut down this program. Or they say the opposite.
We will not shut down the program, because good things we are doing. They are asking us first to look at ridiculous outcomes. If we base it on what you said, we would shut down healthy start. But we know is it not possible for the case management for 20 women to make a difference in infant mortality in these communities. The youth comes they -- outcomes they ask us to be accountable is ridiculous. If we went on your approach, we would be shutting down those programs which may be the only thing going on in those communities. If communities were assured money would be coming for something else, maybe they would be willing to say this is not working. When people are afraid this is the only resource in their community to get women into pre natal care, the only outreach program going, whatever it is, they are not willing to say, it didn't work. They are going to cover up, make fake data, whatever it takes to make sure the dollars keep coming to a disenfranchised communities. In the U.S., where we have in equities in economics particularly, I think programs are protective because there is nothing else happening. We have to deal with that too. That is partly why we keep on doing maybe programs that aren't that effective.
MICHAEL PATTON: A lot there, Arden. I'm going to give you a context here, since the purpose of bringing folks like this together is to have the tough discussions. I'm going to suggest a pro vocative answer and leave town. (Laughter) . But part of what a group like this has to decide is how long you keep playing the game of doing more with less. If you follow basic reinforcement theory, what is the wise thing for policy makers to do if you keep pretending to do more with less. Give you less. A part of what evaluation has to do is document that cutting programs beyond the bone means that kids are dying. We have to be loud about that. We can't just say, we have fewer resources this year.
One of the things we have, know with 30 years of evaluation, it is within our reach. It is with many of the books that show that effective programs work. Effective programs require resources. In the long term, in Arden's passionate presentation, is about trying to get by in the short term in the hopes there will come a better day . But we have to ask ourselves, by delivering programs that have so few resources, that they are barely able to do anything on the margin, if communities are better off, then clearly getting in the press, we said we can't do anything meaningful with the dollars we give and rather than pretend that there is a program here, we are closing it down because there is no program here anywhere. We don't have enough to do anything meaningful. We can't serve enough people to say that there is a program. We have to take folks off the hook at some point and start calling it like it is. There isn't enough there. Because the public hasn't figured that out yet. I was at a program with human service providers a week ago in dull Out, where hard -- Duluth, where hardened cynical old timers were in tears about people dying on the streets. But they are still running a program for home less and the public thinks there is a program for home less. When they have to turn away 90 percent of the people in the Minnesota winter. And people are dying out there. That story hasn't gotten out. Evaluation ought to help tell that story.
Keep in mind that the ability of people to short term the results, some of this is a means/ends problem. What gets measured gets done is one of the mantra s of performance measurement. When there are unreasonable outcome demands, over promised programs, politically, when you are held accountable for things that don't make sense and you go a long with that and play that game, you essentially accept those unrealistic criteria and set your programs up for failure. You are setting your programs up for failure by outcomes that are outrageous from the beginning, because those will come back to bite you. Evaluation is a powerful tool for good and for learning. But like any tool, it can be used for harm. Performance indicators have been used for harm. The mantra what gets measured gets done means that if you measure the wrong thing, you do the wrong thing. If you buy into the wrong outcomes, you will do the wrong thing. We have found lots of pathological reinforcement cycles in programs, where they get rewarded for the wrong things. I looked at foster care programs, where parents get paid more money for taking care of children who are more severe. That makes a lot of sense. Except it reduces any motivation to help those kids function higher, because they lose money if the kids get better. The same thing goes on in nursing homes where they get reimbursed for taking care of people who are less functional. Therefore, they have an incentive to have nursing home residents be less functional because they get more money to have people that are less functional. Understanding these reinforcements cycles in our programs and the way in which evaluations are used is a part of that , of what is going on. Keep in mind, we are coming off with the Iraqi war, the mother of all examples of distorting information to prove a predetermined conclusion. And the Iraqi jail scandal is also an evaluation story. It is a means/end story.
When you read between the lines, apart from the abuses, the logic model, and many of you know logic models, the logic model was we will get better information by torturing people. That is the ancient torture model. That is what was being practiced there. If it had worked, it turns out they didn't get good information. You don't get good information by torturing people. You don't. It doesn't work. It has been demonstrated time and time again. It does not work. But they continue to use it. Partly because they are inherently sadistic perhaps but part of my response to Arden and it is a terribly important question, I hope that you will all be discussing it, is when do you say, enough is enough? When do you put your result out there and say the resources are so puny, that we are doing harm giving the public the impression that there is a program here? And that you document the downsides of that pun y funding, because we know 30 years of evaluation, that effective programs take resources. Intense interventions work. Milquetoast interventions don't work. We are all culpable if we pretend that the stuff that is documented to work and we are not able to do because of resources, we do under a Band-Aid kind of mode. You have your own ethical decisions to make about that and as a profession to make decisions about that. That is a right issue to be tailing about. I hope you spend time during the conference trying to sort that out.
PEG DUBLIN: My name is Peg Dublin. I'm involved with a quality improvement initiative in healthcare. I want to get your opinion about this, these rapid cycles of change that are part of the quality improvement movement now. Plan, do, study, act. I'm sure you are familiar with that. I wanted to get your sense of what are the strengths and weak nesses of that approach to quality improvement.
MICHAEL PATTON: That is a great question. I'm going to, with our time short, turn it back to the importance of evaluating evaluations and evaluating quality improvement processes. I don't assume evaluations are inherently good. In a lot of place s, I argue folks ought not do evaluations. They are not going to use them. They know what they are doing isn't working. They don't need additional evidence for it. They will use it for public relations. The quality improvement cycle have intended outcomes. They are supposed to accomplish some things. Sometimes they do and sometimes they don't. It is important to look at and build in to the dialogues about those, are they accomplishing what they are supposed to accomplish. Are they achieving what they are supposed to achieve? And be able to talk realistically about that. The difficulty with any of these models, including evaluation models, including public health models, maternity care models, early childhood family education models, the models themselves are typically not inherently good or bad. What we find is that most of the success depends upon how they are implemented, it is the people involved and how they approach it. There was a time folks hoped we would develop models that were people-free. Some of you may have remembered colleges of education in the '80s. There was a lot of attention to what they call teacher proof curriculum. Teachers took that as a challenge and demonstrated you could not develop a teacher-proof curriculum. We are looking for people-proof quality improvement programs or people-proof maternity programs. What we find is the way these things are done, who is involved, what they understand the culture is, what the motivations are, how they interact. The relationships piece. Then is it those qualities that determine whether or not the stuff works. It is not the model that works.