A couple of weeks ago, the UK government launched data.gov.uk, a new website designed to make government look a little more transparent and be a little more accountable for results. Data.gov.uk follows on the heels of data.gov, its US counterpart. Both sites actualize the belief that information is power; that more information will drive innovation, increase citizen engagement, and improve decision-making.
It’s hard to argue that putting more information into the hands of citizens is a bad thing.
It’s easy to argue that the quality of the information matters. By quality, I don’t just mean the completeness or consistency of the data. By quality, I mean the kinds of questions the information can help us to answer. Questions like: Do services improve people’s outcomes? What counts as an outcome? How can we capture wellbeing? How do people define concepts like health and safety? What’s the strength of relationships in families and in neighborhoods? How fulfilled are people in their jobs?
Education league tables, health care wait times, employment statistics and crime rates don’t get at these things. They don’t capture people’s sense of where they are and where they want to be going. In our experience, measuring people’s skills, values, motivations, relationships, and ambitions is the key to doing social innovation with social impact. That’s because until we measure things that are meaningful to people—that help them to see the whole of their lives—our services and our systems will keep doing the same reductionist things.
So, how can we start to measure more meaningful things?
I am asked this question a lot. We design and test new metrics over the course of our prototypes, and have learned good measurement is always a work-in-progress.
One way to define good measurement is to contrast it with poor measurement. Poor measurement perverts practice. John Seddon, in his book Systems Thinking in the Public Sector, shows how most targets and metrics inspire a stream of senseless activity: they focus so much on process they lose all sight of purpose. We can all muster up examples of public services where form-filling overtook problem resolution. We recognize the inanity of coupling performance with protocol, rather than with real results. And yet most social innovations don’t do much better. They look at constructs like access to services or user satisfaction, rather than at changes to people’s behaviors or system’s behaviors. Static measurement is poor measurement. It breaks the relationship between ‘means’ and ‘ends’ and focuses us in on the wrong thing.
OK, you say, then what are some of the right things to focus on? Here’s the set of principles we are currently using to decide what counts as good measurement.
Good measures are:
(1) Useful
They start from the premise that users own any data they give, and as such must convey something that is useful, usefully. Useful data comes from the perspective of users and not just institutions. Something like an education league table communicates more to the institution than to the individual user; the league table doesn’t tell the student much about the quality of the teaching and learning they experience; their own strengths or weaknesses; or their progress over time.
(2) Actionable
Good measures tell us what we could do differently. They offer up explanations for things; they don’t just count or describe things. Knowing how many people received a service gives us little insight into the value of that service, or the features that contribute to better or worse outcomes. Indeed a good measure gives users, practitioners and even policymakers some insight, or at least a hunch, into the best solution set. Take a measure like school tardiness. Knowing that a handful of students are late to school most days does not give us any clues as to why they are late: does it have to do with the time they get up, the time they go to bed, the speed of their walking, taking a sibling to school, not wanting to come to school, etc.? Until we know something about the behaviors and attitudes underlying the thing we want to change, real change will remain hit-or-miss.
(3) Purposeful
Good measures combine to tell a bigger story. They help us to test our theory of change. In a previous blog post, I described how we use theories of change or logic models to help us come up with new interventions. So building on the prior example, if you were developing an intervention for school tardiness, not only would you want to change students’ behavior, you would also want to change the things that contribute to students’ behavior. Maybe it’s the lack of an alarm clock; or a job that runs late; or bullying at school. These things are called determinants. We try to measure them, along with the core behaviors and attitudes.
(4) Ecological
Good measures are multi-layered. They don’t just give us insight into what’s going on at an individual level, but at a relational and structural level too. Good measures, in other words, capture context as well as behavior. Here, we draw on the work of Urie Bronfenbrenner who is pretty famous (in geeky academic circles) for his model of ecological development. That model says that our behaviors are not only shaped by our immediate environments (e.g. our homes, our schools, & our workplaces) but by the interactions between our environments, as well as by the broader social, cultural, and economic context. That means if we want to solve something that is going on at home, we may need to look outside of the home, to the workplace, and to the relationship between the workplace and the small business regulations government recently put into place. Yup, it’s complicated. And while we can’t always measure everything (nor would it be cost effective to do so), we can at least widen the viewfinder.
(5) Positive
Good measures treat people as part of the solution, not the problem to be fixed. We start by looking at people’s assets—their strengths and resources—not their deficits and needs. This doesn’t mean we look at the world wearing rose-colored glasses. In many communities, the amount of need is staggering. But wherever there is need, there is also incredible resilience. We think by capturing how and why people are resilient, we can build more of those things up, while reducing the things that stand in the way. In other words, we try and measure people’s capacity to bounce back and to live well. We’ve found that capacity measures acknowledge barriers, while focusing in on the things we can enable.
This list is just a start. Over the coming weeks and months, we’ll add to this list, offer examples, and link to some of the articles and books that have informed our measurement approach. We’re eager to hear what you think, and collate your examples of good and bad social impact measures. What is missing from our list of principles? What measures can we put in place in the short-term to indicate whether we are moving in the right direction, and what measures are better suited for the longer-term? Do you know of any examples of systems that regularly collect qualitative data, at scale? When should we not measure?
Comment from Triachy Press:
http://triarchypress.blogspot.com/2010/02/ecological-thinking-systems-thinking.html
Tags
PLEASE SHARE You're free to take what's ours, use it, improve it and make derivative works under a Creative Commons Attribution 3.0 License© Inwithfor 2009-2011 Worldwide
SAY HELLO We love to get your comments and hear about new things. Send us a mail.
Tweets