Doing a bit of research on Spending Challenge the other night, Stephen Whitehead alerted me to his excellent post on the subject: Three lessons from the Treasury’s Spending Challenge fiasco
The article touches upon three important concepts (great analysis, make sure to read it in full):
- Asking the right questions
- Collaborative brainstorming
- Objective-driven public participation
I wanted to highlight the section that deals with the crowdstorming aspects of the Spending Challenge online consultation:
2. Good ideas are like Lego: some assembly required
In the Spending Challenge, ideas are like property: you’ve got yours and I’ve got mine. Every suggestion made, no matter how bizarre or unworkable, is a complete, perfect whole. It can be commented on, rated, but never improved. And in the end, each idea is in competition with every other for the meaningless title of ‘highest rated’.
In the real world, that’s not how things work. The best ideas are shared, discussed, revised, merged, and split. They grow and evolve as people work together. We brainstorm, we collaborate, and at every stage our ideas get better.
The virtue of holding this kind of conversation online should be that it pools peoples’ [sic] knowledge to produce a whole that is greater than the sum of its parts. [...] But collaboration around ideas – not the ownership of ideas – must be built into the process from the very start.
To me, this is one of the central criticisms of the way large-group online idea generation efforts in the area of policy making have usually been designed to date. It applies to most if not all of the prominent “brainstorming” contests we’ve seen over the last couple of years (e.g. on Change.gov and during the Open Government Dialogue).
Here’s what I wrote back in January 2009:
Having looked at a number of large-scale online input gathering efforts recently (Change.gov, Rebuild the Party, ObamaCTO and others), I believe what’s missing in order for any of them to scale in meaningful ways are processes that allow the participants to spend some of their energy on collaboratively improving (e.g. to review, revise, clarify, categorize etc.) and synthesizing (e.g. to de-dupe, group, merge, summarize etc.) the massive amounts of content that they create.
There are other issues that remain to be solved, for example:
- Who are the participants and how well do they represent the public? Ranked lists of ideas are relatively easy to generate online but may fail the legitimacy test.
- How will ideas be evaluated and by whom? Evaluation process and evaluation criteria should be established and communicated before the brainstorming begins.
- How will participants know their ideas have been reviewed, considered or implemented? At last count, there were 19,630 ideas and 34,568 comments. Given these massive numbers, it’s not unlikely that a lot of necessary follow-up communication won’t happen.
- How do we know if a brainstorming exercise was a success? Once again, criteria and success metrics should be defined upfront rather than after the fact.
I have yet to see a project that has addressed these challenges (and well). If you’re aware of any examples, please leave a note in the comments. Until then, I say we’re just not quite there yet…
Related posts:
{ 7 comments… read them below or add one }
Tim – I raised the question of supporting cooperation at a Federal webinar on online challenges last week, and someone pointed to the Netflix challenge. In particular, Netflix used a leaderboard – http://www.netflixprize.com/leaderboard – so that high-scoring teams could find one another.
It seems to me that an even more important feature of the Netflix Prize – http://en.wikipedia.org/wiki/Netflix_Prize – was that teams were competing as much against “the course” – beating the existing algorithm by at least 10% – as against each other. (And indeed the winning team was a combination of three of the originally separate teams.)
So, encourage cooperation by (1) pitting contestants “against the course”; (2) making “the course” difficult, so it’s possible that no one wins; (3) using a leaderboard to highlight leading teams; (4) running the competition long enough that contestants have time to team up; and (5) having a prize worth winning. (Of course, there are other ways to encourage cooperation as well.)
In a simple “vote up ideas” contest, (1) and (2) might be handled by requiring that the winning ideas garner votes from at least, say, 5% of the voters.
@Chris
Thanks for chiming in!
True, the Netflix Prize is widely regarded as one of the success stories of corporate crowdsourcing.
However, their objective was the improvement of a mathematical algorithm, not the creation of public policy. Two key differences:
1. The competing teams’ algorithms could be compared and ranked applying objective criteria. This is not the case with policy ideas.
2. The leader board may have worked well for Netflix, but looking at previous efforts in the policy arena, exposing the leader board while the voting process is still ongoing always seems to lead to serious bias favoring the early leaders, at least based on what I’ve been able to observe (see http://www.intellitics.com/blog/tag/bias/).
I agree, though, that there are still many options that haven’t been fully explored. More iterative approaches that include the opportunity for merging similar ideas as well as more creative voting models come to mind, for example.
Tim – You’re right about #2 and the early leader bias. Further, the Netflix prize is really an example of “cooperation within competition” rather than cooperation outright (e.g. brainstorm idea piggybacking), since not every team (indeed, in this case, only one team) could win, per the rules.
Re #1 – (a) It’s not inherently impossible to rate policy ideas in a way that’s acceptable to a broad array of people (fiendishly difficult, yes; impossible, no). The US Congressional Budget Office scores legislation for financial and other impact, so, if we agree to go by their estimates, we could have compared the various health care reform proposals “objectively”. (Big “if”, of course.)
(b) Cooperation does not require objective rating. If “winning” ideas must first garner at least 5% of the votes, say, a kind of cooperation and blending of higher scoring ideas might be encouraged. (This is in fact what happens in Congress – factions come together in all sorts of odd groupings to pass legislation.)
Agreed, rating and ranking of policy ideas is doable, and the participants of an online consultation can be part of that evaluation.
However, here’s what usually happens (in no particular order):
* Voting takes place prematurely
* The simple thumbs up or thumbs down voting mechanism provides a very limited form of evaluating idea fragments
* Evaluation criteria are never clearly defined nor agreed upon upfront
* The conversation focuses on (half-baked) positions rather than underlying values and interests
* Lack of demographic information makes it difficult to put participants’ collective choices in perspective
At the end of the day the question is: what do the results mean and what can and will the decision maker do about them? Most projects I’ve seen don’t seem to think that part through to the end.
Agreed. The online challenges have their shortcomings.
But there are also a disruptive phenomena and I’m wary of being the “they” in Gandhi’s ““First they ignore you, then they laugh at you, then they fight you, then you win.”
So I’m aggressively open-minded about how online challenges are, currently and flawed as we see them to be, nonetheless useful to their sponsors, and I’m preoccupied about which of the flaws *we* see would be seen by their current and future sponsors as most useful to address. (E.g. Grandpa tells me that mp3s don’t have the audio range of his CDs, and I don’t care – Coldplay sounds fine to me. Grandpa tells me that DRM-encrypted mp3s could be “repossessed” by a server failure or when I copy my music to a new iPod, and I get that it’s an issue.)
I’ve been looking at this mostly from a public participation standpoint.
Public participation is about sustainable decision making. When I look at projects such as the Spending Challenge, I ask: Is there a decision at stake, first of all? If so, what is the decision? Who is the decision maker? Where in the decision making process are we? What is the timeline? What level of impact will participants have? What are the project objectives and what defines a successful outcome? Etc.
More often than not these questions do not get considered, hardly ever are the answers made explicit (which is a problem to begin with). We can assume, though, that the convener’s core objective is usually to make better decisions. Wether an online consultation is useful depends on how well it achieves that objective.
There may be other aspects of usefulness, but my research is focused on how the tools in combination with the (not always intentionally designed) process can help or hinder good public participation.
I think what’s sometimes forgotten is the user perspective. Do I want to put up the extra effort to revise ideas, or work on better text with others, or even do some research? Consider the amount of work that would go into improving the quality of ideas in such activities. And then contrast it to the possible impact. It does not add up. Not only do most ideation projects on the web today look (I say “look” because not all of them are) like token participation projects, but also the invidiual user may get the impression that his contribution is too minimal and will drown in the masses. These projects are not like wikipedia, their content is not being constantly improved and then public accessible. These ideation projects are short lived, not very well publicised, hidden, and their impact often negligible. As long as these problems are not mitigated, all this work that is being put into them on the micro level is not worth it. Yes I’m playing devil’s advocate here, but too often we paint a way too rosy picture of all these platforms. I wonder how many decisionmakers always know of these activities that the institutions they belong to are carrying out…. does the 80 year old congress member who has not used email once in his life care about something that was produced on a thing called a website? etc…
{ 4 trackbacks }