Tag Archive for 'changegov'

Re: Grading WhiteHouse.gov

Published by Tim on March 24, 2009 in E-participation and Public Participation. 2 Comments Tags: , , .

44: The Obama Presidency, one of the Washington Post’s blogs, today came out with a new monthly feature where they’ll have a group of five experts (for today, that’s Craig Newmark, Andrew Rasiej, Ellen Miller, Jon Henke, and David Weinberger) examine the new WhiteHouse.gov website: Grading WhiteHouse.gov

Excerpt:

For all the innovations of Obama’s WhiteHouse.gov — yesterday, officials announced that it will distribute tickets to the Easter Egg Roll online — online observers, a sometimes prickly, often exacting, let’s-get-ahead-of-the-curve bunch, are left wanting for more. Take the issue of generating comments. Allowing comments on blogs is a given, nothing more than an online SOP. BarackObama.com and Change.gov allowed comments. But WhiteHouse.gov doesn’t — at least not yet.

To which I left the following comment:

Citizen engagement (in the form of public participation) covers a whole range of activities from merely providing citizens with useful and timely information, to soliciting citizen feedback, to collaborative drafting of policies, and last but not least all the way up to granting citizens certain decision making powers.

First and foremost, this is about process: Where can participation be helpful or required, and to what degree? What promises are being made to the public at each level and phase of public participation and how can the organization leading the engagement effort make sure these promises are consistently being kept? Only then does the question of tools come into play.

Anyone serious about public participation must get these basics right for it to achieve the desired outcomes.

With that in mind, I seriously doubt that simply turning on comments on the WhiteHouse.gov official blog would qualify as meaningful participation. Worse yet, in some cases it might even be counter-productive to quality citizen engagement.

The experiments we saw on Change.gov were definitely a step in the right direction. However, from a public participation standpoint there were many best practices the transition team did not yet manage to adhere to. Moreover, none of the tools that were used on Change.gov (IntenseDebate, Google Moderator, Salesforce Ideas) were really built to scale (much less in a public participation environment), and they all struggled with the massive onslaught of user contributions.

So rather than getting impatient with the new administration, my advice to them would be to address the participation piece with great care and caution and to innovate one step at a time. Identify the most promising use cases and work your way up the ladder of public participation. Definitely continue in the spirit of experimentation that was visible on Change.gov, but make sure you don’t fail too badly too often as the participants’ trust, once broken, will be hard to recover.

For all I know, the current linear models of commenting on the web (be it threaded or non-threaded comments, with or without ratings, advanced sorting etc.) do not scale. If the activity we’ve seen on Change.gov is any indication, the WhiteHouse.gov web team might be well-advised to hold off on any general roll-out and only use comments where they absolutely don’t have any better alternatives.

Finding Appropriate Evaluation Criteria for Participant Input

Published by Tim on February 17, 2009 in Research. 0 Comments Tags: , , , , , , .

A common approach to trying to surface the most relevant, highest-quality or most agreed-upon items out of a large quantity of content is to allow participants to rate each other’s contributions and then expose the highest-rated items in a “most popular” list. Very often, a simple binary, up-or-down rating mechanism is used for this purpose.

The three input gathering tools used on Change.gov all offered such mechanisms:

  • Join the Discussion: Threaded comments + “Thumbs up, thumbs down” comment ratings (screenshot, I’m intentionally leaving out the reputation piece here, though that certainly serves as another filter)
  • Open for Questions: Posts only + “Good question? yes/no” post ratings (screenshot)
  • Citizen’s Briefing Book: Posts + non-threaded comments + “Vote up, vote down” post ratings (screenshot)

These rating mechanism look fairly straightforward, though from the way they are named it is not entirely obvious which evaluation criterion each one of them refers to:

  • “Thumbs up, thumbs down” might rate contributions based on relevance, quality or agreement
  • “Good question? yes/no” looks like it’s going after relevance or quality
  • “Vote up, vote down” seems to filter based on agreement (in the context of an idea contest, at least)

It may not always matter, but strictly speaking these three (relevance, quality, agreement) aren’t identical. There may well be occasions where it’s worth differentiating between the three or at least be more explicit as to which one is the intended criterion.

More interestingly, though, there’s room for other, less commonly used criteria as well. If we assume for a second that all input types are not created equal, then surely there may be more appropriate evaluation criteria that can be applied.

Below are a few examples of various input types as well as potential evaluation criteria that might serve well in an early, fairly general phase of an e-participation project:

  • Question: “I share this question (and would like it answered)”
  • Story: “I have had the same experience” or “I know someone who has had the same experience”
  • Resource: “This material was helpful”
  • Idea: “This idea should be investigated in more detail”

Note that negative ratings are being avoided here (it’s really more a flagging mechanism than an up-or-down vote).

More to think about:

  • Other combinations of input type and evaluation criterion are entirely possible, of course.
  • Other rating mechanisms (e.g. 5-star voting, 0-to-100-percent sliding scale etc.) may be more helpful.
  • And finally, negative feedback isn’t always a bad thing: it just depends on where in the process it’s applied, how well it’s introduced and supported (e.g. by community ground rules), how much group cohesion and trust there is among the participants etc.

Again, it may not always be necessary to go to this level of detail, but at the scale of participation we saw on Change.gov it might have provided another much-needed step towards transforming the amorphous mass of unstructured participant contributions into a more meaningful summary.

Structuring Participant Input: Dynamic Facilitation, Brainstorming

Published by Tim on February 15, 2009 in Facilitation and Research. 4 Comments Tags: , , .

While I was assembling my off-the-cuff analysis of input types on Change.gov, I felt compelled to revisit two existing facilitation techniques that help guide participation by adding to the process the kind of structure that I believe could work very well for large-scale efforts.

First, Dynamic Facilitation, a method I first learned about at the 2006 National Conference on Dialogue & Deliberation in San Francisco, CA (emphasis mine):

What is Dynamic Facilitation?

Dynamic Facilitation is a natural way of facilitating that works well with people addressing difficult issues about which they care deeply. Rather than asking them to limit themselves — to hold back their emotions, to stay on the agenda, or to follow the process — the dynamic facilitator frames the conversation so that all comments are helpful and productive. He or she establishes a “zone” of thinking and talking known as “choice-creating,” where shifts, insights and breakthroughs are frequent. The process allows ordinary, untrained people to address difficult issues and reach consensus solutions that are better, faster and which have more support than traditional means of “consensus-building.”.

…

The dynamic facilitator starts by helping people determine an issue they really care about, whether it seems solvable or not. Then he or she helps them to say what is on their minds and hearts. To do this she uses four flip charts for creating lists of: Solutions, Problem-statements, Data, and Concerns. A fifth chart of Decisions is added as group conclusions emerge.

Here, the participants’ input is discovered, captured and structured using content categories or input types. And while Dynamic Facilitation is intended for small groups only and relies heavily on the facilitator role, I believe that in principle this could be applied to large-scale engagements as well.

The other process I wanted to reference here is plain old brainstorming (again, emphasis mine):

Brainstorming is a method for developing creative solutions to problems. It works by focusing on a problem, and then having participants come up with as many deliberately unusual solutions as possible and by pushing the ideas as far as possible.

During the brainstorming session there is no criticism of ideas – the idea is to open up as many possibilities as possible, and break down preconceptions about the limits of the problem. Once this has been done the results of the brainstorming session can be analysed and the best solutions can be explored either using further brainstorming or more conventional solutions.

The key here is to suspend judgment while the brainstorming process is still ongoing. If we were to view the various e-participation efforts on Change.gov as large brainstorms (which, in my view, is a fairly reasonable characterization), then clearly this principle of suspending judgment was not adhered to. Instead, participants were allowed and encouraged to rate each other’s contributions by way of simple up-or-down votes at the same time new contributions were still being added.

In their excellent book Facilitator’s Guide to Participatory Decision-Making (2nd edition, 2007), the authors Sam Kaner et. al. have this to say about “the cost of premature criticism” (p. 118):

Rough-draft thinking is just like rough-draft writing: it needs encouragement, not evaluation. Many people don’t understand this. If they notice a flaw in someone’s thinking, they point it out. They think they’ve been helpful. But rough-draft ideas need to be clarified, researched, and modified before being subjected to critical evaluation. The timing of critical evaluation can make the difference between the life and death of a new idea.

Premature criticism is often inaccurate. And stifling. When ideas are criticized before they are fully formed, many people feel discouraged and stop trying. Furthermore, they may become unwilling to volunteer their rough-draft thinking at future meetings. They anticipate objections and keep quiet unless they can invent a counterargument. Thus, people learn to practice self-censorship. A group is then deprived of access to its most valuable natural resource: the creative thinking of its members.

I’d argue that in the context of e-consultations like the ones seen on Change.gov, the same rule applies not only to ideas but to personal stories, concerns, questions etc. as well.

The key take-away here is that structure can be added to an input gathering process not only by categorizing participant contributions (brainstorming relies on additional processes for category creation and input sorting to handle this), but also by applying time-based activities or phases.

Following this approach, here’s what the process on Change.gov might have looked like at a very basic level:

  1. Invite a lot of raw input (using e.g. forum-type discussions)
  2. Organize and refine participant input (e.g. by way of categorizing along input types, by summarizing, rephrasing or merging content and by removing duplicates)
  3. Evaluate and select (e.g. by way of ranking, prioritizing or voting at the summary level, not the raw input level)

Potentially, this could make for a much more efficient setup.

25 Types of Participant Input on Change.gov

Published by Tim on February 15, 2009 in Research. 0 Comments Tags: , , .

As I pointed out previously, some of the discussions we saw on Change.gov were all over the place even when they were supposed to focus on specific topic-related questions (e.g. “What worries you most about the healthcare system in our country?”) or tasks (suggesting a question or an idea to the president-elect).

At the massive scale of participation we saw on Change.gov, this poses a considerable challenge: there simply ends up being way too much unstructured content for any single participant to digest or make sense of. Here’s what I wrote back in December with regard to the healthcare discussion, which was still ongoing at the time:

Lack of focus in the comments: Instead of simply answering the question (”What worries you…?”), many participants choose to share rich combinations of personal stories, experiences, concerns, assumptions, questions, ideas, solutions, values, priorities, resources, data etc.  While this shows just how much energy the participants bring to the table, it also tends to leave the discussion somewhat directionless. There is no process in place to further organize this input, nor does the forum software support participants in being more disciplined or structured.

I wanted to take a closer look at this phenomenon as I have a hunch that understanding the underlying structures of large-group discussions like these may provide a good first step towards finding a better approach to dealing with large-scale online input gathering and the overwhelming amounts of content it can produce.

Looking at a sample of about 1,000 comments from Join the Discussion: Service (a little less than 25% out of the total 4,199), I tried to extract various common types of inputs the participants shared with each other (see screenshots):

 

Below is a list of 25 input types I was able to identify on a first pass, roughly sorted by frequency (with the more common types listed at the top):

  1. Off-topic remarks
  2. Expressions of approval or disapproval
  3. Personal stories and anecdotes
  4. Ideas
  5. Arguments for or against other ideas
  6. Resources (both online and offline)
  7. Concerns
  8. Questions
  9. Frustrations or rants
  10. Value statements
  11. Hopes
  12. Kudos
  13. Data and statistics
  14. Expressions of empathy, listening or appreciation
  15. Moderator advice or guidance (community management)
  16. Contact information
  17. Quotes
  18. Process feedback
  19. Personal profile information (introductions)
  20. Calls for help or support
  21. Test posts
  22. Personal attacks
  23. People suggestions (expert referrals)
  24. Definitions
  25. Event notifications

Obviously, this isn’t a particularly complete or refined list by any means nor does it claim to be generally applicable. At the same time, however, it seems to include a good portion of input types we can typically expect to find in forum discussions of this sort.

A few additional observations:

  • I didn’t have time to produce exact numbers, but a large majority of comments fall into one or more of the top 5-10 categories, whereas much fewer comments fall into any of the bottom 15-20 categories.
  • As I noted in December, many comments do in fact combine various input types (e.g. a story and an idea, kudos and a supporting argument, an idea and a few resources and a question etc.)
  • While I haven’t done a detailed comparison, from what I remember it seems the same categorization can be applied to the entries and comment discussions in the Citizen’s Briefing Book and — to a lesser extent — to the questions submitted in Open for Questions.

The three input gathering tools used on Change.gov (IntenseDebate, Google Moderator, Salesforce Ideas) presented the participants’ contributions in the form of relatively flat lists (sortable mainly by recency and/or popularity).

What if there was a mechanism in place that allows for content to be processed by input type? What if the participants’ numerous contributions could be aggregated or even synthesized across input types? This might solve a number of problems:

  • Improve navigation across the entire discussion.
  • Greatly reduce the time necessary for participants to gain or maintain an overview of the entire discussion.
  • Lower the number of duplicate entries due to increased visibility into what has already been said by others.
  • Facilitate follow-up by highlighting any loose ends (e.g. questions awaiting an answer).
  • Improve the quality of input evaluation and ratings: Not only become up-or-down votes a lot more meaningful when they are applied to inputs of the same type, but a variety of evaluation criteria and rating mechanisms could be used for different input types depending on what’s most appropriate (e.g. a “thumbs down” may not be an appropriate rating option in the context of participants’ sharing of personal stories).

This kind of summary layer built on top of the input gathering effort or discussion could make large-scale input gathering more manageable and productive.

Clay Shirky on Change.gov and Crowd Wisdom

Published by Tim on February 9, 2009 in Crowdsourcing and E-participation. 3 Comments Tags: , , , .

David Wilcox reports on a talk Clay Shirky gave at the London School of Economics this past week about collective action in a political context and some of the discussions that have since ensued: Clay Shirky: online crowds aren’t always wise

Clay Shirky, leading commentator on internet technologies and author of Here Comes Everybody, last night backed away from his earlier enthusiasm for the online wisdom of crowds in democratic decision-making. He suggested that Government use of social media should focus more on “small groups of smart people arguing with each other”, than national-scale engagement online.

We’ll, that’s my interpretation. You can listen for yourself … and find more on Twitter from last night and today’s at ICA.

A few years back Clay said that the ability of groups to organise online and challenge conventional engagement was “the glory of this medium”. He now believes we need more checks and balances.

Shirky’s examples include the spread of information following the 2008 Sichuan earthquake in China and the viral success of the will.i.am song “Yes We Can”). Starting at around 30:30 into the video, Shirky takes a look at Change.gov as an example of “wisdom of crowds instantiated in a political context”. He sees the fact that the call for legalization of medical marijuana ended up being the top-rated item in the Citizen’s Briefing Book as a sign that

… this is another place where certain kinds of special interests can make their feelings known. But it is not the same as saying anything that’s thrown to the top of Change.gov therefore has or should have priority in the president’s queue.

And so the problem we’ve got now isn’t a problem of capability, it’s a problem of legitimacy. Under what circumstances would you take advice from people primarily coordinated on the internet and headed for political action and under what circumstances would you ignore that advice. [...]

But unless there’s a principle by which you can say that all you’re really doing is saying it’s nice that you have this outlet (?) but we’re not going to take it seriously. But then if you go the other extreme and say you have write privileges to the president’s calendar, you can’t do that either. And the only way when we’re in situations where neither extreme solution works is to set up a set of checks and balances and that’s where I think the conversation is going.

A few thoughts as per my earlier comments here and here:

Not every crowd is wise. That realization shouldn’t really come as a surprise to anyone, especially when you consider that the groups that formed during the various e-participation efforts on Change.gov don’t even fit Surowiecki’s list of conditions:

  1. Diversity of opinion: Each person should have private information even if it’s just an eccentric interpretation of the known facts.
  2. Independence: People’s opinions aren’t determined by the opinions of those around them.
  3. Decentralization: People are able to specialize and draw on local knowledge.
  4. Aggregation: Some mechanism exists for turning private judgments into a collective decision.

By that standard, the groups that formed during the various e-participation efforts on Change.gov clearly weren’t wise. For example, in violation of the second rule above, not only were participants’ choices (aggregate number of votes for, say, an idea) exposed while voting was still in process but ideas were listed in order of popularity. This led to herding, favoring those relatively few ideas that managed to gain an early lead.

Moreover, it is debatable whether the concept of crowd wisdom applies to value judgments in just the same way it seems to apply to factual issues (I don’t remember any examples in Surowiecki’s book that would support the former). Since policy discussions are often comprised of both (they certainly were on Change.gov), I wonder to what extent the theories of crowd wisdom really apply here.

It’s also debatable whether what we saw on Change.gov really amounts to decision making. I’d argue it was some low-level form of general input gathering with ranked preferences for questions and ideas: fairly broad, with little structure and little to no process (and still flawed in many ways). On the actual issues no choices were made.

As far as I know, there was never any commitment on part of the transition team to let the Digg-style voting algorithms employed on Change.gov determine what Obama’s priorities would or should be. Instead, the Citizen’s Briefing Book was intended solely as a compilation of “facts and recommendations to be considered while crafting and enacting policies.” Nothing binding in there, just another channel for public input.

Applying IAP2’s Spectrum of Public Participation, the last two points illustrate that the e-participation efforts on Change.gov can be categorized as inform/consult types, not the more advanced involve/collaborate/empower types, in which case they may have had all the legitimacy they needed.

I generally agree with Shirky  that there are many issues facing efforts such as Change.gov with regard to legitimacy and process, just that a more differentiated analysis is needed to figure out where exactly the challenges lie and what conclusions we should draw. For government to abandon work on large-scale efforts and instead focus on small-group engagement alone seems premature advice at this point.

Participation on the New WhiteHouse.gov Website

Published by Tim on January 20, 2009 in E-participation. 0 Comments Tags: , , , .

Today at noon Eastern Time, WhiteHouse.gov, the official website of the President of the United States of America, underwent its long-expected relaunch. Those who have been following candidate and president-elect Barack Obama’s web efforts over the last year will surely recognize it.

This announcement sounds exciting:

Participation – President Obama started his career as a community organizer on the South Side of Chicago, where he saw firsthand what people can do when they come together for a common cause. Citizen participation will be a priority for the Administration, and the internet will play an important role in that. One significant addition to WhiteHouse.gov reflects a campaign promise from the President: we will publish all non-emergency legislation to the website for five days, and allow the public to review and comment before the President signs it.

Given the various e-participation efforts the transition team has already engaged in on Change.gov since November, my bet is we will see a lot of innovation in this area in the months ahead.