Intellitics, Inc. » changegov

Re: Grading WhiteHouse.gov

Tim — Tue, 24 Mar 2009 23:10:26 +0000

44: The Obama Presidency, one of the Washington Post’s blogs, today came out with a new monthly feature where they’ll have a group of five experts (for today, that’s Craig Newmark, Andrew Rasiej, Ellen Miller, Jon Henke, and David Weinberger) examine the new WhiteHouse.gov website: Grading WhiteHouse.gov

Excerpt:

For all the innovations of Obama’s WhiteHouse.gov — yesterday, officials announced that it will distribute tickets to the Easter Egg Roll online — online observers, a sometimes prickly, often exacting, let’s-get-ahead-of-the-curve bunch, are left wanting for more. Take the issue of generating comments. Allowing comments on blogs is a given, nothing more than an online SOP. BarackObama.com and Change.gov allowed comments. But WhiteHouse.gov doesn’t — at least not yet.

To which I left the following comment:

Citizen engagement (in the form of public participation) covers a whole range of activities from merely providing citizens with useful and timely information, to soliciting citizen feedback, to collaborative drafting of policies, and last but not least all the way up to granting citizens certain decision making powers.

First and foremost, this is about process: Where can participation be helpful or required, and to what degree? What promises are being made to the public at each level and phase of public participation and how can the organization leading the engagement effort make sure these promises are consistently being kept? Only then does the question of tools come into play.

Anyone serious about public participation must get these basics right for it to achieve the desired outcomes.

With that in mind, I seriously doubt that simply turning on comments on the WhiteHouse.gov official blog would qualify as meaningful participation. Worse yet, in some cases it might even be counter-productive to quality citizen engagement.

The experiments we saw on Change.gov were definitely a step in the right direction. However, from a public participation standpoint there were many best practices the transition team did not yet manage to adhere to. Moreover, none of the tools that were used on Change.gov (IntenseDebate, Google Moderator, Salesforce Ideas) were really built to scale (much less in a public participation environment), and they all struggled with the massive onslaught of user contributions.

So rather than getting impatient with the new administration, my advice to them would be to address the participation piece with great care and caution and to innovate one step at a time. Identify the most promising use cases and work your way up the ladder of public participation. Definitely continue in the spirit of experimentation that was visible on Change.gov, but make sure you don’t fail too badly too often as the participants’ trust, once broken, will be hard to recover.

For all I know, the current linear models of commenting on the web (be it threaded or non-threaded comments, with or without ratings, advanced sorting etc.) do not scale. If the activity we’ve seen on Change.gov is any indication, the WhiteHouse.gov web team might be well-advised to hold off on any general roll-out and only use comments where they absolutely don’t have any better alternatives.

Finding Appropriate Evaluation Criteria for Participant Input

Tim — Tue, 17 Feb 2009 21:33:28 +0000

A common approach to trying to surface the most relevant, highest-quality or most agreed-upon items out of a large quantity of content is to allow participants to rate each other’s contributions and then expose the highest-rated items in a “most popular” list. Very often, a simple binary, up-or-down rating mechanism is used for this purpose.

The three input gathering tools used on Change.gov all offered such mechanisms:

Join the Discussion: Threaded comments + “Thumbs up, thumbs down” comment ratings (screenshot, I’m intentionally leaving out the reputation piece here, though that certainly serves as another filter)
Open for Questions: Posts only + “Good question? yes/no” post ratings (screenshot)
Citizen’s Briefing Book: Posts + non-threaded comments + “Vote up, vote down” post ratings (screenshot)

These rating mechanism look fairly straightforward, though from the way they are named it is not entirely obvious which evaluation criterion each one of them refers to:

“Thumbs up, thumbs down” might rate contributions based on relevance, quality or agreement
“Good question? yes/no” looks like it’s going after relevance or quality
“Vote up, vote down” seems to filter based on agreement (in the context of an idea contest, at least)

It may not always matter, but strictly speaking these three (relevance, quality, agreement) aren’t identical. There may well be occasions where it’s worth differentiating between the three or at least be more explicit as to which one is the intended criterion.

More interestingly, though, there’s room for other, less commonly used criteria as well. If we assume for a second that all input types are not created equal, then surely there may be more appropriate evaluation criteria that can be applied.

Below are a few examples of various input types as well as potential evaluation criteria that might serve well in an early, fairly general phase of an e-participation project:

Question: “I share this question (and would like it answered)”
Story: “I have had the same experience” or “I know someone who has had the same experience”
Resource: “This material was helpful”
Idea: “This idea should be investigated in more detail”

Note that negative ratings are being avoided here (it’s really more a flagging mechanism than an up-or-down vote).

More to think about:

Other combinations of input type and evaluation criterion are entirely possible, of course.
Other rating mechanisms (e.g. 5-star voting, 0-to-100-percent sliding scale etc.) may be more helpful.
And finally, negative feedback isn’t always a bad thing: it just depends on where in the process it’s applied, how well it’s introduced and supported (e.g. by community ground rules), how much group cohesion and trust there is among the participants etc.

Again, it may not always be necessary to go to this level of detail, but at the scale of participation we saw on Change.gov it might have provided another much-needed step towards transforming the amorphous mass of unstructured participant contributions into a more meaningful summary.

Structuring Participant Input: Dynamic Facilitation, Brainstorming

Tim — Sun, 15 Feb 2009 22:32:20 +0000

While I was assembling my off-the-cuff analysis of input types on Change.gov, I felt compelled to revisit two existing facilitation techniques that help guide participation by adding to the process the kind of structure that I believe could work very well for large-scale efforts.

First, Dynamic Facilitation, a method I first learned about at the 2006 National Conference on Dialogue & Deliberation in San Francisco, CA (emphasis mine):

What is Dynamic Facilitation?

Dynamic Facilitation is a natural way of facilitating that works well with people addressing difficult issues about which they care deeply. Rather than asking them to limit themselves — to hold back their emotions, to stay on the agenda, or to follow the process — the dynamic facilitator frames the conversation so that all comments are helpful and productive. He or she establishes a “zone” of thinking and talking known as “choice-creating,” where shifts, insights and breakthroughs are frequent. The process allows ordinary, untrained people to address difficult issues and reach consensus solutions that are better, faster and which have more support than traditional means of “consensus-building.”.

…

The dynamic facilitator starts by helping people determine an issue they really care about, whether it seems solvable or not. Then he or she helps them to say what is on their minds and hearts. To do this she uses four flip charts for creating lists of: Solutions, Problem-statements, Data, and Concerns. A fifth chart of Decisions is added as group conclusions emerge.

Here, the participants’ input is discovered, captured and structured using content categories or input types. And while Dynamic Facilitation is intended for small groups only and relies heavily on the facilitator role, I believe that in principle this could be applied to large-scale engagements as well.

The other process I wanted to reference here is plain old brainstorming (again, emphasis mine):

Brainstorming is a method for developing creative solutions to problems. It works by focusing on a problem, and then having participants come up with as many deliberately unusual solutions as possible and by pushing the ideas as far as possible.

During the brainstorming session there is no criticism of ideas – the idea is to open up as many possibilities as possible, and break down preconceptions about the limits of the problem. Once this has been done the results of the brainstorming session can be analysed and the best solutions can be explored either using further brainstorming or more conventional solutions.

The key here is to suspend judgment while the brainstorming process is still ongoing. If we were to view the various e-participation efforts on Change.gov as large brainstorms (which, in my view, is a fairly reasonable characterization), then clearly this principle of suspending judgment was not adhered to. Instead, participants were allowed and encouraged to rate each other’s contributions by way of simple up-or-down votes at the same time new contributions were still being added.

In their excellent book Facilitator’s Guide to Participatory Decision-Making (2nd edition, 2007), the authors Sam Kaner et. al. have this to say about “the cost of premature criticism” (p. 118):

Rough-draft thinking is just like rough-draft writing: it needs encouragement, not evaluation. Many people don’t understand this. If they notice a flaw in someone’s thinking, they point it out. They think they’ve been helpful. But rough-draft ideas need to be clarified, researched, and modified before being subjected to critical evaluation. The timing of critical evaluation can make the difference between the life and death of a new idea.

Premature criticism is often inaccurate. And stifling. When ideas are criticized before they are fully formed, many people feel discouraged and stop trying. Furthermore, they may become unwilling to volunteer their rough-draft thinking at future meetings. They anticipate objections and keep quiet unless they can invent a counterargument. Thus, people learn to practice self-censorship. A group is then deprived of access to its most valuable natural resource: the creative thinking of its members.

I’d argue that in the context of e-consultations like the ones seen on Change.gov, the same rule applies not only to ideas but to personal stories, concerns, questions etc. as well.

The key take-away here is that structure can be added to an input gathering process not only by categorizing participant contributions (brainstorming relies on additional processes for category creation and input sorting to handle this), but also by applying time-based activities or phases.

Following this approach, here’s what the process on Change.gov might have looked like at a very basic level:

Invite a lot of raw input (using e.g. forum-type discussions)
Organize and refine participant input (e.g. by way of categorizing along input types, by summarizing, rephrasing or merging content and by removing duplicates)
Evaluate and select (e.g. by way of ranking, prioritizing or voting at the summary level, not the raw input level)

Potentially, this could make for a much more efficient setup.

25 Types of Participant Input on Change.gov

Tim — Sun, 15 Feb 2009 08:46:40 +0000

As I pointed out previously, some of the discussions we saw on Change.gov were all over the place even when they were supposed to focus on specific topic-related questions (e.g. “What worries you most about the healthcare system in our country?”) or tasks (suggesting a question or an idea to the president-elect).

At the massive scale of participation we saw on Change.gov, this poses a considerable challenge: there simply ends up being way too much unstructured content for any single participant to digest or make sense of. Here’s what I wrote back in December with regard to the healthcare discussion, which was still ongoing at the time:

Lack of focus in the comments: Instead of simply answering the question (”What worries you…?”), many participants choose to share rich combinations of personal stories, experiences, concerns, assumptions, questions, ideas, solutions, values, priorities, resources, data etc. While this shows just how much energy the participants bring to the table, it also tends to leave the discussion somewhat directionless. There is no process in place to further organize this input, nor does the forum software support participants in being more disciplined or structured.

I wanted to take a closer look at this phenomenon as I have a hunch that understanding the underlying structures of large-group discussions like these may provide a good first step towards finding a better approach to dealing with large-scale online input gathering and the overwhelming amounts of content it can produce.

Looking at a sample of about 1,000 comments from Join the Discussion: Service (a little less than 25% out of the total 4,199), I tried to extract various common types of inputs the participants shared with each other (see screenshots):

Below is a list of 25 input types I was able to identify on a first pass, roughly sorted by frequency (with the more common types listed at the top):

Off-topic remarks
Expressions of approval or disapproval
Personal stories and anecdotes
Ideas
Arguments for or against other ideas
Resources (both online and offline)
Concerns
Questions
Frustrations or rants
Value statements
Hopes
Kudos
Data and statistics
Expressions of empathy, listening or appreciation
Moderator advice or guidance (community management)
Contact information
Quotes
Process feedback
Personal profile information (introductions)
Calls for help or support
Test posts
Personal attacks
People suggestions (expert referrals)
Definitions
Event notifications

Obviously, this isn’t a particularly complete or refined list by any means nor does it claim to be generally applicable. At the same time, however, it seems to include a good portion of input types we can typically expect to find in forum discussions of this sort.

A few additional observations:

I didn’t have time to produce exact numbers, but a large majority of comments fall into one or more of the top 5-10 categories, whereas much fewer comments fall into any of the bottom 15-20 categories.
As I noted in December, many comments do in fact combine various input types (e.g. a story and an idea, kudos and a supporting argument, an idea and a few resources and a question etc.)
While I haven’t done a detailed comparison, from what I remember it seems the same categorization can be applied to the entries and comment discussions in the Citizen’s Briefing Book and — to a lesser extent — to the questions submitted in Open for Questions.

The three input gathering tools used on Change.gov (IntenseDebate, Google Moderator, Salesforce Ideas) presented the participants’ contributions in the form of relatively flat lists (sortable mainly by recency and/or popularity).

What if there was a mechanism in place that allows for content to be processed by input type? What if the participants’ numerous contributions could be aggregated or even synthesized across input types? This might solve a number of problems:

Improve navigation across the entire discussion.
Greatly reduce the time necessary for participants to gain or maintain an overview of the entire discussion.
Lower the number of duplicate entries due to increased visibility into what has already been said by others.
Facilitate follow-up by highlighting any loose ends (e.g. questions awaiting an answer).
Improve the quality of input evaluation and ratings: Not only become up-or-down votes a lot more meaningful when they are applied to inputs of the same type, but a variety of evaluation criteria and rating mechanisms could be used for different input types depending on what’s most appropriate (e.g. a “thumbs down” may not be an appropriate rating option in the context of participants’ sharing of personal stories).

This kind of summary layer built on top of the input gathering effort or discussion could make large-scale input gathering more manageable and productive.

Clay Shirky on Change.gov and Crowd Wisdom

Tim — Mon, 09 Feb 2009 10:27:13 +0000

David Wilcox reports on a talk Clay Shirky gave at the London School of Economics this past week about collective action in a political context and some of the discussions that have since ensued: Clay Shirky: online crowds aren’t always wise

Clay Shirky, leading commentator on internet technologies and author of Here Comes Everybody, last night backed away from his earlier enthusiasm for the online wisdom of crowds in democratic decision-making. He suggested that Government use of social media should focus more on “small groups of smart people arguing with each other”, than national-scale engagement online.

We’ll, that’s my interpretation. You can listen for yourself … and find more on Twitter from last night and today’s at ICA.

A few years back Clay said that the ability of groups to organise online and challenge conventional engagement was “the glory of this medium”. He now believes we need more checks and balances.

Shirky’s examples include the spread of information following the 2008 Sichuan earthquake in China and the viral success of the will.i.am song “Yes We Can”). Starting at around 30:30 into the video, Shirky takes a look at Change.gov as an example of “wisdom of crowds instantiated in a political context”. He sees the fact that the call for legalization of medical marijuana ended up being the top-rated item in the Citizen’s Briefing Book as a sign that

… this is another place where certain kinds of special interests can make their feelings known. But it is not the same as saying anything that’s thrown to the top of Change.gov therefore has or should have priority in the president’s queue.

And so the problem we’ve got now isn’t a problem of capability, it’s a problem of legitimacy. Under what circumstances would you take advice from people primarily coordinated on the internet and headed for political action and under what circumstances would you ignore that advice. [...]

But unless there’s a principle by which you can say that all you’re really doing is saying it’s nice that you have this outlet (?) but we’re not going to take it seriously. But then if you go the other extreme and say you have write privileges to the president’s calendar, you can’t do that either. And the only way when we’re in situations where neither extreme solution works is to set up a set of checks and balances and that’s where I think the conversation is going.

A few thoughts as per my earlier comments here and here:

Not every crowd is wise. That realization shouldn’t really come as a surprise to anyone, especially when you consider that the groups that formed during the various e-participation efforts on Change.gov don’t even fit Surowiecki’s list of conditions:

Diversity of opinion: Each person should have private information even if it’s just an eccentric interpretation of the known facts.
Independence: People’s opinions aren’t determined by the opinions of those around them.
Decentralization: People are able to specialize and draw on local knowledge.
Aggregation: Some mechanism exists for turning private judgments into a collective decision.

By that standard, the groups that formed during the various e-participation efforts on Change.gov clearly weren’t wise. For example, in violation of the second rule above, not only were participants’ choices (aggregate number of votes for, say, an idea) exposed while voting was still in process but ideas were listed in order of popularity. This led to herding, favoring those relatively few ideas that managed to gain an early lead.

Moreover, it is debatable whether the concept of crowd wisdom applies to value judgments in just the same way it seems to apply to factual issues (I don’t remember any examples in Surowiecki’s book that would support the former). Since policy discussions are often comprised of both (they certainly were on Change.gov), I wonder to what extent the theories of crowd wisdom really apply here.

It’s also debatable whether what we saw on Change.gov really amounts to decision making. I’d argue it was some low-level form of general input gathering with ranked preferences for questions and ideas: fairly broad, with little structure and little to no process (and still flawed in many ways). On the actual issues no choices were made.

As far as I know, there was never any commitment on part of the transition team to let the Digg-style voting algorithms employed on Change.gov determine what Obama’s priorities would or should be. Instead, the Citizen’s Briefing Book was intended solely as a compilation of “facts and recommendations to be considered while crafting and enacting policies.” Nothing binding in there, just another channel for public input.

Applying IAP2′s Spectrum of Public Participation, the last two points illustrate that the e-participation efforts on Change.gov can be categorized as inform/consult types, not the more advanced involve/collaborate/empower types, in which case they may have had all the legitimacy they needed.

I generally agree with Shirky that there are many issues facing efforts such as Change.gov with regard to legitimacy and process, just that a more differentiated analysis is needed to figure out where exactly the challenges lie and what conclusions we should draw. For government to abandon work on large-scale efforts and instead focus on small-group engagement alone seems premature advice at this point.

Participation on the New WhiteHouse.gov Website

Tim — Tue, 20 Jan 2009 19:06:16 +0000

Today at noon Eastern Time, WhiteHouse.gov, the official website of the President of the United States of America, underwent its long-expected relaunch. Those who have been following candidate and president-elect Barack Obama’s web efforts over the last year will surely recognize it.

This announcement sounds exciting:

Participation – President Obama started his career as a community organizer on the South Side of Chicago, where he saw firsthand what people can do when they come together for a common cause. Citizen participation will be a priority for the Administration, and the internet will play an important role in that. One significant addition to WhiteHouse.gov reflects a campaign promise from the President: we will publish all non-emergency legislation to the website for five days, and allow the public to review and comment before the President signs it.

Given the various e-participation efforts the transition team has already engaged in on Change.gov since November, my bet is we will see a lot of innovation in this area in the months ahead.

Change.gov launches online discussion around healthcare

Tim — Mon, 01 Dec 2008 16:24:17 +0000

Last Tuesday on Change.gov (the official website of President-elect Barack Obama), the transition team launched a public online discussion on the topic of healthcare. From the announcement on their blog:

Join the Discussion

Today we’re trying out a new feature on our website that will allow us get instant feedback from you about our top priorities. We also hope it will allow you to form communities around these issues — with the best ideas and most interesting discussions floating to the top.

The question they are asking:

What worries you most about the healthcare system in our country?

The site uses IntenseDebate, which — at a very basic level — is a threaded discussion forum (with a few added features, such as reputation points, comment ratings, OpenID support).

More than 3,500 comments have been posted so far.

One of the reasons this is interesting is because it may hint at a more participatory approach to government by the new administration, something for which a lot of people have expressed high hopes during the recent US presidential campaign. Accordingly, the initial feedback — from some in the web community, at least — has been very positive: Over on techPresident, Micah Sifry calls it “the beginning of a rebooting of the American political system,” and Eric Eldon of VentureBeat thinks of it as “a great early step in making government more open.”

I wanted to share a few initial observations, mainly with regard to process quality. How to best address some of these issues will be left for future posts but feel free to share your thoughts in the comments. For those interested in this discussion, there are a couple of threads going on on the NCDD main mailing list as well as on one of the groups at Democracies Online.

First off, though, kudos to the people behind Change.gov for experimenting with large-scale e-participation so early in the process (remember this is still just the transition, not the presidency). Despite the many challenges that have yet to be mastered in order to make it work at scale, I believe the opportunities for online dialogue and deliberation to help boost civic engagement and to improve public decision making at all levels of government are tremendous.

That said, a few things that immediately caught my eye:

No clear process model: Judging from the information available on the website, it is not entirely clear how exactly the comments will be processed, what impact realistically they may or may not have on any policy decisions, or what kind of follow-up and follow-through either the transition team or the new administration are willing to commit to. This can become a problem since it risks disappointing participants (e.g. when assumed impact doesn’t match actual impact and participants are left frustrated over the time and energy they spent in vain).
Lack of focus in the comments: Instead of simply answering the question (“What worries you…?”), many participants choose to share rich combinations of personal stories, experiences, concerns, assumptions, questions, ideas, solutions, values, priorities, resources, data etc. While this shows just how much energy the participants bring to the table, it also tends to leave the discussion somewhat directionless. There is no process in place to further organize this input, nor does the forum software support participants in being more disciplined or structured.
Lack of organizer participation: I was able to spot one instance of comment deletion by the forum administrators, presumably according to their comment policy (screenshot). I may be wrong, but other than that the transition team does not seem to actively engage in the discussions (e.g. ask or answer questions, express agreement or disagreement, or otherwise facilitate the process or provide general community management etc.). I only looked at a few sample pages, though, so I may be wrong.
Overwhelming amounts of unstructured data: The discussion on Change.gov was off to a fast start, reaching 2,000 comments in the first 24 hours. As with many online discussion forums that reach a certain activity level, the amounts of content produced by the participants can be quite staggering. For example, total word count on this forum may well be approaching 500,000 words already (for details how I got this number, see my rough calculations). That means it becomes extremely time-consuming to keep up with even a small fraction of overall input (poor navigation adds to the problem). Moreover, the fact that this data is largely unstructured makes further processing very difficult if not impossible.

Again, this is an early assessment of a work in progress, and it wouldn’t at all surprise me if these issues will be addressed over time. The combination of a more demanding public (with regard to opportunities for meaningful e-participation) and a more interested and supportive administration (from what we’ve heard of the past year from some of the people involved) looks quite promising.