A common approach to trying to surface the most relevant, highest-quality or most agreed-upon items out of a large quantity of content is to allow participants to rate each other’s contributions and then expose the highest-rated items in a “most popular” list. Very often, a simple binary, up-or-down rating mechanism is used for this purpose.
The three input gathering tools used on Change.gov all offered such mechanisms:
- Join the Discussion: Threaded comments + “Thumbs up, thumbs down” comment ratings (screenshot, I’m intentionally leaving out the reputation piece here, though that certainly serves as another filter)
- Open for Questions: Posts only + “Good question? yes/no” post ratings (screenshot)
- Citizen’s Briefing Book: Posts + non-threaded comments + “Vote up, vote down” post ratings (screenshot)
These rating mechanism look fairly straightforward, though from the way they are named it is not entirely obvious which evaluation criterion each one of them refers to:
- “Thumbs up, thumbs down” might rate contributions based on relevance, quality or agreement
- “Good question? yes/no” looks like it’s going after relevance or quality
- “Vote up, vote down” seems to filter based on agreement (in the context of an idea contest, at least)
It may not always matter, but strictly speaking these three (relevance, quality, agreement) aren’t identical. There may well be occasions where it’s worth differentiating between the three or at least be more explicit as to which one is the intended criterion.
More interestingly, though, there’s room for other, less commonly used criteria as well. If we assume for a second that all input types are not created equal, then surely there may be more appropriate evaluation criteria that can be applied.
Below are a few examples of various input types as well as potential evaluation criteria that might serve well in an early, fairly general phase of an e-participation project:
- Question: “I share this question (and would like it answered)”
- Story: “I have had the same experience” or “I know someone who has had the same experience”
- Resource: “This material was helpful”
- Idea: “This idea should be investigated in more detail”
Note that negative ratings are being avoided here (it’s really more a flagging mechanism than an up-or-down vote).
More to think about:
- Other combinations of input type and evaluation criterion are entirely possible, of course.
- Other rating mechanisms (e.g. 5-star voting, 0-to-100-percent sliding scale etc.) may be more helpful.
- And finally, negative feedback isn’t always a bad thing: it just depends on where in the process it’s applied, how well it’s introduced and supported (e.g. by community ground rules), how much group cohesion and trust there is among the participants etc.
Again, it may not always be necessary to go to this level of detail, but at the scale of participation we saw on Change.gov it might have provided another much-needed step towards transforming the amorphous mass of unstructured participant contributions into a more meaningful summary.
Related posts:
{ 1 trackback }