Earlier this summer, I happened to come across an excellent new article on online deliberation design. Now that it’s out of draft mode, I can finally share it here.

Online Deliberation Design: Choices, Criteria, and Evidence (PDF, final working version) by Todd Davies and Reid Chandler is currently in press and will appear as chapter 6 in Tina Nabatchi, Michael Weiksner, John Gastil, and Matt Leighninger (Editors), Democracy in Motion: Evaluating the Practice and Impact of Deliberative Civic Engagement, Oxford University Press, Fall 2011:

Abstract: This chapter reviews empirical evidence bearing on the design of online forums for deliberative civic engagement. Dimensions of design are defined for different aspects of the deliberation: its purpose, the target population, the spatiotemporal distance separating participants, the communication medium, and the deliberative process to be followed. After a brief overview of criteria for evaluating different design options, empirical findings are organized around design choices. Research has evolved away from treating technology for online deliberation dichotomously (either present or not) toward nuanced findings that differentiate between technological features, ways of using them, and cultural settings. The effectiveness of online deliberation depends on how well the communicative environment is matched to the deliberative task. Tradeoffs, e.g. between rich and lean media and between anonymous and identifiable participation, suggest different designs depending on the purpose and participants. Findings are limited by existing technologies, and may change as technologies and users co-evolve.

Table 6.1: Summary of Five Design Categories

The authors present a number of design issues, grouped into five multi-dimensional categories (click to enlarge image on the right):

[...] These issues have been selected for inclusion mainly because they have been the subjects of empirical research. As should be apparent, the issues discussed below represent a relatively small subset of all the potential choices a deliberation designer faces. Nevertheless, these issues do provide a way to organize the empirical literature relevant to deliberation design. [...]

Table 6.1 (page 3 in the PDF) illustrates the five design categories and the dimensions they include:

1. Purpose: Why is the deliberation being designed?
   • Outcome (decisions-beliefs-ideas)
   • Collectivity (group-individual)
2. Population: Who will be involved?
   • Recruitment (random-selected)
   • Audience (public-private)
3. Spatiotemporal Distance: Where and when will participants be interacting with each other?
   • Colocation (face to face-telecommunication)
   • Cotemporality (synchronous-asynchronous)
4. Communication Medium: How will communication occur?
   • Modality (speech-text-image-multimodal)
   • Emotivity (impeded-enabled)
   • Fidelity (transformed-unaltered)
5. Deliberative Process: What will occur between participants
   • Facilitation (moderated-unmoderated)
   • Structure (rules-free form)
   • Identifiability (identifiable-anonymous)
   • Incentivization (reward-no reward)

Under “Design Criteria” (page 9), the authors share “some of the [success] criteria that have been applied in the past, especially by researchers evaluating different design choices [...]“. They are:

• Quantity
• Quality
• Inclusiveness
• Preference
• Efficiency
• Efficacy

The article goes on to describe empirical findings related to each of the design dimensions, touching upon some of the key trade-offs involved in choosing either between online and face-to-face or between various online setups, and mentions plenty of opportunities for future research, including:

• Effects of random versus non-random sampling for participant recruitment
• Effects of audience (public versus private deliberations)
• Effects of various modalities
• Effects of structure (specifically, particular techniques)
• Effects of incentives

This research potential is summed up as follows (page 22):

Remarks on Result Specificity and Culturalism

The studies we have cited in this section, together with the design dimensions into which we have organized them, demonstrate that the main question facing future deliberation designers as they contemplate online designs will increasingly be not whether but how to use online tools. Early research in this area tended to treat technology as “a dichotomous variable … either present or not.” But as technologies have evolved, the range of experiences they offer now span a broad spectrum from the simple and truncated text of a Twitter message to the high verisimilitude of the most advanced virtual reality environments, as well as augmented reality systems that attempt to give us the best face-to-face and online experiences simultaneously. Online deliberation and its public version – online deliberative civic engagement – are rich areas of study and design because of the huge space of possibilities within them.

Great stuff! It is clear that given certain objectives, requirements and conditions a variety of design choices may be viable. Furthermore, every tool has its strengths and weaknesses and must generally fit the design approach in order to perform well. There can never be a single most effective web tool for online deliberation (or for public participation, for that matter). That question falls short as long as it’s not based on at least some specifics.

While working on the previous post, I couldn’t help but think how valuable it would be for participants, researchers and everyone else interested if these high-level participation metrics were readily available in real-time on the site that hosts the discussion, rather than having to dig them up manually and deal with incomplete information, assumptions and more or less wild guesses.

Some of the numbers I usually like to track over the course of an e-participation initiative include the following (these are all pretty straightforward and a number of tools recently used by the transition team and the new administration already expose some of them by default):

• Number or registered users
• Number of posts (or ideas, questions etc.)
• Number of comments
• Number of votes
• Number of flags
• Total word count
• Participation per participant (e.g. average number of comments, median)
• Leader boards (where applicable)

Others — both participants and observers — have pointed out other issues with the site that make it harder to use than necessary.

Based on their feedback and some of my own observations, I’ve compiled a list of 26 enhancement ideas that would considerably improve the participation experience during phase 2 of the Open Government Dialogue and increase the overall quantity and quality of participant input.

Some of these refer specifically to the tool used for phase 2 (WordPress blogging engine for threaded discussions and a plug-in for comment rating) but most should be generally applicable across other tools as well.

I’m sure the list is far from complete. Please leave a comment below if you can think of anything else that could be improved.

Registration

1. Improve the usability of the sign-up process (including a better fall-back solution for CAPTCHA, which a lot of users apparently can’t seem to figure out).

User profile

2. Add a prominent link to the user profile self-management page, where participants can edit their name, contact information, password etc.
3. Add a public user profile page that allows participants to voluntarily reveal more background on their real identity (e.g. by sharing their name, affiliation, a brief bio, photo/avatar, link to their personal blog etc.).
4. On each participant’s profile page, list some basic participation metrics for that user and link back to all of her comments (this also makes finding one’s own comments a lot easier than, say, having to browse the entire archive).

Commenting

5. Allow participants to edit or delete their comments for a reasonable time period after posting (e.g. to correct typos or remove duplicate entries).
6. Consider imposing smart limits on the maximum number of characters allowed per comment to avoid overly lengthy submissions.

Navigation

7. Add permalinks to comments for easier referencing and sharing across the web (e.g. using email, blogs, Twitter etc.).
8. Highlight staff contributions more prominently.
9. Display posts and threads in chronological order.
10. Provide a more robust tree structure, one that properly associates replies with the comments they refer to (even if one or more comments from a thread have been hidden or removed) and which supports better browsing and sorting of comments and threads (e.g. show most recent posts or most recently active threads).
11. Highlight recently added comments.
12. Allow tagging of posts and comments.
13. Add a tag cloud.
14. Add a searchable directory of all registered users that supports various filters (e.g. sort by most active users, most recently joined, most highly voted etc.).

Notifications

15. Add a prominent link to the general comment RSS feed as an alternative way to follow the discussions: http://blog.ostp.gov/comments/feed/
16. Enable email notifications for new or updated blog posts, comments and replies.

Comment voting

17. Allow participants to correct (take back or switch) their up or down votes on comments.
18. In the meantime (and at a minimum), make participants better aware of the fact that votes cannot be changed once submitted.
19. In addition, improve the usability of the voting (vote up, vote down), flagging and reply buttons: the icons aren’t clear enough and the buttons are fairly small, both factors that can lead to accidental mis-voting).
20. In addition to net number of votes, expose the total number of positive and negative notes for each comment.
21. Expose controversiality (the ratio of negative to positive votes).

Moderation

22. For comments that have to be removed due to a violation of the terms of participation, leave a note that references the type of violation.
23. In case a comment is removed, notify the original poster of her offense (we don’t know for sure if this is done consistently but judging from user feedback it’s not).

General

24. Add an FAQ or help page.
25. Add a statistics page that shows the number of registered users, number of comments, number of votes and number of flags over time.
26. Add site-wide search (currently comments don’t seem to show up in search).

That’s all I could find over the past week. Please expand on this list as you come across other stumbling blocks.