While working on the previous post, I couldn’t help but think how valuable it would be for participants, researchers and everyone else interested if these high-level participation metrics were readily available in real-time on the site that hosts the discussion, rather than having to dig them up manually and deal with incomplete information, assumptions and more or less wild guesses.
Some of the numbers I usually like to track over the course of an e-participation initiative include the following (these are all pretty straightforward and a number of tools recently used by the transition team and the new administration already expose some of them by default):
- Number or registered users
- Number of posts (or ideas, questions etc.)
- Number of comments
- Number of votes
- Number of flags
- Total word count
- Participation per participant (e.g. average number of comments, median)
- Leader boards (where applicable)
Others — both participants and observers — have pointed out other issues with the site that make it harder to use than necessary.
Based on their feedback and some of my own observations, I’ve compiled a list of 26 enhancement ideas that would considerably improve the participation experience during phase 2 of the Open Government Dialogue and increase the overall quantity and quality of participant input.
Some of these refer specifically to the tool used for phase 2 (Wordpress blogging engine for threaded discussions and a plug-in for comment rating) but most should be generally applicable across other tools as well.
I’m sure the list is far from complete. Please leave a comment below if you can think of anything else that could be improved.
Registration
- Improve the usability of the sign-up process (including a better fall-back solution for CAPTCHA, which a lot of users apparently can’t seem to figure out).
User profile
- Add a prominent link to the user profile self-management page, where participants can edit their name, contact information, password etc.
- Add a public user profile page that allows participants to voluntarily reveal more background on their real identity (e.g. by sharing their name, affiliation, a brief bio, photo/avatar, link to their personal blog etc.).
- On each participant’s profile page, list some basic participation metrics for that user and link back to all of her comments (this also makes finding one’s own comments a lot easier than, say, having to browse the entire archive).
Commenting
- Allow participants to edit or delete their comments for a reasonable time period after posting (e.g. to correct typos or remove duplicate entries).
- Consider imposing smart limits on the maximum number of characters allowed per comment to avoid overly lengthy submissions.
Navigation
- Add permalinks to comments for easier referencing and sharing across the web (e.g. using email, blogs, Twitter etc.).
- Highlight staff contributions more prominently.
- Display posts and threads in chronological order.
- Provide a more robust tree structure, one that properly associates replies with the comments they refer to (even if one or more comments from a thread have been hidden or removed) and which supports better browsing and sorting of comments and threads (e.g. show most recent posts or most recently active threads).
- Highlight recently added comments.
- Allow tagging of posts and comments.
- Add a tag cloud.
- Add a searchable directory of all registered users that supports various filters (e.g. sort by most active users, most recently joined, most highly voted etc.).
Notifications
- Add a prominent link to the general comment RSS feed as an alternative way to follow the discussions: http://blog.ostp.gov/comments/feed/
- Enable email notifications for new or updated blog posts, comments and replies.
Comment voting
- Allow participants to correct (take back or switch) their up or down votes on comments.
- In the meantime (and at a minimum), make participants better aware of the fact that votes cannot be changed once submitted.
- In addition, improve the usability of the voting (vote up, vote down), flagging and reply buttons: the icons aren’t clear enough and the buttons are fairly small, both factors that can lead to accidental mis-voting).
- In addition to net number of votes, expose the total number of positive and negative notes for each comment.
- Expose controversiality (the ratio of negative to positive votes).
Moderation
- For comments that have to be removed due to a violation of the terms of participation, leave a note that references the type of violation.
- In case a comment is removed, notify the original poster of her offense (we don’t know for sure if this is done consistently but judging from user feedback it’s not).
General
- Add an FAQ or help page.
- Add a statistics page that shows the number of registered users, number of comments, number of votes and number of flags over time.
- Add site-wide search (currently comments don’t seem to show up in search).
That’s all I could find over the past week. Please expand on this list as you come across other stumbling blocks.
Related Posts
- June 9, 2009 by Tim:
Open Government Dialogue: Phase 2 Metrics Update - June 8, 2009 by Tim:
Open Government Dialogue: Agenda for Phase 2 Discussion about Citizen Participation - May 31, 2009 by Tim:
Open Government Dialogue: First Look at Site Activity and User Adoption
Good list! Here are a few more, plus some comments.
Registration – add
- Form should indicate what will appear on messages – name or user name.
- CAPTCHA buttons should be labeled (they do have mouseovers – not sure those were there earlier, or that it is sufficient); also, if the CAPTCHA code should be refreshed after an error message, this (and how to do it) should be included in that message along with the instruction to use your “browsers” back button – at least that seemed to help when I was having this problem!
User profile
4. OK, but in Phase 1 I found the voting records excessive and inconsistent.
Commenting
5. Very important.
6. Rather than limiting number of characters, which I think could be a red flag, how about cutoffs with “read the rest…” links, as in the OSTP blog?
Navigation
10. Very important.
11. Recently added comments: add access to a list of “New since your last visit,” perhaps allowing users to sort into new vs. reply comments.
14. “Most highly voted” is questionable, especially in discussions with so few participants and non-rigorous recruitment.
Add:
- In discussions with more than 2 pages, link to “page 1,” “page 2,” etc., not just older vs. more recent.
- Make sure “reload” and messages about “exiting this site” are operating properly.
- Add boring “< to top” buttons.
Notifications – Very important for developing interactions
Comment voting
17 – 19. Very important. I would like to see some discussion of voting, though. I think I understand the attractions, but am concerned about the tradeoffs.
Moderation
22. Reword to acknowledge the various stages, e.g.: When a comment has been flagged by participants and is awaiting moderation, or afterwards, if a moderator has determined that it should be permanently removed due to a violation of the Terms of Participation, leave a note that explains and, after permanent removal, references the type of violation and links to the Terms of Participation. (Since many participants in Phase II, discussion 1 felt that “they” had removed, hidden, etc. a message, I would suggest including a phrase e.g. “flagged by one or more participants; removed due to .…” in the note.)
Add:
- Messages returned by moderators should go back to their original place in the list of comments.
- Clear, easy to find statements of policies regarding when messages will be shrunk or deleted, how the deletion process works, and where to find flagged messages are essential.
- Clarify the use of flagging to indicate duplicate posts; if this is an acceptable use, state this clearly in the Terms of Participation.
- Create more focus on discussion questions – for example:
– continue staff participation
– state questions simply; set off with color; give nuances beneath this frame.
– consider adding a link/popup on comment submission forms: “View discussion questions.”
General
24. FAQs – there is the Guide (http://blog.ostp.gov/guide-to-using-this-blog/); specific details might be helpful, and since “FAQs” is recognizable and short, that may be a better name, but I think one clear information link is better than two. Add this to the links at the top and bottom of each Web page! Might be formatted as short points with links to “Read more.”
26. Search – Yes! This point might appear in your list under Navigation?
27. Make e-mail address(es) for mechanical problems more obvious. (The first place I saw the “webmaster@” address was in the unfortunate messages about leaving the site; I’ve now seen it again, in the Terms of Participation.)
27. Weekends: If the site cannot be maintained on weekends, I would go as far as refusing comments (sending an e-mail to users who try to message). The problems on the first weekend of Phase II (discussion 1) made some users needlessly angry and suspicious.
For the record, some of the features that were not working properly at that time: a “reply” to a post sometimes came up at the end of the page; messages did not always post (when this happened to me on Sunday evening, I wanted to avoid a duplicate post, so I sent it to the “Contact us” address; it was posted on Monday); CAPTCHA error messages seemed to become increasingly frequent (for me, 3 on Sunday as opposed to 0 earlier). Also, moderation of flagged messages was on hold, I believe.
Wow, thanks for the detailed response. Good stuff!
1. I have found I have to save my comment somewhere else and attempt reposting usually two or three times before it will allow me to post on the OSTP Transparency Governance blog. After a while, the site seemed to detect my frustration but did not respond to it well, instead it began to tell me that I was “posting too quickly” and that I should wait longer before the next post.. even though I was only trying to make a single successful post. This occurred despite correct and prompt entry of the CAPTCHA code.
2. The site is overwhelmed with people who have no idea what the site is for. Most of them, across the board, have nothing better to do than tell their version of why they think Obama’s birth certificate was fake or some other birth certificate related issue. I flag these posts because according to the terms of use, they are “off-topic,” as a discussion of someone’s opinion of Obama’s birth certificate has nothing to do with defining transparency or developing transparency principles, but the reviewers seem to let the endless birth certificate comments stay, and as a result, whenever I made a post, the birthers would flag it and give it a pile of negative votes, regardless of my topic.
There is only one remedy that I can see to this problem.
a) Before a person comments, they should be required to select a comment category from a drop-down list. One of the drop-down categories should be “other.” If it is not a transparency definition or transparency principle, the person will need to select “other” and it will take the person to a seperate page where they can post about birth certificates with other birthers to their heart’s delight.
b) Also, when you click on the “post comment” button or its equivalent, there should be something that pops up that says, “Is this a (insert here the dropdown selection)?” For example, if before the post began, the poster selected “transparency definition” from the dropdown menu, the confirming question before the post is made visible should ask the poster, “Is this a transparency defintion you are posting?” If they click “no,” that should cancel the comment.
Thank you for the opportunity to make these suggestions.
Thanks for adding these issues, very helpful!
I just left a comment and was able to find what looks like a permalink in the source code for that page:
http://blog.ostp.gov/2009/06/10/enhancing-citizen-participation-in-decision-making/comment-page-1/#comment-9460
If that’s the case, it should be fairly easy to expose those to the users.
Another thing I noticed is that profile updates don’t seem to affect comments already made. In the comment mentioned above I still show up with just my first name even though I have updated my profile (nickname) to also include my last name.
Once you have user profiles, it’s not too far off to think about social networking as well. During phase 2, I’ve come across at least four comments from people I didn’t know but with whom I wanted to connect immediately (e.g. write a direct message or send an email).
It would be nice if participants could follow each other or “watch” the activity stream of certain users.
Hi Tim
have you posted these all on line to them and consolidated on list? I thought perhaps a link to your list could be included in the IAP2 submission currently under construction – let me know – see you next week! cheers Moira
I only shared a couple of ideas with their webmaster early on. Would be happy to get you an updated list (including what people have posted in the comments).