Tag Archive for 'Metrics'

OpenGov Tracker

Published by Tim on February 15, 2010 in Metrics and Radar. 3 Comments Tags: , , , , .

A few days ago, a new site was launched that helps track activity across the 23 federal agencies that are using IdeaScale for their consultations on Open Government: OpenGov Tracker

This webpage makes use of the IdeaScale API to aggregate data for all agencies which are making use of that platform. USDA and Health and Human Services also have public feedback sites; however they are not using IdeaScale and so unfortunately, are not included here. We encourage you to visit their sites, too!

Totals reflect all ideas including those submitted to the “Site Feedback” category. The “Top Ideas Across Government” section only highlights ideas in the categories of Transparency, Participation, Collaboration, and Innovation– eg., those related to the Open Government Plans themselves.

The site was created by Robbie Schingler and Jessy Cowan-Sharp, who both work at NASA.

This is a most excellent example of how participation tools can expose some of the quantitative information about an ongoing consultation (IdeaScale itself does some of the basics, though does not offer a convenient view across all 23 projects).

It’s especially nice to see how metrics are being made actionable. The section “Needs Some Love” lists those agencies that haven’t received a lot of input yet along with the call to action: “They need some love. Please help!”

This metrics dashboard is a nice addition to my tracking thread on GovLoop where we also try to keep an eye on qualitative data.

There is of course a lot more that the perfect dashboard could include but this is definitely a refreshing step forward.

Please leave a comment below if you know of any tools that offer this kind of eye candy out of the box.

Share Your OpenGov Forum Tracking Data on GovLoop

Published by Tim on February 9, 2010 in E-participation and Research. 3 Comments Tags: , , , .

Following up on my list of ten things to monitor, I’ve started a thread on GovLoop: Monitoring and continuous evaluation of OpenGov forums

Here’s my initial post:

As part of my ongoing efforts to monitor the various feedback channels that have been launched as part of the Open Government Directive, I’ll be tracking some basic metrics. I thought I’d share the raw data here on a regular basis so people can chime in and add any other insights they’ve come across.

23 agencies are using IdeaScale, which exposes the following in real-time:

  • Number of ideas
  • Number of comments
  • Number of votes
  • Number of registered users

So that’s nice. I’ll check in on the other two forums occasionally as well (here, here).

Aside from cold, hard numbers, I will be looking for examples that illustrate how various agencies are doing in terms of the other nine items on my list. Here it is again in short form:

  • Expectation management
  • Community ground rules
  • Level of convener involvement/participation
  • Quality of moderation
  • Quantity of participation over time
  • Outreach and diversity of participants
  • Conclusion and impact
  • Tech support
  • Project communications
  • Mood

I’m using this RSS feed for easy scanning but don’t expect to catch everything. Please share your impressions in the comments.

A set of screenshots will be available on Flickr.

I hope this will attract enough metrics junkies so that we’ll end up with a fairly complete picture in the end. It would certainly be great to see not only a breadth of data points but also to have a discussion about what should be measured and how. After all, the participation piece of the upcoming dashboard still hasn’t been designed in detail yet. Maybe this little exercise will create a few valuable insights.

“No Better Time” Session Notes Now Available

Published by Tim on September 6, 2009 in Deliberation, Events and Research. 0 Comments Tags: , .

Session notes and related resources from the “No Better Time” conference at the University of New Hampshire back in July are now available for download on their resources page.

Measuring and metrics was a hot topic that was discussed at the following learning exchanges:

  • Embedding Deliberative Practices in Local Democracy (session notes, PDF):
    This would need some further digging into, but lots of interesting tidbits and potential starting points for assessing impact and ROI, various ways to measure “embeddedness” over time etc.
  • Making the Case for this Work: Improving the Way We Collect, Report, and Explain Outcomes (session notes, PDF):
    The goals of this session were “to a) Identify where the field/participants are with regard to evaluation; b) Identify where the field/participants want to be with regard to evaluation; and c) Identify strategies to get the field/participants to that place (how are going to get there?)”
  • New Frontiers for Assessment and Measurement
    No session notes available, but check out “Impacts and Outcomes of Participatory Processes in Brazil and Canada” (download, PDF)
  • Renewing the Research Agenda (session notes, PDF)
    Only a brief outline here (“Assessment was the main theme. One view was that questions about what works, under what conditions, and why can be broken down into researchable parts and studied (quantitatively or qualitatively), with the gold standard being a randomized experiment.”), would be nice if someone could add some detail.
  • The “Downside” of Deliberative Democracy (session notes, PDF)
    Very little detail, except “doing ‘diagnostics’ and ‘assessment’ before planning a deliberative project” was discussed, apparently.

Please leave a comment if you have additional notes or comments you can share. Thanks!

Intellitics at IAP2 Mini-Symposium “The Future of Public Participation”

Published by Tim on June 25, 2009 in E-participation, Events, Presentations and Research. 0 Comments Tags: , , , , .

On Monday night, the International Association for Public Participation (IAP2) Northern California chapter hosted a Mini-Symposium on “The Future of Public Participation” in San Francisco, CA.

I had been asked to give a brief talk about social media in public participation. Having recently spent a considerable amount of time and effort monitoring and tracking various e-participation initiatives in the US, I decided to share a few of my observations. I picked five example projects that have relied primarily on using off-the-shelf web 2.0 and social media tools:

  1. Rebuild the Party (UserVoice)
  2. Change.gov: “Join the Discussion: Healthcare” (IntenseDebate)
  3. Change.gov: “Open for Questions” (Google Moderator)
  4. Open Government Dialogue, Phase I (IdeaScale)
  5. #MyIdea4CA (Twitter)

It is important to note that a lot of the insights I present are still a work in progress and are often based on incomplete information. Due to the short time I had available at the event I wasn’t able to go into a lot of detail.

Aside from showing the data and interpreting the results from these specific projects, I was equally interested in drafting what might at some point become a general scorecard system that could be applied to e-participation projects of all kinds, shapes and forms. As I mentioned before, there are quite a few common metrics or key performance indicators that are necessary in order to monitor a project’s success. A standard evaluation framework could be incredibly useful for both conveners as well as tool providers. For example, such a framework could inform an Open Government project directory and knowledge base (and of course, a collection of case studies and project reports would in turn also help improve any framework).

Here are the categories I started out with to describe each project:

  • Title
  • Host
  • Objectives
  • Duration (start and end date)
  • Type (e.g. brainstorm, discussion etc.)
  • Tool(s)
  • Key participation metrics (e.g. number of participants, number of ideas, total word count etc.)
  • Challenges
  • Impact

So without much further ado, here are Monday’s slides:

Off-the-Shelf E-Participation? Five Recent Examples

I’m aware that there’s already work being done in this area and I’ll link to those efforts in one of the next posts on this topic.

What I’d like to work on next:

  • Get more specific regarding the public participation goal (as defined by IAP2 Spectrum of Public Participation)
  • Sanity check some of the assumptions or perceived results (e.g. by way of interviews with the conveners or organizers)

As always, feel free to chime in with comments or suggestions.

Open Government Dialogue: Phase 2 Metrics Update

Published by Tim on June 9, 2009 in E-participation, Events and Research. 1 Comment Tags: , , , .

Phase 2 of the Open Government Dialogue has been under way for almost a week now. Following the brainstorm phase, which ran from May 22 through May 28 (see our coverage here, here and here), this discusson phase was launched June 3 and is being hosted by the Office for Science & Technology Policy (OSTP).

The tool used for this phase is Wordpress, a free and open source blogging engine that powers the OSTP blog. Threaded comments come standard with the latest Wordpress 2.7 release. For comment voting, it appears they added the Comments Vote plug-in:

This plug-in enables users of the website to vote comments up or down. This running tally can be displayed on the comment, and administrators have the ability to activate the collapsing of comments with a designated negative rating. The comment will still be available for viewing by clicking on a “show comment” link, which expands to show the collapsed comment’s contents.

Unlike IdeaScale (the tool used in phase 1), there doesn’t seem to be a page that shows key participation metrics (e.g. number of registered users, number of votes cast etc.). The only thing that’s shown is the number of comments on each blog post.

Based on this limited information plus some plain old manual counting, here’s some of the activity we’ve seen on the four blog posts as of this writing:

That’s a total of 394 comments over 6 days (note that this number doesn’t include comments that have been removed).

Comments that are deemed off-topic (or that otherwise violate the terms of participation) are moved to the flagged comments section. Here’s an overview of how many comments have been demoted since the discussion phase started (flagged comments appear under the date they were first posted):

To date, a total of 142 comments have been removed to the flagged comment section.

The total number of votes is not exposed but since most comments seem to get no more than a handful of net votes and very few comments gain ten or more net votes, it’s safe to assume that the total number of votes cast across all participants lies in the hundreds, rather than in the tens of thousands as was the case in the previous phase.

The number of registered users is also not exposed. Again, my guess would be that only several hundred participants have signed up so far (a lot fewer than during phase 1).

The word count for all 394 comments is approximately 48,000 words, or about 120 words per comment (for comparison, this number is very similar to what we saw during the Change.gov discussions).