Measuring community health: Achieving balanced social media growth

Note: The following is a guest post by Apollo Group’s Jorge Camargo that explores system thinking approaches to cultivating and managing online communities through a real-world case study of the University of Phoenix. I believe it’s a significant story backed by data that shows how strong community management backed by the strategic use of social analytics can increase the business performance of social media. - Dion Hinchcliffe

---

The content contribution pyramid reporting model enabled us to understand variations in context, purpose and participants within each of our communities while keeping an eye on overall growth trends. Conventional wisdom tells us Community Owners should rely on two key metrics to track the success of an online community: Membership (number of registered users) and participation (number of active users in a given time period). That’s well and good but what about measuring the health of the community, not just its size?

Back in 2010, our team at the University of Phoenix created PhoenixConnect, one of the largest academically-focused communities in the world with over half a million registered members, 40% of which are active participants every month. Moreover, students and faculty members who participate in the community spend an incredible 25% more time-on-site than other their peers who are not. This additional time-on-site is incremental, meaning that it does not cannibalize time spent in the classroom. As satisfied as we were with these numbers and their impact on our student’s engagement, we realized early on that we were only measuring the impact of the community on overall student behavior but not the behavior inside the community itself. The first step towards closing this gap was understanding who were these different types of users and what key indicators should we use to track their behavior within the community?

The Contribution Framework

Back during the “dawn” years of social media, people like Bradley Horowitz (then with Yahoo!) and Jakob Nielsen were already tackling this question and pointing out the participation inequalities between community participants, eventually coining the term “participation pyramid” to denote a 1:10:100 ratio between those creating content (creators), those engaging with content created by others (synthesizers) and those simply consuming said content (consumers). The major takeaway both Horowitz and Nielsen left us with back in 2006 was that as social media matured, it was imperative to remove barriers to participation in order to drive growth across all three user segments.

Figures 1 & 2: Horowitz’s Contribution Pyramid and Nielsen’s Pyramid

Our team liked the contribution model outlined by Horowitz and Nielsen so we used it as the basis for our Community Health reporting framework, iterating along the way. The result is encapsulated in the three maturity stages described below.

Stage 1 – Creating the Pyramid KPI and Segmenting Communities

First off, we decided to segment out our KPI reporting into creators, synthesizers and consumers – a simple enough task given the fact that our Jive SBS platform clearly reports the number of discussions, replies and views for each community. During the first iteration of our community we did not have the right instrumentation in place to measure unique contributors so we settled on the number of discussions, messages and views for a community as good enough proxies for creators, synthesizers and consumers respectively. We plotted the data (messages/replies per discussion threads and views per discussion thread) for our four major communities and came up with the following visualization aimed at allowing us to understand the distribution and behavior of users within each community:

Figure 3: November 2010 Contribution “Pyramid” for 4 major PhoenixConnect communities

Which community do you think is the most balanced? This is our interpretation and how we use this data to drive decision making and investment:

• Support Communities seem to be underperforming relative to its peers but we quickly abstracted that this community is clearly transactional in nature (question-and-answer) so we would expect a much shallower degree of engagement evidenced by low replies/views per discussion numbers.
• Academic Communities have the highest synthesizer ratio (messages per discussion thread), which is great since these communities are aimed at encouraging constructive academic discourse between our students and faculty. However the consumer ratio is the highest of the bunch (126 views per discussion), which implies people are viewing the discussions but are not as incented to participate or engage. This insight helps us drill deeper into the behavior inside these communities to understand whether or not we have a content quality problem.

Stage 2 – Removing Barriers to Contribution and Consumption

As we discussed earlier, Horowitz and Nielsen both argued in favor of “removing barriers to participation.” In early 2011 we made two major changes to our community platform that addressed this specific problem:

• We upgraded from Jive 4.0 to Jive 4.5 in order to take advantage of the improved contribution features, particularly the revamped activity feed and the “Like” button (which improves ultra lightweight participation)
• We surfaced the activity feed to the home page of our student portal in order to increase awareness and consumption of community content.

After releasing these changes, we adjusted our KPI metric slightly (to account for “likes” as indicators of synthesizer contribution). The results can be seen below:

Figure 4: August 2011 Extended Contribution “Pyramid” for 4 major PhoenixConnect communities

As you’ll notice at first glance there was a dramatic shift in the ratios: contributions from synthesizers and consumers tripled thanks to these two key enhancements. In short, we were able to see the return on investment in both a Jive upgrade as well as a custom activity feed widget on our student portal homepage.

But aren’t these ratios misaligned? The quick answer is yes. Like many first generation communities out there, PhoenixConnect heavily relied on user-generated content (UGC) to seed conversations. UGC is great to get a community off the ground but it’s far less effective when it comes to creating a steady seeding of relevant, on-topic and insightful conversations at scale. Our ability to generate fresh, engaging content could not keep up with the newfound demand from our community. This data-driven insight was used to inform and adjust our community strategy particularly around content creation through a two-pronged approach:

• Create a community content team responsible for generating monthly editorial calendars that will generate a consistent source of new content our community can congregate around
• Break down barriers to content creation by introducing new objects beyond discussions. These include capabilities such as blogs, ideation and most importantly, the upcoming Jive 5.0 What Matters stream to bring improved discoverability and relevance to status updates

Stage 3 – Disseminating Insights and Keeping your Team Accountable

Constantly pivoting and course-correcting is a must for any Community Owner; however it can be a dizzying prospect for other people in your organization who don’t care or don’t need to care about the details. In order to keep a consistent KPI metric for community health that can be disseminated inside of our organization and easily interpreted, we created a simple “score” based on a contribution-pyramid inspired algorithm:

We made sure we weighed this score against our all-time highest activity day to give us both an easy-to-read percentage as well as a good frame of reference for how active our community is at any given point in time. The graphic below gives you an example of how we report “Health” for a particular PhoenixConnect community:

Figure 5: August 2011 Community Activity Score for a PhoenixConnect community.

Note that the huge jump in activity in March 11 was a result of the enhancements described in Stage 2 above and the ensuing “shiny penny” bump consistent with new features

Each and every community out there will have its own particular intricacies and you organization will surely require you to adhere to its own KPIs and reporting frameworks. In our case, the content contribution pyramid-inspired reporting model was a very valuable addition to our reporting toolbox. This KPI enabled us to understand variations in context, purpose and participants within each of our communities while keeping an eye on overall growth trends.

About The Author

Jorge Camargo is Director of Product Strategy & Development at the Apollo Group. You can find him soon at his session at JiveWorld 11 where he’ll discuss this framework and other metrics for internal/private community.

Apollo Group, Inc. is one of the world’s largest private education providers and has been in the education business for more than 35 years. The Company offers innovative and distinctive educational programs and services both online and on-campus at the undergraduate, master’s and doctoral levels through its subsidiaries: University of Phoenix, Apollo Global, Institute for Professional Development and College for Financial Planning. The Company’s programs and services are provided in 40 states and the District of Columbia; Puerto Rico; Latin America; and Europe, as well as online throughout the world.

cfa|brady hoke|fireworks|chicago tribune|betty broderick