Measuring community -- setting targets
Ben van't Ende - community manager with TYPO3
got into community management from former job
for first couple of years, he didn't have to give statistics or anything... still a relaxed job, but in beginning it was...
now it has advisory board, as part of new structural change
being paid "almost full time"
created some targets, getting more women into the community, getting more metrics
he finds himself... the job is very reactive
all kinds of things happening, and the targets are not being met... finding more reacting to certain needs in the community than being able to make some kinds of graphs about what you have achieved
So question -- do you have to report? how would you report about interactions, soft skills? how would you explain or quantify that?
quick round of introductions
Kyle from Twilio, community content manager
Janet, with Mozilla, technical evangelism
Neil, works in Fidelity investments, in central IT
developing internal communities
Yani, at fetlife.com
Jason Hibbets, works at Red Hat, at opensource.com, and that includes community management
John, not safe for work
Jeff, work for Al Fresco, lead the community, doing community managemetn and technical evangelism.
Jen from WordPress, remarks on how loud it is
Ben has gone to ask people to quiet down, which slightly succeeded -- (editor's note: success through delegation)
Mary Thenevall, community manager for Velocity space at O'Reilly
Heather runs National Novel Writing Month
Martin works for HP on community outreach
Asheesh..
Jen Lopez, director of community at Moz, we make analytics for marketers, the community is one of marketers and constantly trying to find how to measure our community. Community team within Moz is part of marketing team
Megan, works at Moz as well, works on communty team
Krishna, here to learn, community manager for startup creative community
Hi, Pernima (?), works for Neo4J on community team as developer evangelist; we're beside marketing, community team is in Europe, and marketing is in US; it's hard sometimes to collaborate and also what we will be measuring
Ben:
How do I show my results? We increased # of code sprints, # of bar camps... but his stamp is not on it. Also hard to measure your turnover (in the sense of results provided)
Jen:
We're in marketing team and find it difficult. Most of team is focused on getting people into software, taking a trial, then buying the software. We are focused on building the greater community, we have a lot of free education resources, free things, have a big community presence. Difficult to say we know that doing these things is building more paid members. So our number is the things we track back to our greater goals around engagement. Increasing engagement in the community helps to increase awareness and knowledge and education which is what the community team focused..
Q. how do you define engagement
A. # of blog posts, # blog posts submitted by users, have Q&A forum where people answer things, or on Twitter, @ mentions, retweets, tweets we send
Q. How do you gather data?
A. Painstakingly. There's not one that has all the stuff... Google Analytics has some things for this, Moz Pro tool has some data, and have to go to Facebook for some data, Social Engage for Twitter... becomes v difficult.
(open discussion)
more touchy feely. They want a weekly report... what numbers do they want? What tells the story we want to tell?
Asheesh says,
Jen says, working on a satisfaction survey of how excited WordPress affiliated events are with WordPress central. (Editor's note: Asheesh wants this.) + Open ended things like what are we doing well. The @ end of year, you could say, "We ran 68 wordcamps so this will be a focus for how to do better."
Stefano and Erich asks if the community team self-determines metrics? Answer is that yes, it's within the community team.
Ben: Barcamps
Jason says, it's about setting the goal, and then finding the best way to achieve that goal.
Mary says, question -- setting those goals, part of ... my job is to find out what people are looking for, and then talk with editors and conference chairs to find out what people need.
It's touchy feely -- met with a meetup that had 60 people, told them about Velocity... but don't have a way to report
Jason -- you can control how many groups you go to.
Jen -- if you look at traditional marketing, those initial contacts and it being hard to record are just part of life in marketing.
So then the question is, how do you respectfully reset your boss's expectation. It's a brand new position, it's a different role... we're trying to see if it works. I (Mary) need to prove that I'm doing the company some good.
Jen: So if you want to 60 meetups, could you at the end of 3 months, ask meetup organizer to send out a note saying, "Please reach out to me if you want to attend, or did attend the conference, and just chat with me" and then you'll get a qualitative sense. Data in conf reg form might say "My boss told me to" when really Mary planted the seed.
Maybe one of the things is -- if you can't measure it, don't do it. So you have to find a way to measure it. For instance, do you want a 10% off code -- I'll email you the 10% code. Then you have a list of email address and you have created a standard-ish funnel. "You can't leave until you write your email address down!!"
Twilio trying to get people to use a webRTC wheel of fortune demo, with phone numbers.
Can someone explain for me, why do you want people's email addresses?
Two diff reasons -- one is to contact them to push them to sales-y things, and another is to just measure
nanowrimo -- we can match up an email address to later signups, and we can measure conversion rates
If it's a product, you can say, "Do you want a free demo tomorrow?"
We need to A/B test all the community things we do. At the end of the day you want to be lazy, and work on the thing that works best
nanowrimo -- we have a lot of stuff we have to do, and wants to find the most efficient path. This where we post here, and post here, and we have to keep them under some control... and we want to find out how they are using the site, and how to serve them better.
One event is very experimental, and the way it is now is the way they want it to be. Tried to figure out what the community wants... how to translate what I do into making the entire organization more efficient.
Q. What's the output that you want?
A. What we want is year round engagement... hemmorhage users toward the end of the event. Want ways to keep them engaged... why stick around if our novel writing is in november
Q. Do you run surveys.?
A. We do post-event surveys... Asheesh suggests that you email people who left.
Maybe they don't know there's a reason to stay.
Q. Do/could you send them a nudge? Come look what we're doing... and I don't know if those help
Q. Why are they coming?... then why not do that every month?
TO Janet, "community is people coming to each other". These are people who like to write novels. There's people who like to write in November, and those who need to write all the time... focus on those that work all the time...
"We have p
Q. What happens after November? How about New year's Resolution season?
"Not look what we're doing" but "community highlights" -- go in and pick users who are engaged, and showcase them, and "nine times out of ten we get it" and it's pretty low touch if you send them interview questions... you get more success if you highlight what they're doing rather than how it relates to you.
Jason strategically thinks about what content and users to highlight
Ben made "member of the typo3 community" towels!
Q. What are some other highlights? Social media and contributor...
A. (Jason) I'd have to go look that up. "Top contributor of the year", "top commenters", "comment gardeners" for people who are (Ben's friend Robin is a comment gardener)
Asheesh suggests: "Split people up and do different things to them and see if they act differently"
Ben: Could you quantify things are going on in the community? To justify to other people.
Jeff Potts -- there are people in commercial open source who wonder what it is that we do and why. How do educate others about your manager?
Stefano says, your role must not be able to be split from the business objectives. So I would encourage you to anchor your results to the business objectives.
Jeff says, What we do is sow long seeds that take a long time.
Stefano says, Same with everyone in the company, really... one way for me is to look as communities as funnels, borrowed from Lars/Xen... if you think of it that way, then you hook yourself up to the marketing funnel.
Mary: I think one of the hard things if you have a successful community, your name isn't the one out there... then you're still contributing, sure. Probably-it-was-Dawn says, "I know it's a huge problem if my name is in the list of top contributors".
Other person: One of the challenges -- and this is true in any marketing function -- linking to direct sales. But with any svc or software company, maintenance or support... but communities make it harder to move onto a different platform. Makes your platform hard to move away from.
Jen: We think the easier it is for you to leave, the more comfortable you are staying.
Ben: when it comes to community: pillars are growth ("oh shit, we keep on growing, month over month"), retention ("want to see that constantly coming back more and more often") and happiness ("a huge part... I don't know how it can be measured"). We let a community manager go because we couldn't measure what she was doing. As community managers, tell your boss, you can demonstrate that if these are the five competitors, our community is stronger in these ways...
We have "fluffy tools", maybe isn't easily quantifiable on the surface... might not directly correspond to sales, but you can still look at sales as a whole.
Janet: One other thing Mary said -- if community manager is the top contributor, well, it depends on the community, you want to see the proportion of discussions started by other people going up.
What do you when community sense of the mission and the corporate sense of the mission are different.
Ben (the not safe for work person): If the two are super duper different, then you are going after the wrong community or your company is blind. Tell them, "I believe if we do this, we are going to increase sales."
Jason: Part of that is managing expectations. We have users that make blue-sky suggestions that we are never going to do... tell them, our web dev is slow and it might be six months, if ever...
Jen from Moz says, "We are basically the same way" someone said, why don't you just build a tool to do that, and I said, That's not the sort of thing we do.
Ben (not safe for work) person: Be careful about the words you use, too, people might eventually believe their input is not worth it yet.
Ben (typo): Thanks all!