This is an answer to a recent post from Marc Rod, starting a petition to bring back our own commenting system. Marc, I'm glad you've noticed that we changed our comment system. If this would have gone unnoticed, I would have been worried about the job we've done so far. I'm also glad you like our system, I'll pass that on to our design and tech team.
So why did we remove it? We did it solely to generate some data, and we'll shortly bring back the original MP comments. Let me give you a little context, but this might get a bit techie.
Moviepilot has managed to grow massively in the last 12 months. We've grown around 600% year over year. In October 2013 we had around 8 million visits in a month, in August 2014 we recorded almost 55 million, and September looks to be even better. Growing a platform and building momentum is important for any startup, so this feels great. In detail, there are two main drivers for this growth.
First, we see a great increase in articles being published by the community. A year ago we had less than 100 articles published each week, now we see over 2000. Also, as the community activity is growing, we see more comments, more signups, and the general activity on our platform is accelerating. And we couldn't be more proud of that!
Secondly, Facebook has been a vital part in the traffic that arrives at Moviepilot. One dimension of that traffic is our own Facebook pages. The other dimension is sharing and other referral traffic that we get as an echo from people who enjoyed what they found here. Now there's a specific way that Facebook decides when and how to bring this traffic to a website.
A brief overview on how Facebook works
You've probably noticed that your current Facebook stream lists a lot of publisher stories. Internally we refer to this as the 3rd major generation of the Facebook stream. In the beginning, it was all about Friends. The 2nd generation was all about gaming and achievements (anyone remember all the FarmVille stuff in your stream?), and now the 3rd generation is all about publishers. You probably followed the discussion about click-baiting and how Facebook changed its algorithm recently to remove unwanted posts from your timelines.
We're glad Facebook continues to improve the distribution and tries to remove disappointing content. We believe this will help our stories to be seen by even more people. The signal that Facebook uses to generate the distribution of a story (how many people see a specific story in their news stream) is based on the amount of interactions with that story. So the more people click on a story, share a story, or interact with it in any way, the more this will effect its visibility on Facebook. One part of that interaction is the amount of comments and shares of our status updates (e.g. this update), and the second part is the amount of shares by our readers. Facebook calls this Site Engagement. The more Site Engagement, the more visibility. Here is a graph of our Site Engagement from Facebook Insights.
Until now, the only signal that Facebook got from us were the sharing actions. We do not use the 'like' button, and until now we didn't use the Facebook comments plugin. So the only signal we could send back to Facebook were shares. We want to figure out if more signal has a positive effect on the distribution. The easiest way to do that is to remove our commenting system and include the Facebook plugin. You can see the increase of Total Actions on the last day on the graph above.
How the distribution changed
This experiment has been running for a single day now, so it's too early to give a definitive answer on if this change had an effect or not. We could see a slight increase on the distribution on Facebook, but not as much as we estimated. As you can see in the graphs, the sharing and distribution is connected. Usually, the more shares, the higher the distribution. Our theory was that more signal would increase the distribution, even if the amount of shares is not increased. The distribution was slightly higher yesterday, but not very significantly. So, we need to gather more data to see if there would be a long-term effect.
Interestingly, the Insights graph does not allow you to plot "Impressions via Story Comments." So maybe the extra signal that we send has an effect on the distribution, but just not on the distribution graphs on Facebook Insights.
Most importantly we wanted to raise the amount of referral traffic from Facebook to Moviepilot. Usually, the distribution and the referral traffic are connected. In short, when distribution rises, traffic rises. There has been an increase in traffic, and it is above average. So it might be connected to the extra signal that Facebook received. Again, just one day is not enough data to come to a real conclusion here. Nonetheless, it is noteworthy that yesterday's traffic did raise some eyebrows here in our traffic and Facebook distribution team.
So, when is the comment system coming back?
For now, we're going to continue the experiment for another day or two. Please rest assured that traffic is not the only factor in deciding what to do with the comments. All the points raised by Marc are very valid. Also, yesterday's over all number of comments was lower than we would have expected. So it's also fair to assume that the FB Comments plugin might have a negative effect on our community. And quite frankly, we expected that. Expanding our community and providing a great platform for you to share your ideas is our primary goal. So, please bear with us while we finish this experiment. Here we'll outline what we're going to do next:
- We will let the experiment run over the weekend to gather more data
- We will bring back the original commenting system (including the notifications) by Monday, at the latest
- We will continue to be transparent about the data we receive
- With all the information at hand, we'll discuss it with you (the community) and decide internally how to proceed
So, that's the full low-down of the developments. I hope that it has cleared up any questions you had about this change, and if something was still left to be desired, feel free to write us, or create your own article!
*) Note that we actually do send other signals to Facebook. Our own commenting system uses Open Graph actions whenever a user leaves a comment, answers a poll, etc. Ideally, the signal from our own commenting system should be as important to Facebook as the Facebook comments plugin. This is what we're testing right now.