A tweet and DM by @Autom8 on Twitter inspired this post. Autom's tweet linked to a post entitled Sentiment analysis for online content: Honest? which raises some fundamental questions on the reliability and validity of sentiment analysis.

Here is a quote from that post:

Recently a customer asked why we have not yet included sentiment analysis in My.ComMetrics.com. Yes, the customer was fully aware that sentiment analysis (also called opinion mining) involves classifying text using natural language processing, computational linguistics and text analysis to reveal the sentiment (e.g., positive, neutral or negative) of a particular text.

The post goes on to raise a series of interesting questions on the reliability and validity of sentiment analysis and I would suggest reviewing it in its entirety. The one thing that piqued my curiousity was the use of the word "honest" to perhaps describe sentiments place in social media, or as a means to balance the reliability/validity aspects with our own expectations and experiences with sentiment analysis as a valid form of social media measurement.

When I think of describing something as being honest, it normally applies to the dynamics of human interaction. For the most part, vendors in the monitoring and listening space are performing content review and assigning sentiment scores through machine analysis with little, if any, involvement of people. If brand monitoring and listening tools are to make statements on the accuracy of sentiment analysis, then questioning the honesty of those claims is necessary, especially if we place it in the context of questioning whether vendors are doing enough to describe any/all issues relating to the reliability and validity of assigning machines such a daunting task.

I think the takeaway from this post was that sentiment analysis is still at a "test-drive" or early stage, and that bigger and better things are yet to come. Though I wonder if enough questions are being asked at this stage in its evolution to aid in its maturity and progression.

We have been offering sentiment analysis since 2006, and we certainly weren't the first to offer it at the time. We recognized even earlier that there were (and still are) significant limitations in assigning computational linguistics and text analysis the job of distinguishing between things like sarcasm and risk to brand reputation. We believe our approach strikes the correct balance of precision and quality by using the machine processing portion to prescreen online incidents, while allowing the metric validation portion to be handled by human readers.

Has it proven to be reliable? We think it has and our clients have been our biggest believers. And while there are plenty of advantages to assigning human review to the task of finalizing things like context, handling, influence and risk interpretation, has enough been done in the last few years on the machine side to appease audience expectations on quality and reliability?

Moreover, is sentiment analysis regarded as having the kind of overriding measurement authority to justify the required R&D investments to meet or exceed the expectations of audiences? If one of the responses to this question is that it's too early to tell sentiments place in social media monitoring and measurement, then I would have to believe that over the last few years some degree of complacency has already set-in and effected its ability to mature. One thing that is certain is that there are many more questions to ask during the time sentiment analysis goes from a "test-drive' stage to full scale mass production, so lets put sentiment first, and ask questions now rather than later.

Post on Twitter