Social data—the data derived from social media platforms such as Facebook, Twitter, YouTube, Foursquare, LinkedIn, forums, blogs, and so on—are an invaluable resource for those seeking business intelligence, tracking public sentiment, responding to disasters, pushing information out to the public, and analyzing groups and networks of individuals. There are many challenges in making the best use of these data, however.
Technical and Analytical Challenges
By far the greatest challenge of social data is dealing with their sheer volume. Consider the following facts:
• 299 million daily active Facebook users (only 20% U.S. users); 350 million photos uploaded daily on average
• 310 Million Twitter users (33% U.S.; 15% Chinese); 100 million daily active users; on average, 500 million average tweets daily
• 1 billion visitors to YouTube each month; (1 Billion mobile views a day, Youtube Statistics) 6 billion videos watched each month; 100 hours of video uploaded every minute, in 61 countries (only 30% U.S. users)
• LinkedIn has 400 million users. 100 Million users access the site on the monthly basis and only 17% of small businesses use LinkedIn.
A specific analytic focus area narrows the scope, and thus the amount, of data that need to be analyzed, but these numbers demonstrate that in addition to volume, the analysis of social data requires the ability to handle foreign languages, to navigate a constantly shifting landscape, and to run algorithms in real time, or at least daily. While using an off-the-shelf analytic tool minimizes some of the technical challenges, these tools have severe limitations. They were designed to answer specific analytic questions and often fail to address others of equal or greater interest to a client. In addition, most vendors provide black box solutions, which provide no information on data source and validation, algorithms, or filtering methods.
The fast-paced changes in the global social landscape, vast amounts of data, and mixed media format make it a challenge for any organization to harness the power of social data. At Innovative Analytics & Training LLC (IAT) we understand clients’ unique needs and analytic challenges and can help them identify a solution that meets their operational goals. IAT’s solutions focus on the complete analytical cycle: The analytic questions drive the data and technology choices. The capabilities coupled with the analytic questions determine the methodology and training to be employed, and the analysis in turn helps focus the questions and data collection requirements.
Case Studies and Methodology
By executing mission-specific case studies, IAT can work with clients to identify which data sources should be used, how much data should be incorporated, and what kinds of questions can be asked of the data. In addition, we offer the best practices on integrating these data into each client’s current analytic workflow.
The need to understand social media data has spawned literally hundreds of analytic companies offering their own solutions. At IAT we keep up with the latest technology and trends and can provide an independent assessment both of datasets and of how well specific vendor capabilities meet the client’s mission goals.
Infrastructure and Computational Assessment
How many CPUs does it take to process 6 million tweets per day and 10,000 hours of video a month? At IAT we have experts in big data and high-performance computing (HPC) who can help clients understand the computational environment required to handle social data.
Many clients have specific needs that are not readily addressed by commercial solutions. At IAT we leverage our internal capabilities in social media, human language technologies, HPC, and visualization to devise custom solutions to your data needs.