{"id":797,"date":"2015-02-24T06:48:16","date_gmt":"2015-02-24T06:48:16","guid":{"rendered":"https:\/\/bitvore.com\/?p=797"},"modified":"2017-06-28T04:13:47","modified_gmt":"2017-06-28T04:13:47","slug":"extracting-value-from-the-unstructured-internet","status":"publish","type":"post","link":"https:\/\/bitvore.com\/2015\/02\/extracting-value-from-the-unstructured-internet\/","title":{"rendered":"Extracting Value from the Unstructured Internet"},"content":{"rendered":"
\n

\"intelligence-gathering\"<\/p>\n

Gaining insights from unstructured data is valuable but requires a daunting collection of skill and technology.\u00a0 Gartner says “Unstructured data growth is rapidly outpacing structured data and is poorly controlled, stored and managed on file shares, on personal devices and in the cloud\u201d. \u00a0They conclude that it isn’t a storage\u00a0problem, it’s an intelligence problem.<\/p>\n

I shook my head when I read a recent article in Datamation about the “9 Steps to Extract Insight from Unstructured Data”<\/a>. \u00a0 The article makes a great case for why we invented Bitvore. \u00a0 I’ll summarize the challenge: \u00a0figure out where and how to collect the information, establish a data lake, invent a taxonomy, master a variety of natural language analysis methods and algorithms, then begin.\u00a0 It’s no wonder few companies attempt it.\u00a0 Once you get the budget, hire the team, buy and integrate the tools to do this, you are at the starting line. You have to have a strong ROI case to get this class of project budget approved.\u00a0 This article does not address the additional\u00a0challenge of analyzing continuous streams of information for new intelligence rather than a static data set for historical insights. \u00a0 The cost and skill keeps this kind of project out of reach for all but a few companies.<\/p>\n

However solving the problem is very valuable.\u00a0 In the end, you have created a surveillance platform to “listen” to your market and “know” before anyone else.\u00a0 Some of the obvious beneficiaries of this solution are hedge funds, portfolio managers, commodities houses, insurance companies, and the government. \u00a0 These organization all take actions based on knowing that certain external events are happening. \u00a0 Many companies have formed competitive intelligence teams and business development teams chartered with “knowing” what is happening around the company though their toolkit is often manual dredging using search and information services like Lexus Nexus and Hoovers. \u00a0 Marketing and sales also receive great value from understanding changes in markets, messages, feedback and customers.<\/p>\n

Why is Bitvore such a giant leap forward? \u00a0 Bitvore essentially collapses the twisted maze of technologies and techniques into one integrated system. \u00a0 The result is a time-to-value for projects that is measured in days instead of months.\u00a0 This opens the door to attacking use cases that could never justify a positive ROI using a custom-built in-house\u00a0system. Bitvore ingests, stores and analyzes many parallel sources and streams of unstructured and semi-structured information continuously in the cloud.\u00a0 No data lake needs to be established.\u00a0 No new technologies need to be integrated.\u00a0 Even the core natural language processing techniques are built into the system.<\/p>\n

An example of an application of Bitvore being\u00a0put to work on\u00a0Wall Street today helps explain how it works. \u00a0 Today this application of Bitvore is indexing content from 13,000 sources of news, social media and email.\u00a0 To gather the information, the system is “aimed” both using explicit URLs, hashtags and email addresses, plus guided crawling using keyword logic to scan search engines and social media hunting for relevant information. \u00a0 Found data is automatically indexed by\u00a0the system with time stamps for the indexing\u00a0time and publication time. \u00a0 In parallel to the indexing\u00a0engine, analysis processes are running against all the data – old and new.\u00a0 In essence, the indexing\u00a0system hoovers up great slices of the Internet based on the needs of the application and the analysis system inspects and enhances the information.\u00a0 New sources and adjustments can be made on the fly without interrupting the continuous processing.<\/p>\n

In this application, the system has several missions for the analysis. \u00a0 First it is watching for several hundred specific material situations that our customers want to know about right away.\u00a0 These material situations happen across the US dozens of times daily.\u00a0 When they occur, the customers has an information advantage to make better decisions over everyone who is unaware.\u00a0 In addition, the information needs to be sorted by specific geographies down to the town, county and state.\u00a0 The industry sector also has to be identified. \u00a0 Similar to SIC codes, customers need to know if the information is about hospitals, transportation, corporations, schools, or dozens of other sectors.\u00a0 Additional classification determines if the information is about elections, crimes, lawsuits, regulations, taxation, opinion pieces and other topics.\u00a0 And last, the suppression system gates the irrelevant and redundant from reaching the user.\u00a0 Suppression is critical to removing false-positives and ‘echo chamber’ repetition from news syndication and retweeting.<\/p>\n

The analysis techniques come from an extensible pallet of tools built-in to Bitvore.\u00a0 The tools, range from relatively simple keyword logic (similar to Googles “advanced” search), and regular expression analyzers, to built-in trained Naive Bayesian classifiers and Vector Space analysis.\u00a0 One tool allows new algorithms to be added at any time using scripting in a host of popular languages including Javascript, Python, PHP, R, even LISP. \u00a0 The job of the analyzers are to examine the source information and add metadata information to enhance the original indexed information. \u00a0 Like the indexing system, analyzers can be added or modified at any time without stopping the continuous process.\u00a0 Even the script-driven analyzers can be added on the fly.\u00a0 The ability to adapt the system on the fly is paramount. \u00a0Evolutionary discovery, testing and adaptation are a key workflow in operating a system that monitors changing live data streams.<\/p>\n

The metadata tags in Bitvore provide structure to the unstructured. \u00a0 Because of their flexible nature, metadata tags provide a fluid, malleable structure that can be expanded or re-organized as needed. \u00a0 The metadata yields the benefits of a structured system because the tags can be queried and return instant results. \u00a0 A language is included to formulate queries so that any slice of the information can be extracted to create data sets, search results, alerts and trigger actions in other applications. For example, a simple query in the application above can set alerts on “any new lawsuit filed against a hospital in Pennsylvania” or “Any regulatory change impacting the energy companies” in my investment portfolio.<\/p>\n

While Wall Street’s interest in Bitvore is obvious, executives across industries are excited by the ability to immediately “know” about all external events that will impact them from actions at their suppliers, customers, competitors and regulators. By harvesting and analyzing the Internet at scale, Bitvore will bring to an end the ad hoc searching and word of mouth information sharing that is the de facto solution for business intelligence gathering today.<\/p>\n<\/div>\n

\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Gaining insights from unstructured data is valuable but requires a daunting collection of skill and technology.\u00a0 Gartner says “Unstructured data growth is rapidly outpacing structured data and is poorly controlled, stored and managed on file shares, on personal devices and in the cloud\u201d. \u00a0They conclude that it isn’t a storage\u00a0problem, it’s an intelligence problem. I […]<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/bitvore.com\/wp-json\/wp\/v2\/posts\/797"}],"collection":[{"href":"https:\/\/bitvore.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bitvore.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bitvore.com\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/bitvore.com\/wp-json\/wp\/v2\/comments?post=797"}],"version-history":[{"count":0,"href":"https:\/\/bitvore.com\/wp-json\/wp\/v2\/posts\/797\/revisions"}],"wp:attachment":[{"href":"https:\/\/bitvore.com\/wp-json\/wp\/v2\/media?parent=797"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bitvore.com\/wp-json\/wp\/v2\/categories?post=797"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bitvore.com\/wp-json\/wp\/v2\/tags?post=797"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}