Research Article: The most valuable data that is hiding in an organization!

Where is the valuable data hidden? Who can reap it's benefits?

Let's start this idea from the beginning, to understand why data is valuable.

There is always a need to know more. There is scope for knowing more in any subject, similar with data too.

We know that the business transactions create large volumes of data. This data is used in decision making, even at the top levels in an organization.

Why are organizations collecting more and more data?

We know that all the data is collected and used for analysis purposes.

But, lot of data that is collected is thought as not so useful and discarded too.

Why is the field of Data Analytics gaining importance?

Well, we want to know the customer better so that we can cater well to the needs of our customers. Businesses calculate Return on Investment (ROI) to measure how the analysis helps in gaining a competitive advantage.

Unless the businesses can think about how to harnesses all the data is that is thought to be not useful, the potential of deriving value from raw unusable data is lost .Technology should support using this data easily and flexibly for decision making and not wasting time on fixing data issues.

First step here, is to change the way we think about using data to support us. Think about the goal whether you want to optimize the work flows and gain insights on improvement.

Do you want to predict some demand about a product? Use Machine Learning Models and train those models with data previous data to predict future.

Then, encourage change and train everyone who is a part of the business process to use the data responsibly. Communicate with everyone so that data is used as a tool to empower themselves and take action.

When people at all levels realize the usefulness of data, the organization can make smarter decisions. This is important not just for mid level management but also for the top level management executives to drive Business Strategy.

Now, let's talk about the most valuable data that is hiding!

This data is not in the spreadsheets or databases, it is often thought as not useful and discarded or lost in the process of using only structured data for analysis purposes.

The data that is hiding to be found is within the unstructured data!

It is in the form of email attachments, meeting notes, or web server logs. It is in the formats like audio, video, social media posts, images or scanned documents. It is very difficult to analyze all this data using traditional analysis tools and to utilize for a competitive advantage.

Researchers estimate about 80% of the data that is generated in the recent times is unstructured. Organizations are now realizing that they do not use this unstructured data for any purposes. They lack the necessary infrastructure and they are not able to get a holistic picture as to how they could take advantage of all this data resources to make intelligent decisions in the organization.

 How can we harness the unstructured data?

Data Science and Artificial Intelligence(AI) play a major role in utilizing those insights by collecting and analyzing the raw data and integrating with structured analytics. Data Cleaning can be done by extracting texts, names, numbers, dates using Machine Learning systems. The (NLP) helps in understanding human language context and nuances. These learning systems can recognize and learn from data patterns and identify relationships within the raw data.

How to use this unstructured data ?

Based on their research, FMCG(Fast-moving consumer goods) companies are using web camera information to promote marketing ideas to their customers.

Ex. Banking and Insurance use the web activity and social media posts to market products to customers.

Ex. Developing actionable insights into supply chain for Shipping and  Logistics.

Ex. Mechanical equipment service and maintenance failures can be predicted by analyzing the daily sensor feeds.

Ex. Financial Service companies can use Natural Language Processing (NLP) techniques to retrieve numbers from their unstructured data for analysis purposes.

Data Security and Data Ethics:

Hidden data in the organization that was never thought useful could be used for making decisions for the future. But as the saying goes: "With great power comes great responsibility". Priority is to ensure sensitive data is protected and used responsibly. Security and Governance of data are both important.

Generative AI models can read from publicly available data and create new data by creating summaries and insights. Ex. Generative AI Large Language Models(LLMs). It is imperative that these models are trained on unbiased data so to avoid unintentional behavior while delivering results.

Ex. If an LLM is trained to hire candidates for a particular job based on biases, hiring can become unethical.

Some risks to consider about data usage are: the licenses, copyrights, subscriptions, poor data quality and unclear data sources.

Some output may mislead users and lose their trust. LLMs generated output can be defined as "hallucinations", that means, the information generated may be probable results or may be wrong facts that need to be reviewed by humans.

Organizations need to adopt security and ethics strategies and manage legal compliance while using Generative AI capabilities.

Contributions from: "Make Data Indispensable" research article and "Tapping-Power-unstructured-data" from experts at MIT Sloan Management.

Book by David Baum: Snowflake Special Edition.