Unstructured data is a rich source of information that can be used for various purposes and benefits. However, it is also opaque and requires painstaking labour to transform into useful insights. Or it has until now: today, AI techniques and technologies can turn a morass of unstructured data into a key tool for making business decisions.
It’s commonplace today to say that we live in a data-driven economy. “Data is the new oil”, or so the saying goes. It’s a nice turn of phrase as it indicates that in today’s economy information is a crucial resource. Strictly speaking, however, data is not really what drives business: information is, and at the risk of delving into philosophy, specifically what we need is information that we can transform into knowledge.
To give just two simple examples, asking how many did we sell might lead to deciding how many to make, and asking what percentage of people responded might lead to deciding who else to contact.
In other words, what we really want is to use data in order to know things. Of course, we have been doing this for decades. From cash ledgers to sales reports and actuarial tables, businesses have long run on numbers. But just as there is more to life than numbers, there is more to business, too.
It is in exaggeration to say that most of the world’s information does not fit into neat, mathematically-ordered rows and columns. And a lot of it is vital. From social media posts (useful for market research and sentiment analysis) to the knowledge contained in a library or, crucially, locked up in e-mails and sundry business documents, there is more information out there that cannot be easily tabulated than can. In other words, while we have worked on the basis of structured data, there is a vast untapped potential in unstructured data.
The amount of unstructured data is growing, too. Indeed, according to analysts Gartner, unstructured data represents an estimated 80 to 90 per cent of all newly created enterprise data.
What is Structured and Unstructured Data?
At this point some definitions are in order, so just what are structured and unstructured data?
Structured data, which is typically quantitative in nature is data that is stored in fields and is easily recognisable and searchable by traditional computing applications. Examples include names, addresses, dates and figures of all kinds. This kind of data has been mined, using databases and spreadsheets, for decades.
Indeed, SQL, or Structured Query Language, is a core component of popular databases such as Microsoft SQL Server (itself a key part of Microsoft Data Platform), PostgresSQL, Oracle SQL and MariaDB. Created by IBM engineers in 1974, SQL is a programming language specifically designed to manage structured data held in relational databases that organise data into rows and columns to form tables.
Similarly, though much more simply, spreadsheets allow quantitative data to be arranged in the rows and columns to be manipulated and used in calculations.
On the other hand, unstructured data is data that is not organised in a pre-defined manner or using a set data model. Until now, unstructured data has posed serious challenges for data analysis and management: it is difficult to store, process, and, especially, extract meaningful insights from it using traditional methods and tools.
Unlocking knowledge from unstructured data means finding ways to access, organise and analyse it in order to derive value from it. But the potential value of doing so is enormous because unstructured data can help organisations and individuals gain a deeper understanding of their customers, markets, competitors, products, services, processes and performance.
It can also enable businesses to discover new opportunities, trends, patterns, and, most of all, develop insights that can improve decision-making, innovation, efficiency and customer satisfaction.
And it is precisely here that AI can help.
AI Training Needs
When reading about AI today, excitement at its uses can be tempered by concerns about how much work will need to be done to integrate it into business processes and workflows.
Typical questions that arise include ‘Do I need to hire a data scientist?’ and ‘Will the AI need to be trained?’
The answer to both, happily, is no.
In fact, one of the key advantages of generative AIs is that they do not need to be trained for business use. This means that businesses can start using them immediately to improve their operations and efficiency.
Today’s crop of generative AIs comes pre-trained and, therefore, need only be pointed at the right data and set to work.
Generative AIs are based on large language models (LLMs), which are neural networks that can generate natural language texts based on a given input or prompt. LLMs are already trained on massive amounts of text data from various sources, such as books, web sites, news articles, social media posts and more. This enables them to learn the patterns and probabilities of natural language, as well as the general knowledge and information contained in the text data.
This skill set is then used to set the AI to work performing various tasks, such as content creation, summarisation, translation, paraphrasing and question answering. These tasks, in combination with the ability to process and analyse unstructured data, such as text, images, audio and video, open the door to untold knowledge creation because using natural language as an interface, generative AIs can make unstructured data more accessible and understandable for humans.
Information and insights can be extracted from unstructured text data, such as reviews, comments, or e-mails, which can parse and explain the meaning and sentiment of data providing responses also in the form of natural language. Let’s take a look at an example of just that.
Chat with your Documents
At TEKenable, we believe we have already demonstrated how generative AI can have a transformative impact on business.
Millions of words have already been written on how generative AI can be used for content creation, but what about using it for content comprehension? In other words, sourcing and collating disparate information and synthesising it into useful knowledge.
This is precisely what we set our minds to when we created our application Chat With Your Documents.
Initially created as a tool for internal use, Chat With Your Documents does exactly what it sounds like it does. An Azure-based AI application designed to use AI to unlock business value by allowing businesses to capture and explain information locked up in a variety of sources, from e-mails, PowerPoint slides, PDFs and any other kind of document people create or share, Chat With Your Documents does more than merely search.
After all, the ability to search for, or inside, documents is not itself sufficient to unlock that knowledge. The full potential can only be unlocked by using AI to understand your documents and combine content from them intelligently in response to questions.
Imagine being able to ask questions such as “Please summarise the key benefits of product X versus product Y for a potential IT buyer?”, or “What is our stance on ESG in developing markets?”, “What projects have we delivered for Acme Inc and what were the key selling points for them?”.
Clearly applications like Chat With Your Documents – and it is really only the beginning for us at TEKenable – can have an enormous impact on the ability of a business to use the information it already has in order to transform operations. Next week, in the final article in this series, I will look at the potential impact of AI on the workplace and the job market.
