Without massive amounts of data, digital personal assistants such as Alexa and Siri wouldn’t be able to understand what people are saying or develop reasonable responses. The more information you feed an artificial neural network, the faster it learns and the more capable it becomes.
Prior to starting an AI program, you need to consider the role that digital media strategy data will play in that program and how you’ll be using that data; for example, you must decide whether you simply want to analyze data to gain insight or you need a machine to analyze the data for you and make predictions. In order to make these decisions, you need to understand a few key concepts, including big data, data science, and data mining.
This post sheds some light on the role that data plays in AI programs and helps you make informed decisions in how to apply AI to make the most of the massive volumes of data you currently have access to and the growing volumes of data you’ll have access to in the future.
Understanding the Concept of Big Data
Big data has come to be used to describe huge data sets that can be analyzed by computers to reveal patterns, trends and associations. But if you go back to the original report in which that term was coined, you'll see that the authors weren't thinking of “big data” as a term. They used it to describe a problem, as in “We have a BIG data problem” not as in “We have a BIG-DATA problem.”
The problem is that we are having trouble storing and processing the massive amount of data being generated. Soon after a company upgrades its on-premises data warehouse, it’s likely to outgrow that warehouse. The warehouse can’t keep up with the volume and variety of data flowing into it or it doesn’t have sufficient processing power to generate reports from that data. Many companies have to start running their reports at the end of the day, so the report will be done the next morning or afternoon. At other companies, where numerous employees are querying the data at the same time, they have to wait hours for results, and if the system crashes or freezes due to its lack of processing capacity, they have to start over. Many of these businesses (such as a stock exchange) rely on real-time reporting to remain competitive.
The problem is growing. According to one estimate, within the next decade there will be more than 150 billion networked sensors in the world, each of which will be generating data 24/7 365 days a year. And just imagine all the data that humans generate in a single day on Facepost, Twitter, Google, online shopping sites, online gaming sites and more.
The takeaway here is that big data is both a problem and an opportunity. It’s a problem in that you need to determine whether you need to work with huge data sets or have more modest needs. Perhaps you merely need to use smaller data sets to monitor and analyze website usage or gauge the effectiveness of your marketing campaigns. However, if you need to analyze huge data sets (for example, to find a cure for the common cold), then you need to plan for storage and processing. But big data is also an opportunity. Without it, AI wouldn’t have the data it needs to identify patterns from and make predictions on that data.
Teaming Up with a Data Scientist
If you’re developing an AI program that requires big data, you would be wise to team up with or at least consult with a data scientist. A data scientist is trained in various disciplines, including programming, data management and statistics, for the purpose of knowing how to collect, analyze and interpret data, typically to assist a business in its decision-making.
Data scientists may work with or without the assistance of machine learning. For example, a data scientist may ask questions you never thought of asking to help you develop a clearer picture of what you’re trying to extract from the data — perhaps an answer to a question, a solution to a problem or insight into the possible contributing factors of a system failure. The data scientist could then use or recommend tools to gather, analyze and interpret the data to achieve that goal. Or, the data scientist may discover that you really don’t know what you’re looking for in that data and help you develop a machine learning system to identify patterns that may provide insight you never would have thought to seek.
In short, a data scientist can help you size up your data and analytical needs and provide solutions to get the most out of your data.
Prior to starting an AI program, you need to consider the role that digital media strategy data will play in that program and how you’ll be using that data; for example, you must decide whether you simply want to analyze data to gain insight or you need a machine to analyze the data for you and make predictions. In order to make these decisions, you need to understand a few key concepts, including big data, data science, and data mining.
This post sheds some light on the role that data plays in AI programs and helps you make informed decisions in how to apply AI to make the most of the massive volumes of data you currently have access to and the growing volumes of data you’ll have access to in the future.
Understanding the Concept of Big Data
Big data has come to be used to describe huge data sets that can be analyzed by computers to reveal patterns, trends and associations. But if you go back to the original report in which that term was coined, you'll see that the authors weren't thinking of “big data” as a term. They used it to describe a problem, as in “We have a BIG data problem” not as in “We have a BIG-DATA problem.”
The problem is that we are having trouble storing and processing the massive amount of data being generated. Soon after a company upgrades its on-premises data warehouse, it’s likely to outgrow that warehouse. The warehouse can’t keep up with the volume and variety of data flowing into it or it doesn’t have sufficient processing power to generate reports from that data. Many companies have to start running their reports at the end of the day, so the report will be done the next morning or afternoon. At other companies, where numerous employees are querying the data at the same time, they have to wait hours for results, and if the system crashes or freezes due to its lack of processing capacity, they have to start over. Many of these businesses (such as a stock exchange) rely on real-time reporting to remain competitive.
The problem is growing. According to one estimate, within the next decade there will be more than 150 billion networked sensors in the world, each of which will be generating data 24/7 365 days a year. And just imagine all the data that humans generate in a single day on Facepost, Twitter, Google, online shopping sites, online gaming sites and more.
The takeaway here is that big data is both a problem and an opportunity. It’s a problem in that you need to determine whether you need to work with huge data sets or have more modest needs. Perhaps you merely need to use smaller data sets to monitor and analyze website usage or gauge the effectiveness of your marketing campaigns. However, if you need to analyze huge data sets (for example, to find a cure for the common cold), then you need to plan for storage and processing. But big data is also an opportunity. Without it, AI wouldn’t have the data it needs to identify patterns from and make predictions on that data.
Teaming Up with a Data Scientist
If you’re developing an AI program that requires big data, you would be wise to team up with or at least consult with a data scientist. A data scientist is trained in various disciplines, including programming, data management and statistics, for the purpose of knowing how to collect, analyze and interpret data, typically to assist a business in its decision-making.
Data scientists may work with or without the assistance of machine learning. For example, a data scientist may ask questions you never thought of asking to help you develop a clearer picture of what you’re trying to extract from the data — perhaps an answer to a question, a solution to a problem or insight into the possible contributing factors of a system failure. The data scientist could then use or recommend tools to gather, analyze and interpret the data to achieve that goal. Or, the data scientist may discover that you really don’t know what you’re looking for in that data and help you develop a machine learning system to identify patterns that may provide insight you never would have thought to seek.
In short, a data scientist can help you size up your data and analytical needs and provide solutions to get the most out of your data.