Editorial Feature

How Quickly can AI Process Data?

Since the first computers were introduced in the middle of the twentieth century, science and industry alike have learned that processing and analyzing big sets of data can be hugely rewarding. But as datasets grow, the computational challenge of processing them also grows. The key to solving this challenge is the development of artificial intelligence (AI).

How Quickly can AI Process Data?

Image Credit: temp-64GTX/Shutterstock.com

The Enormous Task of Processing Big Data

Big data analytics – the science of transforming large, raw datasets into useful knowledge – has disrupted almost every industry and sector in the global economy today. Data analysis is a fundamental technology underpinning almost every technological development in the last half century, from stock markets to spacecraft.

The value that data owners can glean from these big datasets comes from their size: with a large enough sample, the future behaviors of the sample as a whole become predictable to a surprising degree of accuracy.

But this presents a challenge: the larger the dataset is (and the more valuable it is), the harder it is to process. Data processing includes gathering the data in a usable or computer-readable format, storing it, retrieving it, comparing it with other data points, and performing functions or actions on it or with it.

This is achieved with algorithms: mathematical instructions that tell a computer what task to perform on or with what piece of data and when, and what to do with the resulting information.

A computer's processing power can be thought of as the speed with which it can complete individual tasks: the superfast computers we use today compute almost instantaneously and can perform millions of tasks at once.

Datasets, however, have continued to grow in size and complexity since the first computers were introduced. This is partly due to the realization of how much value they present to their owners; researchers and industry began to actively seek ways to increase their data gathering activities.

In recent years, networks of users have expanded rapidly as companies like Google and Facebook leveraged a useful technological proposition to gain access to data that is valuable to marketers.

Passive sensing, remote facilities, and automated networks have also added more data streams to the processing workload. The Internet of Things (IoT) and Industrial Internet of Things (IIoT) are extracting data from objects and devices that had previously been mute.

At the same time, our technological capacity for transmitting, storing, and processing large amounts of data has exponentially increased. High-bandwidth broadband, mobile data networks, and cloud computing are factors in the growing scale of the data processing need.

As well as increasing the technological capacity for transmitting, storing, and processing data with better hardware and computer architecture, cutting-edge data science employs artificial intelligence (AI) to process data quickly.

AI is the application of algorithms to make computers behave in a way that seems intelligent. Computers that can perform millions of mathematical functions simultaneously can learn from the data they interact with and even change behaviors to respond to what they have learned. This is called machine learning (ML) and is one example of an AI tool used for data processing.

AIDA: The Alan Turing Institute's Artificial Intelligence for Data Analytics Project

The Artificial Intelligence for Data Analytics project was an extended five-year research program from The Alan Turing Institute, a UK research body specializing in AI, that culminated in the summer of 2021.

It filled a research gap for AI solutions for data "wrangling", the laborious tasks of understanding the available data, integrating it from various sources, finding missing, messy, or anomalous data, and extracting metrics for computer modeling.

Researchers said these time-consuming data processing tasks represent as much as 80% of typical data science projects' workloads.

The AIDA project drew on AI and machine learning developments to partially automate data processing tasks known as wrangling. Through over 20 papers, and the code and datasets that accompanied them, the researchers demonstrated their successful achievement of all the main objectives:

  • building AI assistants that can help with tasks,
  • building an open-source platform
  • integrating the assistants with the platform,
  • providing example use cases of cutting-edge real-world data wrangling.

Many of the project outcomes led directly to developments in AI that enabled machines to process data faster.

An early result was the development of a user interface called Data Diff. Data Diff enabled researchers to repeat data analysis tasks on different datasets more easily.

Later, the team developed a family of systems intended to improve AI's so-called "semantic understanding" of data in table format. Semantic AI works by emulating or dealing with language data. The first edition of this family, ColNet was capable of predicting semantic types from a row of data.

In a paper, "Wrangling Messy CSV Files by Detecting Row and Type Patterns" published in the journal Data Mining and Knowledge Discovery, the AIDA team presented a new AI technique to detect formatting parameters automatically in comma separated value (CSV) files. The method automatically standardizes CSV data, rapidly speeding up any kind of data processing.

Continue reading: AI-Enhanced Innovation

References and Further Reading

The Alan Turing Institute. (2022) Artificial intelligence for data analytics (AIDA). [Online] Available at: https:l//www.turing.ac.uk/research/research-projects/artificial-intelligence-data-analytics-aida.

EE Paper (2021). Microsoft will use FPGA to speed up the processing of real-time data. [Online] EE-paper.com. Available at: https://ee-paper.com/microsoft-will-use-fpga-to-speed-up-the-processing-of-real-time-data/.

Space Daily (2022). Lion will roam above the planet - KP Labs to release their "king of orbit". [Online] SpaceDaily.com. Available at: https://www.spacedaily.com/reports/Lion_will_roam_above_the_planet___KP_Labs_to_release_their_king_of_orbit__999.html?utm_source=ground.news&utm_medium=referral 

Disclaimer: The views expressed here are those of the author expressed in their private capacity and do not necessarily represent the views of AZoM.com Limited T/A AZoNetwork the owner and operator of this website. This disclaimer forms part of the Terms and conditions of use of this website.

Ben Pilkington

Written by

Ben Pilkington

Ben Pilkington is a freelance writer who is interested in society and technology. He enjoys learning how the latest scientific developments can affect us and imagining what will be possible in the future. Since completing graduate studies at Oxford University in 2016, Ben has reported on developments in computer software, the UK technology industry, digital rights and privacy, industrial automation, IoT, AI, additive manufacturing, sustainability, and clean technology.


Please use one of the following formats to cite this article in your essay, paper or report:

  • APA

    Pilkington, Ben. (2022, April 20). How Quickly can AI Process Data?. AZoRobotics. Retrieved on May 18, 2022 from https://www.azorobotics.com/Article.aspx?ArticleID=510.

  • MLA

    Pilkington, Ben. "How Quickly can AI Process Data?". AZoRobotics. 18 May 2022. <https://www.azorobotics.com/Article.aspx?ArticleID=510>.

  • Chicago

    Pilkington, Ben. "How Quickly can AI Process Data?". AZoRobotics. https://www.azorobotics.com/Article.aspx?ArticleID=510. (accessed May 18, 2022).

  • Harvard

    Pilkington, Ben. 2022. How Quickly can AI Process Data?. AZoRobotics, viewed 18 May 2022, https://www.azorobotics.com/Article.aspx?ArticleID=510.

Tell Us What You Think

Do you have a review, update or anything you would like to add to this article?

Leave your feedback
Your comment type