How to prepare your data for AI success

AI promises business transformation, but those efforts are likely to fail without the right data foundations. According to Gartner, 63% of organisations either lack or are unsure if they have the right data management practices needed to support AI.

As a result, Gartner predicts 60% of AI projects without AI-ready data will be abandoned by 2026. This calls for a fundamental shift in how data management practices support and enable AI across the organisation.

Make existing data systems work for AI

Organisations can start by building on their existing data management practices by iteratively adding AI-specific data innovations that help extend and improve data management to support new use cases, such as generative AI. These could include vector data stores, chunking, sampling, embedding and retrieval-augmented generation (RAG) integration, among others.

It’s important to remember that AI-ready data is not ‘one and done’. Think of it as a practice where the data management infrastructure needs constant improvement based on existing and upcoming AI use cases. As AI investments are made, it’s equally important to build an AI-ready data practice by strengthening metadata management, data observability and governance across both data and AI initiatives.

Formal data management practices can no longer be solely relied on if organisations want to successfully integrate AI in their data and analytics (D&A) strategy. Traditional practices are too slow, structured and rigid for AI teams.

In addition, uses of data are not well-documented in traditional data management, and data is often collected in silos across various repositories, systems and platforms. As a result, organisations lack the required practice and metadata to assess the readiness of data for AI.

Steps to building AI-ready data foundations

It’s important to define what constitutes AI-ready data. It must be representative of the use case, every pattern, errors, outliers and unexpected emergence that is needed to train or run an AI model for a specific use.

Proving the AI-readiness of data is a process and practice based on the availability of metadata to align, qualify and govern the data. There are five steps that can be taken to make data AI-ready.

1. Align various data sources to AI use cases

When aligning data sources to AI use cases, make sure to include both internal and external data sources.

2. Identify data governance requirements for AI

It is necessary to identify data governance requirements to prevent or mitigate the risks of violating legal requirements and the unethical use of AI products. This can be done by working closely with legal and business leaders to answer questions such as whether the data will be interoperable across many user communities and applications; how sensitive data can be automatically detected; and how to protect data when being fed into AI models.

3. Evolve metadata from passive to active

Evolving metadata from passive to active builds intelligence and provides continuous iterative improvement and automation. Discover, enrich and analyse metadata and infer a recommendation from the results.

4. Prepare data pipelines

Prepare data pipelines to build an AI model dataset for training purposes, as well as for a live data feed to AI production systems based on the requirements gathered.

5. Assure and enhance data

Assure and enhance data by testing and monitoring it to be optimised for AI model training. Implement DataOps and data observability processes to track patterns and changes, adjusting requirements as needed. If the data has issues, then it’s not ready for AI.

Governing AI-ready data in organisations

As organisations move from pilots to fully operational AI, using collaborative and cross-domain strategies to manage and govern AI across the company becomes crucial for continued success. They should consider enterprise governance of AI, a flexible method that helps combine different governance areas to make responsible AI decisions and achieve business goals.

This journey often starts at the executive level, establishing a virtual layer of decision making practices across existing IT, D&A and risk governance domains. For example, if an organisation wants to use AI to improve customer experience and drive growth, the decision process will start at this level. The CIO or CDAO will then shape the governance response, clarify risk, value and cost, as well as orchestrate the necessary decision making to execute and deliver on executive expectations.

While governance practices are critical to data readiness, organisations with basic or manual metadata management practices will face challenges in making their data AI-ready. Maturing their metadata management practices now is a great first step to AI-readiness.

*Roxane Edjlali is a senior director analyst at Gartner focused on data management solutions. She will be presenting on making data AI-ready at the Gartner Data & Analytics Summit, 17‍-‍18 June in Sydney.

Image credit: iStock.com/UniqueMotionGraphics