By Dr Irina Steenbeek, Managing Director, Data Crossroads

This is the first article of the series: “Choosing Data Management IT Tools.” In this article, I will discuss the following:

• The key market trends in developing data management-related IT tools
• The core challenges companies experience while choosing an IT tool

Key Market Trends

The IT tools and technologies market is one of the most rapidly growing industries. Several trends demonstrate the developments in the industry:

1. Automation and AI
   Automation and AI are becoming increasingly important in developing data management-related solutions. Automation can help reduce manual efforts and increase the speed and accuracy of data processing. In contrast, AI can help to identify patterns and trends in large data sets, providing insights that can be used to make informed business decisions.
2. Cloud-based solutions
   The adoption of cloud-based solutions for data management is on the rise. Cloud-based solutions offer several advantages, including scalability, flexibility, cost-effectiveness, and the ability to access data from anywhere with an internet connection. Cloud-based solutions also provide robust security features, which can help to protect sensitive data.
3. Integration with other systems
   Data management is a multidisciplinary discipline. Data management-related solutions are increasingly being integrated. This integration allows for more efficient data sharing and processing, providing a complete view of business operations and customer interactions.
4. Self-service analytics
   Self-service analytics allows users to access and analyze data independently without needing technical skills or assistance from IT. This trend is driven by the growing demand for data-driven decision-making and the need to democratize access to data across organizations.
5. Real-time data processing
   Real-time data processing is becoming more important in developing data management-related solutions, particularly in finance, healthcare, and manufacturing. Real-time data processing allows organizations to respond quickly to changes in data, enabling faster decision-making and more efficient operations.
6. Collaboration and sharing
   Collaboration and sharing are becoming more important in developing data management-related solutions, mainly as more organizations adopt remote and hybrid work models. Data management solutions that facilitate collaboration and sharing can help teams to work more efficiently and effectively, improving productivity and reducing errors.

Several years ago, these features ensured competitive advantages; nowadays, more and more providers embed these features in complex solutions.

However, these trends and the growing number and complexity of solutions challenge companies in choosing appropriate IT tools. The causes of these challenges can be split into two groups: a company’s approach to implementing and exploring IT tools and information about the IT tools.

Let’s consider these two groups one by one.

Challenges Associated with a Company Practice

Years ago, I was a project manager for enterprise resource planning software. I successfully managed large international projects. At that time, I discovered several reasons companies could fail with their good intentions to implement ERP systems. Unfortunately, I see the same situation with data management-related tools years later. Figure 1 demonstrates the key failure reasons. These failure reasons can turn into key success factors.

Figure 1: Key success factors or failure reasons for IT tools implementation.

I listed here three reasons. Let me explain each of them.

Reason 1: Needs First
Lately, I have had two conversations. These conversations were real. They demonstrate what companies SHOULD NOT DO in any circumstances while choosing an IT tool.
One Chief Data Officer told me: “We bought Collibra, and now we will think about how to implement data governance.” Another specialist asked me: “We decided to implement metadata management. Which tool do we need to buy?” My counter questions were: “What do you mean by data governance?”, “What is the scope of metadata?”
Purchasing software without a clear understanding of the needs and goals of its implementation is the road to hell paved with good intentions. A company must clearly understand its current and future needs and specify detailed business requirements.
Decisions to buy solutions without understanding needs lead to another challenge.

Reason 2: 80/20 Pareto Rule
I’ve often seen that 80 percent of companies use only 20 percent of the software functionality.
First, they want to limit the initial implementation costs when they start the implementation.
They also often make a big mistake: they don’t train internal staff and rely on external consultants.
Then, later, they want to extend the functionality.
Because of the need for more knowledge, they start building some workaround solutions.
The third reason resonates with the first one.

Reason 3: Requirements vs. Feasible Scope
A company must align its needs and requirements with the feasible scope. Any software implementation is a long-term initiative. It must finally turn from a program or project into business-as-usual operations. These implementations can be time- and resource-consuming. Therefore, a company must balance its wishes with the resources available.

Challenges Associated with Available Information about IT Tools

I discovered challenges associated with searching for and choosing IT tools while writing my book: “Data Lineage from a Business Perspective.” This year, I extended this analysis to five types of data management-related tools: data and metadata management, data governance, data lineage, and knowledge graphs. This new analysis has confirmed my previous conclusions. In this article, I will only provide a summary of the challenges. In the consequent articles, I will demonstrate the challenges for the groups mentioned above software.

Challenge 1: The tools’ classification terminology (or labels) does not match the generally accepted terminology.
Several data management guidelines and standards exist. However, they differ in their definition of the ground terms like “data management” and “data governance.” A similar situation takes place in the world of data management IT tools. Various vendors mix up the definitions of data management and governance in describing the functionalities of their software.

Challenge 2: Even if the “labeling” of solutions is similar, the functionalities of products may differ significantly.
It means that different “data governance” or “data management” solutions may have quite different functionalities.

Challenge 3: Various third-party information sources label the same IT tools differently.
One IT tool may get various labels. For example, Talend solution has been labeled by different sources as “data management,” “metadata management,” “data lineage,” “data governance,” and “data fabric” tools. Of course, this product integrates multiple functionalities. However, some can be leading; others can be supporting. If a company searches for only a data lineage solution, the presence of other functionalities can be unnecessary.

Challenge 4: Own vendor classification often does not correspond to the third-party classification.
For example, a company defines its product as a data catalog. While a third-party labels it as a data lineage solution. These functionalities can enrich each other, but they are still different functionalities.

Challenge 5: IT tools have multiple functionalities that do not correspond to their labels.
An IT tool labeled “data lineage” can include functionalities related to managing data lifecycle, including data ingestion, preparation, integration, transformation, analytics, etc.

Challenge 6: The terms and definitions to describe IT tools are not aligned between different vendors.
For example, various providers define their tools as “fabric,” “platform,” “solution,” and “product.” However, often these terms have entirely different meanings.

Challenge 7: The criteria of third parties to define software as “the best” is unclear and unaligned.
I used different sources that promote “the best” tools. The criteria for nominating an IT tool as “the best” are often unspecified. Conversely, Gartner and Solutions Review provides criteria for evaluating different software tools.

Summary

1. The IT tools and technologies market is one of the most rapidly growing industries. Several trends demonstrate the developments in the industry.
2. These trends and the growing number and complexity of solutions challenge companies in choosing appropriate IT tools.
3. The causes of these challenges can be split into two groups: a company’s approach to implementing and exploring IT tools and information about the IT tools.

In the following article, I will discuss the high-level approach to choosing an appropriate IT tool to overcome these challenges.