Home »
Data Mining
Data Integration in Data Mining
Data Mining | Data Integration: In this tutorial, we will learn about the data integration in data mining, why is data integration important, data integration problems, data integration tools and techniques.
By Palkesh Jain Last updated : April 17, 2023
What is Data Integration?
Data integration is an integral part of the operations on data because data may be obtained from various sources. Data integration is a technique that integrates data from different sources to make them accessible in a single unified view to users, which reports their status. Communication between systems, there are sources that may contain several databases, data cubes, or flat files.
To gain usable data, data fusion merges data from several heterogeneous sources. Several databases, several files, or data cubes are involved in the source. Consistencies, inconsistencies, redundancies, and inequalities must be exempted from the consolidated results.
Systemic View of Data Integration Process
The below-mentioned diagram is a systemic view of the data integration process –
Data Integration Process
Integration of data is important because it not only gives a coherent view of the fragmented data; it also ensures the consistency of the data. This allows the data-mining software to extract valuable information, which in turn encourages managers and executives to take rational measures to develop the business.
- Extract, Convert and Load: Dataset copies are compiled, harmonized and loaded into a data warehouse or archive from various sources.
- Extract, install and transform: data is loaded into a big data system and converted for specific analytics use at a later time.
- Change Data Capture: identifies in real-time information changes in databases and applies them to a data warehouse or other repository
- Data replication: data is replicated to other databases in one database to keep the information synchronized for operational and backup uses.
Why is Data Integration Important?
Big data and all its advantages and problems are welcomed by companies who want to stay competitive and relevant. In these massive datasets, data integration facilitates requests, benefiting from everything from business intelligence and consumer data analytics to data enrichment and information delivery in real-time. Market and consumer data collection is one of the foremost use cases for data integration services and technologies. To enable enterprise reporting, business intelligence (BI data integration), and predictive analytics, enterprise data integration feeds integrated data into data centers or hybrid data integration architectures.
The incorporation of customer data offers a full understanding of main performance metrics (KPIs), financial risks, clients, production and supply chain activities, regulatory enforcement initiatives, and other facets of market processes to business managers and data analysts.
Data Integration Problems
We have to deal with many issues that are discussed below while integrating the results.
1. Detection and Settling Data Dispute
Data dispute means there is no match between the data combined from multiple sources. As with attribute values, various data sets can vary. Perhaps the difference is that they are depicted differently in the various data sets. Assuming that the price of a hotel room in different currencies will be expressed in different cities, during data integration, this sort of problem is observed and resolved.
2. Redundancy and Study of Correlation
During data integration, redundancy is one of the major challenges. Redundant information is irrelevant data or data that is no longer needed. It may also occur because of attributes in the data set that may be extracted using another attribute.
Example
One data set has the age of the client and another data set has the date of birth of the client, so age will be a redundant attribute since it could be calculated using the date of birth.
Data Integration Tools and Techniques
Techniques for data integration are available, from fully integrated to manual approaches, across a wide variety of organizational levels. Typical data integration tools and strategies include:
1. Manual Integration or Common User Interface
No single data view is usable. Users have access to all the originating systems and all the related details.
2. Application Based Integration
It allows all the integration efforts to be applied by each application; manageable with a limited number of applications.
3. Middleware Data Integration
Moves an application's integration logic to a new middleware layer. The middleware program is used to gather data from multiple sources normalize the data and archive it in the resulting data collection. If the company needs to integrate data from existing systems to new systems, this strategy is adopted.
4. Integration of Uniform Access
This approach incorporates data from a source that is more discrepant. But, the data location is not modified here, the data remains in its original location. This approach only provides a single viewpoint that reflects the combined results. To store the integrated data, no separate storage is needed, as only the integrated view is generated for the end-user.