Bonree Data’s “Bonree Academy” webinar series focuses on practical observability scenarios, exploring four key areas around the Bonree ONE Intelligent Observability Platform: data integration and governance, AI Ops agent development, AI computing cost management, and database observability.
This article is based on the technical session “Data Integration Deep Dive: Data Governance and Fusion.”
Custom integration is far more than a simple data forwarding pipeline. It is a comprehensive data processing mechanism centered on entity identification and alignment — first establishing trusted and governed data, then enabling intelligent analysis through a unified entity model. By eliminating data silos, it helps ensure stable, efficient, and continuous operation of business systems.
The Core Challenge of Enterprise Operations Data:
It’s Not About “How Much Data You Have” — It’s About “Whether Data Can Be Aligned”
Enterprise operations data is inherently heterogeneous. It comes from diverse sources, including monitoring APIs across public cloud, private cloud, and hybrid cloud environments; self-developed monitoring platforms or customized open-source solutions; as well as logs, service chains, and CMDB asset repositories.
Each system evolves independently, resulting in inconsistent entity identifiers, field definitions, and data formats.
In real-world engineering scenarios, the primary challenge is not insufficient data coverage, but three fundamental structural conflicts:
Inconsistent protocols
Different data sources use different packaging formats, requiring batch data to be parsed and transformed into independent records.Non-unique entity identifiers
The same host may be monitored by multiple platforms, but each system assigns different instance IDs, making direct matching impossible.Inconsistent semantic definitions
Even after entity matching, differences may remain in metric units, aggregation methods, and measurement dimensions.
Without dedicated data governance mechanisms, the more data sources an organization integrates, the more issues such as entity duplication, misalignment, and semantic ambiguity accumulate within the observability platform.
Dual-Path Integration Architecture: Standard Integration vs. Custom Integration
Bonree ONE Data Integration provides two approaches: Standard Integration and Custom Integration, designed to address different types of data source characteristics.
1. Standard Integration: For Structured and Predefined Data Sources
Standard integration is designed for structured components such as databases and middleware.
For these resources, field structures and entity relationships can be modeled in advance. With SmartAgent, data collection is automated through predefined capabilities, enabling a standardized out-of-the-box experience.
2. Custom Integration: For Dynamic and Unstructured Data Sources
Custom integration targets scenarios such as self-developed monitoring systems, generic protocol-based data reporting, and data sources with undefined requirements.
Since these sources cannot rely on fixed templates, users can configure entity mappings and relationship rules based on original data structures.
From a technical perspective, custom integration leverages SmartGate for data collection, uses collection tasks to manage scheduling, and relies on data flows to orchestrate processing pipelines.
The design principle is clear:
For scenarios with well-defined requirements, predefined capabilities maximize efficiency.
For scenarios with evolving or uncertain requirements, configurable data flows enable flexible transformation.
Together, these two approaches cover the two major categories of enterprise data sources and ultimately unify them into the same metric model and entity relationship graph.
Core Mechanism of Custom Integration:
Moving Beyond Data Transfer Toward End-to-End Entity Governance
Unlike simple data forwarding, custom integration performs entity normalization throughout the data processing lifecycle.
The complete workflow consists of three layers:
Layer 1: Protocol Parsing and Batch Data Expansion
Cloud monitoring APIs typically return data in batch format.
In a typical scenario, two raw batch records can be parsed and expanded into 232 independent data records after processing.
Batch parsing and transformation provide the foundation for downstream analysis. Any failure at this stage prevents accurate observability insights from being generated.
Layer 2: Entity Uniqueness Identification
Identifying the same operational entity requires reliable feature-based key fields.
In the AWS EC2 demonstration scenario, the system uses a combination of host identifiers and data center information as a composite identity key.
This approach aligns with dynamic CMDB registration mechanisms: entities are dynamically created and registered based on key attributes, avoiding risks caused by duplicate or unstable single identifiers.
Layer 3: Cross-Source Entity Data Merging
When historical monitoring data for a host already exists in the platform, newly collected cloud-side metrics can be matched through entity rules.
The system identifies that both datasets belong to the same entity, preventing duplicate monitoring objects and fragmented operational views.
These three layers work together to establish a complete governance pipeline — transforming raw heterogeneous data into unified, analyzable operational entities.
End-to-End Operations Assurance:
Validation, Capability Reuse, and Layered Troubleshooting
Entity recognition represents only half of the integration journey.
After configuration takes effect, the platform provides operational metrics such as collection traffic, parsed data volume, and task execution status. These indicators help teams quickly identify configuration issues before they impact troubleshooting processes.
Meanwhile, Bonree ONE provides built-in metric models, dashboards, and alert rules. Newly governed data sources can directly reuse these observability resources, reducing the cost of building monitoring systems from scratch and accelerating metric visualization and alerting.
(Preconfigured resources require user selection and do not take effect automatically.)
Because custom integrations for non-standard data sources involve a high degree of customization, achieving a perfect configuration in the first attempt is challenging.
To address this, the platform provides a layered troubleshooting framework aligned with the data processing pipeline, enabling precise issue localization.
1. Connectivity Layer: Verify Data Transmission Paths
Focus on whether raw data can successfully enter the processing pipeline.
For scenarios involving interrupted data collection, teams can verify synchronization status between collectors and probes, as well as network connectivity.
2. Parsing Layer: Validate Data Processing Accuracy
Focus on whether incoming data is correctly processed.
For issues such as malformed data or unexpected record counts, teams can verify raw field structures and validate custom mapping configurations.
3. Semantic Layer: Ensure Data Accuracy and Consistency
Focus on measurement consistency after data ingestion.
For hidden issues where data is successfully stored but metric calculations are inaccurate, teams can review metric units, aggregation dimensions, and entity mapping logic.
This layered troubleshooting model enables end-to-end observability across data ingestion, parsing, and normalization — making the entire integration process manageable and traceable.
From Trusted Data Foundation to Intelligent Operations
Data integration is not merely a preparation step for observability — it is the foundation of the entire observability architecture.
Custom integration is not simply a data pipeline. It is an entity-centric governance mechanism that first establishes trusted data and then enables intelligent analysis at higher layers.
Ultimately, it supports intelligent operations and business continuity through a three-layer value progression:
1. Data Layer · Trusted Foundation
Through entity uniqueness identification and cross-source data consolidation, fragmented and heterogeneous operational data is transformed into trusted data with unified entities and consistent semantics.
This provides a reliable foundation for AI-driven anomaly detection and root cause analysis.
2. Platform Layer · Reusable Capabilities
Governed data sources can directly leverage built-in metric models, dashboards, and alerting capabilities without rebuilding observability systems from scratch.
Standard integration provides out-of-the-box efficiency, while custom integration delivers flexible expansion — balancing deployment speed with broad data coverage and significantly reducing observability implementation costs.
3. Business Layer · Intelligent Operations
By establishing a unified entity perspective across all observability data, Bonree ONE breaks down data silos, enables faster anomaly detection, improves root cause analysis accuracy, and ultimately helps organizations maintain stable, efficient, and continuous business operations.