Microsoft has added a new feature in Microsoft Fabric designed to streamline the integration of external applications with OneLake using a concept called ‘Mirroring.’ This functionality offers a simplified approach to data integration by eliminating the need for complex ETL (Extract, Transform, Load) workflows. With Mirroring, existing data systems can connect directly to OneLake, enabling seamless synchronization across Microsoft Fabric.
The real power of Mirroring in Microsoft Fabric lies in its ability to provide real-time data integration. It enables continuous synchronization across hybrid and multi-cloud environments, effectively reflecting the data from various sources into mirrored databases within Microsoft Fabric. This ensures that data is always up-to-date and readily available for AI and analytics tasks, eliminating the usual delays or complexities associated with traditional data integration processes.
Think of Mirroring as a two-part operation. The first part involves connecting source systems to Microsoft Fabric. The second part is where Fabric takes over by providing the landing zone, which handles all the storage and converts the incoming data into delta tables for efficient processing. These delta tables represent the incremental changes in the data over time, making it easier to track and analyze real-time updates.
Once the data is mirrored and processed into delta tables, it can be consumed in various ways within Fabric. For example, the data can be accessed through SQL Endpoints for querying, explored using Notebooks, or integrated into analytical and AI workflows, enabling valuable insights without the complexities of data integration. This feature opens the door to faster, more efficient data operations, ultimately providing a more seamless experience in managing and utilizing data in Microsoft Fabric.
Key Components of Mirroring in Microsoft Fabric
Fabric provides the following core components:
- Landing Zone – The landing zone provides a dedicated storage area to drop raw files before they are processed. It ensures that incoming data is properly organized and ready for further integration into Microsoft Fabric.
- Storage for Delta Tables – This storage format helps manage large datasets by capturing incremental changes over time. Delta tables are essential for real-time data synchronization and facilitate easy integration with other Lakehouses for scalable analytics.
- SQL Endpoint – The SQL Endpoint allows users to run SQL-based queries on the mirrored database, making it easier to interact with the data. It simplifies the process of retrieving and analyzing data within Microsoft Fabric.
- Semantic Model – The Semantic Model organizes and defines data relationships in business-friendly terms. It enables users to build consistent reports and dashboards, translating complex data into easily comprehensible insights for decision-making.
Creating a Microsoft Fabric Mirroring Database
To set up a Mirroring Database in Fabric:
- Log in to Fabric and browse to a workspace that is allocated to Fabric Capacity.
- Click + New Item, search for Mirrored Database, and select it.
- Enter a name (e.g., SQL_Mirror_Source), and click Create.
Once created, Fabric provisions the Landing Zone, SQL Endpoint, and Semantic Model. The Landing Zone URL is where folders must be created and data must be loaded.
Accessing the Landing Zone
To create folders and upload files, the Azure Storage Explorer can be used.
- Copy the Landing Zone URL from your Open Mirror setup.
- Open Azure Storage Explorer (or any compatible tool) and select Connect to Azure Data Lake Gen2.
- Paste the URL and name the connection.
- Once connected, the schema, tables, and uploaded change files will be visible in real time.
Key Considerations for Mirroring in Microsoft Fabric
- Data is physically stored in the Fabric workspace (unlike shortcuts, which do not involve physical data movement)
- The data stored physically incurs costs as per the SKUs purchased.
- On pausing the Fabric capacity, storage charges will apply based on the regular OneLake pricing model.
- Direct Lake mode for Power BI semantic models is supported for mirrored tables.
- Deleting the mirrored table does not affect the original table in the source database.
- Individual tables can be selected for Mirroring in Microsoft Fabric rather than the entire database.
Conclusion
With OneLake at the core of Microsoft Fabric and the new Open Mirroring feature, organizations can now achieve efficient, near real-time data integration across their entire data estate. This enables businesses to synchronize data from various sources directly into Microsoft Fabric, providing comprehensive analytics and actionable insights.
By continuously Mirroring data from hybrid and multi-cloud environments, organizations ensure their data is always up-to-date and ready for analysis. This simplifies data management and eliminates the complexities of traditional data pipelines, allowing teams to focus more on extracting insights rather than dealing with data movement.
With this additional Mirroring capability, Microsoft Fabric becomes an even more compelling choice for organizations looking to modernize their data estate. It removes barriers to real-time integration and streamlines workflows, helping businesses unlock the full potential of their data for faster, more informed decision-making.
