Skip to content

Azure SQL Data SyncΒΆ

In situations where heavy querying of an Azure archive database is needed it can be beneficial to sync the SQL data to an on-premise database rather than querying the Azure database. To support this the Azure SQL Data Sync functionality can be used. This is available in the Azure portal under the database as the 'Sync to other databases' option.

The sync process is managed by Azure and works by installing an agent on the on-premise machine that connects to the Azure database. A Sync Metadata Database is then used to facilitate the sync process. Once connected a period can be selected for how often the sync process should run or it can be triggered manually. When choosing this sync period consider how often the reporting data is generated, our advice is to not sync faster than this since there will be no data to sync. A list of tables and columns can be selected to be included in the sync. The ARC_Audit and the ARC_Snapshot tables should not be included in the sync process (see the notes below). Consult the Microsoft documentation for information on how to enable Azure SQL Data Sync.

Schema changes are not synced as part of the sync process. When upgrades make changes to the archive database schema ensure that those same changes are applied to the on-premise database to be able to sync them. The database upgrade scripts are included with the installer and can be used to form the basis of the changes needed. If the on-premise schema does not match the sync schema the sync process will start to fail but the scheduling system will continue to work as usual. Consult the Microsoft documentation for more information including recommendations on automatically syncing schema changes.

Warning

The Azure Data Sync process UI has a limit of being only able to load 500 tables. Ensure that the archive database in the Azure installation is separate to the other scheduling databases to allow the schema to be loaded.

Warning

The ARC_Audit and the ARC_Snapshot tables should not be included in the sync process. These tables can include large XML / binary blobs that are not useful for querying. Including these tables in the sync process can cause the performance of the system to deteriorate. If the data in these tables is needed, it can be download from the Archive workspace in the Workbench.

Warning

When creating the Sync Metadata Database do not place it on the same SQL server as the Scheduling / Archive databases. The installer manages this server and if it sees a database that it does not recognize and it is not in use it will plan to remove the database as part of the install process. There can only be one Sync Metadata Database per region and subscription so it is advised to have a dedicated server in a separate resource group.

Warning

Changes must not be synced from the on-premise database to the Azure database.

Note

Enabling Azure SQL Data Sync has been seen to slow the generation of reporting data by 10-20% but has negligible impact on the rest of the system as long as the ARC_Audit and the ARC_Snapshot tables are not included in the sync.