Tuning, Monitoring And Troubleshooting

This page describes how to tune  data synchronization process to gain better performance, how to monitor data synchronization status and trouble shoot it when errors and problems are encountered. 

Contents

• Tuning
• Monitoring
• Troubleshooting
• Installation

Tuning

• HTTP_SENDER - When sending synchronization data from Hub to Satellite or vice versa HTTP_SENDER is used as the transport connector. In a situation where a large backlog of data has to be cleared it is common for the HTTP_SENDER to fail temporarily with transient errors like "http 400-bad request". In such a situation the address line is retried again and the message goes in to "Waiting" state. After a number of retry attempts the error will go away and the message can be delivered successfully. In a situation like this it is important that RETRY_INTERVAL parameter is set to a lower value than the default 60 seconds so that retries occur more frequently. You can set the MAX_RETRIES into a value acceptable for your environment because after the message sending is tried for this amount of times the message delivery will permanently fail and the message will be moved to state "Failed".
• Work manager - Max Thread Constraint -  Batch Processor Message Driven Bean instantiate a maximum of 16 threads by default for Message queue processing. In an installation where there are many Satellite installations this default value will not be enough for efficient queue processing. In order to increase the threads assigned for the batch processor MDB modify the Max_thread_constraint parameter of the Worker manager.
• Caching LOBs - Fndcn_message_body_tab has two columns which are defined as Large Objects (LOBs). When synchronizing large amounts of data the performance can  be improved considerably by opting to cache the LOB columns in the table fndcn_message_body_tab. You can perform this using an alter table command:

   

  alter table FNDCN_MESSAGE_BODY_TAB modify LOB (MESSAGE_VALUE) (cache);
  alter table FNDCN_MESSAGE_BODY_TAB modify LOB (MESSAGE_TEXT) (cache);
  

  To reverse use the keyword nocache.

• Scheduled task for failed message restart: When a high volume of data is received some transactions will fail with transient errors. These errors will be resolved if the message is restarted. There is a  database tasks that can be scheduled to restart failed messages. The task is  REPL_BATCH_UTIL_API.RESTART_REPLICATION_MESSAGES. Schedule this task with a higher frequency of occurrence to speed up data synchronization message receipt.

Monitoring

Data Synchronization Status

Data synchronization monitoring is supported by the framework by collecting and presenting synchronization message queue status of each environment. Message status of each individual satellite installation is collected and sent to Hub installation periodically by using a specific database task called Forward Data Synchronization Statistics.
Collected message queue status of SATELLITE installation are then sent to HUB.
Hub installation, which is the central installation, contain the message queue statuses for the whole network of satellite installations.

You can use below form window to view message queue status.

Lobbies

There are two separate lobby pages for administration  and monitoring.

• Data Synchronization Administrator - HUB : used to monitor message processing and queue status. Include information from satellite installations connected to the hub installation and information from HUB itself.
• Data Synchronization Administrator - SATELLITE : used to monitor message processing and queue status of individual satellite installations.

Troubleshooting

Bundled Message viewer

Data synchronization messages are bundled in to message bundles of BUNDLESIZE parameter. When the message bundle is received at a destination it is not possible to manipulate or examine each individual message in the bundle. Bundled Message Viewer is designed for this purpose. Using the Bundled Message Viewer you can examine and manipulate each individual message in the message bundle.

To view a particular application message provide the application message id in the search dialogue and press enter. The contents of that application message will be shown as in the above diagram.

The following actions are available for each message in the bundle. Select the message from Message Rows and RMB:

1. Exclude : if the message is marked as Excluded it will be excluded from the message bundle when the bundle is processed. Then at a later time this message can be examined and handled as required. This is useful when trying to figure out the cause of a message failure. You can examine the individual messages of the failed application message and exclude the one that causes the error. Then you can process the rest of the message bundle and attend to the failed message at a later time.
2. Execute and exclude : you can mark a message as Execute and exclude to execute that message only from the message bundle. When the rest of the messages are executed this message will not be executed.
3. Include : you can include a message that was marked as Exclude again into the message bundle. This can be done when what ever the reason for excluding the message from the message bundle is successfully resolved.

The new attribute string and the old attribute string of the synchronization message is also visible here. They can be used to identify the reason for a message failure.

When a message is excluded from the message bundle that message bundle will be kept for a time period as defined by  SKIPPEDDAYSOLD parameter in Data Sync Process Settings before being cleaned up. For example if the SKIPPEDDAYSOLD parameter is 30 days then a finished application message that has an Excluded message will be kept for 30 days before that application message is marked for clean up.

Batch processor trouble shooting

Once an application message is created using the message bundle the Batch processor is responsible for processing the application message. Refer to Batch processor troubleshooting guide given here  to debug and manipulate the application message.

Installation