Skip to content

How does the Refresh Schedule work

The Refresh Intervals of a model determine when a model needs to refresh. Or an Explicit Refresh can be done in order to refresh the model without refresh intervals.

When a model needs refreshing, the data sources that are used in the model are determined. After identifying each Parquet Data Source being used by the model,

  1. Determine if a particular Parquet Data Source needs to load - This is done based on the Max-Age and Parquet Data Source Load History. (The effective destination of an Analysis Model is always the Self-Hosted Data Lake.)

i.e. When the End Time of the 'Last Relevant Parquet Data Source Load History' plus the Max Age exceeds the current timestamp, then this means that the Parquet Data Source is considered outdated and needs to load.

The Last Relevant DataSourceLoadHistory is the most recent Load history of a data source considering the end time of the load history.

Note

  1. If a Parquet Data Source has been loaded or is in the process of loading (when the End Time of the Parquet Data Source Load history is empty) by another model, it will not be loaded again during the current instance of Model refreshing.

  2. If a data source has been loaded, after a model has started loading, then it will not be loaded again.

2.When all the outdated data sources in the model are Loaded (data loaded into Data Lake), Model refreshing is triggered.