Legacy Source Data Import¶
Data from csv files generated external sources are loaded to the legacy data from this and user has the freedom to select the mode (load from server or load from client) as he wish and it is highly recommended to use server loading, to a large data bulk. For each data load a load sequence will be generated and it must be locked in order to use for mapping. User will be able to define date format, thousand separator format and decimal format and these will override the formats define in the source.
Legacy Load Locking¶
The first step is to Load the Legacy Data from outside Smart Data Manager. This is done by first define the Legacy Data Header, which is based on the Migration Project ID, the Source from where the data originates from and finally the Legacy Data Table Name.
When the Header is defined, the next step is to define how the Data will be imported. Where the data file is located and how the data is structured in the File, like field delimiters, Date and number formats etc.
When the file is imported, it creates a Load Sequence number value so it is possible to keep track on different imports of the same Table over time. When the File is imported the data Load needs to be Locked before it can be used. Only one Data Load can be locked at the same time for a Legacy Table. This means that, if the Legacy Table has the same structure for each Load, Smart Data Manager can use the same Mapping to handle different loads from the Legacy Table. To make sure that the Data is structured the same way a Legacy Table Definition is created when the Load is locked. Then Smart Data Manager, can compare the different Loads Table Structure and create a warning if they are not the same.
Read more on Legacy Table Definition
Loading from Client¶
- Loading from client will read the whole file to the buffer using the default encoding in the user’s machine and based on the delimiter provided it will break it into lines.
- Later it will read line by line and break the line into columns using the string delimiter provided.
Loading from Server¶
- For better performance data is loaded using the Oracle’s external tables concept
- Database directories must be set at the time of SDM initial installation
- In case of loading failure, error message logged in bad file created is read and added to the legacy load message column.
- When the same file is loaded next time all the temporary files will be deleted and regenerated.