Underlying Services Supporting Data Services Platform¶
Data Services Platform consists primarily of two core services:
- Data Lake Service
- Data Pipeline Service
Together, these services ensure reliable data ingestion, secure storage, transformation, and orchestration, leveraging high‑performance and scalable processing.
Data Lake Service¶
The Data Lake Service is the central storage layer for DSP. It provides secure and tenant‑specific access to Azure Data Lake Storage Gen2 (ADLS Gen2) in IFS.ai Platform, abstracting away the complexities of cloud storage and offering a managed, scalable environment for data.
Key Capabilities¶
Tenant provisioning securely routes operations to the correct tenant storage.
The service supports single-tenant endpoints and multi-tenant endpoints (used by internal services within the IFS.ai Platform). It determines the tenant from the authentication token or based on the subject, maps it to the appropriate storage account, and performs data operations after validating the request.
Upload, download, list & delete files Supports storing and retrieving data files in the Data Lake, typically Parquet files generated from source systems.
Metadata management Allows adding, updating, and retrieving metadata associated with stored files.
Storage hierarchy navigation Enables listing and managing folder structures inside a container and a path.
Abstracted storage layer Users and applications interact with the Data Lake Service instead of directly accessing ADLS Gen2, simplifying security and configuration management.
Role of Data Lake Service in the Data Services Platform¶
The Data Lake Service acts as the primary data store, storing raw ingested data, documents for indexing, or AI-driven features such as Copilot.
Data Lake Services primarily focuses on the ingestion and transfer of data into the Data Lake. It collaborates with Data Pump processes to extract data from Oracle databases, convert it into Parquet files, and securely load it into the Data Lake environment.
Data Pipeline Service¶
The Data Pipeline Service is the orchestration engine that drives data transformation workflows within the Data Services Platform. It automates the triggering of data processing scripts using a scalable infrastructure built on the IFS.ai Platform.
Purpose and Function¶
The service is responsible for starting and managing Data Pipelines, which consist of one or more Python scripts executed on an Apache Spark Cluster. These scripts are typically used to:
- Enrich data
- Cleanse data
- Transform raw data into analytics‑ready outputs
The processing is orchestrated using Argo Workflow, a Kubernetes‑based workflow engine that determines how the Spark applications should run in different conditions.
How It Works¶
- Tenant resolution Similar to the Data Lake Service, the Data Pipeline Service resolves tenant information through access tokens to ensure the pipeline executes in the correct tenant context.
- Pipeline start When triggered, the Data Pipeline Service initiates a pipeline (Argo Workflow) that defines how multiple Spark jobs should be executed sequentially or in parallel.
- Data lake connection handling The service injects Data Lake connection details into the workflow so each script can securely access relevant data stored in the Data Lake.
Spark cluster execution
A Spark cluster is dynamically created to run the pipeline.
Once processing is complete, the cluster is automatically terminated to optimize resource usage.
Status & logging Users can check the status of running pipelines and view logs through IFS Cloud Web.
Integration with Workload Job Definitions¶
Workload Job Definitions allow users to:
- Define data sources to load
- Specify which Argo Workflows(Actions) to run
- Trigger or schedule pipeline execution
When a job is triggered:
- A Workload Run is created.
- Data is loaded into the Data Lake via the Data Lake Service. Once a Data Source is loaded, the relevant Last Refreshed value is updated.
- The Data Pipeline Service initiates the defined Argo Workflow.
- The Workflow executes transformation scripts on Spark.
- Finally, it will get the status of a Pipeline and all related Logs.