Kubernetes Installation¶

Prerequisite Steps¶

Kubernetes installations can only be done as a command line install into an existing prepared cluster. The cluster must be running on either Kubernetes version 1.33 or 1.34.

Kubectl¶

Kubectl must be installed and available in the PATH. It must be setup with valid credentials, the context and the namespace you wish to deploy to. See the official Kubectl install guide for more information. Your cluster administrator will be able to provide advice on how to setup your kubeconfig file with valid credentials.

Helm¶

Helm must be installed and available in the PATH. See the official Helm install guide for more information. The installed Helm version must be compatible with the Kubernetes version. See the Helm version skew policy for version compatibility.

Ingress Controller¶

An ingress controller must be deployed to the cluster. The ingress controller is responsible for routing traffic to the PSO services. PSO supports both Ingress NGINX and Traefik.

Ingress NGINX (Deprecated)
Support for Ingress NGINX is deprecated and will be removed in a future release. Existing installations should migrate to Traefik. During migration, both controllers can be enabled simultaneously to allow for a smooth transition. This is currently the default ingress controller for backwards compatibility reasons, but Traefik is recommended for new installations. See the official Ingress NGINX install guide for more information.

Traefik
Traefik is the recommended ingress controller for new installations. Traefik uses Custom Resource Definitions (CRDs) for ingress routing. The Traefik CRDs must be installed before deploying PSO. See the official Traefik install guide for more information.

KEDA¶

If support for auto-scaling the deployments is required KEDA must be deployed into the cluster. See the official KEDA install guide for more information. The installed KEDA version must be compatible with the Kubernetes version. See the KEDA Kubernetes Compatibility policy for version compatibility.

The Kubernetes Metrics Server must also be available within the cluster to provide support for the KEDA CPU and Memory scalers. See the scaler prerequisites for more information.

config.linkerd.io/proxy-cpu-request: "0.1"
config.linkerd.io/proxy-memory-request: 128Mi

Databases¶

Databases must be created so that the installer can run the relevant database create scripts against them. See the Database Setup section for more information. Uninstalling a Kubernetes installation using the command line installer does not drop or truncate databases.

Installation¶

Installation requires passing an input values.yaml file to the installer containing all the relevant details needed for the install. These values are used by the PSO installer to prepare the database. The values are then passed on to the Helm installer to do the deployment in the cluster. See the Input Values section for the format of this values.yaml file.

From Windows¶

To run the Installer you will need to locate the Setup folder within the release distribution, within which is the executable IFS PSO Installer.exe. You will then need to open a PowerShell command prompt with Administrator privileges and type the following command:

PS> & "IFS PSO Installer.exe" -cli -yaml -s "path_to_values.yaml"

The installer settings can also be written to the standard input stream of the command line installer using the following command:

PS> Write-Output $InstallerValues | & "IFS PSO Installer.exe" -cli -yaml -stdin

From Linux¶

To run the Installer you will need to locate the Linux Setup folder within the release distribution, within which is the executable installer. You will then need to open a command prompt and type the following command:

./installer -cli -yaml -s "path_to_values.yaml"

Or from the standard input stream:

printf '%s\n' "$InstallerSettings" | ./installer -cli -yaml -stdin

Kubernetes Command Line Flags¶

The following table shows the possible flags to include when running the installer in command line.

Input Values¶

This section documents the possible input values settable in the values.yaml. The values.yaml file found with the installer at 'Helm/ifs-pso' includes all the possible values and their defaults.

Database Connections (Mandatory)¶

A list of database connections for the applications. Used by the installer to prepare the databases and then passed to the deployments as a secret. Example:

databaseConnections:
- database: System
  provider: SQLAzure
  connectionString: "system connection string"
- database: Scheduling
  provider: SQLAzure
  connectionString: "scheduling connection string"

• Possible Databases: System, Archive, Scheduling, Modelling, Simulation, HTM
• Possible Providers: SQL, SQLAzure, Oracle

Licence File (Mandatory)¶

A path to the licence file for the software to be installed into the database. Relative paths can be used.

licenceFile: '.\PSO_Licence.xml'

Admin User (Mandatory)¶

Credentials for the PSO admin user used on first installation. These values are not relevant on subsequent installs.

adminUser:
  username: Administrator
  password: ifspassword

Helm Install Name (Mandatory)¶

The name of the Helm install, used by the PSO installer when deploying.

helmInstallName: pso

Kubernetes Context¶

The Kubernetes context to use. Defaults to the context selected in kubectl

kubeContext: default

Kubernetes Namespace¶

The Kubernetes namespace to install into. Defaults to the namespace selected in kubectl

namespace: pso

Installation¶

Used to specify the installation name and role for multi-install deployments. Safe to leave unspecified for single-install deployments. When role is not specified the role will be set to ACTIVE for new installations and left unchanged for upgrades.

installation:
  name: ""
  role: ""

• Possible Roles: ACTIVE, PASSIVE, STANDBY

Sizing Name¶

Used to specify the deployment sizing name for the installation. Sizing files are stored in the 'Helm/ifs-pso-sizing' directory of the installer files. All sizing files have auto-scaling enabled. For advice on the correct sizing please speak to an IFS consultant.

sizing: PSO-50-M-PROD

Sizing name breakdown:

• PSO - Product name
• 50 - Affects the number of replicas deployed. Corresponds to the expected maximum number of total activities the system can handle, calculated by multiplying this number by 1,000 (e.g., 50 = 50,000 activities). Note: This is across the whole system not per dataset.
• M - Affects the amount of memory and CPU applications have. Corresponds to the maximum activities within a single dataset. When set to L the distribution service (DST) is enabled allowing segmentation of datasets. XS = up to 5,000; S = up to 15,000; M = up to 35,000; L = up to 100,000
• PROD - Environment type: PROD (Production) or NPROD (Non-production). Non-production environments are generally much smaller and cheaper to run for development / testing.
• HA - Configured for High Availability. E.g. 'PSO-50-M-PROD-HA' (Dataset availability set to 2). UHA (Ultra High Availability) is also an option though is currently the same as HA.

Encrypt Database Connection Strings¶

Encrypt the database connection strings before passing the values to Helm. Defaults to true.

encryptDatabaseConnectionStrings: true

Database Import / Export¶

Options for importing data on an install or exporting data on an upgrade. Defaults to no import / export.

databaseImportExport:
- database: System
  folderPath: "./Export/System"
- database: Scheduling
  folderPath: "./Export/Scheduling"

• Possible Databases: System, Scheduling, Modelling, Simulation

Languages¶

Languages to install into the database. Defaults to installing all languages.

languages:
- en
- fr
- pt-BR

Wait for Deployments¶

Whether the PSO installer will wait for application deployments to be ready after running the Helm deployment. Defaults to true.

waitForDeploymentsToStart: true
waitForDeploymentsTimeoutSeconds: 600

Resource Multipliers¶

Multiply all the resource requests or limits on deployments that have no specified resources before passing to Helm. For use in development and testing. Defaults to 1.0.

resourceMultipliers:
  requests: 1.0
  limits: 1.0

Image¶

The container image for the deployments are available via the IFS Build Place remote deployment credentials at the following location:

ifscloud.jfrog.io/docker/ifs/ifs-pso:

E.g.

ifscloud.jfrog.io/docker/ifs/ifs-pso:6.10.0.1

The image registry details can be setup under the images section. The tag is optional and will by default use the application version.

images:
  registry: ifscloud.jfrog.io
  repository: docker/ifs
  tag: "6.9.0.22"
  pullPolicy: IfNotPresent

Image Credentials (Mandatory)¶

The image registry credentials can be setup by specifying a username and password or by specifying an existing manually configured secret.

Configure Secret by Username and Password

The images.username and images.password must be specified under the images section:

images:
  ...
  username: ""
  password: ""

Configure Secret Manually

To manually configure the image pull secret, example YAML for a secret is shown below. See https://kubernetes.io/docs/concepts/configuration/secret/#docker-config-secrets for more information on secrets.

apiVersion: v1
kind: Secret
metadata:
    name: my-image-pull-secrets
type: kubernetes.io/dockerconfigjson
data:
    .dockerconfigjson: |
        "(base64 encoded ~/.dockerconfigjson file)"

Once the secret has been created the images.secretName value must be provided in the values file using the newly created secret name.

images:
  ...
  secretName: "my-image-pull-secrets"

Image Validation¶

Validate the container images and version as part of the PSO install before changing the deployment. Additional values can be specified for node selection to guide the image validation job to the correct nodes. Defaults to enabled: false.

imageValidation:
  enabled: false
  nodeSelector: {}
  affinity: {}
  tolerations: []

Ingress (Mandatory)¶

Details for setting up ingress resources to expose the services allowing access from outside the cluster. PSO supports both Ingress NGINX and Traefik. A host name must be specified and the TLS details configured. Additional annotations and labels can be specified to place on the ingress resources.

Ingress NGINX (Deprecated)
Support for Ingress NGINX is deprecated and will be removed in a future release. This is currently the default ingress controller for backwards compatibility reasons, but Traefik is recommended for new installations.

Traefik
Traefik is the recommended ingress controller. Traefik uses Custom Resource Definitions (CRDs) including IngressRoute and Middleware resources for ingress routing.

ingress:
  enabled: true
  host: www.example.host
  extraHosts: []
  annotations: {}
  labels: {}

  ingressNginx:
    enabled: true

  traefik:
    enabled: false

  tls:
    enabled: true
    ...
  extraTls: []

Extra Hosts¶

Additional hosts can be configured to expose the services on multiple hostnames. Each extra host will use the same TLS configuration unless overridden in extraTls.

ingress:
  ...
  extraHosts:
  - www2.example.host
  - www3.example.host

TLS Configuration¶

Creation of the TLS secrets can be done in multiple ways as specified below:

Configure Secret by crt and key data

The crt and key file data can be specified directly under the tls section and a tls secret will be created using this data.

ingress:
  ...
  tls:
    crt: base64 encoded cert
    key: base64 encoded key

Configure Secret by crt file and key file

The location of the .crt file and .key file can be passed to the PSO installer which will set the crt and key data in the values before passing the values to Helm. The certificate file should only contain the public key details and the key file should only contain the private key details. It is expected that the certificate and key file are PEM encoded.

ingress:
  ...
  tls:
    crtFile: ".\example.host.crt"
    keyFile: ".\example.host.key"

Configure secret by pfx file and password

The location of a .pfx file and optionally a password can be provided. The PSO installer will extract the certificate and key information and will set the crt and key data in the values before passing the values to Helm. If the .pfx file is secured by a password then the pfxPassword must be set.

ingress:
  ...
  tls:
    pfxFile: ".\example.host.pfx"
    pfxPassword: "ifspassword"

Configure Secret Manually

Example YAML for a manually configured TLS secret is shown below.

apiVersion: v1
    kind: Secret
    metadata:
        name: my-tls-secrets
    data:
        tls.crt: base64 encoded cert
        tls.key: base64 encoded key
    type: kubernetes.io/tls

Once a TLS secret has been manually created the ingress.tls.secretName value must be provided in the values file using the secret name.

ingress:
  ...
  tls:
    secretName: "my-tls-secrets"

Pause¶

Pause the Helm deployments

pause: false

Logging¶

Configures the log format for the application, which can be set to either text or ecs. The default is text.

logging:
  format: text

Deployments¶

Each application has its own deployment which can be configured under the deployments section. Each deployment can be enabled / disabled and the number of replicas can be changed.

deployments:
  dse:
    replicas: 2
  mls:
    enabled: false

Deployment Resources¶

The deployment resources can be configured under the resources section for each deployment. See the values.yaml in the Helm chart for the defaults.

deployments:
  #
  # Reduce the requests for the DSE
  #
  dse:
    resources:
      requests:
        cpu: 1.5
        memory: 2048Mi

Deployment Base Href¶

The PSO Workbench (PSW) and RESTful Gateway (SRG) deployments are both exposed externally to the cluster by ingress rules. The location at which they are accessed is specified by the baseHref value. By default they are accessed at /pso/workbench and /pso/gateway.

deployments:
  psw:
    baseHref: /pso-test/workbench
  srg:
    baseHref: /pso-test/gateway

Deployment Graceful Shutdown¶

Each of the deployments have a default termination grace period. This allows the applications to shutdown gracefully by giving enough time for work to be moved to other application instances.

deployments:
  abe:
    terminationGracePeriodSeconds: 900

Deployment Options¶

The below options can be specified at a global level to apply to all application deployments.

deployments:
  global:
    labels: {}
    annotations: {}
    nodeSelector: {}
    affinity: {}
    tolerations: []
    podSecurityContext: {}
    securityContext: {}

The same options can also be overridden at the deployment level:

deployments:
  dse:
    labels: {}
    annotations: {}
    nodeSelector: {}
    affinity: {}
    tolerations: []
    podSecurityContext: {}
    securityContext: {}

Deployment Auto-Scaling¶

Auto-scaling of the deployments can be enabled at a global level.

deployments:
  global:
    autoScaling:
      enabled: false

Deployment auto-scaling can also be enabled / disabled individually:

deployments:
  dse:
    replicas: 4
    autoScaling:
      enabled: false

  psw:
    autoScaling:
      enabled: true

The minimum and maximum number of replicas can be specified on each deployment. See the values.yaml in the Helm chart for the defaults.

deployments:
  dse:
    autoScaling:
      minReplicaCount: 1
      maxReplicaCount: 10

Depending on the type of application the target CPU or Memory can be configured.

deployments:
  abe:
    autoScaling:
      targetCpuUtilization: 70
      targetMemoryUtilization: 70

Additionally the following options can be specified on the deployment auto-scaling section and they will be applied to the KEDA Scaled Object. See the KEDA Scaled Object documentation for more information.

deployments:
  dse:
    autoScaling:
      horizontalPodAutoscalerConfig: {}
      cooldownPeriod: 300
      pollingInterval: 30

WCF Gateway¶

The WCF Gateway is disabled by default but can be enabled from the deployments section:

deployments:
  gwy:
    enabled: true

The WCF Gateway is hosted at a different address when running in Kubernetes. It is exposed on https by the ingress controller and can be configured by specifying a baseHref. Kubernetes WCF Gateway address:

https://example.com/pso/wcf-gateway

Compared to the Standard / Azure WCF Gateway address:

https://example.com:8744/IFS/Scheduling/Gateway/Service

Network Policies¶

Network policies are enabled by default and can be configured from the networkPolicy section:

networkPolicy:
  enabled: true
  internetAccess:
    enabled: true

Internet access is allowed to simplify external communication. E.g. database access, broadcasts and OpenID requests. This can be disabled and egress communication specified explicitly using the egress section:

networkPolicy:
  internetAccess:
    enabled: false
  egress:
    - to:
      - ipBlock:
          cidr: 1.2.3.4/32

Admin Database Connections¶

An optional 'adminConnectionString' attribute can be used when specifying each database connection. If this is specified, then the installer will use the adminConnectionString value during installation, while the PSO components will be set up to use the connectionString value instead.

For example, this might be used to run the installer using an integrated security database connection string, while the services use username and password credentials. Alternatively, the PSO components can be set to use Azure Workload Identities, as described below.

Note that the PSO components do not require all the database privileges that are required by the installer, and so the 'connectionString' login can then have more restricted access. See 'Database Setup' for more details.

databaseConnections:
  - database: System
    provider: SQLAzure
    connectionString: "system connection string"
    adminConnectionString: "admin connection string"
  - database: Scheduling
    provider: SQLAzure
    connectionString: "scheduling connection string"
    adminConnectionString: "admin scheduling connection string"

Azure¶

The Azure section can be used to enable Azure specific features. Enabling allows extracting the region, zone and node information from VM instances improving the dataset load balancing and failover capabilities.

azure:
    enabled: true

Azure Workload Identities¶

Azure workload identities can be used to allow pods to connect to the database using a managed identity.

azure:
  workloadIdentity:
    enabled: true
    clientId: "client-id"

Connection strings can then be used without username and password. E.g.

databaseConnections:
  - database: System
    provider: SQLAzure
    connectionString: 'Server=tcp:host.database.windows.net,1433;Initial Catalog=pso;
      Authentication="Active Directory Default";Connection Timeout=30;
      Encrypt=True;TrustServerCertificate=False;'

Dataset Availability¶

The DatasetAvailability system parameter can be configured on installation:

• high: Set the parameter to 2 if it is less than 2
• standard: Set the parameter to 1
• low: Set the parameter to 0
• : Set the parameter to
• '': The parameter value is not changed

datasetAvailability: high

Input Values Examples¶

Basic Install¶

licenceFile: '.\PSO_License.xml'
kubeContext: pso-aks
namespace: pso
helmInstallName: pso-install

adminUser:
  username: Administrator
  password: ifspassword

databaseConnections:
- database: Scheduling
  provider: SqlAzure
  connectionString: '{connection-string}'
- database: System
  provider: SqlAzure
  connectionString: '{connection-string}'
- database: Archive
  provider: SqlAzure
  connectionString: '{connection-string}'
- database: Simulation
  provider: SqlAzure
  connectionString: '{connection-string}'
- database: Modelling
  provider: SqlAzure
  connectionString: '{connection-string}'

images:
  username: cr-user
  password: cr-password

ingress:
  host: pso.example.com
  tls:
    crtFile: 'cert.crt'
    keyFile: 'cert.key'

Auto-Scaling Install with Private Container Registry¶

licenceFile: '.\PSO_License.xml'
kubeContext: pso-aks
namespace: pso
helmInstallName: pso-install

adminUser:
  username: Administrator
  password: ifspassword

databaseConnections:
- database: Scheduling
  provider: SqlAzure
  connectionString: '{connection-string}'
- database: System
  provider: SqlAzure
  connectionString: '{connection-string}'
- database: Archive
  provider: SqlAzure
  connectionString: '{connection-string}'
- database: Simulation
  provider: SqlAzure
  connectionString: '{connection-string}'
- database: Modelling
  provider: SqlAzure
  connectionString: '{connection-string}'

images:
  registry: my-registry.example.com
  repository: ifs/pso
  username: cr-user
  password: cr-password

ingress:
  host: pso.example.com
  tls:
    crtFile: 'cert.crt'
    keyFile: 'cert.key'

deployments:
  global:
    autoScaling:
      enabled: true
  dse:
    autoScaling:
      maxReplicaCount: 3

Testing a Kubernetes Installation¶

After the installer has completed the Kubernetes installation, it is important to perform a number of initial "smoke tests" to establish whether the deployment was successful and that the scheduling system is working as intended.

Accessing the Scheduling Workbench¶

Navigate to the Scheduling Workbench and login to the administrative user account. This will confirm that login is possible, that UI client-server communications are functioning correctly, and that the server-side of the UI is correctly connected to the database.

Workbench URL

https://{ingress-host}/pso/workbench

Verifying the System¶

Once logged on to the Scheduling Workbench locate the Administration Workspace, click on the System Diagram tab to show your complete system. This is important as it shows all the services and databases that are related to this deployment. At this point it is worthwhile checking that all of these components are as expected based on the configuration during installation.

While still in the Administration Workspace click on the System tab and locate Events. This provides a real time event log from each of the services, showing error, fatal, information and warnings events. From here any errors will be reported, for example if a service is unable to connect to a database.

After establishing that the system contains all of the required services and that there are no critical errors, the next process is to send in scheduling input data.

Sending input data to a Kubernetes Installation¶

Data is sent through the RESTful Gateway. The RESTful gateway allows an API client to interact with PSO, and it can be used to send or retrieve data.

RESTful Gateway URL

https://{ingress-host}/pso/gateway