Skip to content

Tips and Tricks for managing the Middle Tier

This page describes the Administrate IFS Cloud remote deployment model tasks that can be used for managing and troubleshooting the middle tier.

Tools

View status of the middle tier server

To see the overall resource usage on the middle-tier server run the command:

ps> .\main.ps1 -resource 'REMOTE-TOP'   

It will display a Linux top output, with CPU, Memory and all processes.

Restart middle tier server

This will restart the middle-tier server and apply the Disable AppArmor Profile (see "How To Setup An Environment"). If needed you can change the default wait time (eg: MaxVMRebootWaitSecs = 1200) for vm restart.

ps> .\main.ps1 -resource 'REBOOT-LINUXBOX'   

Configure kubectl and view the installed containers in the middle-tier

Kubectl is a client that connects to the kubernetes cluster (the middle tier).

Make sure kubectl has been added to your windows path.

Place the config file to its default location

The default path for the kubeconfig file is c:\user\<YourUser>\.kube\config. Copy the file ifsroot\config\kube\config to c:\user\<YourUser>\.kube folder

Set kubeconfig environment variable

It is possible to use the environment variable KUBECONFIG to point to the kubeconfig file instead of copying the file to the default location. This can be a convenient alternative when managing multiple installations from the same machine. It is also possible to merge the contents of multiple config files into one config single.

Powershell

ps> $env:KUBECONFIG = "ifsroot\config\kube\config"  

Cmd

cmd> set KUBECONFIG=ifsroot\config\kube\config  

Verify

cmd> kubectl get ns  

It should list all namespaces in the kubernetes cluster.

Install IFS Cloud deliveries

In the Remote deployment model, the deliveries and build_home are placed in the ifsroot\deliveries folder. The installation of deliveries and build_home (Fresh install) are made from within the delivery or build_home.

The ifscloud-values.yaml file should be placed in the ifsroot\config folder. Note: A delivery should be placed in the deliveries and should not contain any additional folder depth. i.e. "deliveries\delivery-1.0.1\ifsinstaller" os ok but not "deliveries\delivery-1.0.1\InstallationFiles\ifsinstaller" The delivery can deployed like:

CMD> cd c:\myenvironment\ifsroot\deliveries\delivery-1.0.1\ifsinstaller  
CMD> .\installer.cmd  

See documentation for more info.

Remote log client

The remote log client will collect container logs through a windows schedule task named IfsRemoteLogClientSchedule and will store them in c:\myenvironment\ifsroot\logs\remote-log-client location. The logs will be stored in separate folders with the corresponding container name. The windows schedule task is scheduled to run every 15 minutes.

Advanced - Kubernetes Administration

Concepts

Below is a short list of key words used throughout the guide and what it means.

Container

A container image is a ready-to-run software package, containing everything needed to run an application.

Pod

Pods are deployable objects in Kubernetes and may contain one or more containers. When a Pod runs multiple containers, the containers are managed as a single entity and share the Pod's resources.

Readiness Probe

Kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready.

Liveness Probe

Indicates whether the container is running. If the liveness probe fails, the kubelet kills the container, and the container is subjected to its restart policy

Commands

Set default namespace

If you set the default namespace the -n parmater can be omitted in all kubectl and helm commands. - Its good practice to set the default namespace.

ps> kubectl config set-context --current --namespace=<your namespace>  

View all namespaces

List all namespaces that exists in the kubernetes cluster

ps> kubectl get ns  

View status of the middle tier containers

View all pods in a specific namespace and their current status.

ps> kubectl get pods -n <your namespace> 

or if you have set default namespace as above:
ps> kubectl get pods 

View status of all pods

View all pods in all namespaces and their current status.

ps> kubectl get pods -A  

Example output:

NAME READY STATUS RESTARTS AGE
ifs-db-init-hv9j4 0/1 Completed 0 19d
ifsapp-client-d8b7f8955-hsr9k 2/2 Running 0 19d
ifsapp-iam-58fc885994-dpctb 2/2 Running 0 19d
ifsapp-proxy-654d577886-fj55t 2/2 Running 0 19d

Healthy pods displays STATUS RUNNING and READY 2/2. Because Linkerd is used, it automacically adds the data plane to pods (called proxy injection) which means most pods will show READY 2/2. If not it will show 1/1. What is important is that all of the containers in the pod are in RUNNING state and that all the containers in the pod (whether it's 1/1 or 2/2) is ready.

If the pod displays e.g. 1/2 RUNNING it means not all containers in the pod is up and ready to serve requests. There can be multiple reasons for this where the most common are; * the container has not yet initialized and is currently not ready to serve requests. * the container has problems.

If the STATUS shows RUNNING, it usually means that the pod is currently initializing and that the readiness probe has not yet returned the correct status. If the STATUS says something else (e.g CrashLoopBackOff or ImagePullBackOff) then there is a problem with the image or its configuration and the logs needs to be collected in order to find out what might be wrong. If the STATUS shows 'Evicted' the node is running out of memory or disk and will start an eviction process. See further down how to use 'top' commands on the linux box and in kubectl.

Note that it can take several minutes for a pod to start up

View all pods in all namespaces and their current status.

ps> kubectl get pods -A  

See above for more information about pod status.

Describe a pod

ps> kubectl describe pod <pod-name> -n <namespace>  

Gives configuration information about the container(s) and Pod (labels, resource requirements, etc.), as well as status information about the container(s) and Pod (state, readiness, restart count, events, etc.).

Show logs

ps> kubectl logs <pod-name> <container-name> -n <namespace>  

Returns the log for specified container. This is the same logs that goes into remote-log-client in the logs folder.

Tail logs

ps> kubectl logs <pod-name> <container-name> -n <namespace> -f  

Tails the log for specified container. This is the same logs that goes into remote-log-client in the logs folder.

Show metrics for a pod and its containers

ps> kubectl top pod -n <namespace>  

Show metrics for the pods and its containers in the given namespace.

ps> kubectl top pod -A  

Show metrics for the pods and its containers in all namespaces.

Example output

NAMESPACE NAME CPU(cores) MEMORY(bytes)
ifs-ingress ingress-nginx-ingress-controller-5567747bbd-rnrrm 5m 229Mi
ifs-monitoring fluentd-ptrmw 13m 472Mi
kube-system monitoring-influxdb-grafana-v4-6dc675bf8c-whqr2 3m 340Mi
polvodev ifsapp-proxy-848cc94d5d-d8hng 0m 23Mi

Dive deeper into kubectl top

ps> kubectl top pod <podname> -n <namespace> --containers  

This information can be compared to the configured limits for a specific pod (can be retrieved with kubectl describe). Pods close or far from their configured limits might indicate that a system is scaled to tight or too loose but it is not necessarily so.

If pods are getting evicted, verify their actual CPU and memory usages compared to the configured limits. Also verify that the configured limits aren't overcommitting the resources of the node too much.

Support issues in Middle tier

Most of the above kubectl commands that are very good for troubleshooting and should be sent to support upon any Middle tier issues. Please use the mtctl tool to export them all into a folderstructure that can be zipped and attached to the support ticket.