Tips and Tricks for managing the Middle Tier¶
This page describes the Administrate IFS Cloud remote deployment model tasks that can be used for managing and troubleshooting the middle tier.
Tools¶
- Middle tier Controller
Using Middle tier Controller (mtctl) for administrating IFS Cloud instance.
View status of the middle tier server¶
To see the overall resource usage on the middle-tier server run the command:
ps> .\main.ps1 -resource 'REMOTE-TOP'
It will display a Linux top output, with CPU, Memory and all processes.
Restart middle tier server¶
This will restart the middle-tier server and apply the Disable AppArmor Profile (see "How To Setup An Environment"). If needed you can change the default wait time (eg: MaxVMRebootWaitSecs = 1200) for vm restart.
ps> .\main.ps1 -resource 'REBOOT-LINUXBOX'
Configure kubectl and view the installed containers in the middle-tier¶
Kubectl is a client that connects to the kubernetes cluster (the middle tier).
Make sure kubectl has been added to your windows path.
Place the config file to its default location¶
The default path for the kubeconfig file is c:\user\<YourUser>\.kube\config. Copy the file ifsroot\config\kube\config to c:\user\<YourUser>\.kube folder
Set kubeconfig environment variable¶
It is possible to use the environment variable KUBECONFIG to point to the kubeconfig file instead of copying the file to the default location. This can be a convenient alternative when managing multiple installations from the same machine. It is also possible to merge the contents of multiple config files into one config single.
Powershell
ps> $env:KUBECONFIG = "ifsroot\config\kube\config"
Cmd
cmd> set KUBECONFIG=ifsroot\config\kube\config
Verify¶
cmd> kubectl get ns
It should list all namespaces in the kubernetes cluster.
Install IFS Cloud deliveries¶
In the Remote deployment model, the deliveries and build_home are placed in the ifsroot\deliveries folder. The installation of deliveries and build_home (Fresh install) are made from within the delivery or build_home.
The ifscloud-values.yaml file should be placed in the ifsroot\config folder. Note: A delivery should be placed in the deliveries and should not contain any additional folder depth. i.e. "deliveries\delivery-1.0.1\ifsinstaller" os ok but not "deliveries\delivery-1.0.1\InstallationFiles\ifsinstaller" The delivery can deployed like:
CMD> cd c:\myenvironment\ifsroot\deliveries\delivery-1.0.1\ifsinstaller
CMD> .\installer.cmd
See documentation for more info.
Remote log client¶
The remote log client will collect container logs through a windows schedule task named IfsRemoteLogClientSchedule and will store them in c:\myenvironment\ifsroot\logs\remote-log-client location. The logs will be stored in separate folders with the corresponding container name. The windows schedule task is scheduled to run every 15 minutes.
Advanced - Kubernetes Administration¶
Concepts¶
Below is a short list of key words used throughout the guide and what it means.
Container¶
A container image is a ready-to-run software package, containing everything needed to run an application.
Pod¶
Pods are deployable objects in Kubernetes and may contain one or more containers. When a Pod runs multiple containers, the containers are managed as a single entity and share the Pod's resources.
Readiness Probe¶
Kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready.
Liveness Probe¶
Indicates whether the container is running. If the liveness probe fails, the kubelet kills the container, and the container is subjected to its restart policy
Commands¶
Set default namespace¶
If you set the default namespace the -n parmater can be omitted in all kubectl and helm commands. - Its good practice to set the default namespace.
ps> kubectl config set-context --current --namespace=<your namespace>
View all namespaces¶
List all namespaces that exists in the kubernetes cluster
ps> kubectl get ns
View status of the middle tier containers¶
View all pods in a specific namespace and their current status.
ps> kubectl get pods -n <your namespace>
or if you have set default namespace as above:
ps> kubectl get pods
View status of all pods¶
View all pods in all namespaces and their current status.
ps> kubectl get pods -A
Example output:
| NAME | READY | STATUS | RESTARTS | AGE |
|---|---|---|---|---|
| ifs-db-init-hv9j4 | 0/1 | Completed | 0 | 19d |
| ifsapp-client-d8b7f8955-hsr9k | 2/2 | Running | 0 | 19d |
| ifsapp-iam-58fc885994-dpctb | 2/2 | Running | 0 | 19d |
| ifsapp-proxy-654d577886-fj55t | 2/2 | Running | 0 | 19d |
Healthy pods displays STATUS RUNNING and READY 2/2. Because Linkerd is used, it automacically adds the data plane to pods (called proxy injection) which means most pods will show READY 2/2. If not it will show 1/1. What is important is that all of the containers in the pod are in RUNNING state and that all the containers in the pod (whether it's 1/1 or 2/2) is ready.
If the pod displays e.g. 1/2 RUNNING it means not all containers in the pod is up and ready to serve requests. There can be multiple reasons for this where the most common are; * the container has not yet initialized and is currently not ready to serve requests. * the container has problems.
If the STATUS shows RUNNING, it usually means that the pod is currently initializing and that the readiness probe has not yet returned the correct status. If the STATUS says something else (e.g CrashLoopBackOff or ImagePullBackOff) then there is a problem with the image or its configuration and the logs needs to be collected in order to find out what might be wrong. If the STATUS shows 'Evicted' the node is running out of memory or disk and will start an eviction process. See further down how to use 'top' commands on the linux box and in kubectl.
Note that it can take several minutes for a pod to start up
View all pods in all namespaces and their current status.¶
ps> kubectl get pods -A
See above for more information about pod status.
Describe a pod¶
ps> kubectl describe pod <pod-name> -n <namespace>
Gives configuration information about the container(s) and Pod (labels, resource requirements, etc.), as well as status information about the container(s) and Pod (state, readiness, restart count, events, etc.).
Show logs¶
ps> kubectl logs <pod-name> <container-name> -n <namespace>
Returns the log for specified container. This is the same logs that goes into remote-log-client in the logs folder.
Tail logs¶
ps> kubectl logs <pod-name> <container-name> -n <namespace> -f
Tails the log for specified container. This is the same logs that goes into remote-log-client in the logs folder.
Show metrics for a pod and its containers¶
ps> kubectl top pod -n <namespace>
Show metrics for the pods and its containers in the given namespace.
ps> kubectl top pod -A
Show metrics for the pods and its containers in all namespaces.
Example output
| NAMESPACE | NAME | CPU(cores) | MEMORY(bytes) |
|---|---|---|---|
| ifs-ingress | ingress-nginx-ingress-controller-5567747bbd-rnrrm | 5m | 229Mi |
| ifs-monitoring | fluentd-ptrmw | 13m | 472Mi |
| kube-system | monitoring-influxdb-grafana-v4-6dc675bf8c-whqr2 | 3m | 340Mi |
| polvodev | ifsapp-proxy-848cc94d5d-d8hng | 0m | 23Mi |
Dive deeper into kubectl top¶
ps> kubectl top pod <podname> -n <namespace> --containers
This information can be compared to the configured limits for a specific pod (can be retrieved with kubectl describe). Pods close or far from their configured limits might indicate that a system is scaled to tight or too loose but it is not necessarily so.
If pods are getting evicted, verify their actual CPU and memory usages compared to the configured limits. Also verify that the configured limits aren't overcommitting the resources of the node too much.
Support issues in Middle tier¶
Most of the above kubectl commands that are very good for troubleshooting and should be sent to support upon any Middle tier issues. Please use the mtctl tool to export them all into a folderstructure that can be zipped and attached to the support ticket.