Tips and Tricks for managing the Middle Tier¶
This page describes the administrative tasks available through the IFS Remote Deployment CLI (main.exe) for managing and troubleshooting the middle tier.
Tools¶
- Middle tier Controller
Using Middle tier Controller (mtctl) for administrating IFS Cloud instance.
CLI Overview¶
The CLI is a Windows binary that communicates with the middle-tier nodes over SSH. The binary is main.exe.
A main.ps1 PowerShell wrapper is also included. It forwards all arguments to main.exe, so .\main.ps1 -resource … and .\main.exe -resource … are interchangeable.
For the full list of available resources, command-line options and installation steps, see the Custom Installation page.
View status of the middle tier server¶
To see the overall resource usage (CPU, memory and all processes) on the middle-tier nodes:
It will display a Linux top-style output for every configured node.
Deprecation Notice
ps> .\main.ps1 -resource 'REMOTE-TOP'
The above command is deprecated and will be removed in a future release.
Restart middle tier server¶
This will reboot all configured middle-tier nodes. The tool waits for each node to come back online (default wait: MaxVMRebootWaitSecs = 1200).
Deprecation Notice
ps> .\main.ps1 -resource 'REBOOT-LINUXBOX'
The above command is deprecated and will be removed in a future release.
Start MicroK8s¶
If MicroK8s is stopped on the nodes, start it with:
Install security updates¶
Apply OS security updates to all nodes and reboot them afterwards. The nodes need outbound internet access to package sources; this command is not intended for air-gapped deployments.
Manage firewall¶
# Check firewall status
.\main.exe -resource FIREWALL -status STATUS
# Enable firewall
.\main.exe -resource FIREWALL -status ENABLE
# Disable firewall
.\main.exe -resource FIREWALL -status DISABLE
# Open additional ports
.\main.exe -resource FIREWALL -status ENABLE-PORTS
Configure kubectl and view the installed containers in the middle-tier¶
Kubectl is a client that connects to the kubernetes cluster (the middle tier).
Make sure kubectl has been added to your windows path.
Retrieve the kubeconfig file¶
Use the CLI to fetch the kubeconfig from the cluster:
The kubeconfig is saved to ifsroot\config\kube\config.
Place the config file to its default location¶
The default path for the kubeconfig file is c:\user\<YourUser>\.kube\config. Copy the file ifsroot\config\kube\config to c:\user\<YourUser>\.kube folder.
Set kubeconfig environment variable¶
It is possible to use the environment variable KUBECONFIG to point to the kubeconfig file instead of copying the file to the default location. This can be a convenient alternative when managing multiple installations from the same machine. It is also possible to merge the contents of multiple config files into one config single.
Powershell
Cmd
Using set_remote_envs.cmd
On Windows, each IFS Cloud delivery includes a set_remote_envs.cmd script in its ifsinstaller folder. Running it sets the KUBECONFIG environment variable (and other required variables) automatically for that delivery, pointing to the full path of config\kube\config. This is the easiest way to configure the environment when working with deliveries.
Verify¶
It should list all namespaces in the kubernetes cluster.
Remote log client¶
The remote log client will collect container logs through a windows schedule task named IfsRemoteLogClientSchedule and will store them inc:\myenvironment\ifsroot\logs\remote-log-client location. The logs will be stored in separate folders with the corresponding container name. The windows schedule task is scheduled to run every 15 minutes.
Logging¶
The CLI writes detailed logs to:
- logs/main-script/main.log - main log file
- logs/main-script/<nodename>/<function>/<function>.log - per-node function logs
Console output includes colored messages (Red = errors, Yellow = warnings, White = info, Gray = verbose when -debug is enabled).
Running deliveries with encrypted config¶
The CLI can encrypt all files in the config/ directory using the -lock flag. When a resource is run with -lock, the files are decrypted for the duration of that command and automatically re-encrypted on exit.
However, the IFS Cloud installer, deliveries might read files directly from the config/ folder (e.g. ifscloud-values.yaml, kube/config). These external tools are not aware of the encryption, so the config must be in plain text while they run.
The workflow when config encryption is in use:
# 1. Unlock the config folder
.\main.exe -resource UNLOCK -lock <password>
# 2. Set up the environment for the delivery
cd c:\myenvironment\ifsroot\deliveries\delivery-1.0.1\ifsinstaller
.\set_remote_envs.cmd
# 3. Run the installer or any other external tool
.\installer.cmd
# 4. When done, re-encrypt the config folder
cd c:\myenvironment\ifsroot
.\main.exe -resource LOCK -lock <password>
This also applies when using kubectl - the kubeconfig must be decrypted for kubectl to connect to the cluster.
Advanced - Kubernetes Administration¶
Concepts¶
Below is a short list of key words used throughout the guide and what it means.
Container¶
A container image is a ready-to-run software package, containing everything needed to run an application.
Pod¶
Pods are deployable objects in Kubernetes and may contain one or more containers. When a Pod runs multiple containers, the containers are managed as a single entity and share the Pod's resources.
Readiness Probe¶
Kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready.
Liveness Probe¶
Indicates whether the container is running. If the liveness probe fails, the kubelet kills the container, and the container is subjected to its restart policy
Commands¶
Set default namespace¶
If you set the default namespace the -n parmater can be omitted in all kubectl and helm commands. - Its good practice to set the default namespace.
View all namespaces¶
List all namespaces that exists in the kubernetes cluster
View status of the middle tier containers¶
View all pods in a specific namespace and their current status.
kubectl get pods -n <your namespace>
# or if you have set default namespace as above:
kubectl get pods
View status of all pods¶
View all pods in all namespaces and their current status.
Example output:
| NAME | READY | STATUS | RESTARTS | AGE |
|---|---|---|---|---|
| ifs-db-init-hv9j4 | 0/1 | Completed | 0 | 19d |
| ifsapp-client-d8b7f8955-hsr9k | 2/2 | Running | 0 | 19d |
| ifsapp-iam-58fc885994-dpctb | 2/2 | Running | 0 | 19d |
| ifsapp-proxy-654d577886-fj55t | 2/2 | Running | 0 | 19d |
Healthy pods displays STATUS RUNNING and READY 2/2. Because Linkerd is used, it automacically adds the data plane to pods (called proxy injection) which means most pods will show READY 2/2. If not it will show 1/1. What is important is that all of the containers in the pod are in RUNNING state and that all the containers in the pod (whether it's 1/1 or 2/2) is ready.
If the pod displays e.g. 1/2 RUNNING it means not all containers in the pod is up and ready to serve requests. There can be multiple reasons for this where the most common are; * the container has not yet initialized and is currently not ready to serve requests. * the container has problems.
If the STATUS shows RUNNING, it usually means that the pod is currently initializing and that the readiness probe has not yet returned the correct status. If the STATUS says something else (e.g CrashLoopBackOff or ImagePullBackOff) then there is a problem with the image or its configuration and the logs needs to be collected in order to find out what might be wrong. If the STATUS shows 'Evicted' the node is running out of memory or disk and will start an eviction process. See further down how to use 'top' commands on the linux box and in kubectl.
Note that it can take several minutes for a pod to start up.
Describe a pod¶
Gives configuration information about the container(s) and Pod (labels, resource requirements, etc.), as well as status information about the container(s) and Pod (state, readiness, restart count, events, etc.).
Show logs¶
Returns the log for specified container. This is the same logs that goes into remote-log-client in the logs folder.
Tail logs¶
Tails the log for specified container. This is the same logs that goes into remote-log-client in the logs folder.
Show metrics for a pod and its containers¶
Show metrics for the pods and its containers in the given namespace.
Show metrics for the pods and its containers in all namespaces.
Example output:
| NAMESPACE | NAME | CPU(cores) | MEMORY(bytes) |
|---|---|---|---|
| ifs-ingress | ingress-nginx-ingress-controller-5567747bbd-rnrrm | 5m | 229Mi |
| ifs-monitoring | fluentd-ptrmw | 13m | 472Mi |
| kube-system | monitoring-influxdb-grafana-v4-6dc675bf8c-whqr2 | 3m | 340Mi |
| polvodev | ifsapp-proxy-848cc94d5d-d8hng | 0m | 23Mi |
Dive deeper into kubectl top¶
This information can be compared to the configured limits for a specific pod (can be retrieved with kubectl describe). Pods close or far from their configured limits might indicate that a system is scaled to tight or too loose but it is not necessarily so.
If pods are getting evicted, verify their actual CPU and memory usages compared to the configured limits. Also verify that the configured limits aren't overcommitting the resources of the node too much.
Support issues in Middle tier¶
Most of the above kubectl commands that are very good for troubleshooting and should be sent to support upon any Middle tier issues. Please use the mtctl tool to export them all into a folderstructure that can be zipped and attached to the support ticket.