Control plane logs

All log files can be viewed using CLI. Alternatively, most log files can be viewed using the Dashboard tab ( ) on the grafana UI.

Caution

It is not recommended to delete the below log files or the directories they reside in.

Note

Log files are rotated periodically as a storage consideration. To customize how often logs are rotated, edit the /etc/logrotate.conf file on the node.

Below is a list of all logs available to Loki and can be accessed from the dashboard:

Log files

Name	Location	Purpose	Additional Information
Omnia Logs	/var/log/omnia.log	Omnia Log	This log is configured by Default. This log can be used to track all changes made by all playbooks in the omnia directory.
Accelerator Logs	/var/log/omnia/accelerator.log	Accelerator Log	This log is configured by Default.
Monitor Logs	/var/log/omnia/monitor.log	Monitor Log	This log is configured by Default.
Network Logs	/var/log/omnia/network.log	Network Log	This log is configured by Default.
Platform Logs	/var/log/omnia/platforms.log	Platform Log	This log is configured by Default.
Provision Logs	/var/log/omnia/provision.log	Provision Log	This log is configured by Default.
Scheduler Logs	/var/log/omnia/scheduler.log	Scheduler Log	This log is configured by Default.
Security Logs	/var/log/omnia/security.log	Security Log	This log is configured by Default.
Storage Logs	/var/log/omnia/storage.log	Storage Log	This log is configured by Default.
Telemetry Logs	/var/log/omnia/telemetry.log	Telemetry Log	This log is configured by Default.
Utils Logs	/var/log/omnia/utils.log	Utils Log	This log is configured by Default.
Cluster Utilities Logs	/var/log/omnia/utils_cluster.log	Cluster Utils Log	This log is configured by Default.
syslogs	/var/log/messages	System Logging	This log is configured by Default.
Audit Logs	/var/log/audit/audit.log	All Login Attempts	This log is configured by Default.
CRON logs	/var/log/cron	CRON Job Logging	This log is configured by Default.
Pods logs	/var/log/pods/ * / * / * log	k8s pods	This log is configured by Default.
Access Logs	/var/log/dirsrv/slapd-<Realm Name>/access	Directory Server Utilization	This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’).
Error Log	/var/log/dirsrv/slapd-<Realm Name>/errors	Directory Server Errors	This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’).
CA Transaction Log	/var/log/pki/pki-tomcat/ca/transactions	FreeIPA PKI Transactions	This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’).
KRB5KDC	/var/log/krb5kdc.log	KDC Utilization	This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’).
Secure logs	/var/log/secure	Login Error Codes	This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’).
HTTPD logs	/var/log/httpd/ *	FreeIPA API Calls	This log is available when FreeIPA or 389ds is set up ( ie when enable_security_support is set to ‘true’).
DNF logs	/var/log/dnf.log	Installation Logs	This log is configured on Rocky OS.
BeeGFS Logs	/var/log/beegfs-client.log	BeeGFS Logs	This log is configured on BeeGFS client nodes.
Compute Logs	/var/log/xcat/computes.log	Logs system messages from all cluster nodes.	This log is configured by Default.
Cluster deployment logs	/var/log/xcat/cluster.log	Logs deployment messages from all cluster nodes.	This log is configured by Default.

Logs of individual containers

1. A list of namespaces and their corresponding pods can be obtained using: kubectl get pods -A

2. Get a list of containers for the pod in question using: kubectl get pods <pod_name> -o jsonpath='{.spec.containers[*].name}'

3. Once you have the namespace, pod and container names, run the below command to get the required logs: kubectl logs pod <pod_name> -n <namespace> -c <container_name>

Provisioning logs

Logs pertaining to actions taken during discovery_provision.yml can be viewed in /var/log/xcat/cluster.log and /var/log/xcat/computes.log on the control plane.

Note

As long as a node has been added to a cluster by Omnia, deployment events taking place on the node will be updated in /var/log/xcat/cluster.log.

Telemetry logs

Logs pertaining to actions taken by Omnia or iDRAC telemetry can be viewed in /var/log/messages. Each log entry is tagged “omnia_telemetry”. Log entries typically follow this format.

<Date time> <Node name> omnia_telemetry[<Process ID>]: <name of file>:<name of method throwing error>: <Error message>

Grafana Loki

After telemetry.yml is run, Grafana services are installed on the control plane.

1. Get the Grafana IP using kubectl get svc -n grafana.

2. Login to the Grafana UI by connecting to the cluster IP of grafana service via port 5000. That is http://xx.xx.xx.xx:5000/login.

3. In the Explore page, select control-plane-loki.

4. The log browser allows users to filter logs by job, node, user, etc. Example

   (job= "cluster deployment logs") |= "nodename"
   (job="compute log messages") |= "nodename" |="node_username"
   

Custom dashboards can be created as per your requirement.

If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.