switch_based
Pre requisites
IP address for ToR switch needs to be provided.
Switch port range where all BMC NICs are connected should be provided.
SNMP v3 should be enabled on the switch.
Non-admin user credentials for the switch need to be provided.
Note
- To create an SNMPv3 user on S series switches (running OS10), use the following commands:
To create SNMP view:
snmp-server view test_view internet included
To create SNMP group:
snmp-server group testgroup 3 auth read test_view
To create SNMP users:
snmp-server user authuser1 testgroup 3 auth sha authpasswd1
- To verify the changes made, use the following commands:
To view the SNMP views:
show snmp view
To view the SNMP groups:
show snmp group
To view the SNMP users:
show snmp user
To save this configuration for later use, run:
copy running-configuration startup-configuration
For more information on SNMP on S series switch click here
For more information on SNMP on N series switch click here
IPMI over LAN needs to be enabled for the BMC.
racadm set iDRAC.IPMILan.Enable 1 racadm get iDRAC.IPMILan
BMC credentials should be the same across all servers and provided as input to Omnia.
Target servers should be configured to boot in PXE mode with appropriate NIC as the first boot device.
Set the IP address of the control plane with a /16 subnet mask. The control plane NIC connected to remote servers (through the switch) should be configured with two IPs (BMC IP and admin IP) in a shared LOM or hybrid set up. In the case dedicated network topology, a single IP (admin IP) is required.
Caution
Do not use daisy chain ports or the port used to connect to the control plane in
switch_based_details
ininput/provision_config.yml
. This can cause IP conflicts on servers attached to potential target ports.Omnia does not validate SNMP switch credentials, if the provision tool is run with incorrect credentials, use the clean-up script and re-run the provision tool with the correct credentials.
If you are re-provisioning your cluster (that is, re-running the
provision.yml
playbook) after a clean-up, ensure to use a differentadmin_nic_subnet
ininput/provision_config.yml
to avoid a conflict with newly assigned servers. Alternatively, disable any OS available in theBoot Option Enable/Disable
section of your BIOS settings (BIOS Settings
>Boot Settings
>UEFI Boot Settings
) on all target nodes.admin_nic_subnet
,ib_nic_subnet
andbmc_nic_subnet
should have the same subnet mask (Omnia only supports /16 subnet masks currently).
Note
The IP range x.y.246.1 - x.y.255.253 (where x and y are provided by the first two octets of
bmc_nic_subnet
) are reserved by Omnia.- If any of the target nodes have a pre-provisioned IP, do not use a
bmc_subnet
and/orip_start_range
/ip_end_range
that encapsulates the pre-provisioned IP. For example, if there are target nodes hosted at 10.3.0.11 and 10.3.0.12,
bmc_subnet
= 10.3.0.0 withip_start_range
= 10.3.0.1/ip_end_range
= 10.3.0.255 will cause a conflict with newly assigned servers however,bmc_subnet
= 10.3.0.0 withip_start_range
= 10.3.0.100/ip_end_range
= 10.3.0.150 would be accepted. Alternatively, a different subnet would be acceptable,iebmc_subnet
= 10.13.0.0.
- If any of the target nodes have a pre-provisioned IP, do not use a
The following parameters need to be populated in input/provision_config.yml
to discover target nodes using a mapping file.
Caution
Do not remove or comment any lines in the
input/provision_config.yml
file.THE ROCKY LINUX OS VERSION ON THE CLUSTER WILL BE UPGRADED TO THE LATEST 8.x VERSION AVAILABLE IRRESPECTIVE OF THE PROVISION_OS_VERSION PROVIDED IN PROVISION_CONFIG.YML.
Parameter |
Details |
---|---|
network_interface_type
Required |
The network type used on the Omnia cluster. Choices:
|
discovery_mechanism
Required |
The mechanism through which Omnia will discover nodes for provisioning. For more information on how the mechanisms work, go to DiscoveryMechanisms. Choices:
|
provision_os
Required |
The operating system to be provisioned on target nodes in the cluster. Choices:
|
provision_os_version
Required |
OS version of provision_os to be installed. Choices:
|
iso_file_path
Required |
Path where user has placed the iso image that needs to be provisioned on target nodes. Accepted files are Rocky8-DVD or RHEL-8.x-DVD (full OS). iso_file_path should contain the provision_os and provision_os_version values in the filename. Default values: |
timezone
Required |
Timezone to be used during OS provisioning. Available timezones are provided here. Choices:
|
language
Required |
Language to be used during OS provisioning. Default values: |
default_lease_time
Required |
Default lease time for IPs assigned by DHCP. Range: 21600-86400 Default values: |
provision_password
Required |
|
postgresdb_password
Required |
|
node_name
Required |
Default values: |
domain_name
Required |
|
public_nic
Required |
The nic/ethernet card that is connected to the public internet. Default values: |
admin_nic
Required |
Default values: |
admin_nic_subnet
Required |
The subnet within which all Admin IPs are assigned. Default values: |
switch_based_details
Required |
- { ip: 172.96.28.12, ports: ‘1-48,49:3,50’ }
- { ip: 172.96.28.12, ports: ‘1-48,49:3,50’ } - { ip: 172.96.28.14, ports: ‘1,2,3,5’ } |
switch_snmp3_username
Required |
The non-admin SNMPv3 username for the switch. |
switch_snmp3_password
Required |
|
ip_start_range
Required |
|
ip_end_range
Required |
|
ib_nic_subnet
Optional |
|
bmc_nic_subnet
Required |
|
bmc_username
Required |
|
bmc_password
Required |
|
update_repos
Required |
Note By default, AppSteam and BaseOS repos will be configured from the given ISO file. Choices:
|
|
Choices:
|
rhel_repo_local_path
Optional |
Sample value: - { repo: “AppStream”, repo_url: “http://xx.yy.zz/pub/Distros/RedHat/RHEL8/8.8/RHEL-8-appstream.repo”, repo_name: “RHEL-8-appstream-partners” } - { repo: “BaseOS”, repo_url: “http://xx.yy.zz/pub/Distros/RedHat/RHEL8/8.8/RHEL-8-baseos.repo”, repo_name: “RHEL-8-baseos-partners” } |
primary_dns
Optional |
Ex: If the |
secondary_dns
Optional |
The secondary DNS host IP queried to provide Internet access to cluster Node (through DHCP routing) |
disk_partition
Optional |
Default values: |
mlnx_ofed_path
Optional |
Absolute path to a local copy of the .iso file containing Mellanox OFED packages. The image can be downloaded from https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/. Sample value: /root/MLNX_OFED_LINUX-5.8-1.1.2.1-rhel8.6-x86_64.iso |
cuda_toolkit_path
Optional |
Absolute path to local copy of .rpm file containing CUDA packages. The cuda rpm can be downloaded from https://developer.nvidia.com/cuda-downloads. CUDA will be installed post provisioning without any user intervention. Eg: cuda_toolkit_path: “/root/cuda-repo-rhel8-12-0-local-12.0.0_525.60.13-1.x86_64.rpm” |
apptainer_support
Required |
|
Note
The input/provision_config.yml
file is encrypted on the first run of the provision tool:
To view the encrypted parameters:
ansible-vault view provision_config.yml --vault-password-file .provision_vault_keyTo edit the encrypted parameters:
ansible-vault edit provision_config.yml --vault-password-file .provision_vault_key
To clear the configuration on Omnia provisioned switches and ports, click here.
To continue to the next steps:
If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.