switch_based

Pre requisites

  • IP address for ToR switch needs to be provided.

  • Switch port range where all BMC NICs are connected should be provided.

  • SNMP v3 should be enabled on the switch.

  • Non-admin user credentials for the switch need to be provided.

Note

  • To create an SNMPv3 user on S series switches (running OS10), use the following commands:

    • To create SNMP view: snmp-server view test_view internet included

    • To create SNMP group: snmp-server group testgroup 3 auth read test_view

    • To create SNMP users: snmp-server user authuser1 testgroup 3 auth sha authpasswd1

  • To verify the changes made, use the following commands:

    • To view the SNMP views: show snmp view

    • To view the SNMP groups: show snmp group

    • To view the SNMP users: show snmp user

  • To save this configuration for later use, run: copy running-configuration startup-configuration

  • For more information on SNMP on S series switch click here

  • For more information on SNMP on N series switch click here

  • IPMI over LAN needs to be enabled for the BMC.

  • BMC NICs should have a static IP assigned or be configured in DHCP mode.

  • BMC credentials should be the same across all servers and provided as input to Omnia.

  • Target servers should be configured to boot in PXE mode with appropriate NIC as the first boot device.

  • The control plane NIC connected to remote servers (through the switch) should be configured with two IPs in a shared LOM set up. This NIC is configured by Omnia with the IP xx.yy.255.254, aa.bb.255.254 (where xx.yy are taken from bmc_nic_subnet and aa.bb are taken from admin_nic_subnet) when discovery_mechanism is set to switch-based.

../../../_images/ControlPlaneNic.png

Caution

  • Do not use daisy chain ports or the port used to connect to the control plane in switch_based_details in input/provision_config.yml. This can cause IP conflicts on servers attached to potential target ports.

  • Omnia does not validate SNMP switch credentials, if the provision tool is run with incorrect credentials, use the clean-up script and re-run the provision tool with the correct credentials.

  • If you are re-provisioning your cluster (that is, re-running the provision.yml playbook) after a clean-up, ensure to use a different admin_nic_subnet in input/provision_config.yml to avoid a conflict with newly assigned servers. Alternatively, disable any OS available in the Boot Option Enable/Disable section of your BIOS settings (BIOS Settings > Boot Settings > UEFI Boot Settings) on all target nodes.

Note

  • The IP range x.y.246.1 - x.y.255.253 (where x and y are provided by the first two octets of bmc_nic_subnet) are reserved by Omnia.

  • If any of the target nodes have a pre-provisioned IP, do not use a bmc_subnet and/or ip_start_range/ip_end_range that encapsulates the pre-provisioned IP.

    • For example, if there are target nodes hosted at 10.3.0.11 and 10.3.0.12, bmc_subnet = 10.3.0.0 with ip_start_range = 10.3.0.1/ ip_end_range = 10.3.0.255 will cause a conflict with newly assigned servers however, bmc_subnet = 10.3.0.0 with ip_start_range = 10.3.0.100/ ip_end_range = 10.3.0.150 would be accepted. Alternatively, a different subnet would be acceptable,ie bmc_subnet = 10.13.0.0.

The following parameters need to be populated in input/provision_config.yml to discover target nodes using a mapping file.

Caution

Do not remove or comment any lines in the input/provision_config.yml file.

Parameters

Parameter

Details

network_interface_type

string

Required

The network type used on the Omnia cluster.

Choices:

  • dedicated <- default

  • lom

discovery_mechanism

string

Required

The mechanism through which Omnia will discover nodes for provisioning. For more information on how the mechanisms work, go to DiscoveryMechanisms.

Choices:

  • switch_based

  • mapping <-default

  • bmc

  • snmpwalk

provision_os

string

Required

The operating system to be provisioned on target nodes in the cluster.

Choices:

  • rhel <-default

  • rocky

provision_os_version

string

Required

OS version of provision_os to be installed.

Choices:

  • 8.0

  • 8.1

  • 8.2

  • 8.3

  • 8.4

  • 8.5

  • 8.6 <- default

  • 8.7

iso_file_path

string

Required

Path where user has placed the iso image that needs to be provisioned on target nodes. Accepted files are Rocky8-DVD or RHEL-8.x-DVD (full OS). iso_file_path should contain the provision_os and provision_os_version values in the filename.

Default values: "/home/RHEL-8.6.0-20220420.3-x86_64-dvd1.iso"

timezone

string

Required

Timezone to be used during OS provisioning. Available timezones are provided here.

Choices:

  • GMT <- default

  • EST

  • CET

  • MST

  • CST6CDT

  • PST8PDT

language

string

Required

Language to be used during OS provisioning.

Default values: en-US

default_lease_time

integer

Required

Default lease time for IPs assigned by DHCP. Range: 21600-86400

Default values: 86400

provision_password

string

Required

  • Password set for the root account of target nodes during provisioning.

  • Length >= 8 characters

  • Password must not contain -,, ‘,”

postgresdb_password

string

Required

  • Password set for the postgresDB on target nodes during provisioning.

  • Length >= 8 characters

  • Password must not contain -,, ‘,”

node_name

string

Required

  • Prefix for target node names, if dynamically allocated.

  • Hostname = node_name + ‘0000x’ + domain_name

  • Hostname <= 65 characters

  • Example: servernode00001.Omnia.test , where node_name =servernode, domain_name =Omnia.test , 00001 used by Omnia.

Default values: node

domain_name

string

Required

  • Domain name the user intends to configure on the cluster.

  • Hostname = node_name + ‘0000x’ + domain_name

  • Hostname <= 65 characters

  • Please provide a valid domain name according to the domain name standards.

  • Example: servernode00001.Omnia.test , where node_name=servernode, domain_name=Omnia.test , 00001 used by Omnia.

public_nic

string

Required

The nic/ethernet card that is connected to the public internet.

Default values: eno2

admin_nic

string

Required

Admin NIC of Control Plane. This is the shared LOM NIC.

Default values: eno1

admin_nic_subnet

string

Required

The subnet within which all Admin IPs are assigned.

Default values: 10.5.0.0

switch_based_details

JSON List

Required

  • JSON list of switches to query for target nodes.

  • Split port ranges are not accepted here. (Ex: 10:5-10:10 will not be valid).

  • Example:

  - { ip: 172.96.28.12, ports: ‘1-48,49:3,50’ }

  • Example with 2 switches:

  - { ip: 172.96.28.12, ports: ‘1-48,49:3,50’ }

  - { ip: 172.96.28.14, ports: ‘1,2,3,5’ }

switch_snmp3_username

string

Required

The non-admin SNMPv3 username for the switch.

switch_snmp3_password

string

Required

The non-admin SNMPv3 password for the switch.

ip_start_range

string

Required

  • The IP start range for all the NICs within the subnets mentioned above.(Admin, BMC, IB).

  • Example: For the range x.y.0.1 to x.y.0.100 with admin_nic_subnet = 10.5.0.0, target nodes will be assigned admin IPs between 10.5.0.1 and 10.5.0.100.

ip_end_range

string

Required

  • The IP end range for all the NICs within the subnets mentioned above.(Admin, BMC, IB).

  • Example: For the range x.y.0.1 to x.y.0.100 with admin_nic_subnet = 10.5.0.0, target nodes will be assigned admin IPs between 10.5.0.1 and 10.5.0.100.

ib_nic_subnet

string

Optional

  • If provided, Omnia will handle and assign static IPs to compute node’s IB network.

  • Only the last 16 bits/2 octets of IPv4 are dynamic

  • If provided, the DB entry will be in parallel with the pxe_subnet.

  • Example: If admin_ip: 10.5.0.50 and ib_nic_subnet: 10.10.0.0, then ib_ip: 10.10.0.50

bmc_nic_subnet

string

Required

  • If provided, Omnia will assign static IPs to IB NICs on the compute nodes within the provided subnet.

  • If network_interface_type: lom, mandatory for discovery_mechanism: mapping, switch_based and bmc.

  • If network_interface_type: dedicated, optional for discovery_mechanism: mapping, switch_based.

  • Note that since the last 16 bits/2 octets of IPv4 are dynamic, please ensure that the parameter value is set to x.x.0.0.

  • When the PXE range and BMC subnet are provided, corresponding NICs will be assigned IPs with the same 3rd and 4th octets.

bmc_username

string

Required

  • The username for iDRAC.

  • The username must not contain -,, ‘,”

bmc_password

string

Required

  • The password for iDRAC.

  • The password must not contain -,, ‘,”

update_repos

boolean

Required

  • Indicates whether provision.yml will update offline RHEL repos.

  • If update_repos: false, the update repos for BaseOS and AppStream will not be updated to the latest versions available.

  • If update_repos: true, the update repos for BaseOS and AppStream will be updated to the latest versions available.

Note

By default, AppSteam and BaseOS repos will be configured from the given ISO file.

Choices:

false <- Default

true

rhel_repo_alphabetical_folders boolean

Required

  • Indicates whether the packages in local or subscription repos should be ordered in alphabetical directories.

  • This variable should be filled if control plane OS is RHEL and local RHEL repository is available.

Choices:

false <- Default

true

rhel_repo_local_path

JSON list

Optional

  • The repo path and names of the software repository to be configured on the compute nodes.

  • Provide the repo data file path, which ends with .repo extension in repo_url parameter.

  • Provide the url for BaseOS and AppStream repositories.

  • This variable should be filled if control plane OS is RHEL and subscription is not activated.

  • This variable should be filled if the control plane OS is Rocky and the provision_os is rhel.

Default value:

  • { repo: “AppStream”, repo_url: “”, repo_name: “” }

  • { repo: “BaseOS”, repo_url: “”, repo_name: “” }

primary_dns

string

Optional

  • The primary DNS host IP queried to provide Internet access to Compute Node (through DHCP routing).

  • Currently, the primary_dns value stored in input/provision_config.yml cannot be part of any of the subnets (admin_nic_subnet, ib_nic_subnet and bmc_nic_subnet) also defined in input/provision_config.yml.

Ex: If the primary_dns is set to 10.15.0.7, the subnet 10.15.0.0 cannot be used for admin_nic_subnet, ib_nic_subnet or bmc_nic_subnet.

secondary_dns

string

Optional

The secondary DNS host IP queried to provide Internet access to Compute Node (through DHCP routing)

disk_partition

JSON list

Optional

  • User defined disk partition applied to remote servers.

  • The disk partition desired_capacity has to be provided in MB.

  • Valid mount_point values accepted for disk partition are /home, /var, /tmp, /usr, swap.

  • Default partition size provided for /boot is 1024MB, /boot/efi is 256MB and the remaining space to / partition.

  • Values are accepted in the form of JSON list such as: , - { mount_point: “/home”, desired_capacity: “102400” }

Default values: - { mount_point: "", desired_capacity: "" }

mlnx_ofed_path

string

Optional

Absolute path to a local copy of the .iso file containing Mellanox OFED packages. The image can be downloaded from https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/. Sample value: /root/MLNX_OFED_LINUX-5.8-1.1.2.1-rhel8.6-x86_64.iso

cuda_toolkit_path

string

Optional

Absolute path to local copy of .rpm file containing CUDA packages. The cuda rpm can be downloaded from https://developer.nvidia.com/cuda-downloads. CUDA will be installed post provisioning without any user intervention. Eg: cuda_toolkit_path: “/root/cuda-repo-rhel8-12-0-local-12.0.0_525.60.13-1.x86_64.rpm”

Note

The input/provision_config.yml file is encrypted on the first run of the provision tool:

To view the encrypted parameters:

ansible-vault view provision_config.yml --vault-password-file .provision_vault_key

To edit the encrypted parameters:

ansible-vault edit provision_config.yml --vault-password-file .provision_vault_key

To clear the configuration on Omnia provisioned switches and ports, click here.

To continue to the next steps:

If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.