bmc

For automatic provisioning of servers and discovery, the BMC method can be used.

Pre requisites

  • Set the IP address of the control plane with a /16 subnet mask. The control plane NIC connected to remote servers (through the switch) should be configured with two IPs (BMC IP and admin IP) in a shared LOM or hybrid set up. In the case dedicated network topology, a single IP (admin IP) is required.

../../../_images/ControlPlaneNic.png

  • BMC NICs should have a static IP assigned or be configured in DHCP mode.

  • BMC credentials should be the same across all servers and provided as input to Omnia in the parameters explained below.

  • Target servers should be configured to boot in PXE mode with the appropriate NIC as the first boot device.

  • IPMI over LAN needs to be enabled for the BMC.

    racadm set iDRAC.IPMILan.Enable 1
racadm get iDRAC.IPMILan

  • IP ranges (bmc_static_start_range, bmc_static_start_range) provided to Omnia for BMC discovery should be within the same subnet.

Caution

  • To create a meaningful range of discovery, ensure that the last two octets of bmc_static_end_range are equal to or greater than the last two octets of the bmc_static_start_range. That is, for the range a.b.c.d - a.b.e.f, e and f should be greater than or equal to c and d. Ex: 172.20.0.50 - 172.20.1.101 is a valid range however, 172.20.0.101 - 172.20.1.50 is not.

  • If you are re-provisioning your cluster (that is, re-running the provision.yml playbook) after a clean-up, ensure to use a different admin_nic_subnet in input/provision_config.yml to avoid a conflict with newly assigned servers. Alternatively, disable any OS available in the Boot Option Enable/Disable section of your BIOS settings (BIOS Settings > Boot Settings > UEFI Boot Settings) on all target nodes.

  • admin_nic_subnet, ib_nic_subnet and bmc_nic_subnet should have the same subnet mask (Omnia only supports /16 subnet masks currently).

  • All iDRACs should be reachable from the admin_nic.

Note

When iDRACs are in DHCP mode*

  • The IP range x.y.246.1 - x.y.255.253 (where x and y are provided by the first two octets of bmc_nic_subnet) are reserved by Omnia.

  • x.y.246.1 - x.y.250.253 will be the range of IPs reserved for dynamic assignment by Omnia.

  • During provisioning, Omnia updates servers to static mode and assigns IPs from x.y.251.1 - x.y.255.253.

  • Users can see the IPs (that have been assigned from x.y.251.1 - x.y.255.253) in the DB after provisioning the servers.

  • For example:

    If the provided bmc_subnet is 10.3.0.0 and there are two iDRACs in DHCP mode, the IPs assigned will be 10.3.251.1 and 10.3.251.2.

The following parameters need to be populated in input/provision_config.yml to discover target nodes using BMC.

Caution

  • Do not remove or comment any lines in the input/provision_config.yml file.

  • THE ROCKY LINUX OS VERSION ON THE CLUSTER WILL BE UPGRADED TO THE LATEST 8.x VERSION AVAILABLE IRRESPECTIVE OF THE PROVISION_OS_VERSION PROVIDED IN PROVISION_CONFIG.YML.

Parameters

Parameter

Details

network_interface_type

string

Required

The network type used on the Omnia cluster.

Choices:

  • dedicated <- default

  • lom

discovery_mechanism

string

Required

The mechanism through which Omnia will discover nodes for provisioning. For more information on how the mechanisms work, go to DiscoveryMechanisms.

Choices:

  • switch_based

  • mapping <-default

  • bmc

  • snmpwalk

provision_os

string

Required

The operating system to be provisioned on target nodes in the cluster.

Choices:

  • rhel <-default

  • rocky

Caution

THE ROCKY LINUX OS VERSION ON THE CLUSTER WILL BE UPGRADED TO THE LATEST 8.x VERSION AVAILABLE IRRESPECTIVE OF THE PROVISION_OS_VERSION PROVIDED IN PROVISION_CONFIG.YML.

provision_os_version

string

Required

OS version of provision_os to be installed.

Choices:

  • 8.0

  • 8.1

  • 8.2

  • 8.3

  • 8.4

  • 8.5

  • 8.6 <- default

  • 8.7

iso_file_path

string

Required

Path where user has placed the iso image that needs to be provisioned on target nodes. Accepted files are Rocky8-DVD or RHEL-8.x-DVD (full OS). iso_file_path should contain the provision_os and provision_os_version values in the filename.

Default values: "/home/RHEL-8.6.0-20220420.3-x86_64-dvd1.iso"

timezone

string

Required

Timezone to be used during OS provisioning. Available timezones are provided here.

Choices:

  • GMT <- default

  • EST

  • CET

  • MST

  • CST6CDT

  • PST8PDT

language

string

Required

Language to be used during OS provisioning.

Default values: en-US

default_lease_time

integer

Required

Default lease time for IPs assigned by DHCP. Range: 21600-86400

Default values: 86400

provision_password

string

Required

  • Password set for the root account of target nodes during provisioning.

  • Length >= 8 characters

  • Password must not contain -,, ‘,”

  • The first character of the string should be an alphabet.

postgresdb_password

string

Required

  • Password set for the postgresDB on target nodes during provisioning.

  • Length >= 8 characters

  • Password must not contain -,, ‘,”

  • The first character of the string should be an alphabet.

node_name

string

Required

  • Prefix for target node names, if dynamically allocated.

  • Hostname = node_name + ‘0000x’ + domain_name

  • Hostname <= 65 characters

  • Example: servernode00001.Omnia.test , where node_name =servernode, domain_name =Omnia.test , 00001 used by Omnia.

Default values: node

domain_name

string

Required

  • Domain name the user intends to configure on the cluster.

  • Hostname = node_name + ‘0000x’ + domain_name

  • Hostname <= 65 characters

  • Please provide a valid domain name according to the domain name standards.

  • Example: servernode00001.Omnia.test , where node_name=servernode, domain_name=Omnia.test , 00001 used by Omnia.

public_nic

string

Required

The nic/ethernet card that is connected to the public internet.

Default values: eno2

admin_nic

string

Required

Admin NIC of Control Plane. This is the shared LOM NIC.

Default values: eno1

admin_nic_subnet

string

Required

  • The subnet within which all Admin IPs are assigned.

  • The value of this variable cannot be changed after successfully running provision.yml.

Default values: 10.5.0.0

ib_nic_subnet

string

Optional

  • If provided, Omnia will handle and assign static IPs to cluster node’s IB network.

  • Only the last 16 bits/2 octets of IPv4 are dynamic

  • If provided, the DB entry will be in parallel with the pxe_subnet.

  • Example: If admin_ip: 10.5.0.50 and ib_nic_subnet: 10.10.0.0, then ib_ip: 10.10.0.50

bmc_nic_subnet

string

Required

  • If provided, Omnia will assign static IPs to BMC NICs on the cluster nodes within the provided subnet.

  • If network_interface_type: lom, mandatory for discovery_mechanism: mapping, switch_based and bmc.

  • If network_interface_type: dedicated, optional for discovery_mechanism: mapping, snmpwalk.

  • Note that since the last 16 bits/2 octets of IPv4 are dynamic, please ensure that the parameter value is set to x.x.0.0.

  • When the PXE range and BMC subnet are provided, corresponding NICs will be assigned IPs with the same 3rd and 4th octets.

  • The value of this variable cannot be changed after successfully running provision.yml.

bmc_username

string

Required

  • The username for iDRAC.

  • The username must not contain -,, ‘,”

bmc_password

string

Required

  • The password for iDRAC.

  • The password must not contain -,, ‘,”

  • The first character of the string should be an alphabet.

bmc_static_start_range

string

Required

  • The dhcp range for discovering the static IP assigned iDRACs within the given range.

  • For the range 10.3.0.50 to 10.3.4.100 then, bmc_static_start_range: 10.3.0.50, bmc_static_end_range: 10.3.4.100

  • To create a meaningful range of discovery, ensure that the last two octets of bmc_static_end_range are equal to or greater than the last two octets of the bmc_static_start_range. That is, for the range a.b.c.d - a.b.e.f, e and f should be greater than or equal to c and d.

  • The value of this variable cannot be changed after successfully running provision.yml.

  • Ex: 172.20.0.50 - 172.20.1.101 is a valid range however, 172.20.0.101 - 172.20.1.50 is not.

bmc_static_end_range

string

Required

  • The dhcp range for discovering the static IP assigned iDRACs within the given range.

  • For the range 10.3.0.50 to 10.3.4.100 then, bmc_static_start_range: 10.3.0.50, bmc_static_end_range: 10.3.4.100

  • To create a meaningful range of discovery, ensure that the last two octets of bmc_static_end_range are equal to or greater than the last two octets of the bmc_static_start_range. That is, for the range a.b.c.d - a.b.e.f, e and f should be greater than or equal to c and d.

  • The value of this variable cannot be changed after successfully running provision.yml.

  • Ex: 172.20.0.50 - 172.20.1.101 is a valid range however, 172.20.0.101 - 172.20.1.50 is not.

update_repos

boolean [1]

Required

  • Indicates whether provision.yml will update offline RHEL repos.

  • If update_repos: false, the update repos for BaseOS and AppStream will not be updated to the latest versions available.

  • If update_repos: true, the update repos for BaseOS and AppStream will be updated to the latest versions available.

Note

By default, AppSteam and BaseOS repos will be configured from the given ISO file.

Choices:

false <- Default

true

rhel_repo_alphabetical_folders boolean [1]

Required

  • Indicates whether the packages in local or subscription repos should be ordered in alphabetical directories.

  • This variable should be filled if control plane OS is RHEL and local RHEL repository is available.

Choices:

false <- Default

true

rhel_repo_local_path

JSON list

Optional

  • The repo path and names of the software repository to be configured on the cluster nodes.

  • Provide the repo data file path, which ends with .repo extension in repo_url parameter.

  • Provide a valid url for BaseOS and AppStream repositories. Omnia does not validate the repo_url provided. Invalid entries will cause provision.yml to fail.

  • This variable should be filled if control plane OS is RHEL and subscription is not activated.

  • This variable should be filled if the control plane OS is Rocky and the provision_os is rhel.

Sample value:

- { repo: “AppStream”, repo_url: “http://xx.yy.zz/pub/Distros/RedHat/RHEL8/8.8/RHEL-8-appstream.repo”, repo_name: “RHEL-8-appstream-partners” }

- { repo: “BaseOS”, repo_url: “http://xx.yy.zz/pub/Distros/RedHat/RHEL8/8.8/RHEL-8-baseos.repo”, repo_name: “RHEL-8-baseos-partners” }

primary_dns

string

Optional

  • The primary DNS host IP queried to provide Internet access to cluster Node (through DHCP routing).

  • Currently, the primary_dns value stored in input/provision_config.yml cannot be part of any of the subnets (admin_nic_subnet, ib_nic_subnet and bmc_nic_subnet) also defined in input/provision_config.yml.

Ex: If the primary_dns is set to 10.15.0.7, the subnet 10.15.0.0 cannot be used for admin_nic_subnet, ib_nic_subnet or bmc_nic_subnet.

secondary_dns

string

Optional

The secondary DNS host IP queried to provide Internet access to cluster Node (through DHCP routing)

disk_partition

JSON list

Optional

  • User defined disk partition applied to remote servers.

  • The disk partition desired_capacity has to be provided in MB.

  • Valid mount_point values accepted for disk partition are /home, /var, /tmp, /usr, swap.

  • Default partition size provided for /boot is 1024MB, /boot/efi is 256MB and the remaining space to / partition.

  • Values are accepted in the form of JSON list such as: , - { mount_point: “/home”, desired_capacity: “102400” }

Default values: - { mount_point: "", desired_capacity: "" }

mlnx_ofed_path

string

Optional

Absolute path to a local copy of the .iso file containing Mellanox OFED packages. The image can be downloaded from https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/. Sample value: /root/MLNX_OFED_LINUX-5.8-1.1.2.1-rhel8.6-x86_64.iso

cuda_toolkit_path

string

Optional

Absolute path to local copy of .rpm file containing CUDA packages. The cuda rpm can be downloaded from https://developer.nvidia.com/cuda-downloads. CUDA will be installed post provisioning without any user intervention. Eg: cuda_toolkit_path: “/root/cuda-repo-rhel8-12-0-local-12.0.0_525.60.13-1.x86_64.rpm”

apptainer_support

boolean [1]

Required

  • Indicates whether apptainer will be installed on the cluster to enable execution of HPC benchmarks in a containeraized environment.

  • If apptainer_support: false, apptainer will not be installed on the cluster.

  • If apptainer_support: true, apptainer will be installed on the cluster.

Note

The input/provision_config.yml file is encrypted on the first run of the provision tool:

To view the encrypted parameters:

ansible-vault view provision_config.yml --vault-password-file .provision_vault_key

To edit the encrypted parameters:

ansible-vault edit provision_config.yml --vault-password-file .provision_vault_key

Caution

The IP address 192.168.25.x is used for PowerVault Storage communications. Therefore, do not use this IP address for other configurations.

To continue to the next steps:

If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.