BeeGFS bolt on

BeeGFS is a hardware-independent POSIX parallel file system (a.k.a. Software-defined Parallel Storage) developed with a strong focus on performance and designed for ease of use, simple installation, and management.

../../_images/BeeGFS_Structure.jpg

Pre Requisites before installing BeeGFS client

  • Ensure that the BeeGFS server is set up using the linked steps.

  • Ensure that a connAuthFile is configured on the server as explained here

Caution

Configuring a connAuthFile is now mandatory. Services will no longer start if a connAuthFile is not configured

  • Ensure that the following ports are open for TCP and UDP connectivity:

    Port

    Service

    8008

    Management service (beegfs-mgmtd)

    8003

    Storage service (beegfs-storage)

    8004

    Client service (beegfs-client)

    8005

    Metadata service (beegfs-meta)

    8006

    Helper service (beegfs-helperd)

To open the ports required, use the following steps:

  1. firewall-cmd --permanent --zone=public --add-port=<port number>/tcp

  2. firewall-cmd --permanent --zone=public --add-port=<port number>/udp

  3. firewall-cmd --reload

  4. systemctl status firewalld

Note

BeeGFS services over RDMA is only supported on RHEL 8.3 and above due to limitations on BeeGFS. When setting up your cluster with RDMA support, check the BeeGFS documentation to provide appropriate values in input/storage_config.yml.

  • If the cluster runs Rocky, ensure that versions running are compatible by checking our support matrix.

Installing the BeeGFS client via Omnia

After the required parameters are filled in input/storage_config.yml, Omnia installs BeeGFS on all nodes while executing the storage.yml playbook.

Caution

Do not remove or comment any lines in the input/storage_config.yml file.

Parameters for storage

Variables

Details

nfs_client_params

JSON List

Required

  • This JSON list contains all parameters required to set up NFS.

  • For a bolt-on set up where there is a pre-existing NFS export, set nfs_server to false.

  • When nfs_server is set to true, an NFS share is created on the control plane for access by all cluster nodes.

  • For more information on the different kinds of configuration available, click here.
beegfs_rdma_support

boolean Optional

This variable is used if user has RDMA-capable network hardware (e.g., InfiniBand)

Choices:

  • false <- Default

  • true
beegfs_ofed_kernel_modules_path

string Optional

  • The path where separate OFED kernel modules are installed.

  • Ensure that the path provided here exists on all target nodes.

    Default value: "/usr/src/ofa_kernel/default/include"
beegfs_mgmt_server

string Required

BeeGFS management server IP.

Note

The provided IP should have an explicit BeeGFS management server running .
beegfs_mounts

string Optional
Beegfs-client file system mount location. If storage_yml is being used to change the BeeGFS mounts location, set beegfs_unmount_client to true.

Default value: “/mnt/beegfs”
beegfs_unmount_client

boolean Optional

Changing this value to true will unmount running instance of BeeGFS client and should only be used when decommisioning BeeGFS, changing the mount location or changing the BeeGFS version.

Choices:

  • false <- Default

  • true
beegfs_version_change

boolean Optional

Use this variable to change the BeeGFS version on the target nodes.

Choices:

  • false <- Default

  • true

ansible_config_file_path

string

Required

  • Path to directory hosting ansible config file (ansible.cfg file)

  • This directory is on the host running ansible, if ansible is installed using dnf

  • If ansible is installed using pip, this path should be set

    Default values: /etc/ansible
beegfs_secret_storage_filepath

string Required

  • The filepath (including the filename) where the connauthfile is placed.

  • Required for Beegfs version >= 7.2.7

    Default values: /home/connauthfile

Note

  • BeeGFS client-server communication can take place over TCP or RDMA. If RDMA support is required, set beegfs_rdma_support should be set to true. Also, OFED should be installed on all cluster nodes.

  • For BeeGFS communication happening over RDMA, the beegfs_mgmt_server should be provided with the Infiniband IP of the management server.

  • The parameter inventory refers to the inventory file listing all relevant nodes.)

If input/storage_config.yml is populated before running omnia.yml, BeeGFS client will be set up during the run of omnia.yml.

If omnia.yml is not leveraged to set up BeeGFS, run the storage.yml playbook :

cd storage
ansible-playbook storage.yml -i inventory

Note

Once BeeGFS is successfully set up, set enable_omnia_nfs (input/omnia_config.yml) to false and omnia_usrhome_share (input/omnia_config.yml) to an accessible share path in BeeGFS to use the path across the cluster for deployments.

If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.