2017-07-09

[TOC]

什么是OpenHPC

OpenHPC是一个聚合了众多常见部署应用和管理的系统框架.简单来说就是,集群中想要部署什么应用只需要一次制作好,集群中各个节点运行相同的image,完全统一,适合做高性能计算集群的搭建 .官方原话如下:

Welcome to the OpenHPC site. OpenHPC is a collaborative, community effort that initiated from a desire to aggregate a number of common ingredients required to deploy and manage High Performance Computing (HPC) Linux clusters including provisioning tools, resource management, I/O clients, development tools, and a variety of scientific libraries. Packages provided by OpenHPC have been pre-built with HPC integration in mind with a goal to provide re-usable building blocks for the HPC community.

我这是在CentOS7上面搭建的,官方给出了相应的安装文档,不过里面东西很多,没告诉你为什么这么做,照着做也会出很多问题.下面我就结合自己制作的操作过程记录如下:

1.master配置

a.master节点配置本地yum源

首先创建目录存放节点所需文件,以及配置好yum源,我在集群制作了一个本地yum源,方便快速下载安装.

mkdir -pv /atlas/os_images/compute_node_v0.2.0
mkdir  /atlas/os_images/tftpboot_v0.2.0

cat /etc/yum.repos.d/base.repo 
    [development]
    name=development
    baseurl=ftp://172.16.10.10/centos7.2
    gpgcheck=0
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

    [epel]
    name=epel
    baseurl=ftp://172.16.10.10/epel
    gpgcheck=0

    [local-slurm]
    name=slurm
    baseurl=ftp://172.16.10.10/slurm
    gpgcheck=0

yum clean all
yum repolist

b.配置master的openhpc服务

在master上安装所需软件

yum -y groupinstall ohpc-base
yum -y groupinstall ohpc-warewulf
yum -y groupinstall "InfiniBand Support"
yum -y install infinipath-psm

systemctl enable ntpd.service  #节点与master之间通信,时间准确度要求比较高.
systemctl restart ntpd
systemctl start rdma
systemctl status rdma

配置pxe启动目录,以及所使用的网卡.最后启动pxe服务

vim /etc/warewulf/provision.conf
    network device = enp129s0f0  #master上面网卡名
    tftpdir = /atlas/os_images/tftpboot_v0.2.0
vim /etc/xinetd.d/tftp
    server_args   = -s /atlas/os_images/tftpboot_v0.2.0
    disable     = no
systemctl restart xinetd
    systemctl start tftp.socket
    systemctl start tftp.service

系统启动后会针对系统完成一些配置,而这些配置是通过http从master服务器上面down下来的.down下来的文件放在 /warewulf 目录下

export MODFILE=/etc/httpd/conf.d/warewulf-httpd.conf
perl -pi -e "s/cgi-bin>\$/cgi-bin>\n Require all granted/" $MODFILE
perl -pi -e "s/Allow from all/Require all granted/" $MODFILE
perl -ni -e "print unless /^\s+Order allow,deny/" $MODFILE


cat /etc/httpd/conf.d/warewulf-httpd.conf
          LoadModule perl_module modules/mod_perl.so
          PerlSwitches -w
          PerlSwitches -T
          PerlSwitches -I/var/www/stage/cgi-bin

          # This is disabled as RHEL6 perl_mod seems to be missing this support
          #PerlPreConnectionHandler Apache2::Reload

          Alias /WW/static /usr/share/warewulf/www

          ScriptAlias /WW/file /usr/libexec/warewulf/cgi-bin/file.pl
          ScriptAlias /WW/script /usr/libexec/warewulf/cgi-bin/script.pl
          ScriptAlias /WW/nodeconfig /usr/libexec/warewulf/cgi-bin/nodeconfig.pl
          ScriptAlias /WW/vnfs /usr/libexec/warewulf/cgi-bin/vnfs.pl

          <Directory /usr/libexec/warewulf/cgi-bin>
           Require all granted
              SetHandler perl-script
              PerlResponseHandler ModPerl::Registry
              PerlOptions +ParseHeaders
              Options +ExecCGI
          </Directory>

          <Directory /usr/share/warewulf/www>
              Options Indexes MultiViews
              AllowOverride None
              Require all granted
          </Directory>

集群针对不同节点的mac地址设置不同的IP,主机名等信息都是保存在数据库中,因此要用到mariadb

systemctl enable mariadb.service
systemctl restart mariadb
systemctl enable httpd.service
systemctl restart httpd

2.制作node节点所需配置

首先要制作的是node节点的系统文件,然后再安装所需要的应用

安装系统文件

vim  /usr/libexec/warewulf/wwmkchroot/centos-7.tmpl
    YUM_MIRROR="ftp://172.16.10.10/centos7.2"  #这里是启动最小系统所必须用到的软件安装源
#如果不修改这个,系统会从官方源去下载,由于我自己做这个要反复重新安装测试.为了快速所以这样设置
#如果要在线下载,由于centos系统已经到7.3了,上面原有了路径要自己更新

export CHROOT=/atlas/os_images/compute_node_v0.2.0
wwmkchroot centos-7 $CHROOT  #这一步就是将系统文件安装到$CHROOT中

cp -p /etc/resolv.conf $CHROOT/etc/resolv.conf

cd /atlas/os_images/compute_node_v0.2.0/etc/yum.repos.d/
    rm -f CentOS-*
    cat base.repo
    cat OpenHPC.repo.bak
#这里我同样把节点安装软件的源改到了本地,方便快捷(包括openhpc的也已经下载到了本地)

接下来就安装节点所需应用了

安装常用基础应用

yum -y --installroot=$CHROOT install ntp kernel gcc make grub2-tools environment-modules
yum -y --installroot=$CHROOT groupinstall "InfiniBand Support"
yum -y --installroot=$CHROOT install infinipath-psm
chroot $CHROOT systemctl enable rdma
chroot $CHROOT systemctl enable ntpd
echo "server 172.16.10.10" >> $CHROOT/etc/ntp.conf  #这个IP是对应master的IP,同步时间用

安装munge

配置安装munge,为后面slurm安装做好基础准备,这里注意几点,muster和node的munge和slurm用户名的id号码要一样

export MUNGEUSER=1050
groupadd -g $MUNGEUSER munge
useradd  -m -c "MUNGE Uid 'N' Gid Emporium" -d /var/lib/munge -u $MUNGEUSER -g munge  -s /sbin/nologin munge
export SlurmUSER=1051
groupadd -g $SlurmUSER slurm
useradd  -m -c "Slurm workload manager" -d /var/lib/slurm -u $SlurmUSER -g slurm  -s /bin/bash slurm
cat /etc/passwd |grep munge
cat /etc/passwd |grep slurm
cat /etc/group |grep slurm
cat /etc/group |grep munge
yum install munge munge-libs munge-devel  -y
dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key
chown munge: /etc/munge/munge.key
chmod 400 /etc/munge/munge.key
systemctl enable munge
systemctl start  munge
chroot $CHROOT groupadd -g 1050 munge
chroot $CHROOT useradd  -m -c "MUNGE Uid 'N' Gid Emporium" -d /var/lib/munge -u 1050 -g munge  -s /sbin/nologin munge
chroot $CHROOT groupadd -g 1051 slurm
chroot $CHROOT useradd  -m -c "Slurm workload manager" -d /var/lib/slurm -u 1051 -g slurm  -s /bin/bash slurm
yum install --installroot=$CHROOT munge munge-libs munge-devel -y
cp -a /etc/munge/munge.key  $CHROOT/etc/munge/munge.key
chroot $CHROOT systemctl enable munge
chroot $CHROOT chown -R munge: /etc/munge/ /var/log/munge/
chroot $CHROOT chmod 0700 /etc/munge/ /var/log/munge/
vim /etc/warewulf/vnfs.conf  #这里注意一下,好多服务的日志都会记录在/var/log目录下
    #exclude += /var/log/*   #这个exclude 就是制作node镜像的时候那些目录不要做进去,很明显我们需要这个目录,所以要屏蔽掉他.不然munge服务会启动不成功.
    #exclude += /usr/src     #这个在后面nvidia显卡安装驱动的时候需要用到dkms
    #hybridize += /usr/lib/locale  #字符集编码格式需要用到该文件目录

安装slurm

tar xf slurm.tar
cd rpmbuild/RPMS/x86_64/
yum install ./slurm-*
yum  --installroot=$CHROOT install slurm-devel slurm-contribs slurm-munge slurm-openlava slurm-pam_slurm slurm-perlapi slurm-plugins slurm-slurmdbd slurm-sql slurm-torque -y

vim /etc/slurm/slurm.conf                             #slurm的配置文档

cp -a /etc/slurm/slurm.conf /atlas/os_images/compute_node_v0.2.0/etc/slurm/slurm.conf
chown slurm:slurm /var/spool
mkdir /var/spool/slurmctld
chown slurm: /var/spool/slurmctld
chmod 755 /var/spool/slurmctld
touch /var/log/slurmctld.log
chown slurm: /var/log/slurmctld.log
chroot /atlas/os_images/compute_node_v0.2.0/
    mkdir /var/spool/slurmd
    chown slurm: /var/spool/slurmd
    chmod 755 /var/spool/slurmd
    touch /var/log/slurmd.log
    chown slurm: /var/log/slurmd.log
chroot $CHROOT systemctl enable slurmd.service


[yhu@master etc]$sudo chmod +x rc.local
[yhu@master etc]$cat rc.local 
...
nvidia-smi >> /dev/null
systemctl restart slurmd
# 调用gpu的时候没有设备文件/dev/nvidia0,通过openhpc启动起来的系统的确没有这个设备文件,所以我在 /etc.rc.local 文件中加入两行代码,创建出来这个设备文件.并重新启动slurmd服务

singularity

yum -y --installroot=$CHROOT install gcc make
cp singularity-2.2.1.tar.gz /atlas/os_images/compute_node_v0.2.0/tmp/
chroot /atlas/os_images/compute_node_v0.2.0
        cd /tmp
        tar xf singularity-2.2.1.tar.gz 
        cd singularity-2.2.1 
        ./configure 
        make 
        make install 
        cd ..
        rm -rf singularity-2.2.1*

environment-modules

yum -y --installroot=$CHROOT install environment-modules
chroot $CHROOT vi /usr/share/Modules/init/.modulespath
/atlas/gensoft/public_modules  #只留这一个其他都屏蔽掉

nvidia显卡驱动,cuda,cudnn

chroot /atlas/os_images/compute_node_v0.2.0/
# nvidia-kmod安装之后，并不会立即替换掉so动态加载库，所以需要删除原来的，加载新的。这样就不需要重启了。
yum remove nvidia-kmod
yum install nvidia-kmod xorg-x11-drv-nvidia*
rm -rf /var/lib/dkms/nvidia/375.39/
rm -rf /usr/src/nvidia-375.26
rm /usr/lib64/nvidia/*375.26
cp /var/lib/dkms/nvidia/387.26/3.10.0-327.el7.x86_64/x86_64/module/nvidia*  /lib/modules/3.10.0-327.el7.x86_64/extra/
yum reinstall nvidia-kmod xorg-x11-drv-nvidia*
yum clean all

配置信息保存数据库

cat ~/.ssh/cluster.pub >> $CHROOT/root/.ssh/authorized_keys #把秘钥保存到node节点方便直接远程
wwinit ssh_keys  
wwsh file import /etc/passwd        #
wwsh file import /etc/group         #--------------------------
wwsh file import /etc/shadow        #--------------------------

export WW_CONF=/etc/warewulf/bootstrap.conf
echo "drivers += updates/kernel/" >> $WW_CONF
wwbootstrap `uname -r`            #制作bootstrap
echo "GATEWAYDEV=eth0" > /tmp/network.$$
wwsh -y file import /tmp/network.$$ --name network
wwsh -y file set network --path /etc/sysconfig/network --mode=0644 --uid=0
wwvnfs -y --chroot $CHROOT         #将做好的node启动目录制作成vnfs镜像文件
#配置每个node的IP等
wwsh -y node new node5 --ipaddr=172.16.10.15 --hwaddr=0c:c4:7a:85:18:da --network=255.255.255.0 --gateway=172.16.10.1 -D eth0
wwsh -y node new node7 --ipaddr=172.16.10.17 --hwaddr=0c:c4:7a:82:c5:d8 --network=255.255.255.0 --gateway=172.16.10.1 -D eth0
#----------wwsh -y node delete node7 写错了可以删除,重新添加
#系统会根据这些配置,生成一个dhcp.conf文件 在pxe启动时分配IP等信息

#下面这个是系统启动后,要为每个node配置修改哪些file
wwsh -y provision set "node5" --vnfs=compute_node_v0.2.0 --bootstrap=`uname -r` --files=dynamic_hosts,passwd,group,shadow,network         #--------------------------
wwsh -y provision set "node7" --vnfs=compute_node_v0.2.0 --bootstrap=`uname -r` --files=dynamic_hosts,passwd,group,shadow,network         #--------------------------

3.启动服务

systemctl restart dhcpd
wwsh pxe update
systemctl enable slurmctld.service
systemctl start slurmctld.service
systemctl status slurmctld.service

4.重启node节点

这里我直接操作ipmi重启

ipmitool -I lanplus -H 172.16.10.107 -U ADMIN -P ADMIN chassis bootdev pxe options=persistent
ipmitool -I lanplus -H 172.16.10.107 -U ADMIN -P ADMIN chassis power reset
ipmitool -I lanplus -H 172.16.10.105 -U ADMIN -P ADMIN chassis bootdev pxe options=persistent
ipmitool -I lanplus -H 172.16.10.105 -U ADMIN -P ADMIN chassis power reset

启动后简单的检测

这里我直接使用slurm命令检测

sinfo
    PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
    test*        up   infinite      1  down* node5

scontrol 
    scontrol: update nodename=node5 state=resume
    scontrol: exit

sinfo 
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
test*        up   infinite      1   idle node5

#slurm

5.不同node节点配置

wwvnfs -y --chroot /atlas/os_images/compute_node_v0.2.2 -o /atlas/os_images/vnfs/compute_node_v0.2.2.vnfs

wwsh vnfs import /atlas/os_images/vnfs/compute_node_v0.2.2.vnfs --name=compute_node_v0.2.2  --chroot=/atlas/os_images/compute_node_v0.2.2 -y 

wwsh provision set node1 -V compute_node_v0.2.2 -y 
wwsh provision set node2 -V compute_node_v0.2.2 -y 
wwsh provision set node3 -V compute_node_v0.2.2 -y 
wwsh provision set node4 -V compute_node_v0.2.2 -y 
wwsh provision set node5 -V compute_node_v0.2.2 -y 
wwsh provision set node6 -V compute_node_v0.2.2 -y 
wwsh provision set node7 -V compute_node_v0.2.2 -y

wwsh是warewulf的命令,可以直接输入wwsh进入交互.功能主要是,设置node节点启动的时候,pxe的启动设置,网络以及主机名,还有每个节点应该有哪些配置文件,等等.

#wwsh 
Warewulf> help 
Warewulf command line shell interface

Welcome to the Warewulf shell interface. This application allows you
to interact with the Warewulf backend database and modules via a
single interface.

  bootstrap        Manage your bootstrap images
  dhcp             Manage DHCP service and configuration
  events           Control how events are handled
  exit             Exit/leave the Warewulf shell
  file             Manage files within the Warewulf data store
  node             Node manipulation commands
  object           Generically manipulate all Warewulf data store entries
  output           Set the output verbosity level
  provision        Node provision manipulation commands
  pxe              Manage PXE configuration
  quit             Exit/leave the Warewulf shell
  ssh              Spawn parallel ssh connections to nodes.
  vnfs             Manage your VNFS images

Warewulf>

hooby's blog

纸上得来终觉浅，绝知此事要躬行!

openhpc