问：

我正在尝试将显卡添加到 slurm 节点。目前，我有两个 p104 和一个 1080ti。我在设置 slurm.conf 和 gres.conf 时遇到问题。

我想为每个显卡分配一种类型

Slurm.conf （英语）

SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU
GresTypes=gpu
NodeName=tower CPUs=52 Boards=1 SocketsPerBoard=2 CoresPerSocket=26 ThreadsPerCore=1 RealMemory=257389 Features=gpu Gres=gpu:1080_ti:1,gpu:p104:2

gres.conf

AutoDetect=off
#NodeName=amd01 Name=gpu File=/dev/nvidia0 Cores=24-27
#NodeName=amd01 Name=gpu Type=1080_ti File=/dev/nvidia1 Cores=24-27
NodeName=tower Name=gpu Type=p104 File=/dev/nvidia0 Cores=0-25
NodeName=tower Name=gpu Type=p104 File=/dev/nvidia1 Cores=0-25
NodeName=tower Name=gpu Type=1080_ti File=/dev/nvidia2 Cores=26-51

这是slurmd.log

[2023-11-06T16:28:32.162] debug:  Log file re-opened
[2023-11-06T16:28:32.166] debug:  cgroup/v1: init: Cgroup v1 plugin loaded
[2023-11-06T16:28:32.171] GRES: Global AutoDetect=off(4)
[2023-11-06T16:28:32.171] debug:  gres/gpu: init: loaded
[2023-11-06T16:28:32.171] debug:  gpu/generic: init: init: GPU Generic plugin loaded
[2023-11-06T16:28:32.171] GRES: _set_gres_device_desc : /dev/nvidia2 major 195, minor 2
[2023-11-06T16:28:32.171] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, minor 0
[2023-11-06T16:28:32.171] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, minor 1
[2023-11-06T16:28:32.171] GRES: gpu device number 2(/dev/nvidia2):c 195:2 rwm
[2023-11-06T16:28:32.171] GRES: gpu device number 0(/dev/nvidia0):c 195:0 rwm
[2023-11-06T16:28:32.171] GRES: gpu device number 1(/dev/nvidia1):c 195:1 rwm
[2023-11-06T16:28:32.171] Gres Name=gpu Type=1080_ti Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=26-51 CoreCnt=52 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
[2023-11-06T16:28:32.171] Gres Name=gpu Type=p104 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-25 CoreCnt=52 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
[2023-11-06T16:28:32.171] Gres Name=gpu Type=p104 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-25 CoreCnt=52 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
[2023-11-06T16:28:32.172] topology/none: init: topology NONE plugin loaded
[2023-11-06T16:28:32.172] route/default: init: route default plugin loaded
[2023-11-06T16:28:32.172] debug2: Gathering cpu frequency information for 52 cpus
[2023-11-06T16:28:32.176] debug:  Resource spec: No specialized cores configured by default on this node
[2023-11-06T16:28:32.176] debug:  Resource spec: Reserved system memory limit not configured for this node
[2023-11-06T16:28:32.176] debug:  task/cgroup: init: Tasks containment cgroup plugin loaded
[2023-11-06T16:28:32.177] task/affinity: init: task affinity plugin loaded with CPU mask 0xfffffffffffff
[2023-11-06T16:28:32.177] debug:  spank: opening plugin stack /etc/slurm/plugstack.conf
[2023-11-06T16:28:32.178] debug:  /etc/slurm/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf"
[2023-11-06T16:28:32.179] cred/munge: init: Munge credential signature plugin loaded
[2023-11-06T16:28:32.179] slurmd version 23.02.2 started
[2023-11-06T16:28:32.179] debug:  acct_gather_energy/none: init: AcctGatherEnergy NONE plugin loaded
[2023-11-06T16:28:32.179] debug:  acct_gather_profile/none: init: AcctGatherProfile NONE plugin loaded
[2023-11-06T16:28:32.179] debug:  acct_gather_interconnect/none: init: AcctGatherInterconnect NONE plugin loaded
[2023-11-06T16:28:32.179] debug:  acct_gather_filesystem/none: init: AcctGatherFilesystem NONE plugin loaded
[2023-11-06T16:28:32.179] debug2: No acct_gather.conf file (/etc/slurm/acct_gather.conf)
[2023-11-06T16:28:32.180] debug:  jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded
[2023-11-06T16:28:32.180] debug:  job_container/none: init: job_container none plugin loaded
[2023-11-06T16:28:32.181] debug:  switch/none: init: switch NONE plugin loaded
[2023-11-06T16:28:32.181] debug:  switch Cray/Aries plugin loaded.
[2023-11-06T16:28:32.181] debug:  MPI: Loading all types
[2023-11-06T16:28:32.182] debug2: No mpi.conf file (/etc/slurm/mpi.conf)
[2023-11-06T16:28:32.183] slurmd started on Mon, 06 Nov 2023 16:28:32 +0800
[2023-11-06T16:28:32.183] CPUs=52 Boards=1 Sockets=2 Cores=26 Threads=1 Memory=257390 TmpDisk=459103 Uptime=1832 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
[2023-11-06T16:28:32.200] debug:  _handle_node_reg_resp: slurmctld sent back 11 TRES.

似乎没什么不对劲

但节点信息是：

$ scontrol show node tower
NodeName=tower Arch=x86_64 CoresPerSocket=26
   CPUAlloc=0 CPUEfctv=52 CPUTot=52 CPULoad=0.03
   AvailableFeatures=gpu
   ActiveFeatures=gpu
   Gres=gpu:3(S:0-1)
   NodeAddr=tower NodeHostName=tower Version=23.02.2
   OS=Linux 3.10.0-1160.92.1.el7.x86_64 #1 SMP Tue Jun 20 11:48:01 UTC 2023
   RealMemory=257389 AllocMem=0 FreeMem=252880 Sockets=2 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=fast
   BootTime=2023-11-06T15:58:09 SlurmdStartTime=2023-11-06T16:28:32
   LastBusyTime=2023-11-06T16:28:37 ResumeAfterTime=None
   CfgTRES=cpu=52,mem=257389M,billing=52,gres/gpu=3
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 AveWatts=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

为什么没有 tpye 节目？Gres=gpu:3(S:0-1)

当我提交作业时，我收到错误

# root @ controller in ~ [16:34:38]
$ srun -p fast -w tower --gres=gpu:1080ti:1 hostname
srun: error: Unable to allocate resources: Requested node configuration is not available
# root @ controller in ~ [16:34:47] C:1
$ srun -p fast -w tower --gres=gpu:1080_ti:1 hostname
srun: error: Unable to allocate resources: Requested node configuration is not available
# root @ controller in ~ [16:34:50] C:1
$ srun -p fast -w tower --gres=gpu:p104:1 hostname
srun: error: Unable to allocate resources: Requested node configuration is not available
# root @ controller in ~ [16:35:13] C:1
$ srun -p fast -w tower --gres=gpu:1 hostname
tower
# root @ controller in ~ [16:35:18]
$ srun -p fast -w tower --gres=gpu:2 hostname
tower

如果 slurm.conf 设置如下，则没有区别

NodeName=tower CPUs=52 Boards=1 SocketsPerBoard=2 CoresPerSocket=26 ThreadsPerCore=1 RealMemory=257389 Features=gpu Gres=gpu:3

另一个问题，当我使用一个 gpu 提交多个任务时，我可以使用 htop 命令查看只有一个 CPU（通常是第一个 cpu）以 100% 的速度工作。如何让不同的GPU任务使用不同的CPU，以及如何设置CPU亲和力。