I suppose it's a pretty trivial question but nevertheless, I'm looking for the (sacct I guess) command that will display the CPU time and memory used by a slurm job ID.
问题:
回答1:
You're right that the sacct command is what you're looking for. The --format switch is the other key element. If you run this command:
sacct -e
you'll get a printout of the different fields that can be used for the --format switch. The details of each field are described in the Job Account Fields section of the man page. For CPU time and memory, CPUTime and MaxRSS are probably what you're looking for. cputimeraw can also be used if you want the number in seconds, as opposed to the usual Slurm time format.
sacct --format="CPUTime,MaxRSS"
回答2:
sacct
is indeed the command to use for finished jobs. For running jobs, you can look at the sstat
command.
回答3:
@aaron.kizmiller is right, sacct
is the command to use.
One can fetch all of the following fields by passing them into saact --format="field,field"
Fields:
Account AdminComment AllocCPUS AllocGRES
AllocNodes AllocTRES AssocID AveCPU
AveCPUFreq AveDiskRead AveDiskWrite AvePages
AveRSS AveVMSize BlockID Cluster
Comment ConsumedEnergy ConsumedEnergyRaw CPUTime
CPUTimeRAW DerivedExitCode Elapsed ElapsedRaw
Eligible End ExitCode GID
Group JobID JobIDRaw JobName
Layout MaxDiskRead MaxDiskReadNode MaxDiskReadTask
MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask MaxPages
MaxPagesNode MaxPagesTask MaxRSS MaxRSSNode
MaxRSSTask MaxVMSize MaxVMSizeNode MaxVMSizeTask
McsLabel MinCPU MinCPUNode MinCPUTask
NCPUS NNodes NodeList NTasks
Priority Partition QOS QOSRAW
ReqCPUFreq ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov
ReqCPUS ReqGRES ReqMem ReqNodes
ReqTRES Reservation ReservationId Reserved
ResvCPU ResvCPURAW Start State
Submit Suspended SystemCPU Timelimit
TotalCPU UID User UserCPU
WCKey WCKeyID WorkDir
For example, to list all job ids, elapsed time, and max VM size, you can run:
sacct --format='JobID,Elapsed,MaxVMSize'
回答4:
The other answers all detail formats for output of sacct
, which is great for looking at multiple jobs aggregated in a table.
However, sometimes you want to look at a specific job in more detail, so you can tell whether your job efficiently used the allocated resources. For that, seff
is very useful. The syntax is simply seff <Jobid>
. For example, here's a recent job of mine (that failed):
$ seff 15780625
Job ID: 15780625
Cluster: mycluster
User/Group: myuser/mygroup
State: OUT_OF_MEMORY (exit code 0)
Nodes: 1
Cores per node: 16
CPU Utilized: 12:06:01
CPU Efficiency: 85.35% of 14:10:40 core-walltime
Job Wall-clock time: 00:53:10
Memory Utilized: 1.41 GB
Memory Efficiency: 70.47% of 2.00 GB
Note that the key CPU metric, CPU Utilized, corresponds to the TotalCPU field from sacct
, while Memory Utilized corresponds to MaxRSS.