批处理 – PBS作业相互依赖:一个作业开始,取消其他作业
发布时间:2020-12-14 05:02:24 所属栏目:百科 来源:网络整理
导读:我想向我的群集上的几个队列提交模拟.只要一个队列启动它,它就会被其他队列取消.我知道它可能是不明确的,因为几个工作可能会在几个队列上同时开始. 监视队列的bash脚本可能会这样做.提交作业时是否可以直接使用qsub? 编辑:下面是一个使用bash脚本的工作示
我想向我的群集上的几个队列提交模拟.只要一个队列启动它,它就会被其他队列取消.我知道它可能是不明确的,因为几个工作可能会在几个队列上同时开始.
监视队列的bash脚本可能会这样做.提交作业时是否可以直接使用qsub? 编辑:下面是一个使用bash脚本的工作示例.这可能不是最佳的,因为它需要(慢)磁盘访问. #!/bin/bash - # # Exit in case of error set -e # # Command-line argument is the name of the shared file fid=$* if [ -f ${HOME}/.dep_jobs/${fid} ]; then echo "Given name already used,abort." exit 1 else echo "Initialize case." touch ${HOME}/.dep_jobs/${fid} fi # # Submit master job and retrieve the ID echo "Submitting master job" MID=$(qsub -l select=1:ncpus=1:mpiprocs=1 -q queue1 run.pbs) echo ${MID##* } # # Add the ID to the shared file ln -s ${HOME}/.dep_jobs/${fid} ${HOME}/.dep_jobs/${MID##* } echo "M ${MID##* }" >> ${HOME}/.dep_jobs/${fid} # # Submit slave job and retrieve the ID echo "Submitting slave job" SID=$(qsub -l select=1:ncpus=1:mpiprocs=1 -q queue2 run.pbs) echo ${SID##* } # # Add the ID to the shared file ln -s ${HOME}/.dep_jobs/${fid} ${HOME}/.dep_jobs/${SID##* } echo "S ${SID##* }" >> ${HOME}/.dep_jobs/${fid} # # Terminus,finalize case echo "Finalize case" echo "OK" >> ${HOME}/.dep_jobs/${fid} 提交的PBS脚本应该如下所示 #!/bin/bash #PBS -S /bin/bash #PBS -N Parallel # # Define shared file shared_file=${HOME}/.dep_jobs/${PBS_JOBID} # # Read it until it finishes with "OK" while [[ "$(more ${shared_file} | tail -n1)" != "OK" ]]; do sleep 1 done # # Read master and slave job id while read -r line do key=$(echo ${line} | awk '{print $1}') if [ "$key" = "M" ]; then MID=$(echo ${line} | awk '{print $2}') elif [ "$key" = "S" ]; then SID=$(echo ${line} | awk '{print $2}') fi done < ${shared_file} # # Current job is master or slave? if [ ${PBS_JOBID} = ${MID} ]; then key="M" other="${SID}" else key="S" other="${MID}" fi # # Check the status of the other job status="$(qstat ${other} | tail -n1 | awk '{print $5}')" # # I am running,if the other is in queue,qdel it if [ "${status}" = "Q" ]; then $(qdel ${other}) # If the other is running,we have race and only master survives elif [ "${status}" = "R" ]; then if [ "${key}" = "M" ]; then $(qdel ${other}) else exit fi else echo "We should not be here" exit fi # # The simulation goes here 解决方法
这是一个使用SGE调度程序运行的脚本.对于PBS调度程序,您需要进行一些最小的更改,例如使用
#PBS而不是#$并将$JOB_ID更改为$PBS_JOBID. 同样对于SGE调度程序,更好的方法是运行 qstat -u user_name -sp命令只列出挂起的作业,但是我找不到PBS调度程序的类似选项,所以假设它不存在,一种方法可能是使用以下脚本进行模拟作业(你没有需要任何主脚本): #!/bin/bash #$-N myjobName #$-q queueName #... some other options if needed # get the list of all running jobs myjobs="$(qstat -u username | cut -d " " -f1 | tail -n +3| tr 'n' ' ' )" # from the above list remove the current job (use PBS_JOBID for PBS scheduler) deljobs="$(echo "${myjobs/$JOB_ID/}")" echo "List of all jobs: $myjobs" echo "List of jobs to delete: $deljobs" #delete all other jobs qdel $deljobs #run the desired commands/programs date 您需要在用户名的qstat命令中更改上述脚本的用户名. 以下是我在脚本中使用的命令的一些简要说明: qstat -u username # check all running jobs cut -d " " -f1 # extract JOBID for each job from the previous output (first column) tail -n +3 # skip first 2 lines in the above output tr 'n' ' ' # change new line character on space echo "${myjobs/$JOB_ID/}" # from the string contained in $myjobs variable remove $JOB_ID (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |