c – 简单的MPI_Send和Recv使用CUDA给出分段错误(11)和无效权限(

我正在尝试使用CIDA代码进行格子boltzmann建模,并且遇到了MPI_Send和MPI_Recv函数的令人沮丧的问题.我已经验证了我有一个CUDA感知MPI带有一些简单的设备缓冲区到设备缓冲区MPI发送/ recv代码,所以我可以在GPU设备内存之间发送和recv数组,而无需通过CPU /主机.

我的代码用于3D晶格,它在各个节点之间沿z方向分开,Halos在节点之间传递,以确保流体可以在这些分区之间流动. Halos在GPU上.下面的代码是一个简化和编译,给出与我的主代码相同的错误.这里,Rank 0节点上的GPU Halo是MPI_Send()到1级节点,MPI_Recv()就是它.我的问题现在看起来很简单,我无法获得MPI_Send和MPI_Recv调用功能！代码没有进展到“//代码没有到达这里”.行,让我得出结论MPI_etc()调用不起作用.

我的代码基本上如下,删除了大部分代码,但仍然可以编译同样的错误：

#include <mpi.h>
using namespace std; 

    //In declarations:
    const int DIM_X = 30;
    const int DIM_Y = 50;
    const int Q=19;
    const int NumberDevices = 1;
    const int NumberNodes = 2;

    __host__        int SendRecvID(int UpDown,int rank,int Cookie) {int a =(UpDown*NumberNodes*NumberDevices) + (rank*NumberDevices) + Cookie; return a;} //Use as downwards memTrnsfr==0,upwards==1

    int main(int argc,char *argv[])
    {
       //MPI functions (copied from online tutorial somewhere)
       int numprocessors,rank,namelen;
       char processor_name[MPI_MAX_PROCESSOR_NAME];

       MPI_Init(&argc,&argv);
       MPI_Comm_size(MPI_COMM_WORLD,&numprocessors);
       MPI_Comm_rank(MPI_COMM_WORLD,&rank);
       MPI_Get_processor_name(processor_name,&namelen);

       /* ...code for splitting other arrays removed... */

       size_t size_Halo_z   = Q*DIM_X*DIM_Y*sizeof(double);  //Size variable used in cudaMalloc and cudaMemcpy.
       int NumDataPts_f_halo    = DIM_X*DIM_Y*Q;                 //Number of data points used in MPI_Send/Recv calls.
       MPI_Status status;                                        //Used in MPI_Recv.

       //Creating arrays for GPU data below,using arrays of pointers:
       double   *Device_HaloUp_Take[NumberDevices];              //Arrays on the GPU which will be the Halos.
       double   *Device_HaloDown_Take[NumberDevices];            //Arrays on the GPU which will be the Halos.
       double   *Device_HaloUp_Give[NumberDevices];              //Arrays on the GPU which will be the Halos.
       double   *Device_HaloDown_Give[NumberDevices];            //Arrays on the GPU which will be the Halos.

       for(int dev_i=0; dev_i<NumberDevices; dev_i++)   //Initialising the GPU arrays:
       {
          cudaSetDevice(dev_i);

          cudaMalloc( (void**)&Device_HaloUp_Take[dev_i],size_Halo_z);
          cudaMalloc( (void**)&Device_HaloDown_Take[dev_i],size_Halo_z);
          cudaMalloc( (void**)&Device_HaloUp_Give[dev_i],size_Halo_z);
          cudaMalloc( (void**)&Device_HaloDown_Give[dev_i],size_Halo_z);
       }

       int Cookie=0;             //Counter used to count the devices below.
       for(int n=1;n<=100;n++)   //Each loop iteration is one timestep.
       {    
       /* Run computation on GPUs */


          cudaThreadSynchronize();

          if(rank==0)   //Rank 0 node makes the first MPI_Send().
          {
             for(Cookie=0; Cookie<NumberDevices; Cookie++)
             {
                if(NumberDevices==1)            //For single GPU codes (which for now is what I am stuck on):
                {
                   cout << endl << "Testing X " << rank << endl;
                   MPI_Send(Device_HaloUp_Take[Cookie],NumDataPts_f_halo,MPI_DOUBLE,(rank+1),SendRecvID(1,Cookie),MPI_COMM_WORLD);
                   cout << endl << "Testing Y " << rank << endl;   //CODE DOES NOT REACH HERE.
                   MPI_Recv(Device_HaloUp_Give[Cookie],SendRecvID(0,rank+1,0),MPI_COMM_WORLD,&status);     
                   /*etc */
                }
             }

          }
          else if(rank==(NumberNodes-1))
          {
             for(Cookie=0; Cookie<NumberDevices; Cookie++)
             {
                if(NumberDevices==1)
                {
                   cout << endl << "Testing  A " << rank << endl;
                   MPI_Recv(Device_HaloDown_Give[Cookie],(rank-1),rank-1,NumberDevices-1),&status);
                   cout << endl << "Testing  B " << rank << endl;    //CODE DOES NOT REACH HERE.
                   MPI_Send(Device_HaloUp_Take[Cookie],MPI_COMM_WORLD);
                   /*etc*/
                }
            }
         }
      }
      /* Then some code to carry out rest of lattice boltzmann method. */

   MPI_Finalize();
}

由于我有2个节点(代码中的NumberNodes == 2变量),我有一个作为rank == 0,另一个作为rank == 1 == NumberNodes-1.等级0代码进入if(rank == 0)循环,其中它输出“Testing X 0”但从未输出“Testing Y 0”,因为它预先在MPI_Send()函数中断开.此时的变量Cookie为0,因为只有一个GPU /设备,因此SendRecvID()函数采用“(1,0)”. MPI_Send的第一个参数是指针,因为Device_Halo_etc是一个指针数组,而数据发送到的位置是(rank 1)= 1.

类似地,秩1代码进入if(rank == NumberNodes-1)循环,在该循环中,当代码在完成MPI_Recv调用之前停止时,它输出“Testing A 1”而不是“Testing B 1”.据我所知,MPI_Recv的参数是正确的,因为(rank-1)= 0是正确的,发送和接收的数据点的数量是正确的,并且ID是相同的.

我到目前为止所尝试的是确保它们各自具有完全相同的标签(尽管每种情况下的SendRecvID()需要(1,0)因此无论如何都是相同的)手写999左右,但是这样做了没有不同.我还在两个MPI调用中将Device_Halo_etc参数更改为& Device_Halo_etc,以防万一我搞乱指针,但也没有区别.到目前为止我能让它工作的唯一方法是将MPI_Send / Recv()调用中的Device_Halo_etc参数更改为主机上的一些任意数组,以测试它们是否传输,这样做可以让它通过第一次MPI调用当然会陷入下一个,但即使只有当我将变量数量更改为Send / Recv为1(而不是NumDataPts_f_halo == 14250)时才有效.当然,移动主机阵列并不重要.

使用带有其他链接变量的nvcc编译器运行代码(我不太确定这些工作如何,将方法复制到某处,但考虑到更简单的设备到设备MPI调用已经有效,我认为没有问题),通过：

nvcc TestingMPI.cu -o run_Test -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -L/usr/lib/openmpi/lib -lmpi_cxx -lmpi -ldl

并编译：

mpirun -np 2 run_Test

这样做会给我一个错误,通常看起来像这样：

Testing  A 1

Testing X 0
[Anastasia:16671] *** Process received signal ***
[Anastasia:16671] Signal: Segmentation fault (11)
[Anastasia:16671] Signal code: Invalid permissions (2)
[Anastasia:16671] Failing at address: 0x700140000
[Anastasia:16671] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f20327774a0]
[Anastasia:16671] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x147fe5) [0x7f2032888fe5]
[Anastasia:16671] [ 2] /usr/lib/libmpi.so.1(opal_convertor_pack+0x14d) [0x7f20331303bd]
[Anastasia:16671] [ 3] /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so(+0x20c8) [0x7f202cad20c8]
[Anastasia:16671] [ 4] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0x100f0) [0x7f202d9430f0]
[Anastasia:16671] [ 5] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0x772b) [0x7f202d93a72b]
[Anastasia:16671] [ 6] /usr/lib/libmpi.so.1(MPI_Send+0x17b) [0x7f20330bc57b]
[Anastasia:16671] [ 7] run_Test() [0x400ff7]
[Anastasia:16671] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f203276276d]
[Anastasia:16671] [ 9] run_Test() [0x400ce9]
[Anastasia:16671] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 16671 on node Anastasia exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

我在我的笔记本电脑(Anastasia)上运行代码,这是一款联想Y500,配备在Linux Ubuntu 12.04LTS上运行的双GT650m NVIDIA显卡,如果有所作为的话. nvcc –version给出了“5.0版本,V0.2.1221”,而mpirun –version给出了“mpirun(Open MPI)1.5.4”.