【转载】live555学习 --H264数据处理（3）

    H264FUAFragmenter又对数据做了什么呢？

  1: void H264FUAFragmenter::doGetNextFrame() {    
  2:   if (fNumValidDataBytes == 1) {    
  3:       //读取一个新的frame     
  4:     // We have no NAL unit data currently in the buffer.  Read a new one:     
  5:     fInputSource->getNextFrame(&fInputBuffer[1],fInputBufferSize - 1,
  6:                    afterGettingFrame,this,monospace; font-size:12px">  7:                    FramedSource::handleClosure,255)">this);    
  8:   } else {    
  9:       //     
 10:       //现在buffer中已经存在NALU数据了，需要考虑三种情况     
 11:       //1.一个新的NALU,且足够小能投递给RTP sink。     
 12:       //2.一个新的NALU,但是比RTP sink要求的包大了，投递第一个分片作为一个FU-A packet,并带上一个额外的头字节。     
 13:       //3.部分NALU数据,投递下一个分片作为一个FU-A packet，并带上2个额外的头字节。     
 14:     // We have NAL unit data in the buffer.  There are three cases to consider:     
 15:     // 1. There is a new NAL unit in the buffer,and it's small enough to deliver     
 16:     //    to the RTP sink (as is).     
 17:     // 2. There is a new NAL unit in the buffer,but it's too large to deliver to     
 18:     //    the RTP sink in its entirety.  Deliver the first fragment of this data, 
 19:     //    as a FU-A packet,with one extra preceding header byte.     
 20:     // 3. There is a NAL unit in the buffer,and we've already delivered some     
 21:     //    fragment(s) of this.  Deliver the next fragment of this data,monospace; font-size:12px"> 22:      23:     
 24:     
 25:     if (fMaxSize < fMaxOutputPacketSize) { // shouldn't happen     
 26:       envir() << "H264FUAFragmenter::doGetNextFrame(): fMaxSize ("    
 27:           << fMaxSize << ") is smaller than expectedn";    
 28:     }  29:       fMaxSize = fMaxOutputPacketSize;    
 30:     }    
 31:     
 32:     
 33:     fLastFragmentCompletedNALUnit = True; // by default     
 34:     if (fCurDataOffset == 1) { // case 1 or 2     
 35:       if (fNumValidDataBytes - 1 <= fMaxSize) { // case 1     
 36:            37:           //情况1,处理整个NALU     
 38:            39:     memmove(fTo,&fInputBuffer[1],fNumValidDataBytes - 1);    
 40:     fFrameSize = fNumValidDataBytes - 1;    
 41:     fCurDataOffset = fNumValidDataBytes;    
 42:       } else { // case 2     
 43:            44:           //情况2，处理NALU的第1个分片。注意，我们添加FU指示符和FU头字节(with S bit)到包的最前面(     
 45:           //重用已经存在的NAL 头字节作为FU的头字节)     
 46:            47:     // We need to send the NAL unit data as FU-A packets.  Deliver the first     
 48:     // packet now.  Note that we add FU indicator and FU header bytes to the front     
 49:     // of the packet (reusing the existing NAL header byte for the FU header).     
 50:     fInputBuffer[0] = (fInputBuffer[1] & 0xE0) | 28; // FU indicator     
 51:     fInputBuffer[1] = 0x80 | (fInputBuffer[1] & 0x1F); // FU header (with S bit)   重用NALU头字节     
 52:     memmove(fTo,fInputBuffer,fMaxSize);    
 53:     fFrameSize = fMaxSize;    
 54:     fCurDataOffset += fMaxSize - 1;    
 55:     fLastFragmentCompletedNALUnit = False;    
 56:       }    
 57:     } // case 3     
 58:          59:         //情况3，处理非第1个分片。需要添加FU指示符和FU头(我们重用了第一个分片中的字节，但是需要清除S位，     
 60:         //并在最后一个分片中添加E位)     
 61:          62:          63:       // We are sending this NAL unit data as FU-A packets.  We've already sent the     
 64:       // first packet (fragment).  Now,send the next fragment.  Note that we add     
 65:       // FU indicator and FU header bytes to the front.  (We reuse these bytes that     
 66:       // we already sent for the first fragment,but clear the S bit,and add the E     
 67:       // bit if this is the last fragment.)     
 68:  
 69:       fInputBuffer[fCurDataOffset-2] = fInputBuffer[0];  70:       fInputBuffer[fCurDataOffset-1] = fInputBuffer[1]&~0x80; // FU header (no S bit)     
 71:       unsigned numBytesToSend = 2 + fNumValidDataBytes - fCurDataOffset;    
 72:       if (numBytesToSend > fMaxSize) {    
 73:     // We can't send all of the remaining data this time:     
 74:     numBytesToSend = fMaxSize;    
 75:     fLastFragmentCompletedNALUnit = False;    
 76:       }  77:      78:     //最后一个分片，需要在FU头中设置E标志位     
 79:     // This is the last fragment:     
 80:     fInputBuffer[fCurDataOffset-1] |= 0x40; // set the E bit in the FU header     
 81:     fNumTruncatedBytes = fSaveNumTruncatedBytes;    
 82:       }    
 83:       memmove(fTo,&fInputBuffer[fCurDataOffset-2],numBytesToSend);    
 84:       fFrameSize = numBytesToSend;    
 85:       fCurDataOffset += numBytesToSend - 2;    
 86:     }      
 87:     
 88:     if (fCurDataOffset >= fNumValidDataBytes) {    
 89:       // We're done with this data.  Reset the pointers for receiving new data:     
 90:       fNumValidDataBytes = fCurDataOffset = 1;    
 91:     }    
 92:     
 93:     // Complete delivery to the client:     
 94:     FramedSource::afterGetting( 95:   }    
 96: }   
 97: 

   当fNumValidDataBytes等于１时，表明buffer(fInputBuffer)中没有Nal Unit数据，那么就读入一个新的．从哪里读呢？还记得前面讲过的吗？H264FUAFragmenter在第一次读数据时代替了H264VideoStreamFramer，同时也与H264VideoStreamFramer还有ByteStreamFileSource手牵着脚，脚牵着手形成了链结构．文件数据从ByteStreamFileSource读入，经H264VideoStreamFramer处理传给H264FUAFragmenter．ByteStreamFileSource返回给H264VideoStreamFramer一段数据，H264VideoStreamFramer返回一个H264FUAFragmenter一个Nal unit ．
H264FUAFragmenter对Nal Unit做了什么呢？先看注释：
当我们有了nal unit，要处理３种情况：
１有一个完整的nal unit，并且它小到能够被打包进rtp包中．
２有一个完整的nal unit，但是它很大，那么就得为它分片传送了，把第一片打入一个FU-A包，此时利用了缓冲中前面的一个字节的头部．
３一个nal unit的已被发送了一部分，那么我们继续按FU-A包发送．此时利用了缓冲中前面的处理中已使用的两个字节的头部．
fNumValidDataBytes是H264FUAFragmenter缓冲fInputBuffer中有效数据的字节数．可以看到fNumValidDataBytes重置时被置为１，为什么不是０呢？因为fInputBuffer的第一个字节一直被留用作AU-A包的头部．如果是single nal打包，则从fInputBuffer的第二字节开始把nal unit复制到输出缓冲fTo，如果是FU-A包，则从fInputBuffer的第一字节开始复制．

结合本文最后面的H.264视频 RTP负载格式，可以很容易地把此段函数看明白。

---

   最后，来看下parse函数，parse是定义在MPEGVideoStreamParser中的纯虚函数，在子类H264VideoStreamParser中实现。parse主要是从文件的字节流中，分离出一个个的Frame,对于H264而言其实就是对一个个的NALU。*.264文件的格式非常简单，每个NALU以 0x00000001 作为起始符号(中间的NALU也可以以0x000001作为起始符)，顺序存放。

  
  

unsigned H264VideoStreamParser::parse() { 
try { 
  3:     // 
//首先找到起始符号,并跳过。文件流的最开始必需以0x00000001开始，但后续的NALU充许以0x000001(3 bytes)作为分隔 
  5:       6:     // The stream must start with a 0x00000001: 
  7:     if (!fHaveSeenFirstStartCode) { 
  8:       // Skip over any input bytes that precede the first 0x00000001: 
  9:       u_int32_t first4Bytes; 
while ((first4Bytes = test4Bytes()) != 0x00000001) { 
 11:     get1Byte(); setParseState(); // ensures that we progress over bad data 
 12:       } 
 13:       skipBytes(4); // skip this initial code 
 14:        
 15:       setParseState(); 
 16:       fHaveSeenFirstStartCode = True; // from now on 
 17:     } 
//将起始标志也保存到目的缓冲区中 
if (fOutputStartCodeSize > 0) { 
 22:       // Include a start code in the output: 
 23:       save4Bytes(0x00000001);    
 24:     } 
 25:      
 26:      27:     //保存所有数据，直至遇到起始标志，或者文件结束符。需要注意NALU中的第一个字节，因为它指示了NALU的类型 
 28:      29:     // Then save everything up until the next 0x00000001 (4 bytes) or 0x000001 (3 bytes),or we hit EOF. 
 30:     // Also make note of the first byte,because it contains the "nal_unit_type":  
 31:     u_int8_t firstByte; 
 32:     if (haveSeenEOF()) { 
 33:         34:        //已经设置了文件结束标志，将剩下的数据保存也来即可 
 35:         36:       // We hit EOF the last time that we tried to parse this data,monospace; font-size:12px"> 37:       // so we know that the remaining unparsed data forms a complete NAL unit: 
 38:       unsigned remainingDataSize = totNumValidBytes() - curOffset(); 
 39:       if (remainingDataSize == 0) (void)get1Byte(); // forces another read,which will cause EOF to get handled for real this time 
 40: #ifdef DEBUG 
 41:       fprintf(stderr,"This NAL unit (%d bytes) ends with EOFn",remainingDataSize); 
 42: #endif 
 43:       if (remainingDataSize == 0) return 0; 
 44:       firstByte = get1Byte();   //将第一个字节保存下来，其指示了NALU的类型 
 45:       saveByte(firstByte); 
 46:        
 47:       while (--remainingDataSize > 0) { 
 48:     saveByte(get1Byte()); 
 49:       } 
 50:     } else { 
 51:       u_int32_t next4Bytes = test4Bytes(); 
 52:       firstByte = next4Bytes>>24;   //将第一个字节保存下来 
 53:        54:       //将下一个起始符号之前的数据都保存下来 
 55:        56:       while (next4Bytes != 0x00000001 && (next4Bytes&0xFFFFFF00) != 0x00000100) { 
 57:     // We save at least some of "next4Bytes". 
 58:     if ((unsigned)(next4Bytes&0xFF) > 1) {  //一次可以保存4个字节,并不需要一个一个字节对比，除非到了结尾 
 59:       // Common case: 0x00000001 or 0x000001 definitely doesn't begin anywhere in "next4Bytes",so we save all of it: 
 60:       save4Bytes(next4Bytes); 
 61:       skipBytes(4); 
 62:     } // Save the first byte,and continue testing the rest: 
 64:       saveByte(next4Bytes>>24); 
 65:       skipBytes(1); 
 66:     } 
 67:     next4Bytes = test4Bytes(); 
 68:       } 
 69:       // Assert: next4Bytes starts with 0x00000001 or 0x000001,and we've saved all previous bytes (forming a complete NAL unit). 
 70:       // Skip over these remaining bytes,up until the start of the next NAL unit: 
if (next4Bytes == 0x00000001) { 
 72:     skipBytes(4); 
 73:       }  74:     skipBytes(3); 
 75:       } 
 76:     } 
 77:   
 78:     u_int8_t nal_ref_idc = (firstByte&0x60)>>5; 
 79:     u_int8_t nal_unit_type = firstByte&0x1F; 
 80: #ifdef DEBUG 
 81:     fprintf(stderr,0)">Parsed %d-byte NAL-unit (nal_ref_idc: %d,nal_unit_type: %d ("%s"))n",monospace; font-size:12px"> 82:         curFrameSize()-fOutputStartCodeSize,nal_ref_idc,nal_unit_type,nal_unit_type_description[nal_unit_type]); 
 83: #endif 
 84:  
 85:  
 86:      87:     //下面根据NALU的类型来作具体的分析 
 89:     switch (nal_unit_type) { 
 90:       case 6: { // Supplemental enhancement information (SEI) 
 91:     analyze_sei_data(); 
 92:     // Later,perhaps adjust "fPresentationTime" if we saw a "pic_timing" SEI payload??? ##### 
break; 
 94:       } 
 95:       case 7: { // Sequence parameter set (序列参数集) 
 96:        97:       //保存一份SPS的副本到H264VideoStreamFramer中，后面的pps也需要保存，sps中可能还包含了帧率信息 
 98:        99:     // First,save a copy of this NAL unit,in case the downstream object wants to see it: 
100:     usingSource()->saveCopyOfSPS(fStartOfFrame + fOutputStartCodeSize,fTo - fStartOfFrame - fOutputStartCodeSize); 
101:  
102:  
103:     // Parse this NAL unit to check whether frame rate information is present: 
104:     unsigned num_units_in_tick,time_scale,fixed_frame_rate_flag; 
105:     analyze_seq_parameter_set_data(num_units_in_tick,fixed_frame_rate_flag); 
106:     if (time_scale > 0 && num_units_in_tick > 0) { 
107:       usingSource()->fFrameRate = time_scale/(2.0*num_units_in_tick);   //sps中包含了帧率信息 
108: #ifdef DEBUG 
109:       fprintf(stderr,0)">Set frame rate to %f fpsn",usingSource()->fFrameRate); 
110:       if (fixed_frame_rate_flag == 0) { 
111:         fprintf(stderr,0)">tWARNING: "fixed_frame_rate_flag" was not setn"); 
112:       } 
113: #endif 
114:     } else {    //sps中不包含帧率信息，则使用source中设置的默认帧率 
115: #ifdef DEBUG 
116:       fprintf(stderr,0)">tThis "Sequence Parameter Set" NAL unit contained no frame rate information,so we use a default frame rate of %f fpsn",monospace; font-size:12px">117: #endif 
118:     } 
119:     120:       } 
121:       case 8: { // Picture parameter set (图像参数集PPS) 
122:     // Save a copy of this NAL unit,monospace; font-size:12px">123:     usingSource()->saveCopyOfPPS(fStartOfFrame + fOutputStartCodeSize,monospace; font-size:12px">124:       } 
125:     } 
126:      
127:     usingSource()->setPresentationTime();   //设置当前的时间 
128: #ifdef DEBUG 
129:     unsigned long secs = (long)usingSource()->fPresentationTime.tv_sec; 
130:     unsigned uSecs = (unsigned)usingSource()->fPresentationTime.tv_usec; 
131:     fprintf(stderr,0)">tPresentation time: %lu.%06un",secs,uSecs); 
132: #endif 
133:      
134:     135:     //如果这个NALU是一个VCL NALU(即包含的是视频数据)，我们需要扫描下一个NALU的起始符， 
136:     //以判断这个NALU是否是当前的"access unit"(这个"access unit"应该可以理解为一帧图像帧吧)。 
137:     //我们需要根据这个信息去指明何时该增加"fPresentationTime"(RTP 打包时也需要根据这个信息，决定是否设置"M"位)。 
138:     139:     // If this NAL unit is a VCL NAL unit,we also scan the start of the next NAL unit,to determine whether this NAL unit 
140:     // ends the current 'access unit'.  We need this information to figure out when to increment "fPresentationTime". 
141:     // (RTP streamers also need to know this in order to figure out whether or not to set the "M" bit.) 
142:     Boolean thisNALUnitEndsAccessUnit = False; // until we learn otherwise  
143:     144:       // There is no next NAL unit,so we assume that this one ends the current 'access unit': 
145:       thisNALUnitEndsAccessUnit = True; 
146:     } 147:       Boolean const isVCL = nal_unit_type <= 5 && nal_unit_type > 0; // Would need to include type 20 for SVC and MVC ##### 
148:       if (isVCL) { 
149:     u_int32_t first4BytesOfNextNALUnit = test4Bytes(); 
150:     u_int8_t firstByteOfNextNALUnit = first4BytesOfNextNALUnit>>24; 
151:     u_int8_t next_nal_ref_idc = (firstByteOfNextNALUnit&0x60)>>5; 
152:     u_int8_t next_nal_unit_type = firstByteOfNextNALUnit&0x1F; 
153:     if (next_nal_unit_type >= 6) { 
154:         //下一个NALU不是VCL的，当前的"access unit"结束了 
155:       // The next NAL unit is not a VCL; therefore,we assume that this NAL unit ends the current 'access unit': 
156: #ifdef DEBUG 
157:       fprintf(stderr,0)">t(The next NAL unit is not a VCL)n"); 
158: #endif 
159:       thisNALUnitEndsAccessUnit = True; 
160:     } 161:         //下一个NALU也是VCL的，还需要检查一下它们是不是属于同一个"access unit" 
162:       // The next NAL unit is also a VLC.  We need to examine it a little to figure out if it's a different 'access unit'. 
163:       // (We use many of the criteria described in section 7.4.1.2.4 of the H.264 specification.) 
164:       Boolean IdrPicFlag = nal_unit_type == 5; 
165:       Boolean next_IdrPicFlag = next_nal_unit_type == 5; 
166:       if (next_IdrPicFlag != IdrPicFlag) { 
167:         // IdrPicFlag differs in value 
168: #ifdef DEBUG 
169:         fprintf(stderr,0)">t(IdrPicFlag differs in value)n"); 
170: #endif 
171:         thisNALUnitEndsAccessUnit = True; 
172:       } else if (next_nal_ref_idc != nal_ref_idc && next_nal_ref_idc*nal_ref_idc == 0) { 
173:         // nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0 
174: #ifdef DEBUG 
175:         fprintf(stderr,0)">t(nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0)n"); 
176: #endif 
177:         thisNALUnitEndsAccessUnit = True; 
178:       } if ((nal_unit_type == 1 || nal_unit_type == 2 || nal_unit_type == 5) 
179:              && (next_nal_unit_type == 1 || next_nal_unit_type == 2 || next_nal_unit_type == 5)) { 
180:         // Both this and the next NAL units begin with a "slice_header". 
181:         // Parse this (for each),to get parameters that we can compare: 
182:          
183:         // Current NAL unit's "slice_header": 
184:         unsigned frame_num,pic_parameter_set_id,idr_pic_id; 
185:         Boolean field_pic_flag,bottom_field_flag; 
186:         analyze_slice_header(fStartOfFrame + fOutputStartCodeSize,fTo,monospace; font-size:12px">187:                  frame_num,idr_pic_id,field_pic_flag,bottom_field_flag); 
188:          
189:         // Next NAL unit's "slice_header": 
190: #ifdef DEBUG 
191:         fprintf(stderr,0)">    Next NAL unit's slice_header:n"); 
192: #endif 
193:         u_int8_t next_slice_header[NUM_NEXT_SLICE_HEADER_BYTES_TO_ANALYZE]; 
194:         testBytes(next_slice_header,255)">sizeof next_slice_header); 
195:         unsigned next_frame_num,next_pic_parameter_set_id,next_idr_pic_id; 
196:         Boolean next_field_pic_flag,next_bottom_field_flag; 
197:         analyze_slice_header(next_slice_header,&next_slice_header[sizeof next_slice_header],next_nal_unit_type,monospace; font-size:12px">198:                  next_frame_num,next_idr_pic_id,next_field_pic_flag,next_bottom_field_flag); 
199:          
200:         if (next_frame_num != frame_num) { 
201:           // frame_num differs in value 
202: #ifdef DEBUG 
203:           fprintf(stderr,0)">t(frame_num differs in value)n"); 
204: #endif 
205:           thisNALUnitEndsAccessUnit = True; 
206:         } if (next_pic_parameter_set_id != pic_parameter_set_id) { 
207:           // pic_parameter_set_id differs in value 
208: #ifdef DEBUG 
209:           fprintf(stderr,0)">t(pic_parameter_set_id differs in value)n"); 
210: #endif 
211:           thisNALUnitEndsAccessUnit = True; 
212:         } if (next_field_pic_flag != field_pic_flag) { 
213:           // field_pic_flag differs in value 
214: #ifdef DEBUG 
215:           fprintf(stderr,0)">t(field_pic_flag differs in value)n"); 
216: #endif 
217:           thisNALUnitEndsAccessUnit = True; 
218:         } if (next_bottom_field_flag != bottom_field_flag) { 
219:           // bottom_field_flag differs in value 
220: #ifdef DEBUG 
221:           fprintf(stderr,0)">t(bottom_field_flag differs in value)n"); 
222: #endif 
223:           thisNALUnitEndsAccessUnit = True; 
224:         } if (next_IdrPicFlag == 1 && next_idr_pic_id != idr_pic_id) { 
225:           // IdrPicFlag is equal to 1 for both and idr_pic_id differs in value 
226:           // Note: We already know that IdrPicFlag is the same for both. 
227: #ifdef DEBUG 
228:           fprintf(stderr,0)">t(IdrPicFlag is equal to 1 for both and idr_pic_id differs in value)n"); 
229: #endif 
230:           thisNALUnitEndsAccessUnit = True; 
231:         } 
232:       } 
233:     } 
234:       } 
235:     } 
236:      
237:     if (thisNALUnitEndsAccessUnit) { 
238: #ifdef DEBUG 
239:       fprintf(stderr,0)">*****This NAL unit ends the current access unit*****n"); 
240: #endif 
241:       usingSource()->fPictureEndMarker = True;  //这里就是设置RTP打包时用到的M标志了 
242:       ++usingSource()->fPictureCount; 
243:  
244:  
245:       246:       //下一个NALU不再属于当前"access　unit""时，才改变时间 
247:       248:       // Note that the presentation time for the next NAL unit will be different: 
249:       struct timeval& nextPT = usingSource()->fNextPresentationTime; // alias　这里是引用　 
250:       nextPT = usingSource()->fPresentationTime; 
251:       double nextFraction = nextPT.tv_usec/1000000.0 + 1/usingSource()->fFrameRate;  
252:       unsigned nextSecsIncrement = (long)nextFraction; 
253:       nextPT.tv_sec += (long)nextSecsIncrement; 
254:       nextPT.tv_usec = (long)((nextFraction - nextSecsIncrement)*1000000); 
255:     } 
256:     setParseState(); 
257:  
258:  
259:     return curFrameSize(); 
260:   } catch (int /*e*/) { 
261: #ifdef DEBUG 
262:     fprintf(stderr,0)">H264VideoStreamParser::parse() EXCEPTION (This is normal behavior - *not* an error)n"); 
263: #endif 
264:     return 0;  // the parsing got interrupted 
265:   } 
266: }

??????? H264VideoStreamParser::parse()函数除了取出Frame，还对NALU中的部分参数做了解释工作。对于PPS或者SPS类型的NALU，要保存到H264VideoStreamFramer中。"access unit",在这里可以理解为一副图像,一个"access unit"可以包含多个NALU，很显示这些NALU的时间戳应该是相同的。实际上，很多时候一个"access unit"单元只包含一个NALU，这里简单多了。

---

H.264 视频 RTP 负载格式

1. 网络抽象层单元类型 (NALU)

NALU 头由一个字节组成,它的语法如下:

????? +---------------+
????? |0|1|2|3|4|5|6|7|
????? +-+-+-+-+-+-+-+-+
????? |F|NRI|? Type?? |
????? +---------------+

F: 1 个比特.
? forbidden_zero_bit. 在 H.264 规范中规定了这一位必须为 0.

NRI: 2 个比特.
? nal_ref_idc. 取 00 ~ 11,似乎指示这个 NALU 的重要性,如 00 的 NALU 解码器可以丢弃它而不影响图像的回放. 不过一般情况下不太关心

这个属性.

Type: 5 个比特.
? nal_unit_type. 这个 NALU 单元的类型. 简述如下:

? 0???? 没有定义
? 1-23? NAL单元? 单个 NAL 单元包.
? 24??? STAP-A?? 单一时间的组合包
? 25??? STAP-B?? 单一时间的组合包
? 26??? MTAP16?? 多个时间的组合包
? 27??? MTAP24?? 多个时间的组合包
? 28??? FU-A???? 分片的单元
? 29??? FU-B???? 分片的单元
? 30-31 没有定义

2. 打包模式

? 下面是 RFC 3550 中规定的 RTP 头的结构.

?????? 0?????????????????? 1?????????????????? 2?????????????????? 3
?????? 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |V=2|P|X|? CC?? |M|???? PT????? |?????? sequence number???????? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |?????????????????????????? timestamp?????????????????????????? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |?????????? synchronization source (SSRC) identifier??????????? |
????? +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
????? |??????????? contributing source (CSRC) identifiers???????????? |
????? |???????????????????????????? ....????????????????????????????? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

? 负载类型 Payload type (PT): 7 bits
? 序列号 Sequence number (SN): 16 bits
? 时间戳 Timestamp: 32 bits
??
? H.264 Payload 格式定义了三种不同的基本的负载(Payload)结构. 接收端可能通过 RTP Payload?
? 的第一个字节来识别它们. 这一个字节类似 NALU 头的格式,而这个头结构的 NAL 单元类型字段
? 则指出了代表的是哪一种结构,

? 这个字节的结构如下,可以看出它和 H.264 的 NALU 头结构是一样的.
????? +---------------+
????? |0|1|2|3|4|5|6|7|
????? +-+-+-+-+-+-+-+-+
????? |F|NRI|? Type?? |
????? +---------------+
? 字段 Type: 这个 RTP payload 中 NAL 单元的类型. 这个字段和 H.264 中类型字段的区别是,当 type
? 的值为 24 ~ 31 表示这是一个特别格式的 NAL 单元,而 H.264 中,只取 1~23 是有效的值.
???
? 24??? STAP-A?? 单一时间的组合包
? 25??? STAP-B?? 单一时间的组合包
? 26??? MTAP16?? 多个时间的组合包
? 27??? MTAP24?? 多个时间的组合包
? 28??? FU-A???? 分片的单元
? 29??? FU-B???? 分片的单元
? 30-31 没有定义

? 可能的结构类型分别有:

? 1. 单一 NAL 单元模式
???? 即一个 RTP 包仅由一个完整的 NALU 组成. 这种情况下 RTP NAL 头类型字段和原始的 H.264的
? NALU 头类型字段是一样的.

? 2. 组合封包模式
??? 即可能是由多个 NAL 单元组成一个 RTP 包. 分别有4种组合方式: STAP-A,STAP-B,MTAP16,MTAP24.
? 那么这里的类型值分别是 24,25,26 以及 27.

? 3. 分片封包模式
??? 用于把一个 NALU 单元封装成多个 RTP 包. 存在两种类型 FU-A 和 FU-B. 类型值分别是 28 和 29.

2.1 单一 NAL 单元模式

? 对于 NALU 的长度小于 MTU 大小的包,一般采用单一 NAL 单元模式.
? 对于一个原始的 H.264 NALU 单元常由 [Start Code] [NALU Header] [NALU Payload] 三部分组成,其中 Start Code 用于标示这是一个

NALU 单元的开始,必须是 "00 00 00 01" 或 "00 00 01",NALU 头仅一个字节,其后都是 NALU 单元内容.
? 打包时去除 "00 00 01" 或 "00 00 00 01" 的开始码,把其他数据封包的 RTP 包即可.

?????? 0?????????????????? 1?????????????????? 2?????????????????? 3
?????? 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |F|NRI|? type?? |?????????????????????????????????????????????? |
????? +-+-+-+-+-+-+-+-+?????????????????????????????????????????????? |
????? |?????????????????????????????????????????????????????????????? |
????? |?????????????? Bytes 2..n of a Single NAL unit???????????????? |
????? |?????????????????????????????????????????????????????????????? |
????? |?????????????????????????????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |?????????????????????????????? :...OPTIONAL RTP padding??????? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

? 如有一个 H.264 的 NALU 是这样的:

? [00 00 00 01 67 42 A0 1E 23 56 0E 2F ... ]

? 这是一个序列参数集 NAL 单元. [00 00 00 01] 是四个字节的开始码,67 是 NALU 头,42 开始的数据是 NALU 内容.

? 封装成 RTP 包将如下:

? [ RTP Header ] [ 67 42 A0 1E 23 56 0E 2F ]

? 即只要去掉 4 个字节的开始码就可以了.

2.2 组合封包模式

? 其次,当 NALU 的长度特别小时,可以把几个 NALU 单元封在一个 RTP 包中.

??
?????? 0?????????????????? 1?????????????????? 2?????????????????? 3
?????? 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |????????????????????????? RTP Header?????????????????????????? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |STAP-A NAL HDR |???????? NALU 1 Size?????????? | NALU 1 HDR??? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |???????????????????????? NALU 1 Data?????????????????????????? |
????? :?????????????????????????????????????????????????????????????? :
????? +?????????????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |?????????????? | NALU 2 Size?????????????????? | NALU 2 HDR??? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |???????????????????????? NALU 2 Data?????????????????????????? |
????? :?????????????????????????????????????????????????????????????? :
????? |?????????????????????????????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |?????????????????????????????? :...OPTIONAL RTP padding??????? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.3 Fragmentation Units (FUs).

? 而当 NALU 的长度超过 MTU 时,就必须对 NALU 单元进行分片封包. 也称为 Fragmentation Units (FUs).
??
?????? 0?????????????????? 1?????????????????? 2?????????????????? 3
?????? 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? | FU indicator? |?? FU header?? |?????????????????????????????? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+?????????????????????????????? |
????? |?????????????????????????????????????????????????????????????? |
????? |???????????????????????? FU payload??????????????????????????? |
????? |?????????????????????????????????????????????????????????????? |
????? |?????????????????????????????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
????? |?????????????????????????????? :...OPTIONAL RTP padding??????? |
????? +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

????? Figure 14.? RTP payload format for FU-A

?? The FU indicator octet has the following format:

?? The FU header has the following format:

????? +---------------+
????? |0|1|2|3|4|5|6|7|
????? +-+-+-+-+-+-+-+-+
????? |S|E|R|? Type?? |
????? +---------------+

－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－

H264FUAFragmenter中只支持single和FU-A模式，不支持其它模式．

??????? 我们现在还得出一个结论，我们可以看出RTPSink与Source怎样分工：RTPSink只做形成通用RTP包头的工作，各种媒体格式的Source才是实现媒体数据RTP封包的地方，其实按习惯感觉XXXRTPSink才是进行封包的地方．但是，从文件的安排上，H264FUAFragmenter被隐藏在H264VideoRTPSink中，并在程序中暗渡陈仓地把H264VideoStreamFramer替换掉，其实还是按习惯的架构（设计模式）来做的，所以如果把H264FUAFragmenter的工作移到H264VideoRTPSink中也是没问题的．

http://blog.csdn.net/nkmnkm/article/details/6947309

http://www.voidcn.com/article/p-ytchopga-qs.html