加入收藏 | 设为首页 | 会员中心 | 我要投稿 李大同 （https://www.lidatong.com.cn/）- 科技、建站、经验、云计算、5G、大数据,站长网!
当前位置： 首页 > 大数据 > 正文

live555源码分析----H264的数据处理

发布时间：2020-12-14 01:27:51 所属栏目：大数据 来源：网络整理
导读：现在来分析live555中关于H264的处理部分，主要包括从文件中读取数据进行并进行frame(NALU)的分割，然后对frame进行分片，这些工作都是在frame交给RTP sink之前完成的。 [cpp] ? view plain? copy ? print ? void ?MultiFramedRTPSink::packFrame()?{?? ?? if
现在来分析live555中关于H264的处理部分，主要包括从文件中读取数据进行并进行frame(NALU)的分割，然后对frame进行分片，这些工作都是在frame交给RTP sink之前完成的。
[cpp]? view plain? copy
? print ?
  1. void?MultiFramedRTPSink::packFrame()?{??
  2. ??if?(fOutBuf->haveOverflowData())?{??
  3. ...??
  4. ??}?else?{??
  5. ??
  6. ??
  7. ????//??
  8. ????//从source中获取下一个frame??
  9. ????fSource->getNextFrame(fOutBuf->curPtr(),?fOutBuf->totalBytesAvailable(),??
  10. ??????????????afterGettingFrame,?this,?ourHandleClosure,?this);??
  11. ??}??
  12. } ?
?? ?getNextFrame是定义在FramedSource中的非虚函数，从source中获取下一个frame，然后调用回调函数afterGettingFrame。afterGettingFrame被定义为静态函数，因为在C++中类成员函数是不能作为回调用函数的。不过这里为什么要用回调函数回？
?? ?注意，对于H264来说，上面的fSource并不是MPEGVideoStreamFramer类，因为在 H264VideoRTPSink::continuePlaying()函数中改变了fSource的值。 ? ?
[cpp]? view plain ?copy
?print ?
  1. Boolean?H264VideoRTPSink::continuePlaying()?{??
  2. ??//?First,?check?whether?we?have?a?'fragmenter'?class?set?up?yet.??
  3. ??//?If?not,?create?it?now:??
  4. ??if?(fOurFragmenter?==?NULL)?{??
  5. ????//创建一个辅助类H264FUAFragmenter，用于H264按照RFC3984进行RTP打包??
  6. ????fOurFragmenter?=?new?H264FUAFragmenter(envir(),?fSource,?OutPacketBuffer::maxSize,??
  7. ???????????????????????ourMaxPacketSize()?-?12/*RTP?hdr?size*/);??
  8. ????fSource?=?fOurFragmenter;??
  9. ??}??
  10. ??//?Then?call?the?parent?class's?implementation:??
  11. ??return?MultiFramedRTPSink::continuePlaying();??
  12. } ?
?? ?fSource被指向了H264FUAFragmenter类，这个类主要实现了H264按照RFC3984进行RTP分包，不过这里的实现每个RTP中最多只包含一个NALU，没有实现组合封包的情形。这个类的继承关系如下：H264FUAFragmenter->FramedFilter->FramedSource。很明显，这是一个filter,包装了MPEGVideoStreamFramer类的对像。
?? ?先来看来看getNextFrame的实现
[cpp]? view plain ?copy
?print ?
  1. void?FramedSource::getNextFrame(unsigned?char*?to,?unsigned?maxSize,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ????????????????afterGettingFunc*?afterGettingFunc,??
  2. ????????????????void*?afterGettingClientData,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ????????????????onCloseFunc*?onCloseFunc,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ????????????????void*?onCloseClientData)?{??
  3. ??//?Make?sure?we're?not?already?being?read:??
  4. ??if?(fIsCurrentlyAwaitingData)?{??
  5. ????envir()?<<?"FramedSource["?<<?this?<<?"]::getNextFrame():?attempting?to?read?more?than?once?at?the?same?time!n";??
  6. ????envir().internalError();??
  7. ??fTo?=?to;?????????????//buffer地址??
  8. ??fMaxSize?=?maxSize;???//buffer最大长度??
  9. ??fNumTruncatedBytes?=?0;?//?by?default;?could?be?changed?by?doGetNextFrame()??
  10. ??fDurationInMicroseconds?=?0;?//?by?default;?could?be?changed?by?doGetNextFrame()??
  11. ??fAfterGettingFunc?=?afterGettingFunc;?????????????//获取完一个frame后将执行这个函数??
  12. ??fAfterGettingClientData?=?afterGettingClientData;?//这个参数就是MultiFramedRTPSink类型指针??
  13. ??fOnCloseFunc?=?onCloseFunc;??
  14. ??fOnCloseClientData?=?onCloseClientData;??
  15. ??fIsCurrentlyAwaitingData?=?True;??
  16. ??doGetNextFrame();??
  17. } ?
?? ?上面的函数主要是进行一些成员变量的初始化，获取到的frame需要保存到fTo地址中，然后调用fAfterGettingFunc函数，若文件读取完毕，还需要调用fOnCloseFunc函数。重要的工作还是在doGetNextFrame函数中完成，不过它是定义在FramedSource类中的纯虚函数，需要在子类中重新实现。
?? ?现在来看H264FUAFragmenter中对doGetNextFrame的实现
[cpp]? view plain ?copy
?print ?
  1. void?H264FUAFragmenter::doGetNextFrame()?{??
  2. ??if?(fNumValidDataBytes?==?1)?{??
  3. ??????//读取一个新的frame??
  4. ????//?We?have?no?NAL?unit?data?currently?in?the?buffer.??Read?a?new?one:??
  5. ????fInputSource->getNextFrame(&fInputBuffer[1],?fInputBufferSize?-?1,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ???????????????????afterGettingFrame,??
  6. ???????????????????FramedSource::handleClosure,?this);??
  7. ??????//??
  8. ??????//现在buffer中已经存在NALU数据了，需要考虑三种情况??
  9. ??????//1.一个新的NALU,且足够小能投递给RTP?sink。??
  10. ??????//2.一个新的NALU,但是比RTP?sink要求的包大了，投递第一个分片作为一个FU-A?packet,?并带上一个额外的头字节。??
  11. ??????//3.部分NALU数据,投递下一个分片作为一个FU-A?packet，并带上2个额外的头字节。??
  12. ????//?We?have?NAL?unit?data?in?the?buffer.??There?are?three?cases?to?consider:??
  13. ????//?1.?There?is?a?new?NAL?unit?in?the?buffer,?and?it's?small?enough?to?deliver??
  14. ????//????to?the?RTP?sink?(as?is).??
  15. ????//?2.?There?is?a?new?NAL?unit?in?the?buffer,?but?it's?too?large?to?deliver?to??
  16. ????//????the?RTP?sink?in?its?entirety.??Deliver?the?first?fragment?of?this?data,??
  17. ????//????as?a?FU-A?packet,?with?one?extra?preceding?header?byte.??
  18. ????//?3.?There?is?a?NAL?unit?in?the?buffer,?and?we've?already?delivered?some??
  19. ????//????fragment(s)?of?this.??Deliver?the?next?fragment?of?this?data,??
  20. ??
  21. ????if?(fMaxSize?<?fMaxOutputPacketSize)?{?//?shouldn't?happen??
  22. ??????envir()?<<?"H264FUAFragmenter::doGetNextFrame():?fMaxSize?("??
  23. ??????????<<?fMaxSize?<<?")?is?smaller?than?expectedn";??
  24. ????}?else?{??
  25. ??????fMaxSize?=?fMaxOutputPacketSize;??
  26. ????}??
  27. ????fLastFragmentCompletedNALUnit?=?True;?//?by?default??
  28. ????if?(fCurDataOffset?==?1)?{?//?case?1?or?2??
  29. ??????if?(fNumValidDataBytes?-?1?<=?fMaxSize)?{?//?case?1??
  30. ??????????//??
  31. ??????????//情况1,?处理整个NALU??
  32. ????memmove(fTo,?&fInputBuffer[1],?fNumValidDataBytes?-?1);??
  33. ????fFrameSize?=?fNumValidDataBytes?-?1;??
  34. ????fCurDataOffset?=?fNumValidDataBytes;??
  35. ??????}?else?{?//?case?2??
  36. ??????????//??
  37. ??????????//情况2，处理NALU的第1个分片。注意，我们添加FU指示符和FU头字节(with?S?bit)到包的最前面(??
  38. ??????????//重用已经存在的NAL?头字节作为FU的头字节)??
  39. ????//?We?need?to?send?the?NAL?unit?data?as?FU-A?packets.??Deliver?the?first??
  40. ????//?packet?now.??Note?that?we?add?FU?indicator?and?FU?header?bytes?to?the?front??
  41. ????//?of?the?packet?(reusing?the?existing?NAL?header?byte?for?the?FU?header).??
  42. ????fInputBuffer[0]?=?(fInputBuffer[1]?&?0xE0)?|?28;?//?FU?indicator??
  43. ????fInputBuffer[1]?=?0x80?|?(fInputBuffer[1]?&?0x1F);?//?FU?header?(with?S?bit)???重用NALU头字节??
  44. ????fFrameSize?=?fMaxSize;??
  45. ????fCurDataOffset?+=?fMaxSize?-?1;??
  46. ????fLastFragmentCompletedNALUnit?=?False;??
  47. ??????}??
  48. ????}?else?{?//?case?3??
  49. ????????//??
  50. ????????//情况3，处理非第1个分片。需要添加FU指示符和FU头(我们重用了第一个分片中的字节，但是需要清除S位，??
  51. ????????//并在最后一个分片中添加E位)??
  52. ????????//??
  53. ??????//?We?are?sending?this?NAL?unit?data?as?FU-A?packets.??We've?already?sent?the??
  54. ??????//?first?packet?(fragment).??Now,?send?the?next?fragment.??Note?that?we?add??
  55. ??????//?FU?indicator?and?FU?header?bytes?to?the?front.??(We?reuse?these?bytes?that??
  56. ??????//?we?already?sent?for?the?first?fragment,?but?clear?the?S?bit,?and?add?the?E??
  57. ??????//?bit?if?this?is?the?last?fragment.)??
  58. ??????fInputBuffer[fCurDataOffset-2]?=?fInputBuffer[0];?//?FU?indicator??
  59. ??????fInputBuffer[fCurDataOffset-1]?=?fInputBuffer[1]&~0x80;?//?FU?header?(no?S?bit)??
  60. ??????unsigned?numBytesToSend?=?2?+?fNumValidDataBytes?-?fCurDataOffset;??
  61. ??????if?(numBytesToSend?>?fMaxSize)?{??
  62. ????//?We?can't?send?all?of?the?remaining?data?this?time:??
  63. ????numBytesToSend?=?fMaxSize;??
  64. ????fLastFragmentCompletedNALUnit?=?False;??
  65. ??????}?else?{??
  66. ????//最后一个分片，需要在FU头中设置E标志位??
  67. ????//?This?is?the?last?fragment:??
  68. ????fInputBuffer[fCurDataOffset-1]?|=?0x40;?//?set?the?E?bit?in?the?FU?header??
  69. ????fNumTruncatedBytes?=?fSaveNumTruncatedBytes;??
  70. ??????}??
  71. ??????memmove(fTo,?&fInputBuffer[fCurDataOffset-2],?numBytesToSend);??
  72. ??????fFrameSize?=?numBytesToSend;??
  73. ??????fCurDataOffset?+=?numBytesToSend?-?2;??
  74. ????}??
  75. ????if?(fCurDataOffset?>=?fNumValidDataBytes)?{??
  76. ??????//?We're?done?with?this?data.??Reset?the?pointers?for?receiving?new?data:??
  77. ??????fNumValidDataBytes?=?fCurDataOffset?=?1;??
  78. ????//?Complete?delivery?to?the?client:??
  79. ????FramedSource::afterGetting(this);??
  80. } ?
?? ?H264FUAFragmenter::doGetNextFrame函数第一次执行时，执行条件1,需要调用 MPEGVideoStreamFramer::doGetNextFrame读取一个新的frame，获取frame的具体过程稍后再分析。现在先看获取frame之后的工作，afterGettingFrame函数
[cpp]? view plain ?copy
?print ?
  1. void?H264FUAFragmenter::afterGettingFrame(void*?clientData,?unsigned?frameSize,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ??????????????????????unsigned?numTruncatedBytes,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ??????????????????????struct?timeval?presentationTime,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ??????????????????????unsigned?durationInMicroseconds)?{??
  2. ??H264FUAFragmenter*?fragmenter?=?(H264FUAFragmenter*)clientData;??
  3. ??fragmenter->afterGettingFrame1(frameSize,?numTruncatedBytes,?presentationTime,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ?????????????????durationInMicroseconds);??
  4. } ?
没什么好说的，再看afterGettingFrame1函数
[cpp]? view plain ?copy
?print ?
  1. void?H264FUAFragmenter::afterGettingFrame1(unsigned?frameSize,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ???????????????????????unsigned?numTruncatedBytes,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ???????????????????????struct?timeval?presentationTime,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ???????????????????????unsigned?durationInMicroseconds)?{??
  2. ??fNumValidDataBytes?+=?frameSize;??????//保存读到的frame长度??
  3. ??fSaveNumTruncatedBytes?=?numTruncatedBytes;??
  4. ??fPresentationTime?=?presentationTime;??
  5. ??fDurationInMicroseconds?=?durationInMicroseconds;??
  6. ??//?Deliver?data?to?the?client:??
  7. } ?
?? ?上面的代码首先记录几个数据到成员变量中，fNumValidDataBytes很重要，表示读取到的frame长度+1。然后又一次调用了H264FUAFragmenter::doGetNextFrame()，这里将进入H264FUAFragmenter::doGetNextFrame函数中第二个条件分支，这种循环调用很容易把人弄迷糊了。

?? ?H264FUAFragmenter::doGetNextFrame函数中第二个条件分支中，处理H264的RTP分片问题，这里是按照RFC3984进行RTP封装的。你应该注意到，在上篇文章"RTP的打包与发送"中，也出现了分片的代码(MultiFramedRTPSink::packFrame函数中)，那里直接将frame按MTU的长度来拆分。那为什么H264还要自定义一套RTP打包的标准呢？暂时我也不清楚。

?? ?在H264FUAFragmenter::doGetNextFrame()最后调用了 FramedSource::afterGetting
[cpp]? view plain ?copy
?print ?
  1. void?FramedSource::afterGetting(FramedSource*?source)?{??
  2. ??source->fIsCurrentlyAwaitingData?=?False;?????//表示已经获取到数据了，处于非等待状态??
  3. ??????//?indicates?that?we?can?be?read?again??
  4. ??????//?Note?that?this?needs?to?be?done?here,?in?case?the?"fAfterFunc"??
  5. ??????//?called?below?tries?to?read?another?frame?(which?it?usually?will)??
  6. ??//通过回调用进行后续处理??
  7. ??if?(source->fAfterGettingFunc?!=?NULL)?{??
  8. ????(*(source->fAfterGettingFunc))(source->fAfterGettingClientData,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ???????????????????source->fFrameSize,?source->fNumTruncatedBytes,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ???????????????????source->fPresentationTime,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ???????????????????source->fDurationInMicroseconds);??
  9. } ?
?? ?上面的代码主要是调用了FramedSource::getNextFrame函数中传递下来的回调函数，这个回调函数就是MultiFramedRTPSink::afterGettingFrame，处理过程在上一篇文章"RTP的打包与发送"中已经分析过了。
?? ?现在来看MPEGVideoStreamFramer::doGetNextFrame获取Frame的过程。继承关系：H264VideoStreamFramer->MPEGVideoStreamFramer->FramedFilter->FramedSource。在继承路径中存在FrameFilter，这说明H264VideoStreamFramer包装了其它source(包装的是读取文件的字节流source)。doGetNextFrame函数首先在MPEGVideoStreamFramer中实现。
[cpp]? view plain ?copy
?print ?
  1. void?MPEGVideoStreamFramer::doGetNextFrame()?{??
  2. ??fParser->registerReadInterest(fTo,?fMaxSize);?//将目的buffer信息注册到语法分析类中??
  3. ??continueReadProcessing();?????//继续进行读数据??
  4. } ?
?? ?这里的MPEGVideoStreamFramer::fParser,是一个MPEGVideoStreamParser类型指针，作为语法分析器。再来看continueReadProcessing函数
[cpp]? view plain ?copy
?print ?
  1. void?MPEGVideoStreamFramer::continueReadProcessing()?{??
  2. ??unsigned?acquiredFrameSize?=?fParser->parse();????//文件的语法分析(即demux)??
  3. ??if?(acquiredFrameSize?>?0)?{??
  4. ????//?We?were?able?to?acquire?a?frame?from?the?input.??
  5. ????//?It?has?already?been?copied?to?the?reader's?space.??
  6. ????fFrameSize?=?acquiredFrameSize;??
  7. ????fNumTruncatedBytes?=?fParser->numTruncatedBytes();??
  8. ????//?"fPresentationTime"?should?have?already?been?computed.??
  9. ????//?Compute?"fDurationInMicroseconds"?now:??
  10. ????fDurationInMicroseconds??
  11. ??????=?(fFrameRate?==?0.0?||?((int)fPictureCount)?<?0)???0??
  12. ??????:?(unsigned)((fPictureCount*1000000)/fFrameRate);??
  13. #ifdef?DEBUG??
  14. ????fprintf(stderr,?"%d?bytes?@%u.%06d,?fDurationInMicroseconds:?%d?((%d*1000000)/%f)n",?acquiredFrameSize,?fPresentationTime.tv_sec,?fPresentationTime.tv_usec,?fDurationInMicroseconds,?fPictureCount,?fFrameRate);??
  15. #endif??
  16. ????fPictureCount?=?0;??
  17. ??????
  18. ????//调用自身的afterGetting函数，因为这不一个"leaf"?source,?所以可能直接调用，??
  19. ????//而不用担心出现无限递归??
  20. ????//??
  21. ????//?Call?our?own?'after?getting'?function.??Because?we're?not?a?'leaf'??
  22. ????//?source,?we?can?call?this?directly,?without?risking?infinite?recursion.??
  23. ????afterGetting(this);??
  24. ??}?else?{??
  25. ????//?We?were?unable?to?parse?a?complete?frame?from?the?input,?because:??
  26. ????//?-?we?had?to?read?more?data?from?the?source?stream,?or??
  27. ????//?-?the?source?stream?has?ended.??
  28. } ?
?? ?函数中首先调用了MPEGVideoStreamParser::parse函数，将一个完整的Frame分析出来，并copy到了fTo(fTo就是OutPacketBuffer中的缓冲区)中，这其中肯定也实现了从文件中读取数据的过程。这里的fNumTruncatedBytes变量需要注意，fNumTruncatedBytes>0的话，表明Frame的实际长度大于fTo的最大长度，这将导致数据丢失，这时就要考虑增加缓冲区的长度了。成功获取一个Frame后，将调用afterGetting函数处理后续工作。
?? ?先来看parse函数，parse是定义在MPEGVideoStreamParser中的纯虚函数，在子类H264VideoStreamParser中实现。parse主要是从文件的字节流中，分离出一个个的Frame,对于H264而言其实就是对一个个的NALU。*.264文件的格式非常简单，每个NALU以 0x00000001 作为起始符号(中间的NALU也可以以0x000001作为起始符)，顺序存放。
[cpp]? view plain ?copy
?print ?
  1. unsigned?H264VideoStreamParser::parse()?{??
  2. ??try?{??
  3. ????//首先找到起始符号,?并跳过。文件流的最开始必需以0x00000001开始，但后续的NALU充许以0x000001(3?bytes)作为分隔??
  4. ????//?The?stream?must?start?with?a?0x00000001:??
  5. ????if?(!fHaveSeenFirstStartCode)?{??
  6. ??????//?Skip?over?any?input?bytes?that?precede?the?first?0x00000001:??
  7. ??????u_int32_t?first4Bytes;??
  8. ??????while?((first4Bytes?=?test4Bytes())?!=?0x00000001)?{??
  9. ????get1Byte();?setParseState();?//?ensures?that?we?progress?over?bad?data??
  10. ??????skipBytes(4);?//?skip?this?initial?code??
  11. ????????
  12. ??????setParseState();??
  13. ??????fHaveSeenFirstStartCode?=?True;?//?from?now?on??
  14. ????//将起始标志也保存到目的缓冲区中??
  15. ????if?(fOutputStartCodeSize?>?0)?{??
  16. ??????//?Include?a?start?code?in?the?output:??
  17. ??????save4Bytes(0x00000001);?????
  18. ????//保存所有数据，直至遇到起始标志，或者文件结束符。需要注意NALU中的第一个字节，因为它指示了NALU的类型??
  19. ????//?Then?save?everything?up?until?the?next?0x00000001?(4?bytes)?or?0x000001?(3?bytes),?or?we?hit?EOF.??
  20. ????//?Also?make?note?of?the?first?byte,?because?it?contains?the?"nal_unit_type":???
  21. ????u_int8_t?firstByte;??
  22. ????if?(haveSeenEOF())?{??
  23. ???????//??
  24. ???????//已经设置了文件结束标志，将剩下的数据保存也来即可??
  25. ??????//?We?hit?EOF?the?last?time?that?we?tried?to?parse?this?data,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ??????//?so?we?know?that?the?remaining?unparsed?data?forms?a?complete?NAL?unit:??
  26. ??????unsigned?remainingDataSize?=?totNumValidBytes()?-?curOffset();??
  27. ??????if?(remainingDataSize?==?0)?(void)get1Byte();?//?forces?another?read,?which?will?cause?EOF?to?get?handled?for?real?this?time??
  28. #ifdef?DEBUG??
  29. ??????fprintf(stderr,?"This?NAL?unit?(%d?bytes)?ends?with?EOFn",?remainingDataSize);??
  30. #endif??
  31. ??????if?(remainingDataSize?==?0)?return?0;??
  32. ??????firstByte?=?get1Byte();???//将第一个字节保存下来，其指示了NALU的类型??
  33. ??????saveByte(firstByte);??
  34. ??????while?(--remainingDataSize?>?0)?{??
  35. ????saveByte(get1Byte());??
  36. ??????u_int32_t?next4Bytes?=?test4Bytes();??
  37. ??????firstByte?=?next4Bytes>>24;???//将第一个字节保存下来??
  38. ??????//将下一个起始符号之前的数据都保存下来??
  39. ??????while?(next4Bytes?!=?0x00000001?&&?(next4Bytes&0xFFFFFF00)?!=?0x00000100)?{??
  40. ????//?We?save?at?least?some?of?"next4Bytes".??
  41. ????if?((unsigned)(next4Bytes&0xFF)?>?1)?{??//一次可以保存4个字节,并不需要一个一个字节对比，除非到了结尾??
  42. ??????//?Common?case:?0x00000001?or?0x000001?definitely?doesn't?begin?anywhere?in?"next4Bytes",?so?we?save?all?of?it:??
  43. ??????save4Bytes(next4Bytes);??
  44. ??????skipBytes(4);??
  45. ??????//?Save?the?first?byte,?and?continue?testing?the?rest:??
  46. ??????saveByte(next4Bytes>>24);??
  47. ??????skipBytes(1);??
  48. ????next4Bytes?=?test4Bytes();??
  49. ??????//?Assert:?next4Bytes?starts?with?0x00000001?or?0x000001,?and?we've?saved?all?previous?bytes?(forming?a?complete?NAL?unit).??
  50. ??????//?Skip?over?these?remaining?bytes,?up?until?the?start?of?the?next?NAL?unit:??
  51. ??????if?(next4Bytes?==?0x00000001)?{??
  52. ????skipBytes(4);??
  53. ????skipBytes(3);??
  54. ???
  55. ????u_int8_t?nal_ref_idc?=?(firstByte&0x60)>>5;??
  56. ????u_int8_t?nal_unit_type?=?firstByte&0x1F;??
  57. "Parsed?%d-byte?NAL-unit?(nal_ref_idc:?%d,?nal_unit_type:?%d?("%s"))n",153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ????????curFrameSize()-fOutputStartCodeSize,?nal_ref_idc,?nal_unit_type,?nal_unit_type_description[nal_unit_type]);??
  58. ????//下面根据NALU的类型来作具体的分析??
  59. ????switch?(nal_unit_type)?{??
  60. ??????case?6:?{?//?Supplemental?enhancement?information?(SEI)??
  61. ????analyze_sei_data();??
  62. ????//?Later,?perhaps?adjust?"fPresentationTime"?if?we?saw?a?"pic_timing"?SEI?payload????#####??
  63. ????break;??
  64. ??????case?7:?{?//?Sequence?parameter?set?(序列参数集)??
  65. ??????//??
  66. ??????//保存一份SPS的副本到H264VideoStreamFramer中，后面的pps也需要保存，sps中可能还包含了帧率信息??
  67. ????//?First,?save?a?copy?of?this?NAL?unit,?in?case?the?downstream?object?wants?to?see?it:??
  68. ????usingSource()->saveCopyOfSPS(fStartOfFrame?+?fOutputStartCodeSize,?fTo?-?fStartOfFrame?-?fOutputStartCodeSize);??
  69. ????//?Parse?this?NAL?unit?to?check?whether?frame?rate?information?is?present:??
  70. ????unsigned?num_units_in_tick,?time_scale,?fixed_frame_rate_flag;??
  71. ????analyze_seq_parameter_set_data(num_units_in_tick,?fixed_frame_rate_flag);??
  72. ????if?(time_scale?>?0?&&?num_units_in_tick?>?0)?{??
  73. ??????usingSource()->fFrameRate?=?time_scale/(2.0*num_units_in_tick);???//sps中包含了帧率信息??
  74. "Set?frame?rate?to?%f?fpsn",?usingSource()->fFrameRate);??
  75. ??????if?(fixed_frame_rate_flag?==?0)?{??
  76. ????????fprintf(stderr,?"tWARNING:?"fixed_frame_rate_flag"?was?not?setn");??
  77. ????}?else?{????//sps中不包含帧率信息，则使用source中设置的默认帧率??
  78. "tThis?"Sequence?Parameter?Set"?NAL?unit?contained?no?frame?rate?information,?so?we?use?a?default?frame?rate?of?%f?fpsn",?usingSource()->fFrameRate);??
  79. ??????case?8:?{?//?Picture?parameter?set?(图像参数集PPS)??
  80. ????//?Save?a?copy?of?this?NAL?unit,?in?case?the?downstream?object?wants?to?see?it:??
  81. ????usingSource()->saveCopyOfPPS(fStartOfFrame?+?fOutputStartCodeSize,?fTo?-?fStartOfFrame?-?fOutputStartCodeSize);??
  82. ??????
  83. ????usingSource()->setPresentationTime();???//设置当前的时间??
  84. ????unsigned?long?secs?=?(unsigned?long)usingSource()->fPresentationTime.tv_sec;??
  85. ????unsigned?uSecs?=?(unsigned)usingSource()->fPresentationTime.tv_usec;??
  86. "tPresentation?time:?%lu.%06un",?secs,?uSecs);??
  87. ????//如果这个NALU是一个VCL?NALU(即包含的是视频数据)，我们需要扫描下一个NALU的起始符，??
  88. ????//以判断这个NALU是否是当前的"access?unit"(这个"access?unit"应该可以理解为一帧图像帧吧)。??
  89. ????//我们需要根据这个信息去指明何时该增加"fPresentationTime"(RTP?打包时也需要根据这个信息，决定是否设置"M"位)。??
  90. ????//?If?this?NAL?unit?is?a?VCL?NAL?unit,?we?also?scan?the?start?of?the?next?NAL?unit,?to?determine?whether?this?NAL?unit??
  91. ????//?ends?the?current?'access?unit'.??We?need?this?information?to?figure?out?when?to?increment?"fPresentationTime".??
  92. ????//?(RTP?streamers?also?need?to?know?this?in?order?to?figure?out?whether?or?not?to?set?the?"M"?bit.)??
  93. ????Boolean?thisNALUnitEndsAccessUnit?=?False;?//?until?we?learn?otherwise???
  94. ????if?(haveSeenEOF())?{??
  95. ??????//?There?is?no?next?NAL?unit,?so?we?assume?that?this?one?ends?the?current?'access?unit':??
  96. ??????thisNALUnitEndsAccessUnit?=?True;??
  97. ??????Boolean?const?isVCL?=?nal_unit_type?<=?5?&&?nal_unit_type?>?0;?//?Would?need?to?include?type?20?for?SVC?and?MVC?#####??
  98. ??????if?(isVCL)?{??
  99. ????u_int32_t?first4BytesOfNextNALUnit?=?test4Bytes();??
  100. ????u_int8_t?firstByteOfNextNALUnit?=?first4BytesOfNextNALUnit>>24;??
  101. ????u_int8_t?next_nal_ref_idc?=?(firstByteOfNextNALUnit&0x60)>>5;??
  102. ????u_int8_t?next_nal_unit_type?=?firstByteOfNextNALUnit&0x1F;??
  103. ????if?(next_nal_unit_type?>=?6)?{??
  104. ????????//下一个NALU不是VCL的，当前的"access?unit"结束了??
  105. ??????//?The?next?NAL?unit?is?not?a?VCL;?therefore,?we?assume?that?this?NAL?unit?ends?the?current?'access?unit':??
  106. "t(The?next?NAL?unit?is?not?a?VCL)n");??
  107. ????????//下一个NALU也是VCL的，还需要检查一下它们是不是属于同一个"access?unit"??
  108. ??????//?The?next?NAL?unit?is?also?a?VLC.??We?need?to?examine?it?a?little?to?figure?out?if?it's?a?different?'access?unit'.??
  109. ??????//?(We?use?many?of?the?criteria?described?in?section?7.4.1.2.4?of?the?H.264?specification.)??
  110. ??????Boolean?IdrPicFlag?=?nal_unit_type?==?5;??
  111. ??????Boolean?next_IdrPicFlag?=?next_nal_unit_type?==?5;??
  112. ??????if?(next_IdrPicFlag?!=?IdrPicFlag)?{??
  113. ????????//?IdrPicFlag?differs?in?value??
  114. "t(IdrPicFlag?differs?in?value)n");??
  115. ????????thisNALUnitEndsAccessUnit?=?True;??
  116. ??????}?else?if?(next_nal_ref_idc?!=?nal_ref_idc?&&?next_nal_ref_idc*nal_ref_idc?==?0)?{??
  117. ????????//?nal_ref_idc?differs?in?value?with?one?of?the?nal_ref_idc?values?being?equal?to?0??
  118. "t(nal_ref_idc?differs?in?value?with?one?of?the?nal_ref_idc?values?being?equal?to?0)n");??
  119. ??????}?else?if?((nal_unit_type?==?1?||?nal_unit_type?==?2?||?nal_unit_type?==?5)??
  120. ?????????????&&?(next_nal_unit_type?==?1?||?next_nal_unit_type?==?2?||?next_nal_unit_type?==?5))?{??
  121. ????????//?Both?this?and?the?next?NAL?units?begin?with?a?"slice_header".??
  122. ????????//?Parse?this?(for?each),?to?get?parameters?that?we?can?compare:??
  123. ??????????
  124. ????????//?Current?NAL?unit's?"slice_header":??
  125. ????????unsigned?frame_num,?pic_parameter_set_id,?idr_pic_id;??
  126. ????????Boolean?field_pic_flag,?bottom_field_flag;??
  127. ????????analyze_slice_header(fStartOfFrame?+?fOutputStartCodeSize,?fTo,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ?????????????????frame_num,?idr_pic_id,?field_pic_flag,?bottom_field_flag);??
  128. ????????//?Next?NAL?unit's?"slice_header":??
  129. "????Next?NAL?unit's?slice_header:n");??
  130. ????????u_int8_t?next_slice_header[NUM_NEXT_SLICE_HEADER_BYTES_TO_ANALYZE];??
  131. ????????testBytes(next_slice_header,?sizeof?next_slice_header);??
  132. ????????unsigned?next_frame_num,?next_pic_parameter_set_id,?next_idr_pic_id;??
  133. ????????Boolean?next_field_pic_flag,?next_bottom_field_flag;??
  134. ????????analyze_slice_header(next_slice_header,?&next_slice_header[sizeof?next_slice_header],?next_nal_unit_type,153); list-style:decimal-leading-zero outside; line-height:18px; margin:0px!important; padding:0px 3px 0px 10px!important"> ?????????????????next_frame_num,?next_idr_pic_id,?next_field_pic_flag,?next_bottom_field_flag);??
  135. ??????????
  136. ????????if?(next_frame_num?!=?frame_num)?{??
  137. ??????????//?frame_num?differs?in?value??
  138. ??????????fprintf(stderr,?"t(frame_num?differs?in?value)n");??
  139. ??????????thisNALUnitEndsAccessUnit?=?True;??
  140. ????????}?else?if?(next_pic_parameter_set_id?!=?pic_parameter_set_id)?{??
  141. ??????????//?pic_parameter_set_id?differs?in?value??
  142. "t(pic_parameter_set_id?differs?in?value)n");??
  143. ????????}?else?if?(next_field_pic_flag?!=?field_pic_flag)?{??
  144. ??????????//?field_pic_flag?differs?in?value??
  145. "t(field_pic_flag?differs?in?value)n");??
  146. ????????}?else?if?(next_bottom_field_flag?!=?bottom_field_flag)?{??
  147. ??????????//?bottom_field_flag?differs?in?value??
  148. "t(bottom_field_flag?differs?in?value)n");??
  149. ????????}?else?if?(next_IdrPicFlag?==?1?&&?next_idr_pic_id?!=?idr_pic_id)?{??
  150. ??????????//?IdrPicFlag?is?equal?to?1?for?both?and?idr_pic_id?differs?in?value??
  151. ??????????//?Note:?We?already?know?that?IdrPicFlag?is?the?same?for?both.??
  152. "t(IdrPicFlag?is?equal?to?1?for?both?and?idr_pic_id?differs?in?value)n");??
  153. ??????????thisNALUnitEndsAccessUnit?=?True;??
  154. ????????}??
  155. ????if?(thisNALUnitEndsAccessUnit)?{??
  156. "*****This?NAL?unit?ends?the?current?access?unit*****n");??
  157. ??????usingSource()->fPictureEndMarker?=?True;??//这里就是设置RTP打包时用到的M标志了??
  158. ??????++usingSource()->fPictureCount;??
  159. ??????//下一个NALU不再属于当前"access　unit""时，才改变时间??
  160. ??????//?Note?that?the?presentation?time?for?the?next?NAL?unit?will?be?different:??
  161. ??????struct?timeval&?nextPT?=?usingSource()->fNextPresentationTime;?//?alias　这里是引用　??
  162. ??????nextPT?=?usingSource()->fPresentationTime;??
  163. ??????double?nextFraction?=?nextPT.tv_usec/1000000.0?+?1/usingSource()->fFrameRate;???
  164. ??????unsigned?nextSecsIncrement?=?(long)nextFraction;??
  165. ??????nextPT.tv_sec?+=?(long)nextSecsIncrement;??
  166. ??????nextPT.tv_usec?=?(long)((nextFraction?-?nextSecsIncrement)*1000000);??
  167. ????setParseState();??
  168. ????return?curFrameSize();??
  169. ??}?catch?(int?/*e*/)?{??
  170. "H264VideoStreamParser::parse()?EXCEPTION?(This?is?normal?behavior?-?*not*?an?error)n");??
  171. ????return?0;??//?the?parsing?got?interrupted??
  172. }??

?? ?H264VideoStreamParser::parse()函数除了取出Frame，还对NALU中的部分参数做了解释工作。对于PPS或者SPS类型的NALU，要保存到H264VideoStreamFramer中。"access unit",在这里可以理解为一副图像,一个"access unit"可以包含多个NALU，很显示这些NALU的时间戳应该是相同的。实际上，很多时候一个"access unit"单元只包含一个NALU，这里简单多了。分析过程是根据section 7.4.1.2.4 of the H.264 specification进行的。

H264数据处理完成后就可以进行RTP的打包与发送了，见下篇live555源码分析----RTP的打包与发送

原文链接：http://www.voidcn.com/article/p-ytchopga-qs.html

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!
    相关内容
      推荐文章
        站长推荐
        热点阅读

          【免责声明】本站内容转载自互联网，其发布内容言论不代表本站观点，如果其链接、内容的侵犯您的权益，烦请提交相关链接至邮箱bqsm@foxmail.com我们将及时予以处理。

          建议您使用1920×1080分辨率、谷歌浏览器Google Chrome、Microsoft Edge以获得本站的最佳浏览效果

          Copygight © 2008-2022 https://www.lidatong.com.cn/ All Rights Reserved. 李大同