提问人:forlayo 提问时间:10/17/2023 最后编辑:genpfaultforlayo 更新时间:10/18/2023 访问量:68
带有 mediacodec 表面渲染的 FFMpeg (libav) hls demux 播放速度太快
FFMpeg (libav) hls demux with mediacodec surface rendering is playing too fast
问:
我正在使用 libav 解复用 m3u8 并获取数据包来解码它们 + 播放,解码器是 mediacodec,其表面设置为直接渲染。到目前为止,这仍然有效,但视频播放速度太快,速度极快;尝试使用我发送到解码器的 pts/dts 而没有成功。
我看到其他实现正在这样做,但它们没有使用带有 mediacodec 的直接渲染,然后它们像已经解码的图像队列一样进行维护。由于我的实现旨在在没有太多资源的情况下在android上运行,因此我需要尽可能多地节省处理,因此必须直接渲染。
在这里,我放了一个我所做的事情的示例代码:
int MediaParserFFmpeg::init(const std::string &filePath)
{
m_formatCtx = avformat_alloc_context();
int ret = 0;
ret = avformat_open_input(&m_formatCtx, m_mediaPath.c_str(), iFormat, nullptr);
if (ret < 0)
{
return MEDIA_ERROR_INIT_FAILED;
}
ret = avformat_find_stream_info(m_formatCtx, nullptr);
if (ret < 0)
{
return MEDIA_ERROR_INIT_FAILED;
}
m_duration = m_formatCtx->duration;
const AVCodec *vcodec = nullptr;
m_vStreamIdx = av_find_best_stream(m_formatCtx,
AVMEDIA_TYPE_VIDEO,
-1,
-1,
&vcodec,
0);
AVCodecParameters *codecParam = m_formatCtx->streams[m_vStreamIdx]->codecpar;
m_width = codecParam->width;
m_height = codecParam->height;
m_vCodec = codecParam->codec_id;
AVRational fpsRatio = m_formatCtx->streams[m_vStreamIdx]->avg_frame_rate;
m_frameRate = fpsRatio.num / (float)fpsRatio.den;
if (codecParam->codec_id == AV_CODEC_ID_H264)
{
// if stored in ANNEXB type convert to NALU
m_isNal = codecParam->extradata_size > 0 && *(codecParam->extradata) == 1;
if (m_isNal)
{
const AVBitStreamFilter *bsfFilter = av_bsf_get_by_name("h264_mp4toannexb");
if (!bsfFilter)
{
return MEDIA_ERROR_INIT_FAILED;
}
av_bsf_alloc(bsfFilter, &m_bsfcContext);
m_bsfcContext->par_in = codecParam;
av_bsf_init(m_bsfcContext);
m_bsfPacket = av_packet_alloc();
}
}
m_packet = av_packet_alloc();
}
int MediaParserFFmpeg::getNextFrame(MediaType *mediaType, AVEncodedFrame **outFrame)
{
int ret = 0;
ret = av_read_frame(m_formatCtx, m_packet);
if (ret < 0 && ret != AVERROR_EOF)
{
av_packet_unref(m_packet);
return MEDIA_ERROR_PARSER_PARSE_FAILED;
}
if (ret == AVERROR_EOF)
{
av_packet_unref(m_packet);
return MEDIA_INFO_PARSER_END_OF_STREAM;
}
if (m_packet->data && m_packet->size)
{
if (m_vCodec == VIDEOCODEC_H264 && m_isNal)
{
int res = av_bsf_send_packet(m_bsfcContext, m_packet);
if (res != 0)
{
MEDIA_LOG() << "ZError:" << "Error adding NAL to H264 stream";
}
if (res == 0)
{
*outFrame = new EncodedVideoFrame(m_bsfPacket->data,
m_bsfPacket->size,
getTimeStamp(m_bsfPacket, m_formatCtx->streams[m_vStreamIdx]),
m_width,
m_height,
true);
*mediaType = MEDIA_TYPE_VIDEO;
}
av_packet_unref(m_bsfPacket);
res = av_bsf_receive_packet(m_bsfcContext, m_bsfPacket);
}
}
av_packet_unref(m_packet);
return MEDIA_SUCCESS;
}
int64_t MediaParserFFmpeg::getTimeStamp(AVPacket *packet, AVStream *stream)
{
if (packet->pts != (qint64)AV_NOPTS_VALUE)
{
return (1000000 * (packet->pts * ((float)stream->time_base.num / stream->time_base.den)));
}
else if (packet->dts != (qint64)AV_NOPTS_VALUE)
{
return (1000000 * (packet->dts * ((float)stream->time_base.num / stream->time_base.den)));
}
else
{
return 0;
}
}
然后我有一个线程调用 getNextFrame() 获取一个新的 EncodedVideoFrame 并以这种方式将其推送到 MediaCodec 解码器:
AMediaCodec_queueInputBuffer(m_mediaCodec, index, 0, encodedFrame->m_frameSize, encodedFrame->m_timeStamp, 0);
正如我所说,它可以工作,但产生的视频播放速度非常快;已经尝试在没有运气的情况下玩 pts 和 dts(不确定我是否在那里做错了什么)。
关于如何解决这个问题吗?
答:
0赞
forlayo
10/18/2023
#1
多亏了这个人+我办公室里的一个人(感谢我的朋友),我得到了解决方案,正如他们所指出的,您需要根据系统时钟或基于音频时钟创建一个等待,以等待每一帧的持续时间;否则,他会尽可能快地打球。
在这里,您可以根据系统时钟来解决这个问题。
void VideoDecoderMediaCodec::preRender(const int64_t presentationTimeUsec)
{
qDebug() <<"Oh Oh, preRender() pts:"<<presentationTimeUsec<<" mPrevMonoUsec:"<<m_prevClockUsec<<" mPrevPresentUsec:"<<m_prevPtsUsec;
int64_t OneMillion = 1000000;
if (m_prevClockUsec == 0)
{
// Latch current values, then return immediately.
m_prevClockUsec = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();
m_prevPtsUsec = presentationTimeUsec;
}
else
{
int64_t frameDelta = 0;
frameDelta = presentationTimeUsec - m_prevPtsUsec;
if(frameDelta <0)
{
frameDelta = 0;
}
else if(frameDelta > 10 * OneMillion)
{
// 5 secs cap.
frameDelta = 5 * OneMillion;
}
qDebug() <<"Oh Oh, preRender() frameDelta:"<<frameDelta;
int64_t desiredUsec = m_prevClockUsec + frameDelta;
int64_t nowUsec = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();
qDebug() <<"Oh Oh, preRender() desiredUsec:"<<desiredUsec<<" nowUsec:"<<nowUsec;
while (nowUsec < (desiredUsec - 100) /*&& mState == RUNNING*/) {
long sleepTimeUsec = desiredUsec - nowUsec;
if (sleepTimeUsec > 500000) {
sleepTimeUsec = 500000;
}
qDebug() <<"Oh Oh, preRender() sleepTimeUsec:"<<sleepTimeUsec;
std::this_thread::sleep_for(std::chrono::microseconds(sleepTimeUsec));
nowUsec = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();
}
m_prevClockUsec += frameDelta;
m_prevPtsUsec += frameDelta;
}
}
在调用 AMediaCodec_releaseOutputBuffer 之前必须调用的
...
AMediaCodecBufferInfo info;
AMediaCodec_dequeueOutputBuffer(m_mediaCodec, &info, BUFFER_TIMEOUT_US);
...
preRender(info.presentationTimeUs);
AMediaCodec_releaseOutputBuffer(m_mediaCodec, status, m_isDirectRender);
...
评论