带有 mediacodec 表面渲染的 FFMpeg (libav) hls demux 播放速度太快

FFMpeg (libav) hls demux with mediacodec surface rendering is playing too fast

提问人:forlayo 提问时间:10/17/2023 最后编辑:genpfaultforlayo 更新时间:10/18/2023 访问量:68

问:

我正在使用 libav 解复用 m3u8 并获取数据包来解码它们 + 播放,解码器是 mediacodec,其表面设置为直接渲染。到目前为止,这仍然有效,但视频播放速度太快,速度极快;尝试使用我发送到解码器的 pts/dts 而没有成功。

我看到其他实现正在这样做,但它们没有使用带有 mediacodec 的直接渲染,然后它们像已经解码的图像队列一样进行维护。由于我的实现旨在在没有太多资源的情况下在android上运行,因此我需要尽可能多地节省处理,因此必须直接渲染。

在这里,我放了一个我所做的事情的示例代码:

int MediaParserFFmpeg::init(const std::string &filePath)
{
    m_formatCtx = avformat_alloc_context();
    int ret = 0;
    
    ret = avformat_open_input(&m_formatCtx, m_mediaPath.c_str(), iFormat, nullptr);
    if (ret < 0)
    {
        return MEDIA_ERROR_INIT_FAILED;
    }

    ret = avformat_find_stream_info(m_formatCtx, nullptr);
    if (ret < 0)
    {
        return MEDIA_ERROR_INIT_FAILED;
    }

    m_duration = m_formatCtx->duration;
    const AVCodec *vcodec = nullptr;
    m_vStreamIdx = av_find_best_stream(m_formatCtx,
                                        AVMEDIA_TYPE_VIDEO,
                                        -1,
                                        -1,
                                        &vcodec,
                                        0);


    AVCodecParameters *codecParam = m_formatCtx->streams[m_vStreamIdx]->codecpar;
    m_width = codecParam->width;
    m_height = codecParam->height;
    m_vCodec = codecParam->codec_id;
    AVRational fpsRatio = m_formatCtx->streams[m_vStreamIdx]->avg_frame_rate;
    m_frameRate = fpsRatio.num / (float)fpsRatio.den;

    if (codecParam->codec_id == AV_CODEC_ID_H264)
    {
        // if stored in ANNEXB type convert to NALU
        m_isNal = codecParam->extradata_size > 0 && *(codecParam->extradata) == 1;
        if (m_isNal)
        {
            const AVBitStreamFilter *bsfFilter = av_bsf_get_by_name("h264_mp4toannexb");
            if (!bsfFilter)
            {
                return MEDIA_ERROR_INIT_FAILED;
            }
            av_bsf_alloc(bsfFilter, &m_bsfcContext);
            m_bsfcContext->par_in = codecParam;
            av_bsf_init(m_bsfcContext);

            m_bsfPacket = av_packet_alloc();
        }
    }

    m_packet = av_packet_alloc();
}

int MediaParserFFmpeg::getNextFrame(MediaType *mediaType, AVEncodedFrame **outFrame)
{
    int ret = 0;
    ret = av_read_frame(m_formatCtx, m_packet);

    if (ret < 0 && ret != AVERROR_EOF)
    {
        av_packet_unref(m_packet);
        return MEDIA_ERROR_PARSER_PARSE_FAILED;
    }

    if (ret == AVERROR_EOF)
    {
        av_packet_unref(m_packet);
        return MEDIA_INFO_PARSER_END_OF_STREAM;
    }

    if (m_packet->data && m_packet->size)
    {
        if (m_vCodec == VIDEOCODEC_H264 && m_isNal)
        {
            int res = av_bsf_send_packet(m_bsfcContext, m_packet);
            if (res != 0)
            {
                MEDIA_LOG() << "ZError:" << "Error adding NAL to H264 stream";
            }

            if (res == 0)
            {
                *outFrame = new EncodedVideoFrame(m_bsfPacket->data,
                                                    m_bsfPacket->size,
                                                    getTimeStamp(m_bsfPacket, m_formatCtx->streams[m_vStreamIdx]),
                                                    m_width,
                                                    m_height,
                                                    true);
                *mediaType = MEDIA_TYPE_VIDEO;
            }
            av_packet_unref(m_bsfPacket);

            res = av_bsf_receive_packet(m_bsfcContext, m_bsfPacket);
        }
    }
    av_packet_unref(m_packet);
    return MEDIA_SUCCESS;
}

int64_t MediaParserFFmpeg::getTimeStamp(AVPacket *packet, AVStream *stream)
{
    if (packet->pts != (qint64)AV_NOPTS_VALUE)
    {
        return (1000000 * (packet->pts * ((float)stream->time_base.num / stream->time_base.den)));
    }
    else if (packet->dts != (qint64)AV_NOPTS_VALUE)
    {
        return (1000000 * (packet->dts * ((float)stream->time_base.num / stream->time_base.den)));
    }
    else
    {
        return 0;
    }
}

然后我有一个线程调用 getNextFrame() 获取一个新的 EncodedVideoFrame 并以这种方式将其推送到 MediaCodec 解码器:

AMediaCodec_queueInputBuffer(m_mediaCodec, index, 0, encodedFrame->m_frameSize, encodedFrame->m_timeStamp, 0);

正如我所说,它可以工作,但产生的视频播放速度非常快;已经尝试在没有运气的情况下玩 pts 和 dts(不确定我是否在那里做错了什么)。

关于如何解决这个问题吗?

c++ ffmpeg http-live-streaming android-mediacodec libav

评论


答:

0赞 forlayo 10/18/2023 #1

多亏了这个人+我办公室里的一个人(感谢我的朋友),我得到了解决方案,正如他们所指出的,您需要根据系统时钟或基于音频时钟创建一个等待,以等待每一帧的持续时间;否则,他会尽可能快地打球。

在这里,您可以根据系统时钟来解决这个问题。

void VideoDecoderMediaCodec::preRender(const int64_t presentationTimeUsec)
{

    qDebug() <<"Oh Oh, preRender() pts:"<<presentationTimeUsec<<" mPrevMonoUsec:"<<m_prevClockUsec<<" mPrevPresentUsec:"<<m_prevPtsUsec;
    int64_t OneMillion = 1000000;

    if (m_prevClockUsec == 0)
    {
        // Latch current values, then return immediately.
        m_prevClockUsec = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();
        m_prevPtsUsec = presentationTimeUsec;
    }
    else
    {
        int64_t frameDelta = 0;
        frameDelta = presentationTimeUsec - m_prevPtsUsec;
        if(frameDelta <0)
        {
            frameDelta = 0;
        }
        else if(frameDelta > 10 * OneMillion)
        {
            // 5 secs cap.
            frameDelta = 5 * OneMillion;
        }

        qDebug() <<"Oh Oh, preRender() frameDelta:"<<frameDelta;

        int64_t desiredUsec = m_prevClockUsec + frameDelta;
        int64_t nowUsec = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();

        qDebug() <<"Oh Oh, preRender() desiredUsec:"<<desiredUsec<<" nowUsec:"<<nowUsec;
        while (nowUsec < (desiredUsec - 100) /*&& mState == RUNNING*/) {
            long sleepTimeUsec = desiredUsec - nowUsec;
            if (sleepTimeUsec > 500000) {
                sleepTimeUsec = 500000;
            }

            qDebug() <<"Oh Oh, preRender() sleepTimeUsec:"<<sleepTimeUsec;
            std::this_thread::sleep_for(std::chrono::microseconds(sleepTimeUsec));
            nowUsec = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();
        }

        m_prevClockUsec += frameDelta;
        m_prevPtsUsec += frameDelta;
    }
}

在调用 AMediaCodec_releaseOutputBuffer 之前必须调用的

...
AMediaCodecBufferInfo info;
AMediaCodec_dequeueOutputBuffer(m_mediaCodec, &info, BUFFER_TIMEOUT_US);
...
preRender(info.presentationTimeUs);
AMediaCodec_releaseOutputBuffer(m_mediaCodec, status, m_isDirectRender);
...