Discussion:
[PATCH 0/6] Add decoding/encoding API with decoupled input/output
(too old to reply)
wm4
2016-02-27 14:15:29 UTC
Permalink
This has been often discussed, and everyone agreed it was a good idea.
Here are patches which add such an API, but without making proper use
of it yet.

wm4 (6):
lavu: improve documentation of some AVFrame functions
lavc: factor apply_param_change() AV_EF_EXPLODE handling
lavc: introduce a new decoding/encoding API with decoupled
input/output
avconv: remove sub-frame warning
avconv: use new decode API
avconv: use new encode API

avconv.c | 82 ++++++++++-----
avconv.h | 1 -
libavcodec/avcodec.h | 221 ++++++++++++++++++++++++++++++++++++++++
libavcodec/internal.h | 12 +++
libavcodec/utils.c | 276 +++++++++++++++++++++++++++++++++++++++++++++++---
libavutil/frame.h | 14 ++-
6 files changed, 562 insertions(+), 44 deletions(-)
--
2.7.0
wm4
2016-02-27 14:15:30 UTC
Permalink
---
libavutil/frame.h | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/libavutil/frame.h b/libavutil/frame.h
index 5a04177..6f5d733 100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -391,6 +391,10 @@ void av_frame_free(AVFrame **frame);
* If src is not reference counted, new buffers are allocated and the data is
* copied.
*
+ * @warning: dst MUST have been either unreferenced with av_frame_unref(dst),
+ * or newly allocated with av_frame_alloc() before calling this
+ * function, or undefined behavior will occur.
+ *
* @return 0 on success, a negative AVERROR on error
*/
int av_frame_ref(AVFrame *dst, const AVFrame *src);
@@ -410,7 +414,11 @@ AVFrame *av_frame_clone(const AVFrame *src);
void av_frame_unref(AVFrame *frame);

/**
- * Move everythnig contained in src to dst and reset src.
+ * Move everything contained in src to dst and reset src.
+ *
+ * @warning: dst is not unreferenced, but blindly overwritten without checking
+ * or deallocating its contents. Call av_frame_unref(dst) manually
+ * before calling this function to ensure that no memory is leaked.
*/
void av_frame_move_ref(AVFrame *dst, AVFrame *src);

@@ -426,6 +434,10 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src);
* necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf.
* For planar formats, one buffer will be allocated for each plane.
*
+ * @warning: if frame already has been allocated, calling this function will
+ * leak memory. In addition, undefined behavior can occur in certain
+ * cases.
+ *
* @param frame frame in which to store the new buffers.
* @param align required buffer size alignment
*
--
2.7.0
Luca Barbato
2016-02-27 14:21:24 UTC
Permalink
Post by wm4
In addition, undefined behavior can occur in certain
+ * cases.
Would be better to list the cases.

The rest looks fine.

lu
wm4
2016-02-27 15:07:10 UTC
Permalink
On Sat, 27 Feb 2016 15:21:24 +0100
Post by Luca Barbato
Post by wm4
In addition, undefined behavior can occur in certain
+ * cases.
Would be better to list the cases.
The rest looks fine.
The code of these functions is simply written with the assumption that
the dst frame was unreffed. So depending on which fields are set in
dst, arbitrary behavior would result. I don't think documenting what
could happen would make sense.
Luca Barbato
2016-02-27 15:51:21 UTC
Permalink
Post by wm4
On Sat, 27 Feb 2016 15:21:24 +0100
Post by Luca Barbato
Post by wm4
In addition, undefined behavior can occur in certain
+ * cases.
Would be better to list the cases.
The rest looks fine.
The code of these functions is simply written with the assumption that
the dst frame was unreffed. So depending on which fields are set in
dst, arbitrary behavior would result. I don't think documenting what
could happen would make sense.
I'd reword it like

"The destination AVFrame must be non-allocated, calling this function on
already-allocated frames leads to memory leaks and other unwanted
behaviours."

lu
Vittorio Giovara
2016-02-27 15:48:24 UTC
Permalink
Post by wm4
---
libavutil/frame.h | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/libavutil/frame.h b/libavutil/frame.h
index 5a04177..6f5d733 100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -391,6 +391,10 @@ void av_frame_free(AVFrame **frame);
* If src is not reference counted, new buffers are allocated and the data is
* copied.
*
+ * or newly allocated with av_frame_alloc() before calling this
+ * function, or undefined behavior will occur.
don't you just have to make sure that the frame is allocated?
what undefined behaviour are we talking about?
Post by wm4
+ *
*/
int av_frame_ref(AVFrame *dst, const AVFrame *src);
@@ -410,7 +414,11 @@ AVFrame *av_frame_clone(const AVFrame *src);
void av_frame_unref(AVFrame *frame);
/**
- * Move everythnig contained in src to dst and reset src.
+ * Move everything contained in src to dst and reset src.
+ *
it could do without 'blindly' but ok otherwise
Post by wm4
+ * or deallocating its contents. Call av_frame_unref(dst) manually
+ * before calling this function to ensure that no memory is leaked.
*/
void av_frame_move_ref(AVFrame *dst, AVFrame *src);
@@ -426,6 +434,10 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src);
* necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf.
* For planar formats, one buffer will be allocated for each plane.
*
+ * leak memory. In addition, undefined behavior can occur in certain
+ * cases.
ok
--
Vittorio
wm4
2016-02-29 21:44:07 UTC
Permalink
On Sat, 27 Feb 2016 10:48:24 -0500
Post by Vittorio Giovara
Post by wm4
---
libavutil/frame.h | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/libavutil/frame.h b/libavutil/frame.h
index 5a04177..6f5d733 100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -391,6 +391,10 @@ void av_frame_free(AVFrame **frame);
* If src is not reference counted, new buffers are allocated and the data is
* copied.
*
+ * or newly allocated with av_frame_alloc() before calling this
+ * function, or undefined behavior will occur.
don't you just have to make sure that the frame is allocated?
what undefined behaviour are we talking about?
It calls av_frame_copy_props(), which adds side data to dst without
checking that the side data already exists. So you can have duplicate
side data. It assigns dst->buf[i] entries without freeing it if it's
already set, and unneeded entries are not cleared (so they stay
around). Ironically, av_frame_unref(dst) is called on _some_ error
paths.

So, either follow the doxygen strictly, or expect random behavior.
If you don't follow it, you either get memory leaks or "real" C
undefined behavior.
Post by Vittorio Giovara
Post by wm4
+ *
*/
int av_frame_ref(AVFrame *dst, const AVFrame *src);
@@ -410,7 +414,11 @@ AVFrame *av_frame_clone(const AVFrame *src);
void av_frame_unref(AVFrame *frame);
/**
- * Move everythnig contained in src to dst and reset src.
+ * Move everything contained in src to dst and reset src.
+ *
it could do without 'blindly' but ok otherwise
Changed.
Post by Vittorio Giovara
Post by wm4
+ * or deallocating its contents. Call av_frame_unref(dst) manually
+ * before calling this function to ensure that no memory is leaked.
*/
void av_frame_move_ref(AVFrame *dst, AVFrame *src);
@@ -426,6 +434,10 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src);
* necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf.
* For planar formats, one buffer will be allocated for each plane.
*
+ * leak memory. In addition, undefined behavior can occur in certain
+ * cases.
ok
wm4
2016-02-27 14:15:31 UTC
Permalink
Remove the duplicated code for handling failure of apply_param_change().
---
libavcodec/utils.c | 28 +++++++++++++++-------------
1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/libavcodec/utils.c b/libavcodec/utils.c
index a9a7423..866cdfc 100644
--- a/libavcodec/utils.c
+++ b/libavcodec/utils.c
@@ -1390,7 +1390,8 @@ static int apply_param_change(AVCodecContext *avctx, AVPacket *avpkt)
if (!(avctx->codec->capabilities & AV_CODEC_CAP_PARAM_CHANGE)) {
av_log(avctx, AV_LOG_ERROR, "This decoder does not support parameter "
"changes, but PARAM_CHANGE side data was sent to it.\n");
- return AVERROR(EINVAL);
+ ret = AVERROR(EINVAL);
+ goto fail2;
}

if (size < 4)
@@ -1425,13 +1426,20 @@ static int apply_param_change(AVCodecContext *avctx, AVPacket *avpkt)
size -= 8;
ret = ff_set_dimensions(avctx, avctx->width, avctx->height);
if (ret < 0)
- return ret;
+ goto fail2;
}

return 0;
fail:
av_log(avctx, AV_LOG_ERROR, "PARAM_CHANGE side data too small.\n");
- return AVERROR_INVALIDDATA;
+ ret = AVERROR_INVALIDDATA;
+fail2:
+ if (ret < 0) {
+ av_log(avctx, AV_LOG_ERROR, "Error applying parameter changes.\n");
+ if (avctx->err_recognition & AV_EF_EXPLODE)
+ return ret;
+ }
+ return 0;
}

static int unrefcount_frame(AVCodecInternal *avci, AVFrame *frame)
@@ -1492,11 +1500,8 @@ int attribute_align_arg avcodec_decode_video2(AVCodecContext *avctx, AVFrame *pi

avctx->internal->pkt = avpkt;
ret = apply_param_change(avctx, avpkt);
- if (ret < 0) {
- av_log(avctx, AV_LOG_ERROR, "Error applying parameter changes.\n");
- if (avctx->err_recognition & AV_EF_EXPLODE)
- return ret;
- }
+ if (ret < 0)
+ return ret;

av_frame_unref(picture);

@@ -1560,11 +1565,8 @@ int attribute_align_arg avcodec_decode_audio4(AVCodecContext *avctx,
}

ret = apply_param_change(avctx, avpkt);
- if (ret < 0) {
- av_log(avctx, AV_LOG_ERROR, "Error applying parameter changes.\n");
- if (avctx->err_recognition & AV_EF_EXPLODE)
- return ret;
- }
+ if (ret < 0)
+ return ret;

av_frame_unref(frame);
--
2.7.0
Luca Barbato
2016-02-27 14:22:34 UTC
Permalink
Post by wm4
Remove the duplicated code for handling failure of apply_param_change().
---
libavcodec/utils.c | 28 +++++++++++++++-------------
1 file changed, 15 insertions(+), 13 deletions(-)
Looks fine to me.
wm4
2016-02-27 14:15:32 UTC
Permalink
Until now, the decoding API was restricted to outputting 0 or 1 frames
per input packet. It also enforces a somewhat rigid dataflow in general.

This new API seeks to relax these restrictions by decoupling input and
output. Instead of doing a single call on each decode step, which may
consume the packet and may produce output, the new API requires the user
to send input first, and then ask for output.

For now, there are no codecs supporting this API. The API can work with
codecs using the old API, and most code added here it to make them
interopate. The reverse is not possible, although for audio it might.
---
libavcodec/avcodec.h | 221 ++++++++++++++++++++++++++++++++++++++++++++
libavcodec/internal.h | 12 +++
libavcodec/utils.c | 248 +++++++++++++++++++++++++++++++++++++++++++++++++-
3 files changed, 479 insertions(+), 2 deletions(-)

diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
index 33de8ec..6065dba 100644
--- a/libavcodec/avcodec.h
+++ b/libavcodec/avcodec.h
@@ -3090,6 +3090,21 @@ typedef struct AVCodec {
int (*decode)(AVCodecContext *, void *outdata, int *outdata_size, AVPacket *avpkt);
int (*close)(AVCodecContext *);
/**
+ * Decode/encode API with decoupled packet/frame dataflow. The API is the
+ * same as the avcodec_ prefixed APIs (avcodec_send_frame() etc.), except
+ * that:
+ * - never called if the codec is closed or the wrong type
+ * - AVPacket parameter change side data is applied right before calling
+ * AVCodec->send_packet
+ * - if AV_CODEC_CAP_DELAY is not set, drain packets or frames are never sent
+ * - only one drain packet is ever passed down (until the next flush())
+ * - a drain AVPacket is always NULL (no need to check for avpkt->size).
+ */
+ int (*send_frame)(AVCodecContext *avctx, const AVFrame *frame);
+ int (*send_packet)(AVCodecContext *avctx, const AVPacket *avpkt);
+ int (*receive_frame)(AVCodecContext *avctx, AVFrame *frame);
+ int (*receive_packet)(AVCodecContext *avctx, AVPacket *avpkt);
+ /**
* Flush buffers.
* Will be called when seeking
*/
@@ -4080,6 +4095,212 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub,
AVPacket *avpkt);

/**
+ * send/receive API overview (xxx: where does this go?)
+ *
+ * The avcodec_send_packet()/avcodec_receive_frame()/avcodec_send_frame()/
+ * avcodec_receive_packet() provide a encode/decode API, which decouples input
+ * and output.
+ *
+ * The API is very similar for encoding/decoding and audio/video, and works as
+ * follows:
+ * - Setup and open the AVCodecContext as usual.
+ * - Send valid input:
+ * - For decoding, call avcodec_send_packet() to give the decoder raw
+ * compressed data in an AVPacket.
+ * - For encoding, call avcodec_send_frame() to give the decoder an AVFrame
+ * containing uncompressed audio or video.
+ * In both cases, it's recommended that AVPackets and AVFrames are refcounted,
+ * or libavcodec might have to copy the input data. (libavformat always
+ * returns refcounted AVPackets, and av_frame_get_buffer() allocates
+ * refcounted AVFrames.)
+ * - Receive output in a loop. Periodically call one of the avcodec_receive_*()
+ * functions and process their output:
+ * - For decoding, call avcodec_receive_frame(). On success, it will return
+ * a AVFrame containing uncompressed audio or video data.
+ * - For encoding, call avcodec_receive_packet(). On success, it will return
+ * an AVPacket with a compressed frame.
+ * Repeat this call until it returns AVERROR(EAGAIN) or an error. The
+ * AVERROR(EAGAIN) return value means that new input data is required to
+ * return new output. In this case, continue with sending input. For each
+ * input frame/packet, the codec will typically return 1 output frame/packet,
+ * but it can also be 0 or more than 1.
+ *
+ * At the beginning of decoding or encoding, the codec might accept multiple
+ * input frames/packets without returning a frame, until its internal buffers
+ * are filled. This situation is handled transparently if you follow the steps
+ * outlined above.
+ *
+ * End of stream situations. These require "flushing" (aka draining) the codec,
+ * as the codec might buffer multiple frames or packets internally for
+ * performance or out of necessity (consider B-frames).
+ * This is handled as follows:
+ * - Instead of valid input, send NULL to the avcodec_send_packet() (decoding)
+ * or avcodec_send_frame() (encoding) functions. This will enter draining
+ * mode.
+ * - Call avcodec_receive_frame() (decoding) or avcodec_receive_packet()
+ * (encoding) in a loop until AVERROR_EOF is returned. The functions will
+ * not return AVERROR(EAGAIN), unless you forgot to enter draining mode.
+ * - Before decoding can be resumed again, the codec has to be reset with
+ * avcodec_flush_buffers().
+ *
+ * Using the API as outlined above is highly recommended. But it's also possible
+ * to call functions outside of this rigid schema. For example, you can call
+ * avcodec_send_packet() repeatedly without calling avcodec_receive_frame(). In
+ * this case, avcodec_send_packet() will succeed until the codec's internal
+ * buffer has been filled up (which is typically of size 1 per output frame,
+ * after initial input), and then reject input with AVERROR(EAGAIN). Once it
+ * starts rejecting input, you have no choice but to read at least some output.
+ *
+ * Not all codecs will follow a rigid and predictable dataflow; the only
+ * guarantee is that an AVERROR(EAGAIN) return value on a send/receive call on
+ * one end implies that a receive/send call on the other end will succeed. In
+ * general, no codec will permit unlimited buffering of input or output.
+ *
+ * This API replaces the following legacy functions:
+ * - avcodec_decode_video2() and avcodec_decode_audio4():
+ * Use avcodec_send_packet() to feed input to the decoder, then use
+ * avcodec_receive_frame() to receive decoded frames after each packet.
+ * Unlike with the old video decoding API, multiple frames might result from
+ * a packet. For audio, splitting the input packet into frames by partially
+ * decoding packets becomes transparent to the API user. You never need to
+ * feed an AVPacket to the API twice.
+ * Additionally, sending a flush/draining packet is required only once.
+ * - avcodec_encode_video2()/avcodec_encode_audio2():
+ * Use avcodec_send_frame() to feed input to the encoder, then use
+ * avcodec_receive_packet() to receive encoded packets.
+ * Providing user-allocated buffers for avcodec_receive_packet() is not
+ * possible.
+ * - The new API does not handle subtitles yet.
+ *
+ * Mixing new and old function calls on the same AVCodecContext is not allowed,
+ * and will result in arbitrary behavior.
+ *
+ * Some codecs might require using the new API; using the old API will return
+ * an error when calling it.
+ */
+
+/**
+ * Supply raw packet data as input to a decoder.
+ *
+ * Internally, this call will copy relevant AVCodecContext fields, which can
+ * influence decoding per-packet, and apply them when the packet is actually
+ * decoded. (For example AVCodecContext.skip_frame, which might direct the
+ * decoder to drop the frame contained by the packet sent with this function.)
+ *
+ * @warning The input buffer, avpkt->data must be AV_INPUT_BUFFER_PADDING_SIZE
+ * larger than the actual read bytes because some optimized bitstream
+ * readers read 32 or 64 bits at once and could read over the end.
+ *
+ * @warning Do not mix this API with the legacy API (like avcodec_decode_video2())
+ * on the same AVCodecContext. It will return unexpected results now
+ * or in future libavcodec versions.
+ *
+ * @note The AVCodecContext MUST have been opened with @ref avcodec_open2()
+ * before packets may be fed to the decoder.
+ *
+ * @param avctx codec context
+ * @param[in] avpkt The input AVPacket. Usually, this will be a single video
+ * frame, or several complete audio frames.
+ * Ownership of the packet remains with the caller, and the
+ * decoder will not write to the packet. The decoder may create
+ * a reference to the packet data (or copy it if the packet is
+ * not reference-counted).
+ * Unlike with older APIs, the packet is always fully consumed,
+ * and if it contains multiple frames (e.g. some audio codecs),
+ * will require you to call avcodec_receive_frame() multiple
+ * times afterwards before you can send a new packet.
+ * It can be NULL (or an AVPacket with data set to NULL and size
+ * set to 0); in this case, it's considered a flush packet, which
+ * signals the end of the stream. Sending the first flush
+ * packet will return success. Subsequent ones are unnecessary
+ * and will return AVERROR_EOF. If the decoder still has frames
+ * buffered, it will return them after sending a flush packet.
+ *
+ * @return 0 on success, otherwise negative error code:
+ * AVERROR(EAGAIN): input is not accepted right now - the packet must be
+ * resent after trying to read output
+ * AVERROR_EOF: the decoder has been flushed, and no new packets can
+ * be sent to it (also returned if more than 1 flush
+ * packet is sent)
+ * AVERROR(EINVAL): codec not opened, it's an encoder, or requires flush
+ * AVERROR(ENOMEM): failed to add packet to internal queue, or similar
+ * other errors: legitimate decoding errors
+ */
+int avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt);
+
+/**
+ * Return decoded output data from a decoder.
+ *
+ * @param avctx codec context
+ * @param frame This will be set to a reference-counted video or audio
+ * frame (depending on the decoder type) allocated by the
+ * decoder. Note that the function will always call
+ * av_frame_unref(frame) before doing anything else.
+ *
+ * @return
+ * 0: success, a frame was returned
+ * AVERROR(EAGAIN): output is not available right now - user must try
+ * to send new input
+ * AVERROR_EOF: the decoder has been fully flushed, and there will be
+ * no more output frames
+ * AVERROR(EINVAL): codec not opened, or it's an encoder
+ * other negative values: legitimate decoding errors
+ */
+int avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame);
+
+/**
+ * Supply a raw video or audio frame to the encoder. Use avcodec_receive_packet()
+ * to retrieve buffered output packets.
+ *
+ * @param avctx codec context
+ * @param[in] frame AVFrame containing the raw audio or video frame to be encoded.
+ * Ownership of the frame remains with the caller, and the
+ * encoder will not write to the frame. The encoder may create
+ * a reference to the frame data (or copy it if the frame is
+ * not reference-counted).
+ * It can be NULL, in which case it's considered a flush packet.
+ * This signals the end of the stream. If the encoder still has
+ * packets buffered, it will return them after this call. Once
+ * flushing mode has been entered, additional flush packets are
+ * ignored, and sending frames will return AVERROR_EOF.
+ *
+ * For audio:
+ * If AV_CODEC_CAP_VARIABLE_FRAME_SIZE is set, then each frame
+ * can have any number of samples.
+ * If it is not set, frame->nb_samples must be equal to
+ * avctx->frame_size for all frames except the last.
+ * The final frame may be smaller than avctx->frame_size.
+ * @return 0 on success, otherwise negative error code:
+ * AVERROR(EAGAIN): input is not accepted right now - the frame must be
+ * resent after trying to read output packets
+ * AVERROR_EOF: the encoder has been flushed, and no new frames can
+ * be sent to it
+ * AVERROR(EINVAL): codec not opened, refcounted_frames not set, it's a
+ * decoder, or requires flush
+ * AVERROR(ENOMEM): failed to add packet to internal queue, or similar
+ * other errors: legitimate decoding errors
+ */
+int avcodec_send_frame(AVCodecContext *avctx, const AVFrame *frame);
+
+/**
+ * Read encoded data from the encoder.
+ *
+ * @param avctx codec context
+ * @param avpkt This will be set to a reference-counted packet allocated by the
+ * encoder. Note that the function will always call
+ * av_frame_unref(frame) before doing anything else.
+ * @return 0 on success, otherwise negative error code:
+ * AVERROR(EAGAIN): output is not available right now - user must try
+ * to send input
+ * AVERROR_EOF: the encoder has been fully flushed, and there will be
+ * no more output packets
+ * AVERROR(EINVAL): codec not opened, or it's an encoder
+ * other errors: legitimate decoding errors
+ */
+int avcodec_receive_packet(AVCodecContext *avctx, AVPacket *avpkt);
+
+
+/**
* @defgroup lavc_parsing Frame parsing
* @{
*/
diff --git a/libavcodec/internal.h b/libavcodec/internal.h
index 3c1583d..8e9b46f 100644
--- a/libavcodec/internal.h
+++ b/libavcodec/internal.h
@@ -140,6 +140,18 @@ typedef struct AVCodecInternal {
* hwaccel-specific private data
*/
void *hwaccel_priv_data;
+
+ /**
+ * checks API usage: after codec draining, flush is required to resume operation
+ */
+ int draining;
+
+ /**
+ * buffers for using new encode/decode API through legacy API
+ */
+ AVPacket buffer_pkt;
+ int buffer_pkt_valid; // encoding: packet without data can be valid
+ AVFrame *buffer_frame;
} AVCodecInternal;

struct AVCodecDefault {
diff --git a/libavcodec/utils.c b/libavcodec/utils.c
index 866cdfc..c1616c9 100644
--- a/libavcodec/utils.c
+++ b/libavcodec/utils.c
@@ -95,12 +95,12 @@ static av_cold void avcodec_init(void)

int av_codec_is_encoder(const AVCodec *codec)
{
- return codec && (codec->encode_sub || codec->encode2);
+ return codec && (codec->encode_sub || codec->encode2 ||codec->send_frame);
}

int av_codec_is_decoder(const AVCodec *codec)
{
- return codec && codec->decode;
+ return codec && (codec->decode || codec->send_packet);
}

av_cold void avcodec_register(AVCodec *codec)
@@ -882,6 +882,12 @@ int attribute_align_arg avcodec_open2(AVCodecContext *avctx, const AVCodec *code
goto free_and_end;
}

+ avctx->internal->buffer_frame = av_frame_alloc();
+ if (!avctx->internal->buffer_frame) {
+ ret = AVERROR(ENOMEM);
+ goto free_and_end;
+ }
+
if (codec->priv_data_size > 0) {
if (!avctx->priv_data) {
avctx->priv_data = av_mallocz(codec->priv_data_size);
@@ -1220,6 +1226,11 @@ int attribute_align_arg avcodec_encode_audio2(AVCodecContext *avctx,

*got_packet_ptr = 0;

+ if (!avctx->codec->encode2) {
+ av_log(avctx, AV_LOG_ERROR, "This encoder requires using the avcodec_send_frame() API.\n");
+ return AVERROR(ENOSYS);
+ }
+
if (!(avctx->codec->capabilities & AV_CODEC_CAP_DELAY) && !frame) {
av_packet_unref(avpkt);
av_init_packet(avpkt);
@@ -1327,6 +1338,11 @@ int attribute_align_arg avcodec_encode_video2(AVCodecContext *avctx,

*got_packet_ptr = 0;

+ if (!avctx->codec->encode2) {
+ av_log(avctx, AV_LOG_ERROR, "This encoder requires using the avcodec_send_frame() API.\n");
+ return AVERROR(ENOSYS);
+ }
+
if (!(avctx->codec->capabilities & AV_CODEC_CAP_DELAY) && !frame) {
av_packet_unref(avpkt);
av_init_packet(avpkt);
@@ -1498,6 +1514,11 @@ int attribute_align_arg avcodec_decode_video2(AVCodecContext *avctx, AVFrame *pi
if ((avctx->coded_width || avctx->coded_height) && av_image_check_size(avctx->coded_width, avctx->coded_height, 0, avctx))
return -1;

+ if (!avctx->codec->decode) {
+ av_log(avctx, AV_LOG_ERROR, "This decoder requires using the avcodec_send_packet() API.\n");
+ return AVERROR(ENOSYS);
+ }
+
avctx->internal->pkt = avpkt;
ret = apply_param_change(avctx, avpkt);
if (ret < 0)
@@ -1557,6 +1578,11 @@ int attribute_align_arg avcodec_decode_audio4(AVCodecContext *avctx,

*got_frame_ptr = 0;

+ if (!avctx->codec->decode) {
+ av_log(avctx, AV_LOG_ERROR, "This decoder requires using the avcodec_send_packet() API.\n");
+ return AVERROR(ENOSYS);
+ }
+
avctx->internal->pkt = avpkt;

if (!avpkt->data && avpkt->size) {
@@ -1624,6 +1650,217 @@ void avsubtitle_free(AVSubtitle *sub)
memset(sub, 0, sizeof(AVSubtitle));
}

+int attribute_align_arg avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt)
+{
+ AVPacket leftover;
+ int ret;
+
+ if (!avcodec_is_open(avctx) || !av_codec_is_decoder(avctx->codec))
+ return AVERROR(EINVAL);
+
+ if (avctx->internal->draining)
+ return AVERROR_EOF;
+
+ if (!avpkt || !avpkt->size) {
+ avctx->internal->draining = 1;
+ avpkt = NULL;
+
+ if (!(avctx->codec->capabilities & AV_CODEC_CAP_DELAY))
+ return 0;
+ }
+
+ if (avctx->codec->send_packet) {
+ ret = apply_param_change(avctx, (AVPacket *)avpkt);
+ if (ret < 0)
+ return ret;
+ return avctx->codec->send_packet(avctx, avpkt);
+ }
+
+ // Emulation via old API. Assume avpkt is likely not refcounted, while
+ // decoder output is always refcounted, and avoid copying.
+
+ // This is the lesser evil. The field is for compatibility with legacy users
+ // of the legacy API, and users using the new API should not be forced to
+ // even know about this field.
+ avctx->refcounted_frames = 1;
+
+ if (avctx->internal->buffer_pkt.size || avctx->internal->buffer_frame->buf[0])
+ return AVERROR(EAGAIN);
+
+ // Use evil code to avoid evil memcpy. The goal is decoding the first frame
+ // of the packet without memcpy, because the common case is having only 1
+ // frame per packet (especially with video, but audio too). In other cases,
+ // it can't be avoided, unless the user is feeding refcounted frames.
+ av_packet_unref(&avctx->internal->buffer_pkt);
+ av_init_packet(&avctx->internal->buffer_pkt);
+ if (avpkt)
+ avctx->internal->buffer_pkt = *avpkt;
+ avctx->internal->buffer_pkt.buf = NULL;
+
+ ret = avcodec_receive_frame(avctx, avctx->internal->buffer_frame);
+ if (ret >= 0 || ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
+ ret = 0;
+ leftover = avctx->internal->buffer_pkt;
+ avctx->internal->buffer_pkt = (AVPacket){0};
+
+ if (ret >= 0 && leftover.size) {
+ // This will have to be decoded on the next call.
+ ret = av_packet_ref(&avctx->internal->buffer_pkt, &leftover);
+ }
+ return ret;
+}
+
+int attribute_align_arg avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame)
+{
+ int got_frame;
+ int ret;
+
+ av_frame_unref(frame);
+
+ if (!avcodec_is_open(avctx) || !av_codec_is_decoder(avctx->codec))
+ return AVERROR(EINVAL);
+
+ if (avctx->codec->receive_frame) {
+ if (avctx->internal->draining && !(avctx->codec->capabilities & AV_CODEC_CAP_DELAY))
+ return AVERROR_EOF;
+ return avctx->codec->receive_frame(avctx, frame);
+ }
+
+ // Emulation via old API.
+
+ if (avctx->internal->buffer_frame->buf[0])
+ goto return_frame;
+
+ if (!avctx->internal->buffer_pkt.size && !avctx->internal->draining)
+ return AVERROR(EAGAIN);
+
+ if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
+ ret = avcodec_decode_video2(avctx, avctx->internal->buffer_frame,
+ &got_frame, &avctx->internal->buffer_pkt);
+ if (ret >= 0)
+ ret = avctx->internal->buffer_pkt.size;
+ } else if (avctx->codec_type == AVMEDIA_TYPE_AUDIO) {
+ ret = avcodec_decode_audio4(avctx, avctx->internal->buffer_frame,
+ &got_frame, &avctx->internal->buffer_pkt);
+ } else {
+ ret = AVERROR(EINVAL);
+ }
+
+ if (ret < 0)
+ return ret;
+
+ ret = FFMIN(ret, avctx->internal->buffer_pkt.size);
+ avctx->internal->buffer_pkt.data += ret;
+ avctx->internal->buffer_pkt.size -= ret;
+
+ if (!avctx->internal->buffer_pkt.size && avctx->internal->buffer_pkt.buf)
+ av_packet_unref(&avctx->internal->buffer_pkt);
+
+ if (got_frame) {
+ av_assert0(avctx->internal->buffer_frame->buf[0]);
+ goto return_frame;
+ }
+ return avctx->internal->draining ? AVERROR_EOF : AVERROR(EAGAIN);
+
+return_frame:
+ if (frame != avctx->internal->buffer_frame)
+ av_frame_move_ref(frame, avctx->internal->buffer_frame);
+ avctx->frame_number++;
+ return 0;
+}
+
+static int do_encode(AVCodecContext *avctx, const AVFrame *frame, int *got_packet)
+{
+ int ret;
+ *got_packet = 0;
+
+ avctx->internal->buffer_pkt_valid = 0;
+
+ if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
+ ret = avcodec_encode_video2(avctx, &avctx->internal->buffer_pkt,
+ frame, got_packet);
+ } else if (avctx->codec_type == AVMEDIA_TYPE_AUDIO) {
+ ret = avcodec_encode_audio2(avctx, &avctx->internal->buffer_pkt,
+ frame, got_packet);
+ } else {
+ ret = AVERROR(EINVAL);
+ }
+
+ if (ret >= 0 && *got_packet) {
+ // Encoders must always return ref-counted buffers.
+ // Side-data only packets have no data and can be not ref-counted.
+ av_assert0(!avctx->internal->buffer_pkt.data || avctx->internal->buffer_pkt.buf);
+ avctx->internal->buffer_pkt_valid = 1;
+ ret = 0;
+ } else {
+ av_packet_unref(&avctx->internal->buffer_pkt);
+ }
+
+ return ret;
+}
+
+int attribute_align_arg avcodec_send_frame(AVCodecContext *avctx, const AVFrame *frame)
+{
+ if (!avcodec_is_open(avctx) || !av_codec_is_encoder(avctx->codec))
+ return AVERROR(EINVAL);
+
+ if (avctx->internal->draining)
+ return AVERROR_EOF;
+
+ if (!frame) {
+ avctx->internal->draining = 1;
+
+ if (!(avctx->codec->capabilities & AV_CODEC_CAP_DELAY))
+ return 0;
+ }
+
+ if (avctx->codec->send_frame)
+ return avctx->codec->send_frame(avctx, frame);
+
+ // Emulation via old API. Do it here instead of avcodec_receive_packet, because:
+ // 1. if the AVFrame is not refcounted, the copying will be much more
+ // expensive than copying the packet data
+ // 2. assume few users use non-refcounted AVPackets, so usually no copy is
+ // needed
+
+ if (avctx->internal->buffer_pkt_valid)
+ return AVERROR(EAGAIN);
+
+ return do_encode(avctx, frame, &(int){0});
+}
+
+int attribute_align_arg avcodec_receive_packet(AVCodecContext *avctx, AVPacket *avpkt)
+{
+ av_packet_unref(avpkt);
+
+ if (!avcodec_is_open(avctx) || !av_codec_is_encoder(avctx->codec))
+ return AVERROR(EINVAL);
+
+ if (avctx->codec->receive_packet) {
+ if (avctx->internal->draining && !(avctx->codec->capabilities & AV_CODEC_CAP_DELAY))
+ return AVERROR_EOF;
+ return avctx->codec->receive_packet(avctx, avpkt);
+ }
+
+ // Emulation via old API.
+
+ if (!avctx->internal->buffer_pkt_valid) {
+ int got_packet;
+ int ret;
+ if (!avctx->internal->draining)
+ return AVERROR(EAGAIN);
+ ret = do_encode(avctx, NULL, &got_packet);
+ if (ret < 0)
+ return ret;
+ if (ret >= 0 && !got_packet)
+ return AVERROR_EOF;
+ }
+
+ av_packet_move_ref(avpkt, &avctx->internal->buffer_pkt);
+ avctx->internal->buffer_pkt_valid = 0;
+ return 0;
+}
+
av_cold int avcodec_close(AVCodecContext *avctx)
{
int i;
@@ -1636,6 +1873,8 @@ av_cold int avcodec_close(AVCodecContext *avctx)
if (avctx->codec && avctx->codec->close)
avctx->codec->close(avctx);
av_frame_free(&avctx->internal->to_free);
+ av_frame_free(&avctx->internal->buffer_frame);
+ av_packet_unref(&avctx->internal->buffer_pkt);
for (i = 0; i < FF_ARRAY_ELEMS(pool->pools); i++)
av_buffer_pool_uninit(&pool->pools[i]);
av_freep(&avctx->internal->pool);
@@ -1966,6 +2205,11 @@ const char *avcodec_license(void)

void avcodec_flush_buffers(AVCodecContext *avctx)
{
+ avctx->internal->draining = 0;
+ av_frame_unref(avctx->internal->buffer_frame);
+ av_packet_unref(&avctx->internal->buffer_pkt);
+ avctx->internal->buffer_pkt_valid = 0;
+
if (HAVE_THREADS && avctx->active_thread_type & FF_THREAD_FRAME)
ff_thread_flush(avctx);
else if (avctx->codec->flush)
--
2.7.0
Luca Barbato
2016-02-27 14:25:29 UTC
Permalink
Post by wm4
+ AVPacket buffer_pkt;
Maybe use a pointer there as well. (I leave to the other bikeshed
between my pull/push or your send/receive).

lu
Diego Biurrun
2016-02-28 17:05:17 UTC
Permalink
Post by wm4
For now, there are no codecs supporting this API. The API can work with
codecs using the old API, and most code added here it to make them
interopate. The reverse is not possible, although for audio it might.
interop_ER_ate

Diego
wm4
2016-02-29 21:55:45 UTC
Permalink
On Sun, 28 Feb 2016 18:05:17 +0100
Post by Diego Biurrun
Post by wm4
For now, there are no codecs supporting this API. The API can work with
codecs using the old API, and most code added here it to make them
interopate. The reverse is not possible, although for audio it might.
interop_ER_ate
Forgot to fix this in the resend; next time.
Zhang Rui
2016-02-29 03:41:45 UTC
Permalink
Post by wm4
Until now, the decoding API was restricted to outputting 0 or 1 frames
per input packet. It also enforces a somewhat rigid dataflow in general.
This new API seeks to relax these restrictions by decoupling input and
output. Instead of doing a single call on each decode step, which may
consume the packet and may produce output, the new API requires the user
to send input first, and then ask for output.
For now, there are no codecs supporting this API. The API can work with
codecs using the old API, and most code added here it to make them
interopate. The reverse is not possible, although for audio it might.
---
libavcodec/avcodec.h | 221 ++++++++++++++++++++++++++++++++++++++++++++
libavcodec/internal.h | 12 +++
libavcodec/utils.c | 248 +++++++++++++++++++++++++++++++++++++++++++++++++-
3 files changed, 479 insertions(+), 2 deletions(-)
diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
index 33de8ec..6065dba 100644
--- a/libavcodec/avcodec.h
+++ b/libavcodec/avcodec.h
@@ -3090,6 +3090,21 @@ typedef struct AVCodec {
int (*decode)(AVCodecContext *, void *outdata, int *outdata_size, AVPacket *avpkt);
int (*close)(AVCodecContext *);
/**
+ * Decode/encode API with decoupled packet/frame dataflow. The API is the
+ * same as the avcodec_ prefixed APIs (avcodec_send_frame() etc.), except
+ * - never called if the codec is closed or the wrong type
+ * - AVPacket parameter change side data is applied right before calling
+ * AVCodec->send_packet
+ * - if AV_CODEC_CAP_DELAY is not set, drain packets or frames are never sent
+ * - only one drain packet is ever passed down (until the next flush())
+ * - a drain AVPacket is always NULL (no need to check for avpkt->size).
+ */
+ int (*send_frame)(AVCodecContext *avctx, const AVFrame *frame);
+ int (*send_packet)(AVCodecContext *avctx, const AVPacket *avpkt);
+ int (*receive_frame)(AVCodecContext *avctx, AVFrame *frame);
+ int (*receive_packet)(AVCodecContext *avctx, AVPacket *avpkt);
+ /**
* Flush buffers.
* Will be called when seeking
*/
@@ -4080,6 +4095,212 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub,
AVPacket *avpkt);
/**
+ * send/receive API overview (xxx: where does this go?)
+ *
+ * The avcodec_send_packet()/avcodec_receive_frame()/avcodec_send_frame()/
+ * avcodec_receive_packet() provide a encode/decode API, which decouples input
+ * and output.
+ *
+ * The API is very similar for encoding/decoding and audio/video, and works as
+ * - Setup and open the AVCodecContext as usual.
+ * - For decoding, call avcodec_send_packet() to give the decoder raw
+ * compressed data in an AVPacket.
+ * - For encoding, call avcodec_send_frame() to give the decoder an AVFrame
+ * containing uncompressed audio or video.
+ * In both cases, it's recommended that AVPackets and AVFrames are refcounted,
+ * or libavcodec might have to copy the input data. (libavformat always
+ * returns refcounted AVPackets, and av_frame_get_buffer() allocates
+ * refcounted AVFrames.)
+ * - Receive output in a loop. Periodically call one of the avcodec_receive_*()
+ * - For decoding, call avcodec_receive_frame(). On success, it will return
+ * a AVFrame containing uncompressed audio or video data.
+ * - For encoding, call avcodec_receive_packet(). On success, it will return
+ * an AVPacket with a compressed frame.
+ * Repeat this call until it returns AVERROR(EAGAIN) or an error. The
+ * AVERROR(EAGAIN) return value means that new input data is required to
+ * return new output. In this case, continue with sending input. For each
+ * input frame/packet, the codec will typically return 1 output frame/packet,
+ * but it can also be 0 or more than 1.
+ *
+ * At the beginning of decoding or encoding, the codec might accept multiple
+ * input frames/packets without returning a frame, until its internal buffers
+ * are filled. This situation is handled transparently if you follow the steps
+ * outlined above.
+ *
+ * End of stream situations. These require "flushing" (aka draining) the codec,
+ * as the codec might buffer multiple frames or packets internally for
+ * performance or out of necessity (consider B-frames).
+ * - Instead of valid input, send NULL to the avcodec_send_packet() (decoding)
+ * or avcodec_send_frame() (encoding) functions. This will enter draining
+ * mode.
+ * - Call avcodec_receive_frame() (decoding) or avcodec_receive_packet()
+ * (encoding) in a loop until AVERROR_EOF is returned. The functions will
+ * not return AVERROR(EAGAIN), unless you forgot to enter draining mode.
+ * - Before decoding can be resumed again, the codec has to be reset with
+ * avcodec_flush_buffers().
Is there any "blocking mode" for "draining"?
wm4
2016-02-29 07:12:48 UTC
Permalink
On Mon, 29 Feb 2016 11:41:45 +0800
Post by Zhang Rui
Post by wm4
Until now, the decoding API was restricted to outputting 0 or 1 frames
per input packet. It also enforces a somewhat rigid dataflow in general.
This new API seeks to relax these restrictions by decoupling input and
output. Instead of doing a single call on each decode step, which may
consume the packet and may produce output, the new API requires the user
to send input first, and then ask for output.
For now, there are no codecs supporting this API. The API can work with
codecs using the old API, and most code added here it to make them
interopate. The reverse is not possible, although for audio it might.
---
libavcodec/avcodec.h | 221 ++++++++++++++++++++++++++++++++++++++++++++
libavcodec/internal.h | 12 +++
libavcodec/utils.c | 248 +++++++++++++++++++++++++++++++++++++++++++++++++-
3 files changed, 479 insertions(+), 2 deletions(-)
diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
index 33de8ec..6065dba 100644
--- a/libavcodec/avcodec.h
+++ b/libavcodec/avcodec.h
@@ -3090,6 +3090,21 @@ typedef struct AVCodec {
int (*decode)(AVCodecContext *, void *outdata, int *outdata_size, AVPacket *avpkt);
int (*close)(AVCodecContext *);
/**
+ * Decode/encode API with decoupled packet/frame dataflow. The API is the
+ * same as the avcodec_ prefixed APIs (avcodec_send_frame() etc.), except
+ * - never called if the codec is closed or the wrong type
+ * - AVPacket parameter change side data is applied right before calling
+ * AVCodec->send_packet
+ * - if AV_CODEC_CAP_DELAY is not set, drain packets or frames are never sent
+ * - only one drain packet is ever passed down (until the next flush())
+ * - a drain AVPacket is always NULL (no need to check for avpkt->size).
+ */
+ int (*send_frame)(AVCodecContext *avctx, const AVFrame *frame);
+ int (*send_packet)(AVCodecContext *avctx, const AVPacket *avpkt);
+ int (*receive_frame)(AVCodecContext *avctx, AVFrame *frame);
+ int (*receive_packet)(AVCodecContext *avctx, AVPacket *avpkt);
+ /**
* Flush buffers.
* Will be called when seeking
*/
@@ -4080,6 +4095,212 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub,
AVPacket *avpkt);
/**
+ * send/receive API overview (xxx: where does this go?)
+ *
+ * The avcodec_send_packet()/avcodec_receive_frame()/avcodec_send_frame()/
+ * avcodec_receive_packet() provide a encode/decode API, which decouples input
+ * and output.
+ *
+ * The API is very similar for encoding/decoding and audio/video, and works as
+ * - Setup and open the AVCodecContext as usual.
+ * - For decoding, call avcodec_send_packet() to give the decoder raw
+ * compressed data in an AVPacket.
+ * - For encoding, call avcodec_send_frame() to give the decoder an AVFrame
+ * containing uncompressed audio or video.
+ * In both cases, it's recommended that AVPackets and AVFrames are refcounted,
+ * or libavcodec might have to copy the input data. (libavformat always
+ * returns refcounted AVPackets, and av_frame_get_buffer() allocates
+ * refcounted AVFrames.)
+ * - Receive output in a loop. Periodically call one of the avcodec_receive_*()
+ * - For decoding, call avcodec_receive_frame(). On success, it will return
+ * a AVFrame containing uncompressed audio or video data.
+ * - For encoding, call avcodec_receive_packet(). On success, it will return
+ * an AVPacket with a compressed frame.
+ * Repeat this call until it returns AVERROR(EAGAIN) or an error. The
+ * AVERROR(EAGAIN) return value means that new input data is required to
+ * return new output. In this case, continue with sending input. For each
+ * input frame/packet, the codec will typically return 1 output frame/packet,
+ * but it can also be 0 or more than 1.
+ *
+ * At the beginning of decoding or encoding, the codec might accept multiple
+ * input frames/packets without returning a frame, until its internal buffers
+ * are filled. This situation is handled transparently if you follow the steps
+ * outlined above.
+ *
+ * End of stream situations. These require "flushing" (aka draining) the codec,
+ * as the codec might buffer multiple frames or packets internally for
+ * performance or out of necessity (consider B-frames).
+ * - Instead of valid input, send NULL to the avcodec_send_packet() (decoding)
+ * or avcodec_send_frame() (encoding) functions. This will enter draining
+ * mode.
+ * - Call avcodec_receive_frame() (decoding) or avcodec_receive_packet()
+ * (encoding) in a loop until AVERROR_EOF is returned. The functions will
+ * not return AVERROR(EAGAIN), unless you forgot to enter draining mode.
+ * - Before decoding can be resumed again, the codec has to be reset with
+ * avcodec_flush_buffers().
Is there any "blocking mode" for "draining"?
It is all meant to be blocking.
wm4
2016-02-27 14:15:33 UTC
Permalink
It's not practical to keep this with the new decode API.
---
avconv.c | 7 -------
avconv.h | 1 -
2 files changed, 8 deletions(-)

diff --git a/avconv.c b/avconv.c
index 35ea1f5..d878646 100644
--- a/avconv.c
+++ b/avconv.c
@@ -1337,13 +1337,6 @@ static void process_input_packet(InputStream *ist, const AVPacket *pkt, int no_e

ist->last_dts = ist->next_dts;

- if (avpkt.size && avpkt.size != pkt->size &&
- !(ist->dec->capabilities & AV_CODEC_CAP_SUBFRAMES)) {
- av_log(NULL, ist->showed_multi_packet_warning ? AV_LOG_VERBOSE : AV_LOG_WARNING,
- "Multiple frames in a packet from stream %d\n", pkt->stream_index);
- ist->showed_multi_packet_warning = 1;
- }
-
switch (ist->dec_ctx->codec_type) {
case AVMEDIA_TYPE_AUDIO:
ret = decode_audio (ist, &avpkt, &got_output);
diff --git a/avconv.h b/avconv.h
index f0a948f..645dc30 100644
--- a/avconv.h
+++ b/avconv.h
@@ -240,7 +240,6 @@ typedef struct InputStream {
int64_t nb_samples; /* number of samples in the last decoded audio frame before looping */
PtsCorrectionContext pts_ctx;
double ts_scale;
- int showed_multi_packet_warning;
AVDictionary *decoder_opts;
AVRational framerate; /* framerate forced with -r */
--
2.7.0
Luca Barbato
2016-02-27 14:25:56 UTC
Permalink
Post by wm4
It's not practical to keep this with the new decode API.
---
avconv.c | 7 -------
avconv.h | 1 -
2 files changed, 8 deletions(-)
Possibly OK.
wm4
2016-02-27 14:15:34 UTC
Permalink
Rather than refactoring everything, add a function that resembles the
avcodec_decode_audio4() API.
---
avconv.c | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/avconv.c b/avconv.c
index d878646..c74b4b8 100644
--- a/avconv.c
+++ b/avconv.c
@@ -1079,6 +1079,31 @@ static void do_streamcopy(InputStream *ist, OutputStream *ost, const AVPacket *p
write_frame(of->ctx, &opkt, ost);
}

+static int decode(AVCodecContext *avctx, AVFrame *frame, int *got_frame, AVPacket *pkt)
+{
+ int ret;
+ int consumed = 0;
+
+ *got_frame = 0;
+
+ // This relies on the fact that the decoder will not buffer additional
+ // packets internally, but returns AVERROR(EAGAIN) if there are still
+ // decoded frames to be returned.
+ ret = avcodec_send_packet(avctx, pkt);
+ if (ret < 0 && ret != AVERROR(EAGAIN) && ret != AVERROR_EOF)
+ return ret;
+ if (ret >= 0)
+ consumed = pkt->size;
+
+ ret = avcodec_receive_frame(avctx, frame);
+ if (ret < 0 && ret != AVERROR(EAGAIN) && ret != AVERROR_EOF)
+ return ret;
+ if (ret >= 0)
+ *got_frame = 1;
+
+ return consumed;
+}
+
int guess_input_channel_layout(InputStream *ist)
{
AVCodecContext *dec = ist->dec_ctx;
@@ -1109,7 +1134,7 @@ static int decode_audio(InputStream *ist, AVPacket *pkt, int *got_output)
return AVERROR(ENOMEM);
decoded_frame = ist->decoded_frame;

- ret = avcodec_decode_audio4(avctx, decoded_frame, got_output, pkt);
+ ret = decode(avctx, decoded_frame, got_output, pkt);
if (!*got_output || ret < 0)
return ret;

@@ -1200,8 +1225,7 @@ static int decode_video(InputStream *ist, AVPacket *pkt, int *got_output)
return AVERROR(ENOMEM);
decoded_frame = ist->decoded_frame;

- ret = avcodec_decode_video2(ist->dec_ctx,
- decoded_frame, got_output, pkt);
+ ret = decode(ist->dec_ctx, decoded_frame, got_output, pkt);
if (!*got_output || ret < 0)
return ret;

@@ -1216,7 +1240,6 @@ static int decode_video(InputStream *ist, AVPacket *pkt, int *got_output)

decoded_frame->pts = guess_correct_pts(&ist->pts_ctx, decoded_frame->pkt_pts,
decoded_frame->pkt_dts);
- pkt->size = 0;

if (ist->st->sample_aspect_ratio.num)
decoded_frame->sample_aspect_ratio = ist->st->sample_aspect_ratio;
--
2.7.0
wm4
2016-02-27 14:15:35 UTC
Permalink
The flushing case is a bit strange; not simplifying it so the change is
less noisy.
---
avconv.c | 44 ++++++++++++++++++++++++++++----------------
1 file changed, 28 insertions(+), 16 deletions(-)

diff --git a/avconv.c b/avconv.c
index c74b4b8..40152ff 100644
--- a/avconv.c
+++ b/avconv.c
@@ -458,7 +458,7 @@ static void do_video_out(AVFormatContext *s,
AVFrame *in_picture,
int *frame_size)
{
- int ret, format_video_sync, got_packet;
+ int ret, format_video_sync;
AVPacket pkt;
AVCodecContext *enc = ost->enc_ctx;

@@ -508,13 +508,17 @@ static void do_video_out(AVFormatContext *s,

ost->frames_encoded++;

- ret = avcodec_encode_video2(enc, &pkt, in_picture, &got_packet);
- if (ret < 0) {
- av_log(NULL, AV_LOG_FATAL, "Video encoding failed\n");
- exit_program(1);
- }
+ ret = avcodec_send_frame(enc, in_picture);
+ if (ret < 0)
+ goto error;
+
+ while (1) {
+ ret = avcodec_receive_packet(enc, &pkt);
+ if (ret == AVERROR(EAGAIN))
+ break;
+ if (ret < 0)
+ goto error;

- if (got_packet) {
av_packet_rescale_ts(&pkt, enc->time_base, ost->st->time_base);
write_frame(s, &pkt, ost);
*frame_size = pkt.size;
@@ -532,6 +536,12 @@ static void do_video_out(AVFormatContext *s,
* flush, we need to limit them here, before they go into encoder.
*/
ost->frame_number++;
+
+ return;
+error:
+ av_assert0(ret != AVERROR(EAGAIN) && ret != AVERROR_EOF);
+ av_log(NULL, AV_LOG_FATAL, "Video encoding failed\n");
+ exit_program(1);
}

static double psnr(double d)
@@ -946,39 +956,41 @@ static void flush_encoders(void)
if (enc->codec_type == AVMEDIA_TYPE_AUDIO && enc->frame_size <= 1)
continue;

+ if (enc->codec_type != AVMEDIA_TYPE_VIDEO && enc->codec_type != AVMEDIA_TYPE_AUDIO)
+ continue;
+
+ avcodec_send_frame(enc, NULL);
+
for (;;) {
- int (*encode)(AVCodecContext*, AVPacket*, const AVFrame*, int*) = NULL;
- const char *desc;
+ const char *desc = NULL;

switch (enc->codec_type) {
case AVMEDIA_TYPE_AUDIO:
- encode = avcodec_encode_audio2;
desc = "Audio";
break;
case AVMEDIA_TYPE_VIDEO:
- encode = avcodec_encode_video2;
desc = "Video";
break;
default:
- stop_encoding = 1;
+ av_assert0(0);
}

- if (encode) {
+ if (1) {
AVPacket pkt;
int got_packet;
av_init_packet(&pkt);
pkt.data = NULL;
pkt.size = 0;

- ret = encode(enc, &pkt, NULL, &got_packet);
- if (ret < 0) {
+ ret = avcodec_receive_packet(enc, &pkt);
+ if (ret < 0 && ret != AVERROR_EOF) {
av_log(NULL, AV_LOG_FATAL, "%s encoding failed\n", desc);
exit_program(1);
}
if (ost->logfile && enc->stats_out) {
fprintf(ost->logfile, "%s", enc->stats_out);
}
- if (!got_packet) {
+ if (ret == AVERROR_EOF) {
stop_encoding = 1;
break;
}
--
2.7.0
wm4
2016-02-29 21:48:11 UTC
Permalink
---
Changed some minor things that were pointed out.
---
libavutil/frame.h | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/libavutil/frame.h b/libavutil/frame.h
index 5a04177..a39a1ef 100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -391,6 +391,10 @@ void av_frame_free(AVFrame **frame);
* If src is not reference counted, new buffers are allocated and the data is
* copied.
*
+ * @warning: dst MUST have been either unreferenced with av_frame_unref(dst),
+ * or newly allocated with av_frame_alloc() before calling this
+ * function, or undefined behavior will occur.
+ *
* @return 0 on success, a negative AVERROR on error
*/
int av_frame_ref(AVFrame *dst, const AVFrame *src);
@@ -410,7 +414,11 @@ AVFrame *av_frame_clone(const AVFrame *src);
void av_frame_unref(AVFrame *frame);

/**
- * Move everythnig contained in src to dst and reset src.
+ * Move everything contained in src to dst and reset src.
+ *
+ * @warning: dst is not unreferenced, but directly overwritten without reading
+ * or deallocating its contents. Call av_frame_unref(dst) manually
+ * before calling this function to ensure that no memory is leaked.
*/
void av_frame_move_ref(AVFrame *dst, AVFrame *src);

@@ -426,6 +434,10 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src);
* necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf.
* For planar formats, one buffer will be allocated for each plane.
*
+ * @warning: if frame already has been allocated, calling this function will
+ * leak memory. In addition, undefined behavior can occur in certain
+ * cases.
+ *
* @param frame frame in which to store the new buffers.
* @param align required buffer size alignment
*
--
2.7.0
wm4
2016-02-29 21:48:12 UTC
Permalink
Remove the duplicated code for handling failure of apply_param_change().
---
No changes.
---
libavcodec/utils.c | 28 +++++++++++++++-------------
1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/libavcodec/utils.c b/libavcodec/utils.c
index a9a7423..866cdfc 100644
--- a/libavcodec/utils.c
+++ b/libavcodec/utils.c
@@ -1390,7 +1390,8 @@ static int apply_param_change(AVCodecContext *avctx, AVPacket *avpkt)
if (!(avctx->codec->capabilities & AV_CODEC_CAP_PARAM_CHANGE)) {
av_log(avctx, AV_LOG_ERROR, "This decoder does not support parameter "
"changes, but PARAM_CHANGE side data was sent to it.\n");
- return AVERROR(EINVAL);
+ ret = AVERROR(EINVAL);
+ goto fail2;
}

if (size < 4)
@@ -1425,13 +1426,20 @@ static int apply_param_change(AVCodecContext *avctx, AVPacket *avpkt)
size -= 8;
ret = ff_set_dimensions(avctx, avctx->width, avctx->height);
if (ret < 0)
- return ret;
+ goto fail2;
}

return 0;
fail:
av_log(avctx, AV_LOG_ERROR, "PARAM_CHANGE side data too small.\n");
- return AVERROR_INVALIDDATA;
+ ret = AVERROR_INVALIDDATA;
+fail2:
+ if (ret < 0) {
+ av_log(avctx, AV_LOG_ERROR, "Error applying parameter changes.\n");
+ if (avctx->err_recognition & AV_EF_EXPLODE)
+ return ret;
+ }
+ return 0;
}

static int unrefcount_frame(AVCodecInternal *avci, AVFrame *frame)
@@ -1492,11 +1500,8 @@ int attribute_align_arg avcodec_decode_video2(AVCodecContext *avctx, AVFrame *pi

avctx->internal->pkt = avpkt;
ret = apply_param_change(avctx, avpkt);
- if (ret < 0) {
- av_log(avctx, AV_LOG_ERROR, "Error applying parameter changes.\n");
- if (avctx->err_recognition & AV_EF_EXPLODE)
- return ret;
- }
+ if (ret < 0)
+ return ret;

av_frame_unref(picture);

@@ -1560,11 +1565,8 @@ int attribute_align_arg avcodec_decode_audio4(AVCodecContext *avctx,
}

ret = apply_param_change(avctx, avpkt);
- if (ret < 0) {
- av_log(avctx, AV_LOG_ERROR, "Error applying parameter changes.\n");
- if (avctx->err_recognition & AV_EF_EXPLODE)
- return ret;
- }
+ if (ret < 0)
+ return ret;

av_frame_unref(frame);
--
2.7.0
wm4
2016-02-29 21:48:13 UTC
Permalink
Until now, the decoding API was restricted to outputting 0 or 1 frames
per input packet. It also enforces a somewhat rigid dataflow in general.

This new API seeks to relax these restrictions by decoupling input and
output. Instead of doing a single call on each decode step, which may
consume the packet and may produce output, the new API requires the user
to send input first, and then ask for output.

For now, there are no codecs supporting this API. The API can work with
codecs using the old API, and most code added here it to make them
interopate. The reverse is not possible, although for audio it might.
---
Changed the decoding emulation quite a lot. Removed all AVPacket
static allocation.
---
libavcodec/avcodec.h | 223 +++++++++++++++++++++++++++++++++++++++++++
libavcodec/internal.h | 12 +++
libavcodec/utils.c | 258 +++++++++++++++++++++++++++++++++++++++++++++++++-
3 files changed, 491 insertions(+), 2 deletions(-)

diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
index 33de8ec..3c1bda5 100644
--- a/libavcodec/avcodec.h
+++ b/libavcodec/avcodec.h
@@ -2211,6 +2211,8 @@ typedef struct AVCodecContext {
* Otherwise, the decoded frames must not be freed by the caller and are
* only valid until the next decode call.
*
+ * This is always automatically enabled if avcodec_receive_frame() is used.
+ *
* - encoding: unused
* - decoding: set by the caller before avcodec_open2().
*/
@@ -3090,6 +3092,21 @@ typedef struct AVCodec {
int (*decode)(AVCodecContext *, void *outdata, int *outdata_size, AVPacket *avpkt);
int (*close)(AVCodecContext *);
/**
+ * Decode/encode API with decoupled packet/frame dataflow. The API is the
+ * same as the avcodec_ prefixed APIs (avcodec_send_frame() etc.), except
+ * that:
+ * - never called if the codec is closed or the wrong type
+ * - AVPacket parameter change side data is applied right before calling
+ * AVCodec->send_packet
+ * - if AV_CODEC_CAP_DELAY is not set, drain packets or frames are never sent
+ * - only one drain packet is ever passed down (until the next flush())
+ * - a drain AVPacket is always NULL (no need to check for avpkt->size).
+ */
+ int (*send_frame)(AVCodecContext *avctx, const AVFrame *frame);
+ int (*send_packet)(AVCodecContext *avctx, const AVPacket *avpkt);
+ int (*receive_frame)(AVCodecContext *avctx, AVFrame *frame);
+ int (*receive_packet)(AVCodecContext *avctx, AVPacket *avpkt);
+ /**
* Flush buffers.
* Will be called when seeking
*/
@@ -4080,6 +4097,212 @@ int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubtitle *sub,
AVPacket *avpkt);

/**
+ * send/receive API overview (xxx: where does this go?)
+ *
+ * The avcodec_send_packet()/avcodec_receive_frame()/avcodec_send_frame()/
+ * avcodec_receive_packet() provide a encode/decode API, which decouples input
+ * and output.
+ *
+ * The API is very similar for encoding/decoding and audio/video, and works as
+ * follows:
+ * - Setup and open the AVCodecContext as usual.
+ * - Send valid input:
+ * - For decoding, call avcodec_send_packet() to give the decoder raw
+ * compressed data in an AVPacket.
+ * - For encoding, call avcodec_send_frame() to give the decoder an AVFrame
+ * containing uncompressed audio or video.
+ * In both cases, it's recommended that AVPackets and AVFrames are refcounted,
+ * or libavcodec might have to copy the input data. (libavformat always
+ * returns refcounted AVPackets, and av_frame_get_buffer() allocates
+ * refcounted AVFrames.)
+ * - Receive output in a loop. Periodically call one of the avcodec_receive_*()
+ * functions and process their output:
+ * - For decoding, call avcodec_receive_frame(). On success, it will return
+ * a AVFrame containing uncompressed audio or video data.
+ * - For encoding, call avcodec_receive_packet(). On success, it will return
+ * an AVPacket with a compressed frame.
+ * Repeat this call until it returns AVERROR(EAGAIN) or an error. The
+ * AVERROR(EAGAIN) return value means that new input data is required to
+ * return new output. In this case, continue with sending input. For each
+ * input frame/packet, the codec will typically return 1 output frame/packet,
+ * but it can also be 0 or more than 1.
+ *
+ * At the beginning of decoding or encoding, the codec might accept multiple
+ * input frames/packets without returning a frame, until its internal buffers
+ * are filled. This situation is handled transparently if you follow the steps
+ * outlined above.
+ *
+ * End of stream situations. These require "flushing" (aka draining) the codec,
+ * as the codec might buffer multiple frames or packets internally for
+ * performance or out of necessity (consider B-frames).
+ * This is handled as follows:
+ * - Instead of valid input, send NULL to the avcodec_send_packet() (decoding)
+ * or avcodec_send_frame() (encoding) functions. This will enter draining
+ * mode.
+ * - Call avcodec_receive_frame() (decoding) or avcodec_receive_packet()
+ * (encoding) in a loop until AVERROR_EOF is returned. The functions will
+ * not return AVERROR(EAGAIN), unless you forgot to enter draining mode.
+ * - Before decoding can be resumed again, the codec has to be reset with
+ * avcodec_flush_buffers().
+ *
+ * Using the API as outlined above is highly recommended. But it's also possible
+ * to call functions outside of this rigid schema. For example, you can call
+ * avcodec_send_packet() repeatedly without calling avcodec_receive_frame(). In
+ * this case, avcodec_send_packet() will succeed until the codec's internal
+ * buffer has been filled up (which is typically of size 1 per output frame,
+ * after initial input), and then reject input with AVERROR(EAGAIN). Once it
+ * starts rejecting input, you have no choice but to read at least some output.
+ *
+ * Not all codecs will follow a rigid and predictable dataflow; the only
+ * guarantee is that an AVERROR(EAGAIN) return value on a send/receive call on
+ * one end implies that a receive/send call on the other end will succeed. In
+ * general, no codec will permit unlimited buffering of input or output.
+ *
+ * This API replaces the following legacy functions:
+ * - avcodec_decode_video2() and avcodec_decode_audio4():
+ * Use avcodec_send_packet() to feed input to the decoder, then use
+ * avcodec_receive_frame() to receive decoded frames after each packet.
+ * Unlike with the old video decoding API, multiple frames might result from
+ * a packet. For audio, splitting the input packet into frames by partially
+ * decoding packets becomes transparent to the API user. You never need to
+ * feed an AVPacket to the API twice.
+ * Additionally, sending a flush/draining packet is required only once.
+ * - avcodec_encode_video2()/avcodec_encode_audio2():
+ * Use avcodec_send_frame() to feed input to the encoder, then use
+ * avcodec_receive_packet() to receive encoded packets.
+ * Providing user-allocated buffers for avcodec_receive_packet() is not
+ * possible.
+ * - The new API does not handle subtitles yet.
+ *
+ * Mixing new and old function calls on the same AVCodecContext is not allowed,
+ * and will result in arbitrary behavior.
+ *
+ * Some codecs might require using the new API; using the old API will return
+ * an error when calling it.
+ */
+
+/**
+ * Supply raw packet data as input to a decoder.
+ *
+ * Internally, this call will copy relevant AVCodecContext fields, which can
+ * influence decoding per-packet, and apply them when the packet is actually
+ * decoded. (For example AVCodecContext.skip_frame, which might direct the
+ * decoder to drop the frame contained by the packet sent with this function.)
+ *
+ * @warning The input buffer, avpkt->data must be AV_INPUT_BUFFER_PADDING_SIZE
+ * larger than the actual read bytes because some optimized bitstream
+ * readers read 32 or 64 bits at once and could read over the end.
+ *
+ * @warning Do not mix this API with the legacy API (like avcodec_decode_video2())
+ * on the same AVCodecContext. It will return unexpected results now
+ * or in future libavcodec versions.
+ *
+ * @note The AVCodecContext MUST have been opened with @ref avcodec_open2()
+ * before packets may be fed to the decoder.
+ *
+ * @param avctx codec context
+ * @param[in] avpkt The input AVPacket. Usually, this will be a single video
+ * frame, or several complete audio frames.
+ * Ownership of the packet remains with the caller, and the
+ * decoder will not write to the packet. The decoder may create
+ * a reference to the packet data (or copy it if the packet is
+ * not reference-counted).
+ * Unlike with older APIs, the packet is always fully consumed,
+ * and if it contains multiple frames (e.g. some audio codecs),
+ * will require you to call avcodec_receive_frame() multiple
+ * times afterwards before you can send a new packet.
+ * It can be NULL (or an AVPacket with data set to NULL and size
+ * set to 0); in this case, it's considered a flush packet, which
+ * signals the end of the stream. Sending the first flush
+ * packet will return success. Subsequent ones are unnecessary
+ * and will return AVERROR_EOF. If the decoder still has frames
+ * buffered, it will return them after sending a flush packet.
+ *
+ * @return 0 on success, otherwise negative error code:
+ * AVERROR(EAGAIN): input is not accepted right now - the packet must be
+ * resent after trying to read output
+ * AVERROR_EOF: the decoder has been flushed, and no new packets can
+ * be sent to it (also returned if more than 1 flush
+ * packet is sent)
+ * AVERROR(EINVAL): codec not opened, it's an encoder, or requires flush
+ * AVERROR(ENOMEM): failed to add packet to internal queue, or similar
+ * other errors: legitimate decoding errors
+ */
+int avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt);
+
+/**
+ * Return decoded output data from a decoder.
+ *
+ * @param avctx codec context
+ * @param frame This will be set to a reference-counted video or audio
+ * frame (depending on the decoder type) allocated by the
+ * decoder. Note that the function will always call
+ * av_frame_unref(frame) before doing anything else.
+ *
+ * @return
+ * 0: success, a frame was returned
+ * AVERROR(EAGAIN): output is not available right now - user must try
+ * to send new input
+ * AVERROR_EOF: the decoder has been fully flushed, and there will be
+ * no more output frames
+ * AVERROR(EINVAL): codec not opened, or it's an encoder
+ * other negative values: legitimate decoding errors
+ */
+int avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame);
+
+/**
+ * Supply a raw video or audio frame to the encoder. Use avcodec_receive_packet()
+ * to retrieve buffered output packets.
+ *
+ * @param avctx codec context
+ * @param[in] frame AVFrame containing the raw audio or video frame to be encoded.
+ * Ownership of the frame remains with the caller, and the
+ * encoder will not write to the frame. The encoder may create
+ * a reference to the frame data (or copy it if the frame is
+ * not reference-counted).
+ * It can be NULL, in which case it's considered a flush packet.
+ * This signals the end of the stream. If the encoder still has
+ * packets buffered, it will return them after this call. Once
+ * flushing mode has been entered, additional flush packets are
+ * ignored, and sending frames will return AVERROR_EOF.
+ *
+ * For audio:
+ * If AV_CODEC_CAP_VARIABLE_FRAME_SIZE is set, then each frame
+ * can have any number of samples.
+ * If it is not set, frame->nb_samples must be equal to
+ * avctx->frame_size for all frames except the last.
+ * The final frame may be smaller than avctx->frame_size.
+ * @return 0 on success, otherwise negative error code:
+ * AVERROR(EAGAIN): input is not accepted right now - the frame must be
+ * resent after trying to read output packets
+ * AVERROR_EOF: the encoder has been flushed, and no new frames can
+ * be sent to it
+ * AVERROR(EINVAL): codec not opened, refcounted_frames not set, it's a
+ * decoder, or requires flush
+ * AVERROR(ENOMEM): failed to add packet to internal queue, or similar
+ * other errors: legitimate decoding errors
+ */
+int avcodec_send_frame(AVCodecContext *avctx, const AVFrame *frame);
+
+/**
+ * Read encoded data from the encoder.
+ *
+ * @param avctx codec context
+ * @param avpkt This will be set to a reference-counted packet allocated by the
+ * encoder. Note that the function will always call
+ * av_frame_unref(frame) before doing anything else.
+ * @return 0 on success, otherwise negative error code:
+ * AVERROR(EAGAIN): output is not available right now - user must try
+ * to send input
+ * AVERROR_EOF: the encoder has been fully flushed, and there will be
+ * no more output packets
+ * AVERROR(EINVAL): codec not opened, or it's an encoder
+ * other errors: legitimate decoding errors
+ */
+int avcodec_receive_packet(AVCodecContext *avctx, AVPacket *avpkt);
+
+
+/**
* @defgroup lavc_parsing Frame parsing
* @{
*/
diff --git a/libavcodec/internal.h b/libavcodec/internal.h
index 3c1583d..2144987 100644
--- a/libavcodec/internal.h
+++ b/libavcodec/internal.h
@@ -140,6 +140,18 @@ typedef struct AVCodecInternal {
* hwaccel-specific private data
*/
void *hwaccel_priv_data;
+
+ /**
+ * checks API usage: after codec draining, flush is required to resume operation
+ */
+ int draining;
+
+ /**
+ * buffers for using new encode/decode API through legacy API
+ */
+ AVPacket *buffer_pkt;
+ int buffer_pkt_valid; // encoding: packet without data can be valid
+ AVFrame *buffer_frame;
} AVCodecInternal;

struct AVCodecDefault {
diff --git a/libavcodec/utils.c b/libavcodec/utils.c
index 866cdfc..59ccfed 100644
--- a/libavcodec/utils.c
+++ b/libavcodec/utils.c
@@ -95,12 +95,12 @@ static av_cold void avcodec_init(void)

int av_codec_is_encoder(const AVCodec *codec)
{
- return codec && (codec->encode_sub || codec->encode2);
+ return codec && (codec->encode_sub || codec->encode2 ||codec->send_frame);
}

int av_codec_is_decoder(const AVCodec *codec)
{
- return codec && codec->decode;
+ return codec && (codec->decode || codec->send_packet);
}

av_cold void avcodec_register(AVCodec *codec)
@@ -882,6 +882,18 @@ int attribute_align_arg avcodec_open2(AVCodecContext *avctx, const AVCodec *code
goto free_and_end;
}

+ avctx->internal->buffer_frame = av_frame_alloc();
+ if (!avctx->internal->buffer_frame) {
+ ret = AVERROR(ENOMEM);
+ goto free_and_end;
+ }
+
+ avctx->internal->buffer_pkt = av_packet_alloc();
+ if (!avctx->internal->buffer_pkt) {
+ ret = AVERROR(ENOMEM);
+ goto free_and_end;
+ }
+
if (codec->priv_data_size > 0) {
if (!avctx->priv_data) {
avctx->priv_data = av_mallocz(codec->priv_data_size);
@@ -1220,6 +1232,11 @@ int attribute_align_arg avcodec_encode_audio2(AVCodecContext *avctx,

*got_packet_ptr = 0;

+ if (!avctx->codec->encode2) {
+ av_log(avctx, AV_LOG_ERROR, "This encoder requires using the avcodec_send_frame() API.\n");
+ return AVERROR(ENOSYS);
+ }
+
if (!(avctx->codec->capabilities & AV_CODEC_CAP_DELAY) && !frame) {
av_packet_unref(avpkt);
av_init_packet(avpkt);
@@ -1327,6 +1344,11 @@ int attribute_align_arg avcodec_encode_video2(AVCodecContext *avctx,

*got_packet_ptr = 0;

+ if (!avctx->codec->encode2) {
+ av_log(avctx, AV_LOG_ERROR, "This encoder requires using the avcodec_send_frame() API.\n");
+ return AVERROR(ENOSYS);
+ }
+
if (!(avctx->codec->capabilities & AV_CODEC_CAP_DELAY) && !frame) {
av_packet_unref(avpkt);
av_init_packet(avpkt);
@@ -1498,6 +1520,11 @@ int attribute_align_arg avcodec_decode_video2(AVCodecContext *avctx, AVFrame *pi
if ((avctx->coded_width || avctx->coded_height) && av_image_check_size(avctx->coded_width, avctx->coded_height, 0, avctx))
return -1;

+ if (!avctx->codec->decode) {
+ av_log(avctx, AV_LOG_ERROR, "This decoder requires using the avcodec_send_packet() API.\n");
+ return AVERROR(ENOSYS);
+ }
+
avctx->internal->pkt = avpkt;
ret = apply_param_change(avctx, avpkt);
if (ret < 0)
@@ -1557,6 +1584,11 @@ int attribute_align_arg avcodec_decode_audio4(AVCodecContext *avctx,

*got_frame_ptr = 0;

+ if (!avctx->codec->decode) {
+ av_log(avctx, AV_LOG_ERROR, "This decoder requires using the avcodec_send_packet() API.\n");
+ return AVERROR(ENOSYS);
+ }
+
avctx->internal->pkt = avpkt;

if (!avpkt->data && avpkt->size) {
@@ -1624,6 +1656,221 @@ void avsubtitle_free(AVSubtitle *sub)
memset(sub, 0, sizeof(AVSubtitle));
}

+static int do_decode(AVCodecContext *avctx, AVPacket *pkt)
+{
+ int got_frame;
+ int ret;
+
+ av_assert0(!avctx->internal->buffer_frame->buf[0]);
+
+ if (!pkt)
+ pkt = avctx->internal->buffer_pkt;
+
+ // This is the lesser evil. The field is for compatibility with legacy users
+ // of the legacy API, and users using the new API should not be forced to
+ // even know about this field.
+ avctx->refcounted_frames = 1;
+
+ if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
+ ret = avcodec_decode_video2(avctx, avctx->internal->buffer_frame,
+ &got_frame, pkt);
+ if (ret >= 0)
+ ret = pkt->size;
+ } else if (avctx->codec_type == AVMEDIA_TYPE_AUDIO) {
+ ret = avcodec_decode_audio4(avctx, avctx->internal->buffer_frame,
+ &got_frame, pkt);
+ } else {
+ ret = AVERROR(EINVAL);
+ }
+
+ if (ret < 0)
+ return ret;
+
+ if (ret >= pkt->size) {
+ av_packet_unref(avctx->internal->buffer_pkt);
+ } else {
+ int consumed = ret;
+
+ if (pkt != avctx->internal->buffer_pkt) {
+ if ((ret = av_packet_ref(avctx->internal->buffer_pkt, pkt)) < 0)
+ return ret;
+ }
+
+ consumed = FFMIN(consumed, avctx->internal->buffer_pkt->size);
+ avctx->internal->buffer_pkt->data += consumed;
+ avctx->internal->buffer_pkt->size -= consumed;
+ }
+
+ if (got_frame)
+ av_assert0(avctx->internal->buffer_frame->buf[0]);
+
+ return ret;
+}
+
+int attribute_align_arg avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt)
+{
+ int ret;
+
+ if (!avcodec_is_open(avctx) || !av_codec_is_decoder(avctx->codec))
+ return AVERROR(EINVAL);
+
+ if (avctx->internal->draining)
+ return AVERROR_EOF;
+
+ if (!avpkt || !avpkt->size) {
+ avctx->internal->draining = 1;
+ avpkt = NULL;
+
+ if (!(avctx->codec->capabilities & AV_CODEC_CAP_DELAY))
+ return 0;
+ }
+
+ if (avctx->codec->send_packet) {
+ ret = apply_param_change(avctx, (AVPacket *)avpkt);
+ if (ret < 0)
+ return ret;
+ return avctx->codec->send_packet(avctx, avpkt);
+ }
+
+ // Emulation via old API. Assume avpkt is likely not refcounted, while
+ // decoder output is always refcounted, and avoid copying.
+
+ if (avctx->internal->buffer_pkt->size || avctx->internal->buffer_frame->buf[0])
+ return AVERROR(EAGAIN);
+
+ // The goal is decoding the first frame of the packet without using memcpy,
+ // because the common case is having only 1 frame per packet (especially
+ // with video, but audio too). In other cases, it can't be avoided, unless
+ // the user is feeding refcounted packets.
+ return do_decode(avctx, (AVPacket *)avpkt);
+}
+
+int attribute_align_arg avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame)
+{
+ int ret;
+
+ av_frame_unref(frame);
+
+ if (!avcodec_is_open(avctx) || !av_codec_is_decoder(avctx->codec))
+ return AVERROR(EINVAL);
+
+ if (avctx->codec->receive_frame) {
+ if (avctx->internal->draining && !(avctx->codec->capabilities & AV_CODEC_CAP_DELAY))
+ return AVERROR_EOF;
+ return avctx->codec->receive_frame(avctx, frame);
+ }
+
+ // Emulation via old API.
+
+ if (!avctx->internal->buffer_frame->buf[0]) {
+ if (!avctx->internal->buffer_pkt->size && !avctx->internal->draining)
+ return AVERROR(EAGAIN);
+
+ if ((ret = do_decode(avctx, avctx->internal->buffer_pkt)) < 0) {
+ av_packet_unref(avctx->internal->buffer_pkt);
+ return ret;
+ }
+ }
+
+ if (!avctx->internal->buffer_frame->buf[0])
+ return avctx->internal->draining ? AVERROR_EOF : AVERROR(EAGAIN);
+
+ av_frame_move_ref(frame, avctx->internal->buffer_frame);
+ return 0;
+}
+
+static int do_encode(AVCodecContext *avctx, const AVFrame *frame, int *got_packet)
+{
+ int ret;
+ *got_packet = 0;
+
+ avctx->internal->buffer_pkt_valid = 0;
+
+ if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
+ ret = avcodec_encode_video2(avctx, avctx->internal->buffer_pkt,
+ frame, got_packet);
+ } else if (avctx->codec_type == AVMEDIA_TYPE_AUDIO) {
+ ret = avcodec_encode_audio2(avctx, avctx->internal->buffer_pkt,
+ frame, got_packet);
+ } else {
+ ret = AVERROR(EINVAL);
+ }
+
+ if (ret >= 0 && *got_packet) {
+ // Encoders must always return ref-counted buffers.
+ // Side-data only packets have no data and can be not ref-counted.
+ av_assert0(!avctx->internal->buffer_pkt->data || avctx->internal->buffer_pkt->buf);
+ avctx->internal->buffer_pkt_valid = 1;
+ ret = 0;
+ } else {
+ av_packet_unref(avctx->internal->buffer_pkt);
+ }
+
+ return ret;
+}
+
+int attribute_align_arg avcodec_send_frame(AVCodecContext *avctx, const AVFrame *frame)
+{
+ if (!avcodec_is_open(avctx) || !av_codec_is_encoder(avctx->codec))
+ return AVERROR(EINVAL);
+
+ if (avctx->internal->draining)
+ return AVERROR_EOF;
+
+ if (!frame) {
+ avctx->internal->draining = 1;
+
+ if (!(avctx->codec->capabilities & AV_CODEC_CAP_DELAY))
+ return 0;
+ }
+
+ if (avctx->codec->send_frame)
+ return avctx->codec->send_frame(avctx, frame);
+
+ // Emulation via old API. Do it here instead of avcodec_receive_packet, because:
+ // 1. if the AVFrame is not refcounted, the copying will be much more
+ // expensive than copying the packet data
+ // 2. assume few users use non-refcounted AVPackets, so usually no copy is
+ // needed
+
+ if (avctx->internal->buffer_pkt_valid)
+ return AVERROR(EAGAIN);
+
+ return do_encode(avctx, frame, &(int){0});
+}
+
+int attribute_align_arg avcodec_receive_packet(AVCodecContext *avctx, AVPacket *avpkt)
+{
+ av_packet_unref(avpkt);
+
+ if (!avcodec_is_open(avctx) || !av_codec_is_encoder(avctx->codec))
+ return AVERROR(EINVAL);
+
+ if (avctx->codec->receive_packet) {
+ if (avctx->internal->draining && !(avctx->codec->capabilities & AV_CODEC_CAP_DELAY))
+ return AVERROR_EOF;
+ return avctx->codec->receive_packet(avctx, avpkt);
+ }
+
+ // Emulation via old API.
+
+ if (!avctx->internal->buffer_pkt_valid) {
+ int got_packet;
+ int ret;
+ if (!avctx->internal->draining)
+ return AVERROR(EAGAIN);
+ ret = do_encode(avctx, NULL, &got_packet);
+ if (ret < 0)
+ return ret;
+ if (ret >= 0 && !got_packet)
+ return AVERROR_EOF;
+ }
+
+ av_packet_move_ref(avpkt, avctx->internal->buffer_pkt);
+ avctx->internal->buffer_pkt_valid = 0;
+ return 0;
+}
+
av_cold int avcodec_close(AVCodecContext *avctx)
{
int i;
@@ -1636,6 +1883,8 @@ av_cold int avcodec_close(AVCodecContext *avctx)
if (avctx->codec && avctx->codec->close)
avctx->codec->close(avctx);
av_frame_free(&avctx->internal->to_free);
+ av_frame_free(&avctx->internal->buffer_frame);
+ av_packet_free(&avctx->internal->buffer_pkt);
for (i = 0; i < FF_ARRAY_ELEMS(pool->pools); i++)
av_buffer_pool_uninit(&pool->pools[i]);
av_freep(&avctx->internal->pool);
@@ -1966,6 +2215,11 @@ const char *avcodec_license(void)

void avcodec_flush_buffers(AVCodecContext *avctx)
{
+ avctx->internal->draining = 0;
+ av_frame_unref(avctx->internal->buffer_frame);
+ av_packet_unref(avctx->internal->buffer_pkt);
+ avctx->internal->buffer_pkt_valid = 0;
+
if (HAVE_THREADS && avctx->active_thread_type & FF_THREAD_FRAME)
ff_thread_flush(avctx);
else if (avctx->codec->flush)
--
2.7.0
wm4
2016-02-29 21:48:14 UTC
Permalink
It's not practical to keep this with the new decode API.
---
No changes.
---
avconv.c | 7 -------
avconv.h | 1 -
2 files changed, 8 deletions(-)

diff --git a/avconv.c b/avconv.c
index 35ea1f5..d878646 100644
--- a/avconv.c
+++ b/avconv.c
@@ -1337,13 +1337,6 @@ static void process_input_packet(InputStream *ist, const AVPacket *pkt, int no_e

ist->last_dts = ist->next_dts;

- if (avpkt.size && avpkt.size != pkt->size &&
- !(ist->dec->capabilities & AV_CODEC_CAP_SUBFRAMES)) {
- av_log(NULL, ist->showed_multi_packet_warning ? AV_LOG_VERBOSE : AV_LOG_WARNING,
- "Multiple frames in a packet from stream %d\n", pkt->stream_index);
- ist->showed_multi_packet_warning = 1;
- }
-
switch (ist->dec_ctx->codec_type) {
case AVMEDIA_TYPE_AUDIO:
ret = decode_audio (ist, &avpkt, &got_output);
diff --git a/avconv.h b/avconv.h
index f0a948f..645dc30 100644
--- a/avconv.h
+++ b/avconv.h
@@ -240,7 +240,6 @@ typedef struct InputStream {
int64_t nb_samples; /* number of samples in the last decoded audio frame before looping */
PtsCorrectionContext pts_ctx;
double ts_scale;
- int showed_multi_packet_warning;
AVDictionary *decoder_opts;
AVRational framerate; /* framerate forced with -r */
--
2.7.0
wm4
2016-02-29 21:48:15 UTC
Permalink
Rather than refactoring everything, add a function that resembles the
avcodec_decode_audio4() API.
---
No changes.
---
avconv.c | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/avconv.c b/avconv.c
index d878646..c74b4b8 100644
--- a/avconv.c
+++ b/avconv.c
@@ -1079,6 +1079,31 @@ static void do_streamcopy(InputStream *ist, OutputStream *ost, const AVPacket *p
write_frame(of->ctx, &opkt, ost);
}

+static int decode(AVCodecContext *avctx, AVFrame *frame, int *got_frame, AVPacket *pkt)
+{
+ int ret;
+ int consumed = 0;
+
+ *got_frame = 0;
+
+ // This relies on the fact that the decoder will not buffer additional
+ // packets internally, but returns AVERROR(EAGAIN) if there are still
+ // decoded frames to be returned.
+ ret = avcodec_send_packet(avctx, pkt);
+ if (ret < 0 && ret != AVERROR(EAGAIN) && ret != AVERROR_EOF)
+ return ret;
+ if (ret >= 0)
+ consumed = pkt->size;
+
+ ret = avcodec_receive_frame(avctx, frame);
+ if (ret < 0 && ret != AVERROR(EAGAIN) && ret != AVERROR_EOF)
+ return ret;
+ if (ret >= 0)
+ *got_frame = 1;
+
+ return consumed;
+}
+
int guess_input_channel_layout(InputStream *ist)
{
AVCodecContext *dec = ist->dec_ctx;
@@ -1109,7 +1134,7 @@ static int decode_audio(InputStream *ist, AVPacket *pkt, int *got_output)
return AVERROR(ENOMEM);
decoded_frame = ist->decoded_frame;

- ret = avcodec_decode_audio4(avctx, decoded_frame, got_output, pkt);
+ ret = decode(avctx, decoded_frame, got_output, pkt);
if (!*got_output || ret < 0)
return ret;

@@ -1200,8 +1225,7 @@ static int decode_video(InputStream *ist, AVPacket *pkt, int *got_output)
return AVERROR(ENOMEM);
decoded_frame = ist->decoded_frame;

- ret = avcodec_decode_video2(ist->dec_ctx,
- decoded_frame, got_output, pkt);
+ ret = decode(ist->dec_ctx, decoded_frame, got_output, pkt);
if (!*got_output || ret < 0)
return ret;

@@ -1216,7 +1240,6 @@ static int decode_video(InputStream *ist, AVPacket *pkt, int *got_output)

decoded_frame->pts = guess_correct_pts(&ist->pts_ctx, decoded_frame->pkt_pts,
decoded_frame->pkt_dts);
- pkt->size = 0;

if (ist->st->sample_aspect_ratio.num)
decoded_frame->sample_aspect_ratio = ist->st->sample_aspect_ratio;
--
2.7.0
wm4
2016-02-29 21:48:16 UTC
Permalink
The flushing case is a bit strange; not simplifying it so the change is
less noisy.
---
No changes.
---
avconv.c | 44 ++++++++++++++++++++++++++++----------------
1 file changed, 28 insertions(+), 16 deletions(-)

diff --git a/avconv.c b/avconv.c
index c74b4b8..40152ff 100644
--- a/avconv.c
+++ b/avconv.c
@@ -458,7 +458,7 @@ static void do_video_out(AVFormatContext *s,
AVFrame *in_picture,
int *frame_size)
{
- int ret, format_video_sync, got_packet;
+ int ret, format_video_sync;
AVPacket pkt;
AVCodecContext *enc = ost->enc_ctx;

@@ -508,13 +508,17 @@ static void do_video_out(AVFormatContext *s,

ost->frames_encoded++;

- ret = avcodec_encode_video2(enc, &pkt, in_picture, &got_packet);
- if (ret < 0) {
- av_log(NULL, AV_LOG_FATAL, "Video encoding failed\n");
- exit_program(1);
- }
+ ret = avcodec_send_frame(enc, in_picture);
+ if (ret < 0)
+ goto error;
+
+ while (1) {
+ ret = avcodec_receive_packet(enc, &pkt);
+ if (ret == AVERROR(EAGAIN))
+ break;
+ if (ret < 0)
+ goto error;

- if (got_packet) {
av_packet_rescale_ts(&pkt, enc->time_base, ost->st->time_base);
write_frame(s, &pkt, ost);
*frame_size = pkt.size;
@@ -532,6 +536,12 @@ static void do_video_out(AVFormatContext *s,
* flush, we need to limit them here, before they go into encoder.
*/
ost->frame_number++;
+
+ return;
+error:
+ av_assert0(ret != AVERROR(EAGAIN) && ret != AVERROR_EOF);
+ av_log(NULL, AV_LOG_FATAL, "Video encoding failed\n");
+ exit_program(1);
}

static double psnr(double d)
@@ -946,39 +956,41 @@ static void flush_encoders(void)
if (enc->codec_type == AVMEDIA_TYPE_AUDIO && enc->frame_size <= 1)
continue;

+ if (enc->codec_type != AVMEDIA_TYPE_VIDEO && enc->codec_type != AVMEDIA_TYPE_AUDIO)
+ continue;
+
+ avcodec_send_frame(enc, NULL);
+
for (;;) {
- int (*encode)(AVCodecContext*, AVPacket*, const AVFrame*, int*) = NULL;
- const char *desc;
+ const char *desc = NULL;

switch (enc->codec_type) {
case AVMEDIA_TYPE_AUDIO:
- encode = avcodec_encode_audio2;
desc = "Audio";
break;
case AVMEDIA_TYPE_VIDEO:
- encode = avcodec_encode_video2;
desc = "Video";
break;
default:
- stop_encoding = 1;
+ av_assert0(0);
}

- if (encode) {
+ if (1) {
AVPacket pkt;
int got_packet;
av_init_packet(&pkt);
pkt.data = NULL;
pkt.size = 0;

- ret = encode(enc, &pkt, NULL, &got_packet);
- if (ret < 0) {
+ ret = avcodec_receive_packet(enc, &pkt);
+ if (ret < 0 && ret != AVERROR_EOF) {
av_log(NULL, AV_LOG_FATAL, "%s encoding failed\n", desc);
exit_program(1);
}
if (ost->logfile && enc->stats_out) {
fprintf(ost->logfile, "%s", enc->stats_out);
}
- if (!got_packet) {
+ if (ret == AVERROR_EOF) {
stop_encoding = 1;
break;
}
--
2.7.0
Anton Khirnov
2016-03-04 07:16:42 UTC
Permalink
No important objections from me now. Since this is a rather large
change, I'd wait until next week for more comments, then push it if
there are none.

Are you planning to convert any actual encoders/decoders to the new API?
Also, what about deprecating the old one?
--
Anton Khirnov
wm4
2016-03-04 09:36:09 UTC
Permalink
On Fri, 04 Mar 2016 08:16:42 +0100
Post by Anton Khirnov
No important objections from me now. Since this is a rather large
change, I'd wait until next week for more comments, then push it if
there are none.
I also have some follow up patches to make. But would be nice if
patches 1, 2, 4 could be applied now already.
Post by Anton Khirnov
Are you planning to convert any actual encoders/decoders to the new API?
No, other than maybe MMAL (i.e. the thing nobody cares for). I hope that
eventually multithreaded decoding could use this API, but since that
would preclude using that with the "old" API, and since it's
complicated, it will probably take a LONG time until this happens.
Post by Anton Khirnov
Also, what about deprecating the old one?
I'd argue it shouldn't be done now.
Anton Khirnov
2016-03-04 10:07:02 UTC
Permalink
Quoting wm4 (2016-03-04 10:36:09)
Post by wm4
On Fri, 04 Mar 2016 08:16:42 +0100
Post by Anton Khirnov
No important objections from me now. Since this is a rather large
change, I'd wait until next week for more comments, then push it if
there are none.
I also have some follow up patches to make. But would be nice if
patches 1, 2, 4 could be applied now already.
Ok, will do.
Post by wm4
Post by Anton Khirnov
Are you planning to convert any actual encoders/decoders to the new API?
No, other than maybe MMAL (i.e. the thing nobody cares for). I hope that
eventually multithreaded decoding could use this API, but since that
would preclude using that with the "old" API, and since it's
complicated, it will probably take a LONG time until this happens.
Post by Anton Khirnov
Also, what about deprecating the old one?
I'd argue it shouldn't be done now.
Why not? As you say above, it's problematic to convert any codecs to the
new API when no callers support it. E.g. I think QSV would benefit from
converting to the new API, but that breaks all current callers.

So we want everyone to switch ASAP, and deprecating the old API right
now will help that IMO. That doesn't mean we have to remove it right at
the next bump, it can live on life support for a while.
--
Anton Khirnov
Continue reading on narkive:
Loading...