Claims
1 . A video encoding method using fractionalaccuracy motion compensation, the method comprising:
a step of generating multiple weighted filter coefficients from multiple interpolation filter sets using weight parameters which designate weight values of interpolation filter coefficients; a step of selecting weighted filter coefficients which minimize prediction error energy in the fractionalaccuracy motion compensation from among the weighted filter coefficients; a step of interpolating fractionalaccuracy pixels for a reference picture by application of an interpolation filter with the selected weighted filter coefficients and encoding an input picture with the fractionalaccuracy motion compensation; and a step of encoding the weight parameters used for generating the selected weighted filter coefficients and the interpolation filter coefficients and adding encoded weight parameters and encoded interpolation filter coefficients to an encoded bitstream.
2 . The video encoding method according to claim 1 , wherein selection of the weighted filter coefficients is performed for each interpolation position group of the fractionalaccuracy pixels calculated by a predetermined method or for each designated interpolation position group of the fractionalaccuracy pixels, and
an interpolation filter with the weighted filter coefficients is switched for each interpolation position group of the fractionalaccuracy pixels.
3 . A video encoding apparatus using fractionalaccuracy motion compensation, the apparatus comprising:
a weighted filter coefficient generation unit which generates multiple weighted filter coefficients from multiple interpolation filter sets using weight parameters which designate weight values of interpolation filter coefficients; a weighted filter coefficient selection unit which selects weighted filter coefficients which minimize prediction error energy in the fractionalaccuracy motion compensation from among the weighted filter coefficients; a first encoding unit which interpolates fractionalaccuracy pixels for a reference picture by application of an interpolation filter with the selected weighted filter coefficients and encodes an input picture with the fractionalaccuracy motion compensation; and a second encoding unit which encodes the weight parameters used for generating the selected weighted filter coefficients and the interpolation filter coefficients and adds encoded weight parameters and encoded interpolation filter coefficients to an encoded bitstream.
4 . The video encoding apparatus according to claim 3 , wherein selection of the weighted filter coefficients is performed for each interpolation position group of the fractionalaccuracy pixels calculated by a predetermined method or for each designated interpolation position group of the fractionalaccuracy pixels, and
an interpolation filter with the weighted filter coefficients is switched for each interpolation position group of the fractionalaccuracy pixel.
5 . A video decoding method using fractionalaccuracy motion compensation, the method comprising,
a step of decoding interpolation filter coefficients for generating fractionalaccuracy pixels and weight parameters which designate weight values of the interpolation filter coefficients; a step of generating weighted filter coefficients determined by a weighted sum of the decoded interpolation filter coefficients using the decoded weight parameters; a step of generating a prediction signal using an interpolation filter determined by the generated weighted filter coefficients; a step of decoding a residual signal; and a step of generating a decoded picture using the decoded residual signal and the generated prediction signal.
6 . The video decoding method according to claim 5 , wherein decoding of the interpolation filter coefficients for generating the fractionalaccuracy pixels and the weight parameters which designate the weight values of the interpolation filter coefficients is performed for each interpolation position group designated by an encoding end or for each predetermined interpolation position group, and
an interpolation filter determined by the weighted filter coefficients is switched for each interpolation position group.
7 . A video decoding apparatus using fractionalaccuracy motion compensation, the apparatus comprising:
a first decoding unit which decodes interpolation filter coefficients for generating fractionalaccuracy pixels and weight parameters which designate weight values of the interpolation filter coefficients; a weighted filter coefficient generation unit which generates weighted filter coefficients determined by a weighted sum of the decoded interpolation filter coefficients using the decoded weight parameters; a prediction signal generation unit which generates a prediction signal using an interpolation filter determined by the generated weighted filter coefficients; a second decoding unit which decodes a residual signal; and a decoded picture generation unit which generates a decoded picture using the decoded residual signal and the generated prediction signal.
8 . The video decoding apparatus according to claim 7 , wherein decoding of the interpolation filter coefficients for generating the fractionalaccuracy pixels and the weight parameters which designate the weight values of the interpolation filter coefficients is performed for each interpolation position group designated by an encoding end or for each predetermined interpolation position group, and
an interpolation filter determined by the weighted filter coefficients is switched for each interpolation position group.
9 . A video encoding program which causes a computer to execute the video encoding method according to claim 1 .
10 . A video decoding program which causes a computer to execute the video decoding method according to claim 5 .
11 . A video encoding program which causes a computer to execute the video encoding method according to claim 2 .
12 . A video decoding program which causes a computer to execute the video decoding method according to claim 6 .
TECHNICAL FIELD
[0001] The present invention relates to video encoding/decoding technologies that achieve an improvement in the performance of interpolation filters in video coding and improve the coding efficiency.
[0002] Priority is claimed on Japanese Patent Application No. 2011141725, filed Jun. 27, 2011, the content of which is incorporated herein by reference.
BACKGROUND ART
[0003] In video coding, in a case of interframe prediction (motion compensation) coding, which performs prediction between different frames, an already decoded frame is referred to, a motion vector which minimizes the prediction error energy is determined, and a prediction error signal (also called a residual signal) thereof is subjected to orthogonal transform. Thereafter, quantization is applied, entropy encoding is performed, and finally binary data, i.e., a bitstream is obtained. In order to increase the coding efficiency, it is indispensable to reduce the prediction error energy, and thus a prediction scheme that provides high prediction accuracy is required.
[0004] A great number of tools for increasing the accuracy of interframe prediction have been introduced into video coding standard schemes. For example, if there is occlusion in the nearest frame, the prediction error energy can be further reduced by referring to a frame that is distant in the time domain to some extent, and thus, in H.264/AVC, multiple frames can be referred to. This tool is called multiple reference frame prediction. In addition, in order to be able to deal with motions having complex shapes, a block size can be subdivided, such as 16×8, 8×16, 8×4, 4×8, and 4×4, in addition to 16×16 and 8×8. This tool is called variable block size prediction.
[0005] Similar to these, ½ accuracy pixels are interpolated from integeraccuracy pixels of a reference frame using a 6tap filter, and then ¼ accuracy pixels are generated by linear interpolation using these pixels. Accordingly, it becomes possible to realize accurate prediction for motions of fractional accuracy. This tool is called ¼ pixel accuracy prediction.
[0006] In order to develop the nextgeneration video coding standard scheme that provides higher coding efficiency than that of H.264/AVC, International Organization for Standardization/International Electrotechnical Commission “Moving Picture Experts Group” (the international organization for standardization ISO/IEC “MPEG”) and International Telecommunication UnionTelecommunication Standardization Sector “Video Coding Experts Group” (ITUT “VCEG”) collaboratively established an investigation team (Joint Collaborative Team for Video Coding: JCTVC). The nextgeneration standard scheme is called high efficiency video coding: HEVC, various novel coding technologies are now gathering from all over the world, and they are under discussion in the JCTVC meetings.
[0007] Among them, in particular, many proposals related to interframe prediction (motion compensation) have been presented, and reference software for HEVC (HEVC test Model: HM) employs tools for improving the prediction efficiency of motion vectors and tools for extending the block size to 16×16 or larger.
[0008] Moreover, tools for increasing the interpolation accuracy of fractionalaccuracy pixels have also been proposed, and a DCTbased interpolation filter: DCTIF, in which interpolation filter coefficients are derived from basis of discrete cosine transform (DCT) coefficients, is highly effective and it is adopted in HM. In order to further increase the interpolation accuracy, interpolation filters which adaptively change interpolation filter coefficients on a framebyframe basis are also proposed, which are called adaptive interpolation filters: AIFs. The adaptive interpolation filter is highly effective in terms of an improvement in the coding efficiency, and it is also adopted in reference software for the nextgeneration video coding (key technical area: KTA) that was developed under the leadership of VCEG. Because of a high contribution to an improvement in the coding efficiency, an improvement in the performance of interpolation filters is a very expectative domain.
[0009] Conventional interpolation filters will be described in greater detail.
[Fixed Interpolation]
[0010] FIG. 8 is a diagram illustrating an interpolation method of a fractionalaccuracy pixel in H.264/AVC. In H.264/AVC, as shown in FIG. 8 , when a ½ pixel position is interpolated, interpolation is performed using six integer pixels in total including three points on the left side of the interpolation target pixel and three points on the right side of the interpolation target pixel. With respect to the vertical direction, interpolation is performed using six integer pixels in total including three points on the upper side and three points on the lower side. Filter coefficients are [(1, −5, 20, 20, −5, 1)/32]. After ½ pixels positions have been interpolated, ¼ pixels positions are interpolated using a mean filter of [½, ½]. Since it is necessary to interpolate all the ½ pixels positions, the computational complexity is high, but highperformance interpolation is possible, so that the coding efficiency is improved. NonPatent Document 1 and so on disclose the above interpolation technology using a fixed filter.
[0011] Filters which use the same coefficient values for all the input pictures and for all the frames, such as a onedimensional 6tap filter of H.264/AVC, are called fixed interpolation filters.
[0012] As a scheme for further improving the performance of an interpolation filter adopted in H.264/AVC, the reference software HM for HEVC adopts a DCTbased interpolation filter (DCTIF). FIG. 9 illustrates an interpolation method of a fractionalaccuracy pixel by the DCTbased interpolation filter. As shown in FIG. 9 , it is assumed that p denotes an interpolation target pixel at a fractionalaccuracy position, p x denotes an integer position pixel, and a (0≦α≦1) denotes a parameter indicating the position of p between integer position pixels. At this time, it is assumed that the number of integer position pixels to be used for interpolation, i.e., a tap length, is 2M (M is an integer that is greater than or equal to 1). From the definitional equation of DCT transform, Equation (1) holds.
[0000]
[
Equation
1
]
C
k
=
1
M
∑
l
=

M
+
1
M
p
(
l
)
cos
(
(
2
l

1
+
2
M
)
k
π
4
M
)
(
1
)
[0013] Moreover, from the definitional equation of inverse DCT transform, Equation (2) holds.
[0000]
[
Equation
2
]
p
(
x
)
=
C
0
2
∑
k
=
1
2
M

1
C
k
cos
(
π
(
2
x

1
+
2
M
)
k
4
M
)
(
2
)
[0014] When x is regarded as a position, an equation for interpolating a pixel at a fractional position α is represented by the following Equation (3).
[0000]
[
Equation
3
]
p
(
α
)
=
C
0
2
+
∑
k
=
1
2
M

1
C
k
cos
(
π
(
2
α

1
+
2
M
)
k
4
M
)
(
3
)
[0015] From Equation (3), it is possible to uniquely derive coefficients once the tap length 2M to be used for interpolation and the interpolation target position α are determined. Examples of an interpolation filter obtained from the above discussion are collected in Table 1 and Table 2. The details of the above are disclosed in NonPatent Document 2.
[0000]
TABLE 1
Fractional Position α
Filter Coefficient Values (6Tap Filter, 2M = 6)
− 1/12
{−4, 19, 254, −19, 8, −2}
1/12
{4, −16, 252, 22, −8, 2}
⅙
{6, −28, 242, 48, −17, 5}
¼
{9, −37, 227, 75, −25, 7}
2/6
{11, −42, 208, 103, −33, 9}
5/12
{12, −44, 184, 132, −39, 11}
½
{11, −43, 160, 160, −43, 11}
7/12
{11, −39, 132, 184, −44, 12}
⅔
{9, −33, 103, 208, −42, 11}
¾
{7, −25, 75, 227, −37, 9}
⅚
{5, −17, 48, 242, −28, 6}
[0000]
TABLE 2
Fractional
Position α
Filter Coefficient Values (12Tap Filter, 2M = 12)
− 1/12
{1, −3, 5, −10, 22, 253, −19, 10, −6, 4, −2, 1}
1/12
{−1, 3, −5, 9, −19, 253, 23, −10, 6, −4, 2, −1}
⅙
{−2, 5, −9, 16, −34, 244, 49, −21, 12, −7, 4, −1}
¼
{−1, 6, −12, 21, −43, 229, 75, −30, 17, −10, 5, −1}
2/6
{−3, 8, −15, 26, −50, 211, 105, −40, 22, −13, 7, −2}
5/12
{−3, 9, −16, 28, −53, 188, 134, −47, 26, −15, 8, −3}
½
{−2, 7, −15, 28, −52, 162, 162, −52, 28, −15, 7, −2}
7/12
{−3, 8, −15, 26, −47, 134, 188, −53, 28, −16, 9, −3}
⅔
{−2, 7, −13, 22, −40, 105, 211, −50, 26, −15, 8, −3}
¾
{−1, 5, −10, 17, −30, 75, 229, −43, 21, −12, 6, −1}
⅚
{−1, 4, −7, 12, −21, 49, 244, −34, 16, −9, 5, −2}
[0016] DCTbased interpolation filters are capable of dealing with any filter length and any interpolation accuracy and they are highperformance interpolation filters, so that they are adopted in the test model HM for HEVC.
[Adaptive Interpolation]
[0017] In H.264/AVC, the values of filter coefficients are constant, irrespective of conditions of an input picture (the type of a sequence, the size of a picture, and a frame rate) and coding conditions (the block size, the structure of a group of pictures (GOP), and quantization parameters (QP)). When the values of the filter coefficients are fixed, for example, effects that vary over time, such as aliasing, a quantization error, an error resulting from motion estimation, and camera noise, are not taken into consideration. Therefore, it is considered that an improvement in the performance is limited in terms of the coding efficiency. Accordingly, NonPatent Document 3 proposes a scheme of adaptively changing interpolation filter coefficients, which is called a nonseparable adaptive interpolation filter.
[0018] In NonPatent Document 3, a twodimensional interpolation filter (6×6=36 filter coefficients in total) is assumed, and the filter coefficients are determined so as to minimize the prediction error energy. Although it is possible to realize higher coding efficiency than that obtained by a onedimensional 6tap fixed interpolation filter used in H.264/AVC, the computational complexity for determining filter coefficients is very high, and thus NonPatent Document 4 introduces a proposal for reducing the computational complexity.
[0019] The technique introduced in NonPatent Document 4 is called a separable adaptive interpolation filter (SAIF), and it uses a onedimensional 6tap interpolation filter rather than a twodimensional interpolation filter.
[0020] FIG. 10A to FIG. 10C are diagrams illustrating a method for interpolating a fractionalaccuracy pixel in the separable adaptive interpolation filter (SAIF). Its procedure is such that, first, as shown by step 1 in FIG. 10B , pixels in the horizontal direction (a, b, and c) are interpolated. Integeraccuracy pixels C1 to C6 are used for determining filter coefficients. Filter coefficients in the horizontal direction that minimize a prediction error energy function E h 2 of Equation (4) are analytically determined by the commonly known least square method (see NonPatent Document 3).
[0000]
[
Equation
4
]
E
h
2
=
∑
x
,
y
(
S
x
,
y

∑
c
i
w
c
i
·
P
x
~
+
c
i
,
y
~
)
2
(
4
)
[0021] Here, S denotes an original picture, P denotes an already decoded reference picture, and x and y respectively denote positions in the horizontal direction and the vertical direction in a picture. Moreover, ˜x (˜ is a symbol placed above x; the same is also applied to the others) satisfies ˜x=x+MV x −FilterOffset, where MV x denotes the horizontal component of a motion vector that has been obtained beforehand, and FilterOffset denotes an offset for adjustment (the value obtained by dividing a filter length in the horizontal direction by 2). With respect to the vertical direction, ˜y=y+MV y is satisfied, where MV y denotes the vertical component of the motion vector. w ci denotes a group of filter coefficients in the horizontal direction c i (0≦c i <6) that is to be determined.
[0022] Linear equations the number of which is equal to the number of the filter coefficients determined by Equation (4) are obtained, and minimizing processes are performed for fractionalpixel positions in the horizontal direction independently of one another. Through the minimizing processes, three groups of 6tap filter coefficients are determined, and fractionalaccuracy pixels a, b, and c are interpolated using these filter coefficient groups.
[0023] After the interpolation of the pixels in the horizontal direction has been completed, as shown by step 2 in FIG. 10C , an interpolation process in the vertical direction is performed. Filter coefficients in the vertical direction are determined by solving a linear problem similar to that in the horizontal direction. Specifically, filter coefficients in the vertical direction that minimize a prediction error energy function E v 2 of Equation (5) are analytically determined.
[0000]
[
Equation
5
]
E
v
2
=
∑
x
,
y
(
S
x
,
y

∑
c
j
w
c
j
·
P
^
x
~
,
y
~
+
c
j
)
2
(
5
)
[0024] Here, S denotes an original picture, ̂P (̂ is a symbol placed above P) denotes a picture which has been decoded and then interpolated in the horizontal direction, and x and y respectively denote positions in the horizontal direction and the vertical direction in a picture. Moreover, ˜x is represented as 4·(x+MV x ), where MV x denotes the horizontal component of a motion vector that has been rounded off to the nearest whole number. With respect to the vertical direction, ˜y is represented as y+MV y −FilterOffset, where MV y denotes the vertical component of the motion vector, and FilterOffset denotes an offset for adjustment (the value obtained by dividing a filter length by 2). w cj denotes a group of filter coefficients in the vertical direction c j (0≦c j ≦6) that is to be determined.
[0025] Minimizing processes are performed for fractionalaccuracy pixels independently of one another, and 12 groups of 6tap filter coefficients are obtained. The remaining fractionalaccuracy pixels are interpolated using these filter coefficients.
[0026] From the above, it is necessary to encode 90 (=6×15) filter coefficients in total and transmit them to a decoding end. In particular, since the overhead becomes large in low resolution coding, filter coefficients to be transmitted are reduced using the symmetry of a filter. For example, as show in FIG. 10A , viewed from integeraccuracy pixels, b, h, i, j, and k are positioned at the centers with respect to interpolation directions, and with respect to the horizontal direction, coefficients obtained by inverting coefficients to be used for three points on the left side can be applied to three points on the right side. Similarly, with respect to the vertical direction, coefficients obtained by inverting coefficients to be used for three points on the upper side can be applied to three points on the lower side (c 1 =c 6 , c 2 =c 5 , and c 3 =c 4 ).
[0027] Additionally, since the relationship between d and l is symmetric about h, inverted filter coefficients can be used. That is, by transmitting 6 coefficients for d, their values can be applied to l. c(d) 1 =c(l) 6 , c(d) 2 =c(l) 5 , c(d) 3 =c(l) 4 , c(d) 4 =c(l) 3 , c(d) 5 =c(l) 2 , and c(d) 6 =c(l) 1 are satisfied. This symmetry is also used for e and m, f and n, and g and o. Although the same theory holds for a and c, since the result for the horizontal direction affects interpolation in the vertical direction, a and c are transmitted separately without using symmetry. As a result of the use of the symmetry described above, the number of filter coefficients to be transmitted for each frame is 51 (15 for the horizontal direction and 36 for the vertical direction).
[0028] In the above adaptive interpolation filter of NonPatent Document 4, the processing unit of the minimization process of the prediction error energy is fixed to a frame. 51 filter coefficients are determined per one frame. If an encoding target frame is roughly divided into two types of texture regions A and B (or multiple types), the optimum filter coefficients are a group of coefficients in which both of them (all the textures) are taken into consideration. Under a situation in which characteristic filter coefficients are essentially obtained only in the vertical direction with respect to the region A and filter coefficients are obtained only in the horizontal direction with respect to the region B, filter coefficients are derived as the average of both of them.
[0029] NonPatent Document 5 proposes a method for achieving a reduction in the prediction error energy and realizing an improvement in the coding efficiency by performing division into regions in accordance with the local property of a picture and generating interpolation filter coefficients for each divided region, without being limited to one group of filter coefficients (51 coefficients) per one frame.
[0030] Moreover, in order to improve the performance of the adaptive interpolation filter of NonPatent Document 4, a technology of grouping interpolation positions, selecting a fixed interpolation filter or an adaptive interpolation filter for each group so that the prediction error energy can be reduced, and generating an interpolated picture is proposed (see NonPatent Document 6).
PRIOR ART DOCUMENTS
Patent Document
[0000]
Patent Document 1: Japanese Unexamined Patent Application, First Publication No. 201182725
NonPatent Documents
[0000]
NonPatent Document 1: Sakae Okubo, Shinya Kadono, Yoshihiro Kikuchi, and Teruhiko Suzuki, “H.264/AVC Textbook: 3rd Revised Edition”, Impress R&D, pp. 119123, 2009
NonPatent Document 2: Ken McCann, WooJin Han, IlKoo Kim, JungHye Min, Elena Alshina, Alexander Alshin, Tammy Lee, Jianle Chen, Vadim Seregin, Sunil Lee, YoonMi Hong, MinSu Cheon, and Nikolay Shlyakhov, “Samsung's Response to the Call for Proposals on Video Compression Technology”, JCTVCA124 r2, pp. 1214, 1st JCTVC Meeting, Dresden, April 2010
NonPatent Document 3: Y. Vatis, B. Edler, D. T. Nguyen, and J. Ostermann: “Motion and aliasingcompensated prediction using a twodimensional nonseparable adaptive Wiener interpolation filter”, Proc. ICIP2005, IEEE International Conference on Image Processing, pp. II 894897, Genova, Italy, September 2005
NonPatent Document 4: S. Wittmann and T. Wedi: “Separable adaptive interpolation filter for video coding”, Proc. ICIP2008, IEEE International Conference on Image Processing, pp. 25002503, San Diego, Calif., USA, October 2008
NonPatent Document 5: Shohei Matsuo, Yukihiro Bandoh, Seishi Takamura, and Hirohisa Jozawa: “Enhanced regionbased adaptive interpolation filter”, Proc. PCS2010, IEEE Picture Coding Symposium, pp. 526529, Nagoya, Japan, December 2010
NonPatent Document 6: Faouzi Kossentini, Nader Mandi, Hsan Guermazi, and Mohammed Ali Ben Ayed: “An Adaptive Interpolation Filtering Technique”, JCTVCE284, 5th JCTVC Meeting, Geneva, March 2011
SUMMARY OF INVENTION
Problems to be solved by the Invention
[0038] The interpolation filters disclosed in NonPatent Document 4 and NonPatent Document 5 are not provided with a function of switching an interpolation filter depending on each interpolation position, and thus there is room for improvement in terms of an increase in the performance of motion compensation.
[0039] The interpolation filter disclosed NonPatent Document 6, which has adaptability with respect to interpolation positions, determines whether a fixed interpolation filter which has been previously defined for each interpolation position is to be used or an adaptive interpolation filter which has been derived on a framebyframe basis is to be used. This is a scheme in which selection of the filters is performed from the viewpoint of minimizing the prediction error energy and either one of them is necessarily selected.
[0040] In order to achieve an increase in the coding efficiency by selection from multiple interpolation filters, it is considered that the prediction error energy can be minimized if there is a wide range of choice for an interpolation filter. However, the method disclosed in NonPatent Document 6 employs a choice between two alternatives, and thus it is considered that there is a limit on an improvement in performance. If it is possible to design an interpolation filter which realizes a further reduction in the prediction error energy by incorporating the weighted sum of multiple interpolation filters into the range of the choice, it is possible to realize an improvement in the coding efficiency.
[0041] In order to solve the above problems, an object of the present invention is to provide a novel scheme which extends the range of choice for an interpolation filter to thereby reduce the prediction error energy in interframe prediction with motion compensation than those of conventional technologies and improve the coding efficiency.
Means for Solving the Problems
[0042] As a method for achieving the above object, the present invention employs a weighted interpolation filter as an alternative in generation of an interpolated picture under an assumption that the weighted sum of multiple interpolation filters can further reduce the prediction error energy. If it is determined that the weighted interpolation filter minimizes the prediction error energy, weight coefficients are transmitted. With the present means, it is possible to more flexibly generate an interpolated prediction picture, and an improvement in the coding efficiency can be realized as a result of a reduction in the prediction error energy.
[0043] For example, the present invention performs the following processing in video encoding using fractionalaccuracy motion compensation.
[0044] Interpolation positions of fractionalaccuracy pixels are divided into multiple groups in accordance with a previously designated method or some another method.
[0045] Weighted filter coefficients are derived from multiple sets of interpolation filters using multiple parameters for each interpolation position group.
[0046] From among the weighted filter coefficients, weighted filter coefficients which minimize the prediction error energy are selected.
[0047] An interpolation filter with the weighted filter coefficients is applied to interpolate fractionalaccuracy pixels for a reference picture and an input picture is encoded using fractionalaccuracy motion compensation.
[0048] Weight parameters which designate weight values of the selected weighted filter coefficients and the interpolation filter coefficients used for the motion compensation are encoded and added to an encoded bitstream.
[0049] Moreover, for example, the present invention performs the following processing in video decoding using fractionalaccuracy motion compensation.
[0050] Weight parameters which designate weight values of interpolation filter coefficients and interpolation filter coefficients for generating fractionalaccuracy pixels are decoded for each interpolation position group.
[0051] Weighted filter coefficients determined by the weighted sum of the decoded interpolation filter coefficients are generated using the decoded weight parameters.
[0052] A prediction signal is generated using the generated weighted filter coefficients.
[0053] A residual signal is decoded and a decoded picture is generated using the residual signal and the prediction signal.
[0054] The following is the operation of the present invention. Conventional interpolation filters having adaptability with respect to interpolation positions employ a scheme of a choice between two alternatives with respect to a switching unit of an interpolation filter, and thus there is a limit on an improvement in performance. In contrast, for example, the present invention performs motion compensation by calculating the weighted sum of multiple filter coefficients to obtain the optimum weight coefficients for each interpolation position group and applying an interpolation filter using the weight coefficients. As a result, it is possible to more flexibly represent an interpolation filter, achieve an improvement in the performance of motion compensation by generating a filter that reduces the prediction error energy, and improve the coding efficiency.
Advantageous Effects of the Invention
[0055] The present invention can represent the weighted sum of interpolation filters, which cannot be taken into consideration by conventional interpolation filters having adaptability with respect to interpolation positions, extend the range of choice for an interpolation filter, and achieve an improvement in the coding efficiency as a result of a reduction in the prediction error energy.
BRIEF DESCRIPTION OF DRAWINGS
[0056] FIG. 1 is a diagram illustrating an example of a configuration of a video encoding apparatus in accordance with an embodiment of the present invention.
[0057] FIG. 2 is a diagram illustrating a configuration example 1 of an interpolation filter coefficient determination unit.
[0058] FIG. 3 is a diagram illustrating a configuration example 2 of the interpolation filter coefficient determination unit.
[0059] FIG. 4 is a flowchart of an encoding process in accordance with an embodiment of the present invention.
[0060] FIG. 5 is a diagram illustrating an example of a configuration of a video decoding apparatus in accordance with an embodiment of the present invention.
[0061] FIG. 6 is a flowchart of a decoding process in accordance with an embodiment of the present invention.
[0062] FIG. 7 is a diagram illustrating an example of a configuration of a system when an embodiment of the present invention is implemented using a computer and a software program.
[0063] FIG. 8 is a diagram illustrating a method for interpolating a fractionalaccuracy pixel in a video coding standard scheme (H.264/AVC).
[0064] FIG. 9 is a diagram illustrating a method for interpolating a fractionalaccuracy pixel in a DCTbased interpolation filter (DCTIF).
[0065] FIG. 10A is a diagram illustrating a method for interpolating a fractionalaccuracy pixel in a separable adaptive interpolation filter (SAIF).
[0066] FIG. 10B is a diagram illustrating a method for interpolating a fractionalaccuracy pixel in a separable adaptive interpolation filter (SAIF).
[0067] FIG. 10C is a diagram illustrating a method for interpolating a fractionalaccuracy pixel in a separable adaptive interpolation filter (SAIF).
MODES FOR CARRYING OUT THE INVENTION
[0068] Hereinafter, an embodiment of the present invention will be described with reference to drawings.
Configuration Example of Video Encoding Apparatus
[0069] FIG. 1 is a diagram illustrating an example of a configuration of a video encoding apparatus in accordance with an embodiment of the present invention.
[0070] In a video encoding apparatus 10 , an interpolation filter coefficient calculation unit 11 calculates interpolation filter coefficients of fractionalaccuracy pixels used for a reference picture in predictive encoding. An interpolation filter coefficient determination unit 12 calculates the weighted sum of multiple filter coefficients to obtain the optimum weight coefficients using a motion vector MV detected by a motion detection unit 132 and outputs weight parameters and interpolation filter coefficients. Moreover, the interpolation filter coefficient determination unit 12 outputs interpolation position group information when interpolation position groups which are switching units of the weighted filter coefficients are dynamically determined.
[0071] A prediction signal generation unit 13 is provided with a reference picture interpolation unit 131 and the motion detection unit 132 . The reference picture interpolation unit 131 applies an interpolation filter having interpolation filter coefficients selected by the interpolation filter coefficient determination unit 12 to an already decoded reference picture stored in a reference picture memory 17 . The motion detection unit 132 performs a motion search on an interpolated reference picture to calculate a motion vector. The prediction signal generation unit 13 performs motion compensation using the fractionalaccuracy motion vector calculated by the motion detection unit 132 to generate a prediction signal.
[0072] A predictive encoding unit 14 performs predictive encoding by, for example, calculating a residual signal between an input video signal and the prediction signal, performing orthogonal transform thereon, and quantizing transform coefficients. Moreover, a decoding unit 16 performs decoding on the result of the predictive encoding and stores a decoded picture in the reference picture memory 17 for later predictive encoding. At this time, an inloop filter for removing coding noise, such as a deblocking filter or an adaptive loop filter (ALF), may be applied and then the storage may be performed.
[0073] A variablelength encoding unit 15 performs variablelength encoding on quantized transform coefficients and motion vectors as well as weight parameters and interpolation filter coefficients, which are outputs of the interpolation filter coefficient determination unit 12 , and outputs them as an encoded bitstream. Moreover, when an interpolation position group of fractionalaccuracy pixels, which is a switching unit of weighted filter coefficients, is dynamically determined, the variablelength encoding unit 15 also encodes information on the interpolation position group and adds it to the encoded bitstream.
Configuration Example 1 of Interpolation Filter Coefficient Determination Unit
[0074] FIG. 2 is a diagram illustrating a first configuration example of the interpolation filter coefficient determination unit. The interpolation filter coefficient determination unit 12 is remarkably different from conventional technologies.
[0075] In the interpolation filter coefficient determination unit 12 , an interpolation filter information acquisition unit 121 outputs the values of fixed interpolation filter coefficients, which are prepared in advance of encoding, to a weighted filter coefficient generation unit 123 . For example, 6tap coefficients defined in H.264/AVC, or 8tap coefficients or 12tap coefficients in DCTIF may be used.
[0076] A weight parameter designation unit 122 sets weight parameter(s) and outputs the parameter(s) to the weighted filter coefficient generation unit 123 . For example, when two kinds of filter coefficients are weighted, α is given as a weight parameter (it is sufficient that a single weight is given because there is a constraint of α+β=1. Although two kinds of weights, α and β, may be transmitted, one kind of weight is transmitted in order to reduce an overhead). Similarly, when weights for three kinds of filters are calculated, α and β are given because there is a constraint of α+β+γ=1. The same is applied to a case of m (an integer greater than or equal to 4) kinds of weights.
[0077] The weighted filter coefficient generation unit 123 takes (a) an adaptive interpolation filter coefficient value: f A (x) generated on a framebyframe basis or on a regionbyregion basis, (b) the weight parameter: a output from the weight parameter designation unit 122 , and (c) a fixed interpolation filter coefficient value: f B (x) output from the interpolation filter information acquisition unit 121 , as inputs, and performs the following calculation.
[0000] α× f A ( x )+β× f B ( x ) (where, 0 ≦x≦t, t is a tap length, and β=1−α)
[0078] Here, the weighted sum of two kinds of filter coefficients is assumed, and the weighted sum of three or more kinds of filter coefficients can be set in a similar manner. The weighted filter coefficient generation unit 123 outputs the obtained weighted filter coefficients to an MSE calculation unit 124 described below.
[0079] The MSE calculation unit 124 takes (a) motion vectors MV obtained by the motion search process, (b) a decoded signal (a local decoded picture), and (c) the weighted filter coefficients output from the weighted filter coefficient generation unit 123 , as inputs, generates an interpolated picture at positions designated by the motion vectors MV using integer position pixels of the decoded picture and the weighted filter coefficients, and calculates a mean square error (MSE) with an original picture, i.e., prediction residual energy.
[0080] For example, the following equation can be used for an MSE.
[0000] MSE={total sum of (original signal−prediction signal)}/number of pixels
[0081] The calculated MSE is output to a minimum MSE storage unit 125 .
[0082] The minimum MSE storage unit 125 takes the MSE obtained by the MSE calculation unit 124 as an input and stores its value. The minimum MSE storage unit 125 compares a previously stored minimum value or a previously defined minimum value with the input MSE, determines that (a) the minimum MSE is the value of the input MSE if the value of the input MSE is smaller than the already stored minimum value and that (b) the minimum MSE is the already stored minimum value if the already stored minimum value is smaller than the value of the input MSE, and stores and updates the minimum value. Moreover, when the minimum value is stored, the minimum MSE storage unit 125 stores weight parameters that realize the MSE value.
[0083] In the interpolation filter coefficient determination unit 12 , the MSE calculation unit 124 processes possible combinations related to an interpolation filter and then outputs a combination that realizes the minimum MSE, i.e., (1) weight parameters and (2) interpolation filter coefficients.
Configuration Example 2 of Interpolation Filter Coefficient Determination Unit
[0084] FIG. 3 is a diagram illustrating a second example of the configuration of the interpolation filter coefficient determination unit. An interpolation filter coefficient determination unit 12 ′ shown in FIG. 3 is different from the interpolation filter coefficient unit 12 shown in FIG. 2 in that an interpolation target position grouping unit 126 is provided, the setting of the abovedescribed weight parameters is performed for each of groups of interpolation positions, which are fractional pixel positions, and interpolation filter coefficients are switched for each of the groups of the interpolation positions.
[0085] Although the abovedescribed configuration example 1 uses, for example, a weighted filter for all the pixel interpolation positions, the configuration example 2 sets weighted filter coefficients used for an interpolation process by an interpolation filter for each designated interpolation position group or each interpolation position group calculated using a given method. Whether to calculate the weighted sum of filter coefficients for a given interpolation position group or not may be selected. For example, it is possible to perform an adaptive process on interpolation position groups; e.g., a weighted filter is used for important interpolation positions and interpolation positions that are selected with a high probability, and a filter that is not subjected to weighing is used for the other positions.
[0086] The functions of the interpolation filter information acquisition unit 121 , the weight parameter designation unit 122 , and the weighted filter coefficient generation unit 123 are the same as those of the abovedescribed configuration example 1.
[0087] The interpolation target position grouping unit 126 performs grouping for each interpolation position which is a fractional pixel position and outputs group information for each interpolation position. The interpolation position groups may be previously determined for fractional pixel positions, or they may be dynamically determined using, for example, a probability distribution of motion vectors.
[0088] The MSE calculation unit 124 generates an interpolated picture at positions designated by the motion vectors MV from integer position pixels of the decoded picture and the weighted filter coefficients using the weighted filter coefficients output from the weighted filter coefficient generation unit 123 , for each interpolation position group designated by the interpolation position group information output by the interpolation target position grouping unit 126 , and calculates an MSE with the original picture, i.e., prediction residual energy. The calculated MSE is output to the minimum MSE storage unit 125 . Moreover, when the minimum value is stored, the minimum MSE storage unit 125 stores weight parameters and interpolation position group information that realize the MSE value.
[0089] The interpolation filter coefficient determination unit 12 ′ examines possible combinations related to an interpolation filter and then outputs a combination which realizes the minimum MSE, i.e., (1) weight parameters, (2) interpolation position group information, and (3) interpolation filter coefficients. It is to be noted that when interpolation position groups are fixed and interpolation position group information can be shared by an encoding end and a decoding end, it is not necessary to output the interpolation position group information.
[Encoding Processing Flow]
[0090] FIG. 4 is a flowchart of the processing of the video encoding apparatus shown in FIG. 1 . Hereinafter, a flow of the processing when the video encoding apparatus encodes one frame will be described in accordance with FIG. 4 . Here, an example of a process when the interpolation filter coefficient determination unit is configured with the configuration example 2 will be described, but a process in a case of the configuration 1 is almost the same. The following description assumes processing on luminance signals unless otherwise specified.
[0091] First, in step S 101 , a frame of an original picture required for an encoding process is input. Subsequently, in step S 102 , for example, a fixed interpolation filter provided in an encoder, such as a onedimensional 6tap filter adopted in H.264/AVC or a onedimensional 8tap or 12tap filter as a DCTbased interpolation filter, is set as a previously defined interpolation filter, and motion vectors (MV) of the entire frame are derived using the interpolation filter. Here, the previously defined interpolation filter is not limited to a fixed interpolation filter, and an adaptive interpolation filter calculated, for example, in the previous frame may be adopted.
[0092] Subsequently, in step S 103 , the values of adaptive interpolation filter coefficients for an encoding target frame are calculated using the motion vectors obtained in step S 102 . In the calculation of the interpolation filter coefficients in the present step, a commonlyknown method for minimizing the prediction error energy (linear regression) is used.
[0093] Subsequently, in step S 104 , grouping of interpolation target pixels to which interpolation filters are to be applied is performed. Specifically, the interpolation target pixels are divided into multiple groups depending on interpolation positions in accordance with a predetermined table. It is to be noted that interpolation position groups may be determined dynamically using a given method. When the interpolation position groups are determined dynamically, it is necessary to encode interpolation position group information and transmit it to a decoding end.
[0094] Subsequently, in step S 105 , weighted filter coefficients are generated for each of the interpolation position groups divided in step S 104 . Weight parameters are read from a table and are set, and weighted interpolation filter coefficients are generated from the weight parameters as shown in the following Equation (6).
[0000] α× f A ( x )+β× f B ( x ) (6)
[0095] Here, α and β are the weight parameters (α+β=1), f A (x) is a previously defined interpolation filter coefficient, and f B (x) is an interpolation filter coefficient determined in the present frame. x denotes a coefficient number and falls within a range of 0≦x<n (n denotes a tap length). Here, although a weighted filter coefficient of two kinds of filter coefficients is assumed, a weighted filter coefficient can be represented in a similar manner in a case of m (>2) kinds of interpolation filters.
[0096] Subsequently, in step S 106 , an interpolation filter which is actually applied to encoding is determined for each of the groups determined in step S 104 using the weighted filter coefficients generated in step S 105 . For example, prediction error energy is set as an evaluation criterion (a cost function), prediction pictures are generated using the original picture and interpolation filters with the weight parameters selected in step S 105 , and weight parameters that minimize the sum of squared difference thereof are calculated. For example, as for the weight parameters, the value of α is changed in increments of 0.1 or 0.01, and α which realizes the smallest cost value is calculated.
[0097] Subsequently, in step S 107 , an interpolation process is performed using an interpolation filter defined by the weighted filter coefficients determined in step S 106 .
[0098] Subsequently, in step S 108 , the weight parameters calculated in step S 106 is encoded. Subsequently, in step S 109 , interpolation filter information, such as the interpolation filter coefficients calculated in step S 103 , is encoded. When the interpolation position group information is to be transmitted, it is encoded in step S 109 . Subsequently, in step S 110 , all the remaining pieces of information to be encoded, such as prediction error signals (texture component information) and the motion vectors, are encoded.
[0099] Subsequently, in step S 111 , a determination as to whether an encoded frame is the final frame is performed. If the processed frame is not the final frame, the processing returns to step S 101 in order to process the next frame. If the processed frame is the final frame, the encoding process is completed.
[0100] It is to be noted that a function of performing an encoding process by optimizing weight parameters of an interpolation filter for each interpolation position group stated in the present embodiment is not limited to application to luminance signals, and it is also applicable to chrominance signals in a similar manner.
Configuration Example of Video Decoding Apparatus
[0101] FIG. 5 is a diagram illustrating an example of a configuration of a video decoding apparatus in accordance with an embodiment of the present invention.
[0102] In a video decoding apparatus 20 , a variablelength decoding unit 21 inputs an encoded bitstream and decodes quantized transform coefficients, motion vectors, weight parameters, interpolation filter coefficients, and so on. Moreover, when interpolation position group information is encoded, the variablelength decoding unit 21 also decodes the interpolation position group information.
[0103] A weighted filter coefficient generation unit 22 generates weighted filter coefficients from the weight parameters and the interpolation filter coefficients decoded by the variablelength decoding unit 21 . An interpolation filter coefficient determination unit 23 determines interpolation filter coefficients used for each grouped interpolation position using the weighted filter coefficients generated by the weighted filter coefficient generation unit 22 .
[0104] A reference picture interpolation unit 241 in a prediction signal generation unit 24 applies an interpolation filter with the interpolation filter coefficients received from the interpolation filter coefficient determination unit 23 to an already decoded reference picture stored in a reference picture memory 26 to restore fractionalaccuracy pixels of a reference picture. The prediction signal generation unit 24 generates a prediction signal of a decoding target block from the reference picture, for which the fractionalaccuracy pixels have been restored.
[0105] A predictive decoding unit 25 performs inverse quantization of the quantized coefficients decoded by the variablelength decoding unit 21 , inverse orthogonal transform, and so on, adds a resultant calculated prediction error signal to the prediction signal generated by the prediction signal generation unit 24 to generate a decoded picture, and outputs it as an output picture. Moreover, the decoded picture decoded by the predictive decoding unit 25 is stored in the reference picture memory 26 for later predictive decoding. At this time, an inloop filter for removing coding noise, such as a deblocking filter or an adaptive loop filter (ALF), may be applied and then the storage may be performed.
[Decoding Processing Flow]
[0106] FIG. 6 is a flowchart of the processing of the video decoding apparatus shown in FIG. 5 . Hereinafter, a flow of the processing in which the video decoding apparatus decodes one frame will be described in accordance with FIG. 6 . The following description assumes the processing for luminance signals unless otherwise specified.
[0107] In step S 201 , information on a frame header (or a slice header) is acquired. Subsequently, in step S 202 , weight parameters are decoded and information thereon is stored. Subsequently, in step S 203 , information on interpolation filter coefficients is decoded. When interpolation position group information is encoded, the interpolation position group information is also decoded. Subsequently, in step S 204 , all the other pieces of information necessary for decoding (e.g., motion vectors and a prediction error signal) are decoded.
[0108] Subsequently, in step S 205 , weighted filter coefficients necessary for interpolating each interpolation position group are generated from the weight parameters decoded in step S 202 and the interpolation filter coefficients decoded in step S 203 .
[0109] Subsequently, in step S 206 , an interpolation process is performed using the motion vectors decoded in step S 204 and the weighted filter coefficients obtained in step S 205 and a prediction signal is generated.
[0110] Subsequently, in step S 207 , the prediction error signal decoded in step S 204 is combined with the prediction signal determined in step S 206 to restore a decoded signal.
[0111] Subsequently, in step S 208 , a determination as to whether all the frames to be decoded have been decoded is performed, the processing returns to step S 201 and transfers to decoding of the next frame if not all the frames have been decoded, and the decoding process is completed if all the frames have been decoded.
[0112] Although the above description is targeted for luminance signals, the present flow can also be applied to chrominance signals in a similar manner.
Configuration Example when Software Program is Used
[0113] The above video encoding and decoding processes can also be realized using a computer and a software program, the program may be recorded on a computerreadable recording medium, and the program may be provided through a network.
[0114] FIG. 7 illustrates an example of a configuration of a system when an embodiment of the present invention is implemented using a computer and a software program.
[0115] The present system has a configuration in which a central processing unit (CPU) 50 which executes a program, a memory 51 which stores the program and data accessed by the CPU 50 , such as a random access memory (RAM), a video signal storage unit 52 which stores an encoding target video signal or a video signal of decoded pictures, a program storage apparatus 53 which stores a program for causing the CPU 50 to execute the processes described in the embodiment of the present invention, and an encoded stream storage unit 54 which stores a bitstream of an encoded result or a decoding target bitstream are connected with a bus.
[0116] The program storage apparatus 53 stores either a video encoding program 531 for encoding a video signal using an embodiment of the present invention or a video decoding program 532 for performing decoding on an encoded bitstream using an embodiment of the present invention. The program storage apparatus 53 may store both of these programs.
[0117] Moreover, when the present system is used as the video encoding apparatus, the video encoding program 531 is loaded on the memory 51 , and the CPU 50 sequentially fetches and executes instructions of the video encoding program 531 loaded on the memory 51 , the CPU 50 encodes a video signal stored in the video signal storage unit 52 using the technique described in an embodiment of the present invention, and the CPU 50 stores a bitstream of an encoded result in the encoded stream storage unit 54 . Alternatively, the bitstream may be output to an external apparatus through an interface such as a network adapter.
[0118] Furthermore, when the present system is used as the video decoding apparatus, the video decoding program 532 is loaded on the memory 51 , the CPU 50 sequentially fetches and executes instructions of the video decoding program 532 loaded on the memory 51 , the CPU 50 performs decoding on a bitstream stored in the encoded stream storage unit 54 using the technique described in an embodiment of the present invention, and the CPU 50 stores a video signal of a decoded result in the video signal storage unit 52 . Alternatively, the video signal of the decoded result is output to an external reproduction apparatus.
[0119] Although embodiments of the present invention have been described above with reference to the drawings, these embodiments are exemplification of the present invention, and it is apparent that the present invention is not limited by these embodiments. Therefore, addition, omission, replacement, and/or other modifications of structural elements that do not depart from the gist of the present invention may be made. That is, the present invention is not limited by the above description and is only limited by the claims stated below.
INDUSTRIAL APPLICABILITY
[0120] The present invention is applicable to, for example, video encoding and video decoding using interframe prediction with motion compensation. The present invention can extend the selectable range of an interpolation filter and achieve an improvement in the coding efficiency as a result of a reduction in the prediction error energy.
DESCRIPTION OF REFERENCE SIGNS
[0000]
10 video encoding apparatus
11 interpolation filter coefficient calculation unit
12 , 12 ′, 23 interpolation filter coefficient determination unit
121 interpolation filter information acquisition unit
122 weight parameter designation unit
123 , 22 weighted filter coefficient generation unit
124 MSE calculation unit
125 minimum MSE storage unit
126 interpolation target position grouping unit
13 , 24 prediction signal generation unit
131 , 241 reference picture interpolation unit
132 motion detection unit
14 predictive encoding unit
15 variablelength encoding unit
16 decoding unit
17 , 26 reference picture memory
20 video decoding apparatus
25 variablelength decoding unit
25 predictive decoding unit