T 1526/20 (Liveness testing/SAMSUNG) 14-03-2023
Download and more information:
LIVENESS TESTING METHODS AND APPARATUSES AND IMAGE PROCESSING METHODS AND APPARATUSES
Sufficiency of disclosure - (no)
Correction of error - (yes)
I. The appeal is against the decision of the Examining Division.
II. The application was refused inter alia
(a) for extension beyond the subject matter of the application as originally filed, Article 123(2) EPC, in view of the independent claims and of dependent claim 5 of then main request and auxiliary requests 1 and 2, and
(b) for lack of sufficient disclosure of the subject-matter of the independent claims of all requests (decision point 7) and of a feature of the mentioned dependent claim 5 (decision point 8).
III. The Examining Division cited inter alia the following documents (numbering by the Board):
D1: Tan, "Face Liveness Detection from a Single Image with Sparse Low Rank Bilinear Discriminative Model", 2010
D2: Wonjun, "Face Liveness Detection From a Single Image via Diffusion Speed Model", 2015
D3: Chen, "Total Variation Models for Variable Lighting Face Recognition", 2006
IV. With the grounds of appeal the appellant requested that the decision of the Examining Division be set aside and that a patent be granted on the basis of claims according to a main request or of one of three auxiliary requests, all filed with the grounds of appeal.
V. In a communication accompanying a summons to oral proceedings the Board provided its provisional opinion, which included that, for all requests, the claims were not clear and the invention was not sufficiently disclosed within the meaning of Article 83 EPC. The Board also indicated a discrepancy between the statements of the Appellant, indicating the deletion of dependent claim 5 for all requests, and the text of the second auxiliary request, which still contained said claim.
VI. With the reply of 7 February 2023, the Appellant submitted a new second auxiliary request, where claim 5 was deleted, to replace the corresponding one filed with the grounds of appeal, and a fourth auxiliary request. During the oral proceedings, the Appellant filed a fifth auxiliary request.
VII. Claim 1 of the main request defines:
A computer-implemented liveness testing method comprising:
testing a liveness (1020) of an object included in a received input image (1010) based on whether an image of the object has a characteristic indicative of a flat surface or a characteristic indicative of a three-dimensional (3D) structure, characterized by
determining whether the image of the object has the characteristic indicative of the flat surface or the characteristic indicative of the 3D structure based on statistical information related to diffusion speeds of a plurality of pixels corresponding to the image of the object.
VIII. Claim 1 of the first auxiliary request differs from that of the main request by the characterising portion, which now defines:
characterized by
diffusing a plurality of pixels corresponding to the object in the received input image by iteratively update values of the plurality of pixels using a diffusion equation;
determining diffusion speeds of the plurality of pixels by calculating differences between first values of the plurality of pixels before diffusion and second values of the plurality of pixels after diffusion; and
determining whether the image of the object has the characteristic indicative of the flat surface or the characteristic indicative of the 3D structure based on statistical information related to the diffusion speeds of the plurality of pixels.
IX. Claim 1 of the second auxiliary request differs from that of the first auxiliary request in that the diffusion equation is now further defined as a
nonlinear diffusivity function-based diffusion equation
X. Claim 1 of the third auxiliary request differs from the previous requests in the characterizing portion, which now defines:
characterized by
diffusing a plurality of pixels corresponding to the object in the received input image by iteratively calculating (1240) values of the plurality of pixels based on a diffusion equation;
calculating the diffusion speeds for each of the plurality of pixels based on a difference between a pixel value before each iterative calculation and a pixel value after each iterative calculation; and
determining whether the image of the object has the characteristic indicative of the flat surface or the characteristic indicative of the 3D structure based on statistical information related to the diffusion speeds of the plurality of pixels.
XI. Claim 1 of the fourth auxiliary request differs from that of the third auxiliary request by further defining the calculation of the diffusion speeds as follows:
wherein the diffusion speed of a pixel at coordinates (x,y) is defined as
FORMULA/TABLE/GRAPHIC
wherein u**(0)(x, y) denotes a value of the pixel at coordinates (x, y) the received input image, and u**(L)(x, y) denotes a value of the pixel at coordinates (x, y) in a final diffusion image;
XII. Claim 1 of the fifth auxiliary request defines:
A computer implemented liveness testing method comprising:
testing a liveness (1020) of an object included in a received input image (1010) based on whether an image of the object has a characteristic indicative of a flat surface or a characteristic indicative of a three dimensional (3D) structure, characterized by
diffusing a plurality of pixels corresponding to the object in the received input image by iteratively calculating (1240) values of the plurality of pixels based on a nonlinear diffusivity function based diffusion equation;
calculating the diffusion speeds for each of the plurality of pixels based on a difference between a pixel value before each iterative calculation and a pixel value after each iterative calculation, wherein the diffusion speed of a pixel at coordinates (x, y) is defined as
FORMULA/TABLE/GRAPHIC
wherein u**(0)(x, y) denotes a value of the pixel at coordinates (x, y) the received input image, and u**(L)(x, y) denotes a value of the pixel at coordinates (x, y) in a final diffusion image; and
determining whether the image of the object has the characteristic indicative of the flat surface or the characteristic indicative of the 3D structure based on statistical information based on the diffusion speeds of the plurality of pixels by extracting a small scale region map SR(x,y), being representative for a small scale region, from the image in accordance with
FORMULA/TABLE/GRAPHIC
wherein SR(x,y) is an indicator indicating whether the pixel at coordinates (x,y) belongs to the small scale region, wherein my corresponds to an average of diffusion speeds of pixels included in the entire image, and sigma corresponds to a standard deviation of the diffusion speeds of the pixels included in the entire image; and wherein it is determined that:
the image of the object has the characteristic indicative of the flat surface if
FORMULA/TABLE/GRAPHIC is satisfied; or that
the image of the object has the characteristic indicative of the 3D structure if
FORMULA/TABLE/GRAPHIC is satisfied,
wherein SRM(x,y) denotes a value of a pixel at coordinates (x,y) in an image acquired by applying median filtering to the small scale region map, and xi denotes a threshold value.
XIII. At the end of the oral proceedings, the chairman announced the Board's decision.
The application
1. The application relates to liveness testing in the framework of face recognition methods; it is in particular concerned with the detection of impersonations using a picture of a face (paragraphs 1 and 2). The proposed method relies on the observation that the image of a 3D structure (i.e. of a real person) is less "uniform" than that of a 2D structure (of a picture of the person). To detect "uniformity" it applies first a diffusion filter and then measures the diffusion speeds, i.e. the magnitude of the change in pixel values before and after filtering (see e.g. paragraphs 7 to 14, 102, 103). These "speeds" are binarized and if the number of non-zero pixels is low, then the imaged object is considered to be a fake (paragraphs 108-117). Alternatively, the binarized image is median filtered, and if the number of (by the filter) changed pixels is low, then the imaged object is considered to be a fake, on the basis that the image is not noisy enough to be one of a real object (paragraphs 118-119).
2. The application further describes (paragraph 143 et seq.) a method for face recognition, also based on diffusion, which includes liveness testing as previously described. In that context it is explained (see e.g. paragraphs 163-164, 194-195) that the diffused image contains (mostly) the illumination component of the given image. Another image is formed by the ratio, or equivalently the log difference, of the original image to the diffused image. This image is said to be related to the "non-illumination component", i.e. to the structure of the imaged object, and is the one used for recognition.
The cited documents
3. Document D1 describes liveness testing by decomposing the input face image into a reflectance image (albedo) and an illuminance image, using the Logarithmic Total Variation (LTV) method of D3 (section 2.2). These images are used either separately, or together, as features in a classifier, e.g. the sparse logistic classifier (section 2.3).
4. Document D2 is not prior art, having been published after the priority date. It is a paper authored by three of the designated inventors and describes a method of liveness testing using diffusion, which is in concept similar to (see section II.A), but in implementation different from, the one described in the current application. In particular, the diffusion speed is computed using the log difference (II.B), and the feature extraction step employs a concatenated histogram of local speed patterns (II.C).
Second auxiliary request
Correction, Rule 139 EPC
5. The second auxiliary request filed with the grounds of appeal still contained the dependent claim 5, to which the Examining Division had objected under various provisions, although the Appellant declared to have deleted it (statement of grounds of appeal, page 9, middle paragraph) like in its other requests. The Appellant also did not submit any arguments against the corresponding grounds for refusal.
6. In its provisional opinion, the Board indicated its understanding that the intention of the Appellant was to delete this claim and asked the Appellant to clarify its requests. In response, the Appellant filed the current second auxiliary request as a "corrected" version, which differs from the previous second auxiliary request only by the deletion of claim 5. The Board accepts that this reflects the Appellant's original intention and considers the corresponding amendment as an allowable correction under Rule 139 EPC.
Fourth auxiliary request
Admittance, Article 13 RPBA 2020
7. The fourth auxiliary request was filed in response to the Board's preliminary opinion. It addresses, and overcomes, the lack of clarity objection raised by the Board (for the first time) in its preliminary opinion at point 7.2, without raising any other issues. In view of this, the Board decides to admit the amended request (see also T 1294/16 reasons 8.2 to 8.4).
Main and first to fourth auxiliary requests
Sufficiency of disclosure, Article 83 EPC
8. In the oral proceedings the Board indicated that it would first examine the fourth auxiliary request, which defined the term "diffusion speed" in a clear manner. The Appellant had no objections.
9. In its decision, the Examining Division objected to the independent claims of all requests (see especially the decision, point 7.1 of the reasons) because the claimed techniques, "relying only on diffusion filtering" were not robust against illumination changes and lead to "totally unreliable results". For this reason other techniques, as presented in D1 or D2, had to include further steps. A similar objection was made in view of the computation of the diffusion speeds (decision, point 7.2), which in D2 (as well as in D1 and D3, though they do not use the term "diffusion speed") was made in logarithmic space, so as to obtain consistent results.
10. The Appellant argued (see for instance the grounds of appeal, pages 5 to 7) that the invention might not work in some hand-picked examples, but worked in "the majority of practical use cases". The approach was not dependent on the illumination used, because it was the shape of the surface considered that determined the diffusion speeds.
10.1 The logarithmic differences provided robustness to changes in illumination for face recognition, but were not necessary for liveness testing. The Appellant further emphasised this distinction during the oral proceedings. Face recognition was a more complex task than liveness testing, which could be carried out based only on the normal difference of the two images. The Examining Division did not sufficiently appreciate this distinction. With reference to figure 12, the Appellant explained that liveness testing did not need the computation of the third image, i.e. the logarithmic difference image.
10.2 The skilled person, at the time of filing, had no reasons to doubt that the proposed method worked. Even if the authors later used the logarithmic space for differencing, the diffusion speeds computed in the "Cartesian" space were also sufficient to determine liveness.
10.3 This could be seen from section 2.2 in document D1. Although a logarithmic difference was used in the part using diffusion filtering, entitled "Variational Retinex-based Method", this was not the case in the following part, entitled "Difference of Gaussian (DoG)-based Method": There, a DoG filter was used, which implemented a normal difference. The idea expressed in that part, namely that because
"the image of a photograph taken through a webcam is essentially an image of a real face but passes through the camera system twice and the printing system once [...] the imposter image tends to be more seriously distorted by the imaging system and hence has lower image quality (i.e., missing more high frequency details) under the same imaging conditions"
was the same as that of the current application (see paragraphs 76 and 77). Essentially, high frequency patterns, i.e. referred to as non-uniform in the present application, could be used to distinguish a live image, i.e. an image of a live face, from a fake one, i.e. the image of a 2D photograph of a face: due to its 3D profile, the live image contained more, (and different) high frequency information than the image of a 2D photograph.
10.4 The application disclosed a clear sequence of steps which the skilled person could implement without undue burden. It explained that, in an image of a real face, small scale regions with a pronounced 3D structure had higher diffusion speeds than large and flat scale regions such as cheeks (paragraph 105). So a live image could be distinguished from a fake one based on the diffusion speeds using equations 6 or 7. In particular the embodiment using median filtering (equation 7) was a robust implementation of this idea and made it credible to the skilled person that the method could provide for reliable liveness testing.
10.5 The Appellant also argued that the sufficiency of disclosure of an invention in a patent application should not be assessed according to the stricter criteria applicable to a scientific publication.
11. The Board notes that the application, as the Appellant also submitted, discusses two distinct scenarios, the first relating to liveness detection and the second to face recognition. In both scenarios, the first step is to apply image diffusion to the original image.
11.1 For face recognition, the application provides a multiplicative image formation model: the acquired image is said to be the product between illumination and object information (reflectance). According to the application, the illumination information (second image) can be extracted by diffusion filtering. The object-related information (third image) is then extracted by a log-difference between the original image and the diffused image - or, equivalently, the logarithm of the ratio of the first two images , i.e. by taking the logarithm of the object-related information according to the multiplicative model.
12. The Board also notes that this multiplicative model is well known to the skilled person as the Lambertian reflectance model for image formation (see e.g. D1 equation 1) and is used in D1, section 2.2, to justify the use of the same log-difference, namely between the original image and a diffused image, to obtain an image used for liveness testing (the albedo image).
13. The application uses the "diffusion speeds" for liveness testing. They constitute a (third) image obtained by a (normal) difference between the original and the diffused image (see the application, equation 4). This rather corresponds to an additive model for image formation, in apparent contradiction with the generally accepted multiplicative theoretical model. The application contains nothing to dismiss that contradiction. Therefore, on the basis of the application and the common general knowledge, which does not include D2, the skilled person must have had doubts that this model successfully extracted object-related information and, thus, provided the information necessary for reliable liveness testing.
14. The application provides no reasons as to why this model was correct, nor results to show that the proposed way of extracting object-related information correctly distinguishes live from fake objects.
14.1 That a detailed implementation method is provided, as the Appellant pointed out, does not withstand this objection, as the implementation cannot establish that the claimed effect exists.
14.2 Beyond that, neither the allegation in the application that the diffusion speeds would be characteristic for 3D "small scale" regions and thus contain the relevant discriminatory information nor the similar argument reproduced from D1 (point 12.3 above) can be accepted without further evidence.
14.3 This is because the purpose requires information distinguishing between images obtained from 3D and 2D structures. Although some filters may be able to extract such information (as it was shown for the DoG filter in D1), a generic low pass filter, (e.g. an average filter) or a generic diffusion filter cannot be presumed to do the same, as they are designed for a different purpose, namely general denoising. Depending on the filter and its parameters, the identified "noise" may or may not capture the needed information. Additional evidence is required to show which image acquisition conditions und which filters and filter parameters may be suitable for the task at hand. This is missing in the application at hand.
14.4 The Board concludes that the application does not dispel the skilled person's doubts that any one of the provided implementation examples is able to determine liveness in a reliable manner. The claimed invention is therefore not sufficiently disclosed in the sense of Article 83 EPC.
15. This conclusion applies a fortiori to the main and first to third auxiliary requests as well, which are effectively broader in scope. The Appellant did not provide any counter-arguments in this respect.
Fifth auxiliary request
Admittance, Article 13 RPBA 2020
16. This request details on the basis of equation 7 the manner in which the discrimination between a fake and a live image takes place. The Appellant argued that the skilled person, in view of the explanations in paragraphs 76 and 77 of the description, would find credible that the theoretical model so defined worked. The claim now provided a clear teaching of the steps to be executed, which could be implemented without any difficulties by the skilled person.
17. The Board notes that the input to the discrimination method is still the "diffusion speed" image, obtained as the difference between the original image and the diffused one (n.b. the specification "nonlinear diffusivity function-based" does not effectively restrict the claim, because all image diffusion methods, as defined by equation 1, are non-linear). As noted above, the Appellant could not dispel the doubts that this image contains the information necessary for liveness discrimination.
18. Thus the request, filed very late during the oral proceedings, does not overcome the objection as to a lack of sufficient disclosure. The Board therefore decided not to admit this request (Article 13 RPBA 2020).
For these reasons it is decided that:
The appeal is dismissed.