T 0814/20 (Adapted Visual Vocabularies/CONDUENT) 20-03-2023

The application

1. The application relates to a method of image processing. The context is that of re-identification, i.e. the matching of an object in an image to a previously seen object, such as a vehicle in traffic surveillance (paragraphs 2 and 3).

1.1 These images may come from different cameras operating under different imaging conditions, for instance lighting. Image representations may be "shifted" as a function of these varying conditions (paragraphs 3-5 and 27). To account for this, the application proposes to start from a universal generative model in the form of Gaussian Mixture Model (GMM) and to adapt it (through e.g. a MAP adaptation process) to individual cameras (paragraphs 27, 36-38, and 71-101).

1.2 As opposed to building dedicated models for each camera from scratch, this has the advantage that the different components of the generative model (the Gaussians) are in known correspondence. An image representation based on the generative model and maintaining this component correspondence, like the Fisher Vectors, allows the image matching to be performed in a simple manner, compo­nent by component (paragraph 82).

1.3 Results are provided in a context of license plate matching using images from different cameras in parking areas (paragraphs 106-118).

The prior art

2. Document D3 describes a method wherein a GMM universal model is adapted to a first and a second image (using MAP adaptation), and then the two probability models are compared component by component to derive a simila­rity measure between the two images (paragraphs 15 to 17, 26). This similarity metric is used for object classification purposes (paragraphs 40 to 44).

3. Document D4 discusses the Fisher Vector image repre­sen­tation. Inter alia, it explains that this represen­ta­tion, defined on the basis of the gradient of the like­lihood of the test data (image patch statistics) given the GMM, bears a close relationship with the MAP adaptation procedure of a GMM to a given single image (section 2.5, around equations 35 to 37).

Objections as to support and added matter

4. The Examining Division had raised (decision point 3.2) an objection of a lack of support and one of added matter in respect of auxiliary request 1 underlying its decision, stating that "the solution to the problem of object matching (re-identification) announced in par.2 requires image similarity computation", for which the Fisher Vector representation was essential, and further

that "in technical terms, the only combination that the skilled person reads directly and unambiguously as having the essential features of the invention is that of MAP adaptation and FVs."

5. The Board's objections as to a lack of support were essentially the same.

6. The current claim 1 defines image similarity computation on the basis of Fisher Vectors and MAP adaptation of the GMM models. The objections therefore no longer apply.

7. The Board found added matter in the definition of the MAP adaptation, considering the (previously) claimed generalizations to lack basis in the description. This objection has been overcome by amendment in the last (current) set of claims.

Other objections under Article 84 EPC

8. In its last two communications, the Board further raised a number of objections under Article 84 EPC regarding the dependent claims (conciseness and clarity) and the description, which required adaptation. These have been overcome by amendment.

Novelty

9. In establishing the equivalence with D3 (decision 1.1. to 1.5), the Examining Division considered that the adaptation to a first and second camera was anticipated by the application of the teachings of D3 to a video object. Further it considered that the first and second image might be ones already used during adaptation, and that the image descriptor was the result of the adaptation, i.e. the mixture model. Under this interpretation "a difference in concrete technical terms" could not be identified on the basis of the further step of extracting image-level descriptors using the adapted models (decision 1.4).

10. The Board does not find this to be a technically reasonable interpretation of the claim.

10.1 For the skilled person, the claim recites a standard sequence comprising a "training" phase, when the models are adapted, and a "use" phase, when the already adapted models are used to provide descriptors for test images. This is clear from the wording of the claim, by itself and in the context of the description (see figure 2A). Thus, even if the adapted generative models themselves may be considered image (or video) descriptors, a further step of extracting an image-level descriptor from an image using the adapted models is not derivable from D3.

10.2 At least for this reason the subject matter of claim 1 is new in view of D3.

Inventive step

Technical effect

11. In its preliminary opinion, in view of the requests on file at that time, the Board indicated that a technical effect, in the sense of obtaining predictable results for image matching could not be acknowledged.

11.1 In particular, although the description disclosed evi­dence that object matching was improved in the specific context of license plate identification, the claims could not profit from this disclosure since they were not limited to this context. The claims only spe­cified measuring image similarity, which purpose was vague (images may be deemed similar for vastly different reasons, e.g. similar luminosity, or, as here, contai­ning the same objects), and could not be said to serve a technical purpose in the absence of any claimed further technical use.

12. Current claim 1 defines a method for the re-identifi­ca­tion of objects captured by image cameras. The Board considers this to be a technical purpose, because it is tantamount to an objective measurement in physical reality: is the object observed now the same as the one observed earlier?

13. It remains to be appreciated whether the claimed method provides a technical effect over substantially the whole scope of the claim (see G 1/19, reasons 82).

13.1 The claim defines that two objects, based on two images taken by two respective cameras, match if their simi­la­rity, computed using Fisher Vectors descriptors from adapted GMMs, is above a threshold.

13.2 The description explains that the method aims at com­pen­sating for different imaging conditions, by reflec­ting the shift in conditions (e.g. illuminations or camera angles) by a corresponding shift in the adapted GMM models (paragraph 27). The description also shows that in a case concerning license plate re-identi­fi­ca­tion in parking facilities using in-house datasets the MAP adaptation claimed provides an improvement over a baseline no-adaptation method (paragraphs 111 to 116). The Board has no reason to doubt these results.

13.3 The theoretical assumptions appear sound to the Board. Variations in illumination and geometry should be cap­tured by the mixture model, and this is confirmed by the license plates example. In principle, the method should also provide improved results for other object matching cases, where the imaging conditions between two cameras differ in a similar manner.

13.4 The claimed method will not "work" under all imaginable circumstances. It is probably safe to say that no com­pu­ter vision method does. For instance, the present method may fail to re-identify objects largely changing appearance. However, the skilled person will under­stand, from the present claims and the description, the kind of situations and its parameters (such as illumi­na­tion and geometry) for which the method is designed. The method credibly works over that range of situa­tions.

13.5 In the Board's judgment, this is sufficient to satisfy the requirement that, in the present case, a technical effect is present over substantially the whole scope of the ­­­­claims (see again G 1/19, reasons 82).

Obviousness

14. The Examining Division acknowledged in regard of the second auxiliary request underlying the decision, see point 5, that an additional step of extracting Fisher Vectors was not disclosed in D3, but stated that it would be obvious in the context of D3 to add a step of extracting Fisher Vectors as an image descriptor.

15. In this respect the Board agrees with the appellant that, starting from D3, even in view of D4, it is not obvious to add a step of Fisher Vector extraction.

15.1 This is because, in D3, the adapted GMM themselves are considered to accurately describe one image. The Fisher Vectors are described in D4 rather as replacements to the image based MAP adaptation previously developed, because they capture the same information as that obtained by adaptation starting from the same GMM (see point 3 above).

15.2 In D3, the MAP adaptation provides the object de­scrip­tor; in the application, the MAP adaptation provides camera invariance for the model (encoding the shift in conditions), and the single image object descriptor is provided by the Fisher Vectors derived from the model. Such use of adaptation to provide camera invariance is neither described, nor hinted at, by D3.

15.3 Thus, in the Board's view, the skilled person, starting from D3, may have considered Fisher Vectors to measure image similarity for classification purposes instead of the MAP adaptation method, but would not combine the two in the claimed manner.

15.4 Thus the claimed matter is not obvious in view of the prior art at hand.

16. In view of the foregoing, the Board concludes that the subject matter of claim 1 involves an inventive step in the sense of Article 56 EPC.