T 2156/17 (Detecting suspicious activity using video analysis / NCR Corporation) 10-06-2022

New main request - Admissibility (Article 13(2) RPBA 2020)

1. The new main request was filed after notification of the summons to oral proceedings, in reaction to the Board's preliminary opinion. It addresses, specifically, the objection of lack of inventive step on the basis of document D1, raised by the Board for the first time. Its admission depends on the conditions set out in Article 13(2) RPBA 2020, which reads:

Any amendment to a party's appeal case made [...] after notification of a summons to oral proceedings shall, in principle, not be taken into account unless there are exceptional circumstances, which have been justified with cogent reasons by the party concerned.

2. The appellant's reasons are cogent. Although the objection of lack of inventive step based on D6, relied upon by the Examining Division, was essentially endorsed by the Board in its preliminary opinion, the new objection, based on D1, relied on a completely new approach towards inventive step.

3. The Board considers that the new objection constitutes exceptional circumstances in the sense of Article 13(2) RPBA 2020. It considers, further, that the amendments define additional technical limitations over D1, thus directly addressing the new objection.

4. The new main request is, therefore, admitted into the proceedings.

New main request - Inventive step (Article 56 EPC)

5. Document D6 was cited by the Examining Division as evidence for a method relying on time stamps to establish correlations between two sets of data items.

6. D6 is particularly relevant to the inventive merits of the claimed invention. It discloses a surveillance method for reviewing behavioural and transaction events which occur at one or more point-of-sale (POS) terminals in a retail environment. It comprises the steps of generating video image and transaction signals, generating a synchronising signal, and correlating these three signals (page 5, lines 12-19). It can be used to monitor or identify suspicious activity (page 10, lines 1-4; page 25, lines 7-9).

7. D6 is not limited just to a method of time stamping a sequence of events synchronously recorded on a video tape and a floppy disk, in order, later, to allow video data to be aligned with transaction data.

8. In the terms of claim 1, D6 discloses

A [deleted: computer-implemented] method for detecting a transaction outcome, the method comprising:

obtaining video data associated with a transaction area;

analyzing at least a portion of the video data to obtain at least one video parameter concerning at least a portion of a transaction associated with the transaction area, [deleted: wherein the step of analyzing comprises automatically analyzing frames of video from at least one region of interest in the at least a portion of the video data to identify a respective event indicating the presence of an item associated with the transaction];

obtaining at least one transaction parameter originated from a transaction terminal associated with the transaction area, wherein the step of obtaining comprises analyzing at least a portion of transaction data from a transaction terminal to identify a sequence of items identified as being transacted in the transaction; and

[deleted: automatically] comparing the at least one video parameter to the at least one transaction parameter to identify a transaction outcome, wherein the step of [deleted: automatically] comparing comprises analyzing the sequence of items identified as being transacted in the transaction from the transaction data in comparison to the events produced from analysis of the video data to determine if the item represented by at least one event produced from the analysis of the video data is indicated as an item for transacting in the at least a portion of transaction data, and correlating video timestamps of events from the analysis of the video data to transaction timestamps of items reflected as having been transacted in the transaction data to identify events indicating an item in the video data that does not have a corresponding record in the transaction data,

obtaining video data further comprising:

obtaining video data originating from at least one [deleted: elevated] video camera that monitors a transaction area defining the region of interest; and

wherein analyzing at least a portion of the video data comprises:

analyzing the video data to track items involved in the transaction in the transaction area; and

wherein [deleted: automatically] comparing the at least one video parameter to the at least one transaction parameter to identify a transaction outcome comprises:

comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity, the comparison including associating scanned item data with video data such that the timestamp of the scan is synchronized with the timestamp of the video.

9. Document D6 describes a manual process of comparing the video data with the transaction data, and it does not disclose the features added to claim 1 in response to the Board's provisional opinion, according to which analyzing at least a portion of the video data comprises:

defining at least one region of interest within the video data;

calculating an object map that identifies a change between a current image of the at least one region of interest and an updated background image of the at least one region of interest;

isolating at least one operator object within the at least one region of interest;

detecting if an analysis of the object map and the operator object identifies at least one of the removal and introduction of an object to the at least one region of interest; and

applying image adjustment processing to the video data to compensate for an autogain characteristic across an entire area of the video data, the image adjustment processing restricting its application to a portion of the region of interest captured within the video data;

wherein applying image adjustment processing comprises:

restricting focus to objects in the transaction area that remain stationary and that exhibit at least one of a color shift and a brightness shift.

10. The analysing steps define, in relation with the recited steps of "automatically analysing frames of video" and of "automatically comparing", further technical contributions of the claimed subject-matter over D6.

11. The appellant's view, that D6 neither disclosed the step of comparing multiple records of the transaction area at the transaction time nor the step of identifying items found in the video data but not in the transaction data, is rejected. The disclosure of D6 concerns surveillance systems that record transaction events for review at a later date (page 1, lines 4, 5). The very purpose of the system disclosed in D6 is to allow such review of video data and corresponding transaction data by an operator (claim 1). The fact that the comparison is made by the operator, when reviewing the synchronised video and transaction data, does not affect this.

12. The distinguishing features identified above are the consequences of significant differences between the claimed method and the disclosure of D6. In effect, the claimed method differs from the method disclosed in D6, essentially, in that it is computer implemented and performed automatically, and in that the video data originates from an elevated video camera.

13. The selection of an elevated camera merely reflects the need for a complete and unhindered view of the regions to be monitored. By itself, it does not justify the existence of an inventive step. The skilled person would have selected the most adequate location for the camera, depending on the circumstances and the scene to be monitored.

14. The main problem addressed by the distinguishing features identified above is the adaptation of the system of D6 to make it autonomous, so that it does not require the presence of a human operator. Solving this main problem entails addressing further technical issues resulting from the need to exclude false interpretations of video data, as may result from a POS-operator being present in the region of interest or from autogain corrections.

15. As observed by the Examining Division, with reference to decisions T 775/90 and T 1175/02, the mere automation of functions previously performed by human operators (supervisor) is in line with the general trend in technology (Decision, paragraph 19). Although the decision to automate the process of D6 undoubtedly requires a very different approach and involves a multiplicity of secondary issues as to the processing strategy best adapted to the intended purpose, there is no obstacle, in principle, to such a step.

16. In D6, the sets of video and transaction data are both recorded and time stamped. This permits a precise alignment of the two, when replayed afterwards by the respective playback means. Contrary to the appellant's view, no improvement in terms of alignment of video data and transaction data result from the claimed process compared to the manual process of D6.

17. However, the steps of focusing on a stationary area, in order to compensate for the autogain correction, contribute to the reliability of the automated process, insofar as the determination of (moving) transaction items is concerned.

18. In the Board's judgment, the initial decision to automate the process of D6, to make it fully autonomous, would have inevitably led the skilled person to consider subordinated issues associated with the treatment of digital video data. The skilled person would, in particular, have been confronted with the problem of discriminating between (moving) transaction items, a moving operator and (stationary) background objects.

19. The Board has no doubt that comparison by way of subtraction within a sequence of images, in order to identify changes within a region of interest, was well-known at the priority date (2005). The same applies to autogain correction techniques. The Board has also no doubt that the techniques consisting of modifying an image on the basis of statistics obtained from a selected area of it, was also well known.

20. This view is confirmed by the very content of the application, that does not elaborate on these specific aspects of the invention, but refers to them merely in passing. The Board observes that a different finding would have led, in the absence of specific indications as to the content of these measures, to an objection of lack of sufficiency under Article 83 EPC.

21. Nothing that was not obvious results from the step of comparing images by way of subtraction, in order to identify changes, that is, motion within the field of view. Being aware of autogain correction techniques, the skilled person would have recognised that it would directly interfere with the reliability of the comparison process. Faced with wrong interpretations resulting from autogain corrections when a large object, not part of the transaction, appears within the field of view of the camera (page 27, lines 30-35), the skilled person would have considered compensating for this detrimental effect.

22. The solution entails quantifying the effect to be compensated. It implies determining on a statistical basis (histogram analysis) how pixels in a a portion of the image that remained stationary have been affected. Concretely, the portion of the image to be considered was a portion of the image the content of which, in the absence of correction, would not have been modified. This reflects the necessity not to incorporate in the statistical analysis required for the compensation the moving object at the origin of the problem (application, page 27, line 36 - page 28, line 9).

23. The claimed solution did not require any inventive skills. It resulted in a straightforward manner from the application of the known techniques referred to above, and from mere logical considerations as to the use of these well known techniques to solve predictable problems.

24. The subject-matter of claim 1 according to the new main request is thus not inventive in the sense of Article 56 EPC.

Auxiliary request 1a - Admissibility

25. Auxiliary request 1a was filed during the oral proceedings before the Board, so its admission is subject to Article 13(2) RPBA 2020 (cf. point 1, above), and also the criteria of Article 13(1) RPBA 2020.

26. Auxiliary request 1a is an attempt to parry the Board's decision not to admit the new first and fourth auxiliary requests on the ground of lack of convergence, when compared to the new main request. (The admissibility of the fourth auxiliary request is discussed, below).

27. The appellant's arguments are not sufficient to decide favourably on the admissibility of auxiliary request 1a. The circumstances referred to by the appellant are not exceptional, but relate, on the contrary, to ordinary aspects of the appeal proceedings.

28. There is nothing exceptional in the board applying the criterion of convergence to question the admissibility of an auxiliary request freshly filed in appeal proceedings (Article 13(2) RPBA 2020). Although convergence, as stressed by the appellant, is not the sole criterion, there is also nothing surprising in the Board applying it to the new first and fourth auxiliary requests filed in response to the communication of the Board. It was up to the appellant to consider the rules of procedure and the criteria applied according to the case law, and the possible consequences resulting from this new request being filed in reaction to the Board's preliminary opinion.

29. The right to be heard under Article 113 EPC does not entail the right to be given further possibilities to file new requests if earlier attempts fail. In the Board's judgment, the appellant's right to be heard is met under the circumstances, since they were aware of the objections to the admissibility of the earlier requests, and had the opportunity to contest this and present their arguments (T 732/21).

30. Auxiliary request 1a is thus not admitted into the proceedings (Article 13(2) RPBA 2020).

First auxiliary request - Inventive step (Article 56 EPC)

31. The claimed invention is a mixture of technical and non-technical features. The approach generally followed to assess inventive step of the subject-matter of a claim comprising a mixture of technical and non-technical features derives from T 641/00, Two identities/COMVIQ, OJ EPO 2003, 352. This approach was generally endorsed by the Enlarged Board of Appeal in G 1/19 Pedestrial simulation, OJ EPO 2021, A77, points 31-34, and points 35 and 36 with explicit reference to G 3/08, Programs for computers, OJ EPO 2011, point 10.13.2).

32. The distinguishing feature of obtaining video data originating from at least one elevated video camera that monitors a transaction is not inventive for the reasons explained above with regard to the new main request.

33. The claimed method is further distinguished from the method of D6 in that the process is automatic, in that it incorporates a step of comparing the set of transaction events with the transaction data in order to identify a discrepancy in the respective number of items, and in the further step of identifying an overall "suspicion level" for the transaction.

34. Contrary to the appellant's view, the step of comparing the set of detection events with the set of transaction data to identify a discrepancy is not technical. Independently of the nature of the events, it must be stressed that the comparison carried out simply compares data. All in all, the claimed step of comparing said data amounts to a mere comparison of lists by computer means.

35. The origin of the data are without bearing on the claimed step of comparing. The technicality of the transaction, as it manifests itself in the movement of items in a sequence of video frames, is lost when it comes to the step of comparing the data. The Board further rejects the view that the count that results from the comparison is technical in the context of the invention. The mere fact that the identification of a discrepancy is used to generate a "suspicion level" is also not sufficient to confer technical character to said method. The notion of "suspicion" is essentially subjective and as such not technical. The same applies to any parameters that may be derived therefrom.

36. It follows that the only feature of a technical nature, beyond the step of obtaining video data by means of an elevated camera, that distinguishes the claimed method from that of D6 resides in the automation of the process.

37. As explained above, this distinguishing feature reflects a general trend in technology and is not sufficient for an inventive step.

38. The subject-matter of claim 1 according to the first auxiliary request is thus not inventive in the sense of Article 56 EPC.

Second auxiliary request - Inventive step (Article 56 EPC)

39. Claim 1 of the second auxiliary request includes limitations regarding the step of comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity. Concretely, the step is performed by comparing a first frame and an empty base image, by subtraction and thresholding and the creation of a "binary object map", to identify new objects in the image that are not part of the empty base image, including an operator "object" and other objects within the region of interest.

40. The appellant stressed that it was not required to automate the process of D6 in order to identify transaction items. The claimed subject-matter extended beyond a mere automation of the process, since it required additional steps to arrive at the determination of the transaction items. There was no indication, in the prior art, that identification of objects could be obtained by subtraction of images followed by thresholding and the generation of a binary object map. Document D4 referred in general terms to techniques of thresholding in order to generate binary images. D4 was concerned with the identification of items based on characteristics that could be derived from video images. It did not suggest using these techniques to create object maps in order to identify objects entering or leaving a region of interest.

41. In the Board's judgement, the claimed subject-matter results in a straightforward manner from the initial decision of automating the process of D6. As stressed above, with regard to claim 1 of both the new main request and the first auxiliary request, no inventive step is involved in this initial step.

42. It would have been obvious for the skilled person, seeking to identify objects entering or leaving the region of interest, to rely on the dynamic inherent to the process, as it manifests itself in changes in the content between successive frames. Hence, it would have been obvious for the skilled person to use image processing techniques adapted for identifying such changes.

43. The Board has no doubt that the basic operations of subtracting images, and thresholding to create binary images were well-known in 2005. As stressed above, if this were not so, and in the absence of any details as to the implementation of these steps in application, an objection of insufficiency under Article 83 EPC would have been justified.

44. The issue of obviousness under Article 56 EPC appears thus to hinge on the question whether it would have been obvious, in the context of automating the process of D6, to use such known processes.

45. The comparison of frames by way of subtraction is straightforward in that it serves exactly the purpose of identifying changes between successive frames. It is particularly adapted to the present purpose of identifying objects entering, moving within, or leaving the region of interest. The comparison of an image with an empty base image (that is, an updated version of the background image (application, page 25, lines 18-23)) is a direct consequence of this finding. In order to allow reliable identification of transaction items, that is of objects entering, moving within, or leaving the region of interest, the skilled person would have recognised that processes of thresholding, to obtain a binary image of the region of interest, would have been particularly adapted to reducing the noise component in the resulting image.

46. The subject-matter of claim 1 according to the second auxiliary request is thus not inventive in the sense of Article 56 EPC.

Third auxiliary request - Inventive step (Article 56 EPC)

47. Similar considerations apply to the steps defined in claim 1 according to the third auxiliary request in order to obtain the video parameter.

48. The terminology is somewhat different, claim 1 defines a more general version of the comparison with a base image in claim 1 of the second auxiliary request. This is apparent in the following step:

defining a region of interest within the transaction area;

automatically identifying a first frame of video data that indicates a first set of items in the region of interest;

automatically identifying a second frame of video data that indicates a second set of items in the region of interest, the first set of items being visually distinct from the second set of items: and

automatically indicating the visual distinctness of the first set of items from the second set of items as an event indicating an item existed within the region of interest of the video data ....

In particular, the Board sees no difference in meaning between an "updated background image" (or "empty base image" in the terms of claim 1 according to the second auxiliary request) and a first frame of video data that indicates a first set of items in the region of interest.

49. The step of automatically indicating the visual distinctness of the first set of items from the second, as an event indicating that an item existed within the region of interest, refers to the actual result of the subtracting and thresholding processes envisaged above.

50. For the reasons developed above with regard to the second auxiliary request, the recited features would have been obvious.

51. The manual process of D6 also relies on the step of obtaining transaction data associated with the transaction terminal associated with the transaction area. The transaction data are indicative of items being registered as transaction items with the transaction terminal. The step of obtaining one transaction parameter means, in the context of the claimed invention, the automation of this step. It is thus a step already envisaged when deciding on automating the process of D6.

52. In D6, the step of identifying suspicious activity, when transaction data are missing for an item indicated as being within the region of interest, is performed by the operator. The method of the invention simply automates this step. Independently of those two facts, however, the Board stresses that identifying suspicious activity does no more than reflect the discrepancy between the two sets of data. As emphasised above, it is the result of a mere comparison of lists of events deprived of any technical character.

53. The subject-matter of claim 1 according to the third auxiliary request is thus not inventive in the sense of Article 56 EPC.

New fourth auxiliary request - Admissibility

54. The new fourth auxiliary request was filed after notification of the summons to oral proceedings, in reaction to the accompanying preliminary opinion. Thus, its admission is subject to Article 13(2) RPBA 2020 (cf. point 1 above) and the criteria of Article 13(1) RPBA 2020.

55. The new fourth auxiliary request attempts to address the Board's preliminary view that the subject-matter of claim 1 according to the fourth auxiliary request was not inventive.

56. The features added to claim 1 stem from original claims 5, 9, and 10. Claim 1, however, does not incorporate the features of original claims 15, 26, and 27 which were added in the new main request. While the newly added features indeed address the objection of lack of inventive step, claim 1 of the new fourth auxiliary request focuses on aspects different than those pursued in claim 1 of the new main request. The two sets of claims are not convergent.

57. The new fourth auxiliary request is thus not admitted into the appeal proceedings.

Fourth auxiliary request - Inventive step (Article 56 EPC)

58. Claim 1 of the fourth auxiliary request includes the features of identifying and transmitting to a reviewer for review a segment of the video data where the video event that is not identified as a transaction event exists.

59. The added steps contribute, in the appellant's opinion, to a reduction of data transmission or data storage. They define a technical feature of the claimed method.

60. The primary purpose of the added step is to allow a review of mismatches.

61. In the Board's judgment, the review suspicious activity identified in the course of the automated process of surveillance can be achieved in two alternative ways. Whether the segment of video data identified as referring to a possibly suspicious event is to be transmitted or consulted by the reviewer on the video terminal is merely a matter of preference. Neither of the alternatives is inventive.

62. Although transmission bandwidths were certainly more limited in 2005 than now, this would not constitute an obstacle for such transmission taking place. The Board is persuaded that the claimed alternative of identifying and transmitting a segment of the video data, would have been envisaged by the skilled person.

63. For this reason, the subject-matter of claim 1 according to the fourth auxiliary request is considered to be obvious in the sense of Article 56 EPC.

Auxiliary request 5 - Admissibility

64. Auxiliary request 5 was filed during oral proceedings before the Board, in reaction to the non-admission or non-allowance of all the previous requests. Thus, its admission is subject to Article 13(2) RPBA 2020 (cf. point 1 above) and the criteria of Article 13(1) RPBA 2020.

65. The arguments put forward by the appellant are not sufficient to decide favourably on the admission of auxiliary request 5. The circumstances referred to by the appellant are not exceptional.

66. The objections to lack of inventive step, set out in the Board's preliminary opinion, were along the same lines as in the Examining Division's refusal. Both the Examining Division (point 19 of the impugned decision) and the Board (points 16 and 34 of the preliminary opinion) referred to common general knowledge with regard to the general trend of automation and to certain aspects of image processing. It is not surprising that the debate during the oral proceedings concentrated on these aspects in more detail.

67. The Board has not provided direct evidence for the existence of the common general knowledge referred to. This is justified under the circumstances, in view of the content of the description, which refers to the image processing techniques in question without explaining them or showing any implementation. This is an unambiguous indication that they were well-known. As mentioned above, a different conclusion would have led to an objection under Article 83 EPC, since the skilled person would have been at a loss as to what the terms used to define the techniques actually implied, and would have been unable to implement them.

68. It follows that the arguments based on common general knowledge were to be expected. The appellant cannot rely on the element of surprise to justify the admissibility of auxiliary request 5.

69. The present circumstances are not exceptional, and do not justify the admission of auxiliary request 5 which is thus not admitted into the proceedings (Article 13(2) RPBA 2020).

70. In the absence of an allowable claim request, the appeal must be dismissed.