T 1294/16 (Image data arrangement/OMRON) 10-03-2021

The application and the prior art

1. The application relates to the detection of patterns on products in manufacturing context (paragraph 2).

1.1 Starting from the known technique of using normalized cross correlation (NCC) to detect known patterns in grayscale images (paragraphs 3, 4), the application aims at providing a robust NCC method for color images, improving on known prior art (paragraphs 5 to 8).

1.2 For this purpose, the 3-color image is transformed into a 2D or 1D matrix by rearranging the values according to predetermined rules. The NCC is then computed on this matrix (paragraph 9 and, further in detail, paragraphs 68 to 73 and 103 to 110, in particular equations (1) and (4)).

2. The prior art documents relevant for this decision are briefly summarised here.

2.1 Document D1 teaches a method of defect detection. Its contribution to the art is the extension of the NCC for use on a 3-color image (abstract; page 2526, right column, 2nd paragraph; Section 2).

2.2 Document D2 is from the same authors as D1. It teaches a method of defect detection with NCC on a grayscale image (abstract). This document also explains that NCC is commonly used for pattern detection by template matching (page 2626, left column).

2.3 Document D3 is an excerpt from a textbook on object recognition explaining the usage of cross-correlation for template matching (section "Matching by correlation").

2.4 Document D6 is a section of the MATLAB documentation explaining the RGB format.

2.5 D4 teaches the different color acquisition arrangements, including single CCD and 3-CCD (Figure 12 and enclosing section "Color cameras"; Figures 41 to 45 and enclosing section "Digital camera limitations").

2.6 Document D7 is a textbook excerpt on arrays and records and their storage in memory.

Main request: inventive step

Starting point (closest prior art)

3. The Examining Division considered that claim 1 of the main request lacked an inventive step starting from document D1.

4. The appellant (page 4 of the grounds of appeal) disagrees with the choice of D1 as "closest prior art" and submits that D2 should instead be the "closest prior art", because D2 has, or at least mentions, the same purpose, i.e. a search for a model in an image, whereas D1 is only about defect detection. In support, the appellant quotes the "Case Law of the Boards of Appeal" (I.D.3.1, seventh edition; same section in the ninth edition) stating:

The boards have repeatedly pointed out that the closest prior art for assessing inventive step is normally a prior art document disclosing subject-matter conceived for the same purpose or aiming at the same objective as the claimed invention and having the most relevant technical features in common.

5. This Board endorses the opinion that a document with a different purpose can be selected as a starting point in an inventive step analysis (see, e.g., T 1742/12, points 9 and 10). Under Article 56 EPC 1973, the only relevant question is whether the skilled person, having regard to the state of the art, would find the claimed invention to be obvious. This does not exclude the skilled person starting their considerations from any piece of prior art they might be aware of. Thus, the choice of the starting point is not restricted. The Board notes that the above passage quoted by the applicant is a rule which states which document is "normally" (not "always") to be chosen as closest prior art. This rule is employed for efficiency reasons, but it cannot be taken to exclude the selection of different prior art as a starting point and thus as "closest prior art" (see T 0694/15, point 13; T 0405/14, point 19; T 0816/16, point 3.7.1).

6. Considering the situation as put forward by the appellant, the fact that a document has a different purpose than the invention does not mean that it cannot lead to a finding of obviousness. There can be obvious reasons for using the teachings of the document for the claimed purpose, e.g. when the skilled person reading it knows that the teaching can be used, as it is or analogously, for that purpose.

Preliminary assessment of differences

7. In the present case, the finding in respect of ob­viousness does not differ if one starts from D1 or D2.

7.1 Starting from D1, the skilled person would immediately recognize, on the basis of their common knowledge (as acknowledged in D2; see point 2.2 above), that the NCC proposed can also be used for pattern detection with template matching and would use it accordingly (at least in order to evaluate its effectiveness).

7.2 Starting from D2, the skilled person would replace the grayscale NCC with a color one, for reasons of improved accuracy, as explained in D1 ("alleviates false alarms", page 2526, right column, second paragraph).

7.3 So, either way, the skilled person arrives at a pattern detection method based on template matching using the color NCC formula (equation 2 of D1).

8. The Board further agrees with the Examining Division (point 17.1.4, second paragraph) that a pattern detection method based on template matching in its standard form implies the features of shifting, re-computing the correlation value and selecting a match at the highest correlation point, as shown by D2 and D3 (see points 2.2 and 2.3 above).

9. The appellant submits that D1 and D2 taken together, (even when considering D6) would not disclose the transformation step itself, and its repetition for every shift (pages 7 and 8 of the grounds).

10. The Board is of the opinion that the repetition is intrinsic to the default implementation of the template matching procedure, which requires that at each position in the large image the corresponding subimage (D2, page 2626, left column) is fetched and matched with the template. The repetition of this fetching process requires the repetition of all necessary steps to transform the subimage in a form that can be matched with the template, thus including a transforming step if necessary.

11. Hence the only question that remains to be answered in respect of the transformation and matching steps taken together is whether, when implementing equation 2 of D1, the transformation step as claimed would be obvious.

11.1 The wording of this feature is: a first data array transformation means adapted to arrange at least two of the three color variables of each pixel contained in the determination target region in a single first data array according to a predetermined rule.

11.2 As the predetermined rule is not specified at all, this wording says nothing more than that the color values of the pixels are arranged in a single data array, which is what fetching the subimage also does by copying the pixels from the original color image to a single array, i.e. the subimage. Hence this feature is implied by the template matching procedure as discussed above.

12. The appellant contested that the fetching of the subimage could be equated to the claimed transformation step. The skilled person understood this step as defining a transformation of the form of the data matrix e.g. at the mathematical or software (programming) level, not simply the copying of the data from the image region into a subimage, i.e. to another memory, during execution.

13. The Board disagrees with this argument because the claim does not specify any such reshaping, it merely requires transformation means to arrange the data of a target region in a single data array (all of it, when three color variables are considered), i.e. the subimage to be matched with the template.

14. Thus claim 1 of this request lacks inventive step starting from D1 in view of the common knowledge in the art, or from D2 in view of D1 and the common knowledge in the art, Article 56 EPC 1973.

Auxiliary requests: admittance

(Article 13(1) and (2) RPBA 2020)

15. Article 13(1) RPBA 2020 states that any amendment to a party's appeal case after it has filed its grounds of appeal or reply may be admitted only at the discretion of the board. Article 13(2) RPBA 2020 further provides that any amendment to a party's appeal case made after notification of a summons to oral proceedings shall, in principle, not be taken into account, unless there are exceptional circumstances, which have been justified by cogent reasons by the party concerned.

16. The amendments to the first two requests with respect to the auxiliary requests as filed with the grounds of appeal consisted in the addition of the following specification of the form of single data array resulting from the first alternative predetermined rule: which is two-dimensional.

16.1 The appellant explained that the amendments were a reaction to the Board citing document D7. It clarified that the first transformation rule lead to data being transformed into a two-dimensional matrix, and not a linear array for storage into memory, which was the teaching of D7.

16.2 The Board accepts that this is the case but notes that it had introduced D7 only as evidence that an argument by the Examining Division was based on common general knowledge (see the summons, point 17). This is a development that the appellant should have foreseen and which does not change the objection in substance.

17. The third auxiliary request was filed during the oral proceedings before the Board. The appellant stated that the filing was caused by the explanations provided by the Board during the oral proceedings as to why no technical effect could be acknowledged, in particular the distinction made between mathematics, software and hardware. The amendment aimed to make clear the difference with D1 in terms of the mathematical equation used and the usage of just two loops for computation instead of three.

17.1 However, the Board's finding of a lack of inventive step is a confirmation of the same finding of the Examining Division. Furthermore, it was already set out in the summons to oral proceedings that the mathematical result was the same and that no technical effect could be identified in view of the lack of software and hardware specification. That the Board is not persuaded by the argumentation of the appellant is a foreseeable possibility.

17.2 The amendments to the third auxiliary request limit the claims in such a way that an argument important to the appellant finds basis in the claim. The appellant's interest to make a claim fit its arguments is not, in the Board's view, sufficient to justify the late filing of amendments.

17.3 As explained in points 16.2, 17.1 and 17.2 the Board holds that the reasons provided by the appellant have not established "exceptional circumstances" in the meaning of Article 13(2) RPBA 2020 for filing the three auxiliary requests at late stages in the proceedings.

18. It needs to be decided whether, in view of the very stringent wording of Article 13(2) RPBA 2020, the Board may admit the new requests nonetheless.

18.1 The Board finds Article 13(2) RPBA 2020 to be ambiguous. It does not merely state that submissions filed after notification of a summons to oral proceedings shall not be taken into account, unless there are exceptional circumstances (that are justified with cogent reasons), but that this should apply only "in principle". "In principle not" is frequently construed as "only under extraordinary, i.e. exceptional, circumstances". This means that, under a literal reading, Article 13(2) RPBA 2020 could be paraphrased to read in pertinent part:

Any amendment to a party's appeal case made... after notification of a summons to oral proceedings shall[deleted: , in principle, not ]be taken into account only under exceptional circumstances, unless there are exceptional circumstances, which have been justified with cogent reasons by the party concerned.

It is not clear whether the term "in principle" is

(i) effectively redundant over the express requirement of exceptional circumstances,

(ii) whether it is in opposition with that express requirement, as the literal reading of Article 13(2) RPBA 2020 might suggest, thus making this provision contradictory in itself and, as a consequence, not applicable, or

(iii) whether the term means something different and what this could be. In this latter respect, it is unclear inter alia whether the term "in principle" is meant to provide a residue of discretion for the Board to admit a request even in the absence of exceptional circumstances (or where no cogent reasons were supplied for their justification).

The Board is of the view that the second ("literal") option would invalidate Article 13(2) RPBA 2020 and thus cannot reflect the lawmakers' intention. Taking either the first or the third option means that no further conclusions can be drawn on the basis of the qualification of "in principle". The Board concludes that the term "in principle" should be ignored.

18.2 The Board also notes that neither the Article nor the explanatory remarks contained in CA/3/19 give an explanation of how to determine in general whether the circumstances are "exceptional". However, the example provided in the explanatory remarks to Article 13(2) RPBA 2020, according to which the Board raising a new objection can be seen to constitute such exceptional circumstances, suggests that, in view of procedural fairness vis-à-vis the concerned party, considerations similar to those in Article 13(1) as to "the suitability of the amendment to resolve the issues ... which were raised by the Board" should (exceptionally) prevail over considerations of procedural economy, although the Board raising a new objection is in a situation that may not necessarily be qualified as exceptional in the (dictionary) sense of unusual or uncommon. The exceptionality is hence not necessarily linked to events being exceptional in the sense of deviating from the expected, but can also be caused by considerations related to the legal framework, notably the principles underlying the rules of procedure.

18.3 Articles 12 and 13 RPBA 2020 implement what the explanatory remarks refer to as "convergent approach", according to which it should be the more difficult for parties to have their submissions considered the later in the appeal proceedings they are made. The major motivation for this principle is the procedural economy of the appeal proceedings. If admittance of a (late-filed) submission is not detrimental to procedural economy this Board considers it appropriate to accept that "exceptional circumstances" within the meaning of Article 13(2) RPBA 2020 are present, and justified to admit the submission, provided that this does not adversely affect any other party. The exceptionality of this situation resides in that considerations related to procedural economy are not present and thus the interests of the party in overcoming objections by amendment may prevail without running counter to the principles of the convergent approach.

18.4 Moreover, the Board sees that there are circumstances which are beyond the submitting party's control, namely the Board's judgment as to whether it can, without undue delay, deal with the proposed submission, but also any other party's agreement to have the submission taken into account. In particular, if the Board, of its own motion, finds the circumstances exceptional in view of the purpose of the convergent approach, then cogent legal reasons need not be brought forward by the party.

19. The Board considers the principles set out in the preceding paragraph to be complied with in respect of all three auxiliary requests. These being ex parte proceedings, no other party is adversely affected, and procedural economy was not affected as the Board was able to deal with the amendment without undue delay during the oral proceedings. That is because the submissions were only further specifications aiming to make sure that all of the appellant's arguments already brought forward are taken into account (see points 16.1, 17, 17.2 in view of the appellant's arguments as summarised in points 12 and 23). The Board therefore finds the auxiliary requests to be filed under "exceptional circumstances" within the meaning of Article 13(2) RPBA 2020 and takes them into account.

With this conclusion, the Board is in agreement with the decision T 2135/18 (reasons 2).

20. The Board would like to make it clear that it would have come to the same conclusion by exercising its discretion under Article 13(1) RPBA 2020 in view of the general approach applying to the admittance of late-filed submissions which can be briefly summarised by the expression "the later, the stricter". The considerations under point 19 above would then apply mutatis mutandis.

First auxiliary request: inventive step

21. This request differs from the main request in that it defines two alternative rules of arranging the color values in a single data array.

22. This arranging does not change the result of the color NCC, which remains identical to that of D1, as the order of the elements is irrelevant for the result of the computation (compare equation 2 of D1 with equations 1 and 4 in the application). This was not contested by the appellant. There is therefore no technical effect in terms of the results of the detection procedure.

23. The appellant argued that the arrangement simplified the addressing of the data values, because it required the use of just one or two indexes, instead of three. It was sufficient to use one or two loops to go through the data instead of three, which made the method faster. This also made the programming more convenient for the programmer.

23.1 If one took the objective technical problem of speeding up the computations, it would have to be seen that D1 did not address this problem. D2 did, but solved it in a different manner, using sum-tables (D2, abstract).

23.2 D7 was also not concerned with this problem, but even if D7 were to be considered by the skilled person, it would not lead to the claimed arrangements. Although D7 taught that one can vary the indexes in different order to create the linear memory array, this was just a memory arrangement, the matrix itself would still be three-dimensional, using three indices. Furthermore, when considering the NCC formula from D1, the obvious way would be to go through the position indexes first, following the way this was defined in D1.

24. Moreover, there was no requirement under Article 56 EPC of a technical effect. Even if a technical effect was not acknowledged, the claim should be recognized as compliant with Article 56 EPC (1973) because it was not obvious. Starting from D1 or D2 there was no pointer to the specific data arrangements claimed.

25. In response to this second point, the Board recalls that it is of constant jurisprudence that only features contributing to the solution of a technical problem by providing a technical effect can contribute towards a finding of inventive step (T 0641/00 Two identities/COMVIK, point 6). This is because the EPC provides for protection of inventions, which require a technical solution to a technical problem (Rule 27(1)(c) EPC 1973). It is the (claimed) invention, i.e. the technical solution, that needs to involve an inventive step (Articles 52 EPC and 56 EPC 1973) for a patent to be granted.

26. In response to the first set of arguments (point 23. above), the Board first remarks that, whichever way the data is arranged, the number of operations (data read-outs, additions, multiplications) cannot change, because the computation of the NCC requires all data to be read and the same products to be computed and added.

26.1 So, if there is any speed-up, it must come from a faster data read-out, caused by the allegedly simpler structure. This, however, cannot be assessed without information on the software and hardware architecture. Such information is not provided, neither in the claim, nor in the description. A simpler indexing structure does not necessarily translate to a faster read-out. This depends on a number of factors, including the data structures allowed by the programming language, the memory management, the compiler design etc. If, for instance, the programming language is optimized for treating RGB images in the standard 3-plane format (see D6), then forcing a rearrangement of the data matrix will actually reduce performance. There is no information in the description allowing to assess this.

26.2 The Board does not dispute that a technical effect might be obtained under some circumstances. But this is not sufficient for acknowledging an inventive step, because those circumstances are not claimed (nor are they in fact made clear by the description), which means that the effect is not obtained over the full breadth of the claim, which in turn leads to the conclusion that at least a subset of the claimed matter is not to be seen as a technical solution to a technical problem and hence cannot be acknowledged as an invention involving an inventive step (see also T 0939/92, points 2.4 to 2.6).

27. Regarding the argument as to the convenience for the programmer, i.e. easing the programming effort, this, in principle, cannot be considered to define an invention at all (T 1539/09, Catchword). Arguendo, even if that could be the case, the technical effect cannot be objectively assessed here, because on the one hand the programming language is not specified, and because on the other hand the answer is a matter of subjective preference: while writing code with only two loops may be more convenient, the indexes will no longer intuitively directly represent the standard RGB format, making code reading less convenient.

28. Thus the claimed data transformation means does not solve any technical problem at all, and hence cannot contribute to a finding of inventive step. As in the case of the main request, claim 1 of this request lacks inventive step starting from D1 in view of the common knowledge in the art or from D2 in view of D1 and the common knowledge in the art.

Second auxiliary request: inventive step

29. Claim 1 of this request defines an image sensor, comprising an image processing device as per the previous request, where the transformation rule is defined as being according to the first rule only. The added features are those of using a display, and of using a 3CCD sensor.

30. A display is implicit in D1.

31. In view of document D4 (passages as cited above), it is clear that the 3CCD configuration is common knowledge, and its advantages (better image quality) are known. Thus the skilled person would use it for template matching if needed.

32. In support of inventive step, the appellant argues synergy between the 3CCD acquisition set-up and the transformation step. Thus was explained in paragraph 99 (description as filed), wherein it is stated:

... since the data array is generated by combining the red gray image, the green gray image, and the blue gray image configuring one image, transformation to the data array is facilitated by using the picture signal from each CCD if the imaging section including three CCDs for acquiring the gray value for each color is used.

33. The Board is not convinced of the presence of synergy. Neither the claim nor the description specify any specific hardware arrangements that would provide for an effect of the usage of the 3 CCD sensor going beyond improved image quality. Paragraph 99 only describes a desired effect, without describing how this is actually achieved. The claim is not specific either as to how the signals from the CCDs are used. In fact, the straightforward interpretation thereof is that the CCDs are used to obtain a standard RGB image, as e.g. in D6, having no further influence on the following processing.

34. Claim 1 of this request lacks inventive step starting from D1 in view of the common knowledge in the art, or from D2 in view of D1 and the common knowledge in the art.

Third auxiliary request: inventive step

35. The Board has already considered the argument as to the number of loops, so this cannot help the appellant's case. The mathematical formula itself is different to the one in D1, but this is immaterial for the inventive-step assessment in the present context.

Differences in mathematics can lead to a finding of inventive step only if they contribute to defining a technical solution in a field of technology not excluded under Articles 52(2) and (3) EPC. This is not the case here, the proposed formula is only a mathematical equivalent of that in D1 causing no ascertainable technical effect, as discussed in the framework of the previous requests.

36. Claim 1 of this request lacks inventive step starting from D1 in view of the common general knowledge in the art, or from D2 in view of D1 and the common general knowledge in the art.

Alleged procedural violation

37. The appellant submits that in the oral proceedings before the Examining Division only D2 was considered as the "closest prior art" in the framework of the "problem and solution approach", whereas the decision is based on document D1 as "closest prior art". The applicant was not heard on the grounds that led to the refusal, in breach of Article 113(1) EPC 1973.

38. The minutes of those oral proceedings mention that the "preliminary opinion of 23-11-2015" was discussed (1st page, paragraph 4), especially its "section 1.3.2" (pa­­ra­graph 10), and state (1st page, last sentence, to 2nd page, 1st paragraph) that the "Art 52 objection of the preliminary opinion under point 1.3.2 was withdrawn" and that "the Art 52 objections raised in sec­tion 1.5 [...] were maintained". There is no point 1.5 in the preliminary opinion of 23 November 2015. The summons to oral proceedings dated 5 June 2015 contain objections starting from both D1 and D2 (in combination with D1) in points 1.4 and 1.5. The discussion during the oral proceedings then focused on the teachings of D1 (2nd page, top: "fills the gaps of D1"; further down "The contiguous data is not explicitly disclosed in D1").

39. Thus, on the basis of the examination file itself, it appears that it was at least implicit during the oral proceedings, i.e. it should have been known to the applicant that both D1 and D2 were considered as "closest prior art".

40. The appellant disagreed. It was clear during the oral proceedings before the Examination Division that the starting point was D2. The arguments were different if one started from D1 or D2. The case could then have been argued differently.

41. The Board is not in a position to verify this allegation. It would have been for the members of the Examining Division to do so had the applicant contested the minutes of the oral proceedings before the division. Thus, the Board cannot find that a (substantial) procedural violation took place.

The request for referral to the Enlarged Board

42. The appellant argued that the Examining Division had based its decision on D1 as closest prior art that had not as such been discussed during the oral proceedings. In its preliminary opinion (see points 5.3 and 5.4), the Board expressed its doubts, stating, with reference to passages from the minutes, that it seemed to be "at least implicit" that both D1 and D2 had been considered as closest prior art by the Examining Division. The appellant takes issue with the idea that an only "impli­cit discussion" could be sufficient to satisfy the appellant's right to be heard under Article 113(1) EPC 1973 and asks the Board to submit that question to the Enlarged Board.

43. The board takes the view that a potential violation of the appellant's right to be heard by the Examining Division would have no impact on the present decision, as the appellant did not contest that the right to be heard on the matter was respected by the Board. The appellant acknowledged to have had ample opportunity to present its case before this Board. In particular, it was discussed what is the appropriate starting point ("closest prior art") for the assessment of inventive step, the Board explicitly considered both D1 and D2 as possible "closest prior art" documents, and the appellant was given several opportunities to file amendments at a very late stage of the proceedings.

44. Hence, the answer to the question the appellant asks the Board to refer would have no consequences on the present case, so that a decision is not required (Article 112(1)(a) EPC 1973).