T 2077/19 (Specimen areas/SONY) 07-02-2023

Reasons for the Decision

The patent

1. The patent relates to a processing method and system to control display of data obtained by a microscope in the fields of medical treatment, pathology, biology etc. (pa­ragraph 1). It addresses the case where a plurality of sections are placed on one slide and acquired within one image. If the sections are related to each other, e.g. multiple slices cut from a single specimen, their association needs to be maintained, as well as the possibility to analyse the individual sections separately (paragraphs 6 to 8).

2. The system acquires an image of the entire slide and detects the positions of the individual sections; this detection takes place in a lower resolution image (paragraphs 34 to 37, 52). A viewer is used to display the different sections in one of three modes (paragraph 58). In the first two modes, the display controller switches the display to show different individual sections using the obtained position information (these modes are referred to as "discrete" and "jump" display, respectively). In the third mode, multiple sections can be shown in a tiled manner (this is referred to as the "synchronous" display; see paragraph 58). In the second and third modes, the detected sections are aligned by extracting image features (e.g. Harris corner features) and matching the sets of features using an affine transform (paragraphs 42, 44 to 48). They are then displayed in alignment, using the translation parameters (offset) obtained from the affine transformation (paragraphs 53 to 56, 96, 97, 109 to 112, 117, 119, 120).

Admissibility of the opposition

3. The Proprietor contested the admissibility of the opposition as it did not comply with Rule 76(2)(c) EPC. The Opposition Division decided that the opposition was admissible (decision, point 1), noting that the notice of opposition clearly indicated the grounds for opposition and the extent to which the European patent was opposed. This was not contested by the Proprietor.

4. Regarding the third requirement according to Rule 76(2)(c) EPC of an indication of the facts and evidence on which the opposition is based, the Opposition Division stated that the Opponent

"filed eight documents of the prior art as well as several pages of argumentation explaining why in his opinion the facts and evidence presented support the grounds of opposition. The different features of the claims were addressed and the opponent gave explanation why in his opinion said features were disclosed or rendered obvious by the prior art. These pages of argumentation define a complete and coherent logical chain of argumentation and can reasonably be considered as an indication of the facts and evidence presented in support of the grounds of opposition. From these facts and submissions the opposition division was able to form an opinion on the grounds of opposition presented. The fact that the proprietor or the opposition division is convinced or not by this argumentation is irrelevant".

5. The Proprietor contested this part of the decision because (grounds of appeal, page 3, 3rd paragraph):

"studying the original facts and evidences provided by

the Opponent it was not possible to form an opinion on the opposition without further investigation. The Opponent did not for a single attack address all features of the independent claims in such a manner as to make it understandable why these features are considered disclosed in or obvious from the cited prior art.[..] For all attacks it was necessary to fill in gaps in the argumentation left open by the opponent." The Proprietor also pointed out that the Opposition Division did not explain which chain of argumentation was complete or coherent.

6. Considering the ground of opposition stating lack of novelty over D1, the Proprietor argued that the Oppo­nent did not indicate where in D1 features 5/V and 6/VI were disclosed, and argued that, even for implicit disclosure, passages had to be indicated demonstrating the implicit disclosure. That was not the case here. Furthermore, as D1 was a complex scientific paper, it was not straightforward to understand which passages might be relevant.

7. The Board agrees with the principle that the third requirement of Rule 76(2)(c) EPC is satisfied "if the contents of the notice of opposition [are] sufficient for the opponent's case to be properly understood, which has to "be assessed on an objective basis, from the point of view of a reasonably skilled person in the art to which the opposed patent relates" (see, e.g. T 222/85, headnote 1 and reasons 4, and T 204/91, reasons 5).

8. The Proprietor and the Opponent agree that the notice of opposition indicates passages for features 1/I to 4/IV, in particular by indicating that specimen areas are detected by blocks 2 and 3 of D1. The notice of opposition (at 2.1) also states: "Anschließend werden die individuellen Probenbereiche für die Anzeige neu arrangiert, was bedingt, dass die Positionslage zuvor abgespeichert wurde (Merkmale 5/V). Sie stehen dann zur individuellen Analyse zur Verfügung, es kann also zwischen den einzelnen Probenbereichen umgeschaltet werden (Merkmale 6/VI)". (Board's translation: "Subsequently, the individual sample areas are rearranged for display, which implies that the position location has been previously saved (features 5/V). They are then available for individual analysis, so that it is possible to switch between the individual sample areas (features 6/VI)").

9. These statements are to be interpreted in view of D1 and, according to the cited principle, by reference to the "reasonably skilled person" in the relevant field.

9.1 Contrary to the Proprietor's submission, the Board does not believe that the "reasonably skilled person" in image processing would find document D1 to be parti­cu­larly complex. It is a scientific document presenting an image processing method with a rather typical degree of complexity. The skilled person is able to appreciate the contents of D1 and see whether the sample areas are, as the Opponent submits, rearranged after detec­tion, and whether they are individually analysed or displayed.

9.2 In particular, the Opponent referred to the passages describing processing blocks 2 and 3 for the detection of specimen area, which passages make reference to figures 4 and 5, and these figures show a display of one individual cell. This establishes an understandable case, namely that the two features in question are (implicitly) disclosed by the description of processing blocks 2 and 3 in conjunction with figures 4 or 5.

10. The ground for opposition stating lack of novelty over D1 is therefore sufficiently supported by an indication of facts and evidence as required by Rule 76(2)c) EPC. For an opposition to be admissible, only one ground for opposition must be sufficiently substantiated (see, for instance T 653/99, reasons 2). As a consequence, the opposition is admissible.

Patent as granted: novelty and inventive step

The Opponent's arguments

11. The Opponent argued that system claim 1 (and correspon­dingly method claim 8) lacked novelty in view of D1. There was common ground between the parties that fea­tures 1 to 3 were disclosed by D1.

12. Regarding feature 4.1, the Opponent was of the opinion (see in particular its reply of 24 February 2020, section 2.1.1) that the proper interpretation of this feature was that the detection and calculation of the position is done in the reduced image. In particular, because the detector operated in the reduced image, it could only have information on the position in the reduced image (paragraphs 35 and 36 of the opposed patent). Via the known correspondence between the low and high resolution images, positions in the reduced images also "indicated" corresponding positions in the high resolution images. No calculation of the position in the high resolution image was actually implied by the feature formulation.

12.1 In D1 the detection of the specimen areas was the result of the Hough Transform on the reduced image (block 2, stage 2.3, as described on page 1172, right column). This provided the position in the reduced image and, via the known correspondence, the position in the high resolution image coordinates.

12.2 The Opponent also argued (letter of 23 May 2022, section 3) that in D1 the reduced image was obtained by JPEG compression (D1, page 1171, right column, bottom), which did not change the resolution of the image. The detection of the specimen areas in the reduced image therefore also provided the position in the original image.

13. Regarding features 4.2. and 4.3 (see e.g. letter of 23 May 2022, page 4), the Opponent was of the opinion that they were disclosed by the use of the Hough Trans­formation which detected the cores by approxi­ma­ting them with circles and calculating their centre. In support, it referred to the last phrases in section Block 2 of D1. The Opponent also stated that claim 1 only required a detector able to detect specimen areas of the same shape, and that the specimen itself was not part of the claimed matter.

13.1 The detection of the core positions also disclosed fea­ture 5, because the further use of the core coordinated for gridding implied the storage of their positions. A usage merely "on the fly" was not reasonable given that D1 was concerned with the analysis of 8900 cores (see e.g. letter of 23 May 2022, page 4).

14. Regarding feature 6 (see e.g. letter of 23 May 2022, page 4), the Opponent considered that D1, concerned with checking the quality of the segmentation and of the arrangement of many (8900) cores, implied a display controller able to switch between the segmented areas, as so many cores could not be simultaneously displayed. The Opponent also made reference to the images of individual cores depicted in the "inserts" of figures 2 and 4. These were individual displays, which must allow a switching of the observed cores, for otherwise the user of the system of D1 could only observe one predetermined core.

14.1 But even if not implied, the implementation of such a measure reflected customary practice and would there­fore have been obvious for the skilled person (reply of 24 February 2020, section 2.2.1).

15. The Opponent further argued that, if one were to take the position that feature 4.1 is to be interpreted to say that positions are calculated in the high resolu­tion image, this would be an obvious alternative for the skilled person, given that the skilled person only has two options: calculate either in the low or in the high resolution image (reply of 24 February 2020, section 2.2.1).

16. The Opponent also pointed to page 1176 of D1, last paragraph, wherein it is stated that

"the correct assignment of cores on the TMA to patients is vital for individual patient based diagnosis and large-scale investigations alike".

This suggested an individual analysis of cores. The declared purpose implied a visualization of any individual core in high resolution, which in turn required both knowledge of their position in high resolution and means to display them individually, i.e. a controller to switch display as claimed.

The Proprietor's arguments

17. The Proprietor noted (letter of 24 June 2020, section 7) that the claim interpretation proposed by the Opponent with its reply of 24 February 2020 was neither discussed during the opposition proceedings nor contained in the Opponent's grounds of appeal, and should therefore not be taken into consideration.

18. In any case, it was clear from the claim wording that the positions were calculated in the high resolution image. This was also in accordance with the description of the opposed patent at paragraphs 36 and 37 wherein a difference was made between the detection of the speci­men areas in the reduced thumbnail image and the cal­cu­lation of the position of the specimen areas in the slide, i.e. high resolution image data (letter of 24 June 2020, section 8).

19. By contrast, it was clear that in D1 all processing took place in the overview image/low resolution data (Block 1 on page 1172). The JPEG compression was per­formed before the overview image was extracted, so it did not define the working resolution. There was no calculation of the position of the specimen areas in the high resolution data.

20. D1 did not disclose features 4.2 or 4.3 either. These features defined the detection of specimen areas of the same shape, the areas including the individual sections.

20.1 In D1, the detected cores did not have the same shape. Specifically, they were not circles, but had irregular contours. The Hough Transform detected cores which were not truly circular but which could be approximated by a circle (letter of 24 January 2023, 2.2). There was a difference between the sections and the specimen areas which included them. The sections could have any shape, but the specimen areas were all of the same shape: rectangular in the patent (figure 13 and paragraph 36). The circles depicted in the thumbnail image of figure 4d were there only for "highlighting", as D1 expressly stated, and they did not represent specimen areas or the outcome of the Hough Transform. Anyway, they were not all circular, as the small image in figure 4d was rather an ellipsis.

20.2 The claim required detection of specimen areas having all the same, but arbitrary shape. D1 was restricted to circular shapes.

20.3 Furthermore, in D1, the positions of the sections were computed, as the centres of gravity (centroids), before the Hough Transform was applied, so they were not the positions of the circles computed by the Hough Trans­form. As stated by D1, it was the centroids that con­sti­tuted the output of Block 2, not the centers of the circles. There was thus no calculation of the positions of the specimen areas, but only of the cores, i.e. sections (letter of 24 June 2020, section 9).

20.4 Even if the detected cores were considered specimen areas as claimed, feature 4.3 required that all sections be included in the specimen areas ("the" individual sections). D1 detected sections, but not all of them, as could be seen from figure 4 (see e.g. letter of 24 January 2023, section 2.3).

21. There was also no reason to assume that in D1 the coordinates of the found sections were stored. D1 could use them "on the fly" to obtain the logical grid, which was its purpose. Even if the skilled person needed the coordinates of some of the high resolution images there were multiple ways of obtaining them, e.g. by using the logical coordinates, or by calculating from the low resolution coordinates. Feature 5 was thus neither disclosed nor rendered obvious by D1.

22. Regarding feature 6, D1 was oriented towards high-throughput automatic analysis (see abstract), and not to analysis of individual cores. There was thus no reason to display individual cores and there was also no motivation to provide a controller able to switch between the display of the different cores. It was not clear that the inserts in the various figures showed the high-resolution image. The reference to individual patients was not to be equated to one to individual cores.

22.1 But even if a switching between individual cores for display purposes was considered implied or obvious, the controller did not need the positions in the high resolution image, it was possible to switch between images using the logical coordinates.

22.2 Feature 6 was thus neither disclosed nor rendered obvious by D1.

The Board's opinion

Claim interpretation: admittance issues

23. The ground for opposition based on lack of novelty and inventive step over D1, alone or in view of common know­ledge, are part of the appealed decision, and the discussion about the correct interpretation of feature 4.1 is reported in the minutes of the oral proceedings before the Opposition Division (bottom of page 2). In view of Article 12(2) RPBA 2020, they hence do not con­stitute amendments, admittance of which would be within the Board's discretion under Article 12(4) RPBA 2020.

23.1 In its appeal against the decision to maintain the patent in amended form, the Opponent did not have to provide its case against the patent as granted. It was timely for the Opponent to present it only in response to the Proprietor's appeal; see Article 12(3) RPBA 2020. The Board thus has no discretion not to admit the corresponding submissions under Article 12(5) RPBA 2020.

Claim interpretation: feature 4.1

24. Even though the claim wording, on its own, might also allow the interpretation proposed by the Opponent (see point 12 above), in the sense that a calculation of the positions in the reduced image may be enough to "in­dicate" the positions in the original image, the Board is convinced that it is not the one that the skilled person, having the description in mind, would adopt.

24.1 The opposed patent is concerned with the positions of the specimen areas in the slide image, for display purposes (see figure 15). The same language used in the claim ("indicating") is also used in paragraph 10 where no "reduced" image is generated.

24.2 The reduced image is introduced to improve the effi­cien­cy of the detection process (paragraph 12). The detection of the specimen area then takes place in the reduced image (paragraph 36), but paragraph 37 makes clear that the position information is effectively a set of coordinates in "a coordinate space" (having an origin and two axial directions) "of the slide image", which slide image is not the thumbnail image (see paragraph 34).

24.3 Thus the Board interprets this feature to refer to a calculation of the position of the specimen areas in a coordinate space of the original, non-reduced, image, on the basis of the positions provided by the detection in the reduced image, and not to a provision of the position of the specimen areas in the reduced image.

The Board's understanding of D1

25. Document D1 teaches a method of automatic gridding for tissue microarrays (TMAs) slides to reduce errors in automatic image analysis. Because the arrays are manu­ally prepared, there are frequent TMA layout issues comprising "low staining, debris, core displacement, nonuniform background, missing cores, and rotated sub­arrays" (abstract). Gridding is "assigning each tissue core to its proper position on the microarray grid", so that the "intensity levels of the protein staining can be analyzed for each individual core by the means of image processing" (page 1170, left, below figure 1).

25.1 The acquired TMA slide is first compressed by JPEG compression (page 1171, right, bottom). The method then carried out comprises 4 blocks. In preprocessing (block 1), a low resolution image is obtained from the "virtual slides". Block 2 comprises multiple stages. In blocks 2.1 and 2.2 the overview image is cleaned and binarized by a sequence of operations to obtain segmented objects. For these objects, in stage 2.1, the centers of gravity (centroids) are computed. D1 states (end of block 2):

"The following stage (2.3) decides which of the segmented objects are cores. Objects with a circular shape are classified as TMA tissue cores. A standard Hough-Transformation hereby determines the roundness of the objects. Outputs of the stage are the x and y coordinates of the center of gravity of each object detected as core. Figure 4d shows an exemplary resulting image with the detected cores highlighted".

The legend of figure 4 states:

"(d) image showing the objects which were recognized as circular cores (bordered with a circle)".

25.2 Thus, at the last stage, the cores are identified by the Hough Transforms. All objects not recognized as circular are not considered to be cores. For all the circular objects, i.e. presumed cores, a corresponding circular area is identified and displayed in figure 4d, and the corresponding circle "borders", i.e. includes, the presumed core. The method also outputs the calculated centroids of the presumed cores.

25.3 These positions are used in block 3 to perform the logical assignment to a grid position in each subarray, by matching with a grid template. Unique numbers are then assigned depending on the subarray position in the whole TMA (block 4).

Novelty and inventive step

26. On the basis of the above, the Board is of the following opinion:

(a) Feature 4.1 is not disclosed by D1, which only detects and calculates coordinates of objects or cores in the overview image, i.e. not in the original image space.

(b) Features 4.2 and 4.3 are disclosed in D1 by the circular areas of figure 4d). Though D1 does not mention it explicitly, the only way these circles can be obtained in D1 is on the basis of the Hough Transform. These areas have all the same shape and include all (relevant) sections, i.e. all sections detected as cores, not all sections actually present on the slide. The claimed method cannot do more either. That the shape of the cores themselves (sections) is irregular is of no importance. The position of the (circular) specimen areas is "indicated" by the centroids of the sections included in the circular areas.

(c) Feature 5 is not disclosed by D1. Block 3 uses the centroids positions, so they must be "stored", i.e. written to memory, at least temporarily. Moreover, as the Proprietor indicated, the centroids were computed in stage 2.1, before the Hough Transform stage 2.3, so they must have been stored waiting for stage 2.2 and 2.3 to execute. However, the stored positions are not in the original image space, but in the overview "reduced" image space.

(d) Feature 6 is not disclosed by D1. While D1 shows inserts containing individual cores, it is not clear that those are of high resolution images, nor is it clear that a controller to switch display between the specimen areas is actually implemented in D1.

27. Claim 1 is therefore new.

28. However, the Board is of the opinion that the skilled person has an obvious interest in being able to view individual cores, as the opponent submitted in its reply to the statement of grounds (2.2.1). This may be either for testing purposes, which per se are consi­dered obvious, but which might also be suggested by the thumbnails in figure 4 of D1, or, as the Opponent argued, by the desire to study individual patients' ca­ses. This does not contradict the fact that the me­thod of D1 is generally aimed at high-throughput au­to­matic image analysis; the option of viewing indi­vi­dual cores is just an additional option. The images of interest are the original high-resolution ones, which contain most information. If they were not used, there would be no point in acquiring the image at high resolution.

28.1 To enable the individual viewing of cores, the skilled person has to implement a controller that is able to switch display between the different individual core images, identified in D1 as circular areas. The straightforward way of selecting them for viewing is based on their position in the acquired TMA image (feature 6), which must therefore be calculated (feature 4.1) - using the known correspondence between the low and high resolution images being obvious to do that - and stored (feature 5) so as to enable access by the controller.

28.2 The Board notes that even if the cores may be addressed by their logical position, as the Proprietor submitted, the displaying of images requires an association of that logical position with image coordinates. The Board also accepts that there may be other ways of displaying the images without (permanently) storing their image coordinates. However, this does not change the fact that displaying them based on the image coordinates is obvious.

28.3 As a side note, the Board considers that, even if a display of individual cores is not considered, features 4.1 and 5 are obvious in view of any future use, such as automatic analysis. That is because gridding must in the end serve to associate the high resolution core images with their logical coordinates.

29. In conclusion, claim 1 lacks inventive step in view of common general knowledge of the skilled person implementing the method of D1.

Patent as maintained

Article 123(2) EPC

30. Note: All the passages cited in this section refer to the original application documents.

The parties' arguments

31. The Opponent (grounds of appeal, section 2) was of the opinion that the inclusion of features 7 and 8 added matter by way of intermediate generalization. In the original claims, they were recited in combination with claims 2 to 5 and further with other features of claim 6. In item 6 of the list of "items" at the end of the description (paragraphs 122 to 130), they were combined with the remaining features of originally filed claim 6. In the remainder of the description they are described as part of the alignment unit, itself part of the display unit, which also included other features. The computation (and storage) of the offset was not disclosed in isolation, it was always associated with its use for displaying, be it "jump" or "synchronous" (pa­ragraph 88 of the original description), and so it could not be claimed independently.

32. The Proprietor was of the opinion (reply to appeal, section II.1) that a basis for the current claim could be found in the combination of items 1, 2 and 6 (paragraphs 122, 123, 127 as filed). The skilled person would realise that the features are not inextricably linked to the remainder of the features in item 6 related to the "jumping display". For instance, the offset was also used in an embodiment related to a different type of display (the synchronous display - paragraph 88). The effect of the offset computation is that "the display area to be used can be set" (reply of 21 February 2020, 1.2 3), with reference to paragraph 19), and this did not require the features of the jumping display embodiment.

33. The Proprietor also referred to paragraphs 8, 10, and in particular 18, as basis. This latter paragraph states:

"[18] The information processing system may further include: an alignment unit to detect feature points of the plurality of specimen areas and to calculate an offset amount between coordinates of the feature points in a coordinate space of the specimen area; and a second storage unit to store the calculated offset amount, in which the controller may calculate, when the controller receives, from a user, an instruction for jumping the display area to another specimen area, the display area at a jumping destination based on the stored position information and the stored offset amount such that positions of the feature points correspond to each other in the display area before and after jumping."

The Proprietor submitted that in this paragraph, "alone from the grammar" (reply of 21 February 2020, section 1.2.2), the use of the offset by the controller was clear­ly optional, as it was introduced by a "may": "in which the controller may calculate". This provided ba­sis for calculating and storing the offset indepen­dent­ly of the controller. The skilled person would under­stand that the offset can be used for other purposes that the jump display of paragraph 18, for instance automatic image analysis.

34. In response, during the oral proceedings, the Opponent submitted that the term "may" does not always introduce purely optional features. "May" has multiple meanings, and in this particular case it should rather be read in a temporal sense, that the offset is stored so that it "may" be used when needed, i.e. it is available for later use by the controller. This did not render the controller optional.

The Board's opinion

35. The Board agrees with the Proprietor that, conceptu­ally, the computation and storage of an offset between two specimen areas is a step that can be executed by itself, in the sense that the way the computation is performed is not dependent on its use for displaying purposes. However, this is not decisive. What is decisive is whether the application as originally filed discloses, directly and unambiguously, such computation and storing by itself, and not as a sub-step in a display method.

35.1 The original claims and the items define the offset computation and storage only in combination with display steps. This is also the case for the rest of the description, with the possible exception of paragraph 18, using the term "may".

35.2 The Board is not convinced by the arguments of the Proprietor in this respect. The skilled person reads a patent application or specification in order to extract the relevant technical information, and does not limit him- or herself to linguistic considerations. Paragraph 18 provides two technical relevant bits of information: that an offset is computed and stored and that a dis­play controller uses the offset to perform a jumping display operation. Paragraph 19 states then that the offset allows for a display of two specimen areas in corresponding positions. In the Board's view, the tech­nical information that the skilled person will unambi­guously extract from these two adjacent paragraphs is that the offset computation plays an essential role in the display method and that it is for this purpose that it is computed. This corresponds to the "temporal" in­terpretation according to the Opponent. Hence, although the offset can be computed independently of its pur­pose, the application discloses that the offset is com­pu­ted for its use by the display controller.

35.3 The skilled person will also understand from other parts of the description that the display method, while still using the offset, may be different (synchronous, paragraph 88). But even if this allows a choice, the disclosure still implies a combination of the offset computation and storage with one of the two display methods.

36. The Board concludes that the subject matter of claim 1 of the patent as maintained extends beyond the content of the application as originally filed.

37. Therefore, the decision under appeal must be set aside.

Remittal

38. The auxiliary requests on file have been presented, but not discussed in the first instance proceedings. The Opponent has, for all these requests, raised a number of objections related to Article 123(2) and Article 56 EPC, and also filed new documents which may be pertinent to these requests, and which may require a decision on admittance or an analysis in substance. The Board finds appropriate in this case to remit the case to the Opposition Division for further prosecution (Article 111(1) EPC, Article 11 RPBA 2020).