improving heuristic reading order

[bertsky]

WIP, starting off with regressions from 0.5.0 and old issues (IndexError etc)

TODO:

• modify return_boxes_of_images_by_order_of_reading_new such that it becomes mildly recursive, in order to avoid cutting through regions: if (for some y slice) some columns have much higher peaks than others, then pick those first and search for new y splitters within the others

[bertsky]

Sorry for the force-push! I had accidentally rebased back to a2a06a8 (which now became cd35241).

Still have not addressed the big TODO (which is coming shortly), but found some more useful changes along the way:

• b2a79cc was clearly a bug
• 66a0e55 is an improvement: find_num_col should not enforce the overall num_col_classifier result for every part of the page, but rather (as in find_number_of_columns_in_document) only for the big parts, so there would still be room for sections with headings stretching multiple columns
• 19b2c3f improves on the previous change to elongate horizontal separators if that does not introduce any overlap: now this is done later, when the column split is already determined, so one can also try partial spans; it also covers headings now, but single-column separators/headings are exempt from this now

There are also lots of new plotting directives (commented out). I'll upload some explanatory images next.

[bertsky]

Done! Let me explain…

• 1a76ce1 is needed because sometimes contours centers in float were exactly "between" bboxes in integer, e.g. cy=400.3 where one box ends on 400 and the next starts on 401, preventing a match
• 95f7608 is merely cosmetic
• 4abc2ff turned out to be a much bigger change than I initially thought; using recursion now makes the box/column ordering much simpler (and thus easier to manage going forward); I'll illstrate how it works below
• 4475183 is a mixed bag of adapting various heuristics that control splitting each vertical strip into columns – the old rules often did not fit well;
  in particular, they were too rigorous in not allowing cuts near the left and right page margin or near each other; also, the minimum for gaps was defined solely based on the "thickness" of the text columns, but not on the "thinness" of the gaps, which (in the presence of noise regions) sometimes yielded suboptimal results
• 3c15c4f and 5a77800 is a follow-up to new attempt at #173 (valid polygons, faster deskewing, various fixes) #192, where the matching between original contours and deskewed contours had been improved;
  however, it turned out that
  • I had confused and with or within the negation (so it would not look for a match on both sides, but on either side)
  • the deskewing method (rotation_not_90_func or rotation_image_new) implicitly created a scaled (and not just rotated and translated) coordinate space, which I was unable to reproduce with the coordinate calculations in shapely, so the matching algorithm would have a hard time, esp. when joining or splitting
• 72d059f is a simplification for the aggregation of local box orderings for the writer, but also fixes a subtle bug which had caused left-right inversion within boxes

[bertsky]

Examples:

original	final RO boxes	final RO

For reference, I put the old version's final RO boxes result of the first example in the top row. There, notice that many regions were cut right through, which in turn gave the contour-box-matching algorithm a hard time, and made it necessary to have workarounds (like "assign the left-most box").

[bertsky]

@vahidrezanezhad thanks to your regression test, I was able to address remaining issues with another series of commits:

• ee59a68 fixes a severe typo I made in 0.6 – me bad!
• 38d9167 just now surfaced as problematic: it allowed holes instead of exteriors to serve as separator candidates (I noticed uncritical use of cv2.RETR_TREE throughout the code base, but have not checked elsewhere if this is actually problematic)
• 06cb9d1 likewise created spurious separators (preventing merges of hseps)
• 5c12b6a is only cosmetic
• a527d7a fixes various issues with the function responsible for merging horizontal separators (I never looked at this before)
• b71bb80 is only relevent for right2left
• 5abf0c1 improves on the robustness of the new facility of multi-column region contours introduced in bertsky@4abc2ff
• 84d1096 fixes the criterion for making boxes span multiple columns (min→max, as originally intended)
• 4dd40c5 allows vertical separators to help finding columns (in addition to the other 2 criteria), increasing robustness
• 5a3de3b makes use of this in both callers; it also fixes 2 previously undetected bugs for the case when a part of the page has too few columns
• adcea47 this is a rather radical change, but seems compelling in hindsight: IMHO it does not make sense to apply erosion of the text mask only for pages with 3 or more columns (because 2-column pages can be very tight as well, and the erosion_hurts / multiplier heuristics of the downward functions already expect a certain "spaciousness")
• 56e73bf this also a big change – we talked about it already, I find it very helpful (sacrificing a few percent in speed for gaining some in precision) for finding optimal column separators and reading order – in addition to all of the above

All of the regressions are gone, and I have not yet found any new ones (but I will search more intensively this time). Here are some diagnostic example of the above:

← This shows how a missed out horizontal separator merge caused suboptimal RO

← This shows how in certain cases (despite more or less correct deskewing, but in this case still without erosion) the gaps are just a little to weak to meet the grenze criterion. Hence the idea to utilise the foreground separators, which will make these gaps pass →

[bertsky]

@vahidrezanezhad the new recursive RO algorithm is described briefly here, and its implementation follows in just a few lines.

Regarding the question of whether we should (as a general rule) A: "finish top mothers first" (as required by the anno_1.jpg example) …

… or B: "recurse into next mothers first" (as illustrated by the 4310063.jpg example) …

Here is the change needed to make my implementation behave effectively as yours did in that circumstance:

--- a/src/eynollah/utils/__init__.py
+++ b/src/eynollah/utils/__init__.py
@@ -1881,7 +1881,7 @@ def return_boxes_of_images_by_order_of_reading_new(
                                       y_mid[nxt]])
                         # dbg_plt(boxes[-1], "recursive column %d:%d box [%d]" % (column, last, len(boxes)))
                         column = last
-                        if last == x_ending[nxt] and x_ending[nxt] <= x_ending[cur] and nxt in args:
+                        if last == x_ending[nxt] and x_ending[nxt] <= x_ending[cur] and x_starting[nxt] >= x_starting[cur] and nxt in args:
                             # child – recur
                             # print("recur", nxt, y_mid[nxt], "%d:%d" % (x_starting[nxt], x_ending[nxt]))
                             args.remove(nxt)

And this is what it would then looks like:

… instead of the current result:

So let's collect more data on that, so perhaps we can make a better decision.