#499 provide better support for link checking (Emacs 29)

[Mike Kupfer]

In Emacs 29, Gnus/shr includes code to verify that the URL in a link matches the link text. When they don't match, the link is given a red background, and a caution emoji is added. If you mouse over the caution emoji, you get a message in the echo area about the mismatch. You can see the link URL by putting point somewhere in the link and pressing w. (This was tested with changeset 4ae315f7c3; I don't remember exactly when this functionality was added.)

MH-E/shr displays the red background and caution emoji, but the mouse-over functionality is missing, and typing w just gives you an error message about the buffer being read-only. We should at least add the mouse-over functionality, though I'd actually like to see both features supported. And we should double-check what Gnus does in tty mode in this situation, and make sure MH-E has equivalent usability.

[Mike Kupfer]

Here's a screenshot from Gnus, with the mouse over the warning emoji.

[Mike Kupfer]

The fontification of the suspicious link and the assignment of the mouse-over text to the warning emoji are done in #'shr-tag-a:

    (when url
      (shr-urlify (or shr-start start) (shr-expand-url url) title)
      ;; Check whether the URL is suspicious.
      (when-let ((warning (or (textsec-suspicious-p
                               (shr-expand-url url) 'url)
                              (textsec-suspicious-p
                               (cons (shr-expand-url url)
                                     (buffer-substring (or shr-start start)
                                                       (point)))
                               'link))))
        (add-text-properties (or shr-start start) (point)
                             (list 'face '(shr-link textsec-suspicious)))
        (insert (propertize "⚠️" 'help-echo warning))))))

Tracing through #'mh-display-msg shows that the mouse-over text is clobbered by #'mh-show-addr, which is just a wrapper over #'goto-address. And it looks like #'goto-address is responsible for losing the ability to use w to get the link text.

When shr is used, the link is already highlighted and active (the user can click on it) before #'mh-show-addr is called. I would suggest not using #'mh-show-addr in combination with shr, but #'mh-show-addr still adds some value by fontifying email addresses and turning them into mailto links. Maybe the email linkification should be done by shr? Or maybe the email linkification should be factored out, so that it can be done separately from the URL linkification?

[Mike Kupfer]

Or maybe goto-address should leave text alone if it's already been propertized?

[Mike Kupfer]

Raising the priority, as this is a new issue in Emacs 29 that we really ought to fix before Emacs 29 goes out the door.

[Bill Wohler]

You thought that "goto-address should leave text alone if it's already been propertized." And Eli seems to agree, but it seems that it might be risky at this late date. I might tend to agree.

In your initial analysis, you suggested not using mh-show-addr if shr was in use. This seems much safer. You mention that shr allows one to click on the email address, so perhaps not too much is lost. Indeed, MH-E defers to other packages in most cases, so perhaps mh-show-addr should only be used as a last resort.

[Bill Wohler]

Guessing that this finally annoyed you for the last time. Good luck!

What did you think of my thoughts about deferring as much as possible to Gnus?

[Mike Kupfer]

I've actually been poking at the mouse-over problem for several weeks, but I had forgotten to assign the bug to me.

The problem with the warning emoji not getting a tooltip is that the emoji was getting included in the match for goto-address-url-regexp. If I moved the emoji to come before the link text, it did not get included in the regexp match, and the mouse-over functionality worked the same as in Gnus. (This left the problem with "w" unaddressed.) I'm attaching a copy of the draft patch for the record.

This prompted me to create some test cases for other types of links, like mailto links, links where the text is just an image, links inside <b></b>, etc. While checking my test cases against a relatively recent version of master, I noticed that the original failure with mouse-over is no longer reproducible. I don't know what's changed since I originally filed this bug, and I don't plan to track it down.

There's still some work to do here (more on that in a new bug update), but I'm going to move the priority back to 5, since this is now an old bug and it doesn't appear to be a priority for any users.

[Mike Kupfer]

By the way, I noticed that shr uses text properties for things like changing the color, whereas goto-address creates an overlay. Overlays can be prioritized, so I thought about changing shr to use an overlay instead of just setting text properties, and giving the shr overlay a higher priority than the goto-address overlay. But looking at the history for shr, I saw that shr had used overlays at some time in the past. They weren't performant, so Lars changed the code to use text properties.

[Mike Kupfer]

I think that leaves the following issues:

1. "w" doesn't work (as described in the original bug report)
2. the shr textsec code does not flag mailto links where the link text does not match the link address (discovered during testing)
3. goto-address still messes with font settings that were made in the HTML.
4. In some cases involving image links, MH-E will display whitespace where Gnus displays the image alt text, effectively disappearing the link (discovered during testing)

Issue #2 isn't an MH-E problem. I don't know if it's deliberate or an oversight. Since the mailto link just brings up a mail composition window (in whatever application Emacs is configured to use), and doesn't actually send an email, it could be that the user is expected to notice who the mail will actually go to before clicking on Send.

Issue #3 is a relatively minor problem, I think. But it could be confusing if the HTML has text like "email Product Support at the address listed above in bold", and there's no bold text because goto-address changed it.

I think issues #1, #3, and #4 all go away if MH-E doesn't use goto-address when shr is the renderer. So yes, I like the idea of deferring to shr.

Since that would be an incompatible change, I'm inclined to add a user option in case someone wants to continue using goto-address. I'm still pondering what exactly the choices should be. One possible slate is

• always use goto-address
• never use goto-address
• use goto-address unless the renderer is shr

This could be generalized to

• t: use goto-address
• nil: don't use goto-address
• Lisp list: evaluate and use goto-address if it evaluates to non-nil

For either setup, I'd be inclined to make the third setting the default. For the Lisp list, the default sexp would evaluate to nil for shr and non-nil for everything else.

What do folks think?

[Bill Wohler]

Since neither mh-show-addr nor goto-addr are documented in the manual, this change seems to be an implementation detail and therefore not an incompatible change. I'd be inclined to simply avoid using goto-address when shr is the renderer unconditionally unless such an option is requested.

Ah, I found this in the mh-show-mode docstring and manual:

Email addresses and URLs in the message are highlighted if the option goto-address-highlight-p is on, which it is by default. To view the web page for a highlighted URL or to send a message using a highlighted email address, use mouse-2 or C-c RET (goto-address-at-point).

If your change would mean that this statement would be come true only if shr was not in play, we could update this documentation and provide the shr equivalents. That would still be less work than introducing a new option that may not see any use.

[Mike Kupfer]

I'm okay with making the change not to use goto-address with shr, and then add a user option if we get complaints from users. shr already supports using mouse-2 and C-c RET to follow a link. So I think the only user-visible change will be that text strings that look like a URL or email address will not get linkified. Hopefully that's minor enough that nobody will demand an option.

I think it would be cleaner if the mh-show-mode docstring does not get into the user-experience details for the different rendering engines. Instead, I'd like for the docstring to note that MH-E can use mhl or Emacs MIME for rendering emails and that Emacs MIME supports multiple rendering methods for HTML; provide cross references to mm-text-html-renderer and mh-mhl-format-file; refer the reader to the MH-E Manual (maybe specific sections) and the Emacs MIME documentation for additional details.

For changes to the MH-E Manual, I'm thinking about tweaking Section 6.3 (HTML) to note that goto-address is not used with shr. And update Section 6.1 (Viewing Your Mail) to say something about goto-address not always being used, with a cross-reference to Section 6.3.

Oh, I've found at least one thing in the MH-E Manual that is incorrect: Section 6.1 (Viewing Your Mail) says

Note that ‘mhl’ is always used for printing and forwarding; in this case, the value of ‘mh-mhl-format-file’ is consulted if you have specified a format file.

That's not true for forwarding if mh-compose-forward-as-mime-flag is non-nil. I'll have to see what the scoop is for printing. I'll fix this and any other inaccuracies that I come across.

What do you think?