#338 abnormal pressure / bad strokes on Lenovo X1 Yoga AES pen

[Denis Auroux]

There is a calibration problem for Lenovo's new-ish tablet PCs with AES pens (in my case an X1 Yoga 2nd gen): the driver reports significant positive pressure when the pen is still approaching the screen and far from it if approaching fast enough.

With the default Threshold of 27 (or even worse if decreased), ButtonPress tends to happen when the pen is still about 5 mm from the screen; and at that time the parallax is considerable so the xy coordinates are about 5-8 mm off from the place where the stylus will actuall contact the screen. So the xy location of the button press events is completely inaccurate.

This is very apparent in xournal or gimp. Depending on the setting of RawSample, one gets either a spurious zigzag or a spurious dot before the actual stroke (connecting the xy location reported when the driver first thought contact occurred, a few mm away).
Increasing Threshold to 50 and drecreasing RawSample to 1 makes the problem less immediately visible, but still present on occasion when writing fast.

xinput/xsetwacom devices:
Wacom Co.,Ltd. Pen and multitouch sensor Finger touch id: 9 type: TOUCH
Wacom Co.,Ltd. Pen and multitouch sensor Pen stylus id: 10 type: STYLUS
Wacom Co.,Ltd. Pen and multitouch sensor Pen eraser id: 16 type: ERASER

lsusb:
Bus 001 Device 005: ID 056a:50b8 Wacom Co., Ltd

xorg-x11-drv-wacom-0.34.2-1.fc25.x86_64

To reproduce: get a Lenovo tablet with AES pen, install gimp or xournal; draw quick long diagonal lines (or anything that will cause you to bring the stylus towards the screen quickly and at an angle); many of the lines drawn on screen will have a "hook" at their start.

Is there any way to fix this, or is there an inherent technological issue with the AES pens?
Thanks in advance!
Denis

[Denis Auroux]

With Threshold=400 and RawSample=1, the spurious portions of strokes are rare enough and short enough that handwriting in xournal remains legible, but they remain present; and some strokes are missed (pressure doesn't reach the threshold) + the xy coordinates are noisy (due to no averaging). If I decrease threshold or increase rawsample then the spurious portions of strokes become too much of a problem.

Not sure why rawsample makes things qualitatively so much worse. Perhaps it is because, when button press occurs at the wrong xy coordinates, the N following samples all include these incorrect xy coordinates in their averaging ?

I am surprised though -- with Threshold=400 I am unable to trigger a ButtonPress event without hitting the screen, but there are still a fair number of ButtonPress events that seem to get reported at the wrong xy location (from when the pen was still up in the air, or in any case not where the xy coordinates end up being once the pen is firmly in contact with the screen).

[Jason Gerecke]

Wacom devices use an actual pressure sensor inside the stylus to detect when you're touching the screen. Occasionally (almost always with old "worn out" pens) the sensor can report non-zero values when the pen is not touching the screen, but even then it should be possible to determine a Threshold value that works correctly. Your tablet isn't old enough for me to suspect a "worn out" pen, but there could potentially be an issue with the driver.

Please see the instructions at https://bentiss.github.io/hid-replay-docs/ for how to install the "hid-recorder" tool. Additionally, please also install the evemu and xorg-x11-server-utils packages with yum. Finally, download our capture.sh script (gist.githubusercontent.com).

Once all the required utilities are on your system, open up GIMP. Next, open a terminal and run sudo /path/to/capture.sh. It will ask to to select a device -- choose your pen. Next, make a few strokes in GIMP, noting which ones show a problem (e.g. the 3rd, 4th, and 5th strokes). Finally, switch back to the terminal and hit CTRL+C to stop the capture. Combine the three log files that are listed in the terminal output into a .tar.gz and upload the file for review along with your notes.

[Denis Auroux]

Capture files and gimp output attached: a few diagonal strokes (with erroneous "check marks" and/or sporadic dots at the beginning), then dots generated just by waving the pen above the screen.

(Sorry for the translated x coordinates, I have an external monitor connected to the left of the laptop screen so the x valuator maps tablet range [0,30935] to rescaled [13257,30935]; y is not rescaled).

[Jason Gerecke]

Thanks for the quick reply :) From a cursory overview, it looks like the sporadic clicks are occuring when the sensor sends its first "MSC_SERIAL" event after coming into proximity. These events shouldn't trigger clicks on their own, but the change from an unknown serial number could easily be tickling something within the X driver.

The checkmark artifacts might be explained by the sensor's accuracy being poor at large distances. You don't normally notice this, but since the driver is sending a click event while the pen is still far away you see the effect of the offset decreasing as the pen comes closer and eventually touches the display.

We'll do some digging to determine why these MSC_SERIAL events are causing sporadic clicks...

[Nico Schottelius]

Hello Jason,

it seems my last post did not make it here (probably connection issues) - I was wondering if I can contribute any measurements / debug information to this problem?

I do observe that the problem is not coherent - sometimes the stylus works normal, without jumping / adding hooks, but it seems to intensify over time (though this might be subjective)

Best,

Nico

[Jason Gerecke]

My apologies, Nico. I've been caught up in a wave of bugs (lots of GNOME ones, in particular) and haven't had a moment to look at this quite yet. The logs provided by Denis should be sufficient for me to start in-depth debugging once I get the time to do so. I'll let you know if I do end up needing more information though :)

[Nico Schottelius]

As my stylus is currently unavailable, I am using the Bamboo at the moment and it seems that after suspending similar issues arise (jumping / erasing prior to touching the tablet).

This just as a hint - maybe this is related.

[Nico Schottelius]

Oh, and here's the xinput + lsub output for it, just in case:

[11:18] manager:~% xinput list
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ Wacom HID 484C Pen stylus id=14 [slave pointer (2)]
⎜ ↳ Wacom HID 484C Finger touch id=15 [slave pointer (2)]
⎜ ↳ Wacom HID 484C Pen eraser id=22 [slave pointer (2)]
⎜ ↳ Alps Alps Touchpad Touchpad id=10 [slave pointer (2)]
⎜ ↳ Wacom Bamboo 16FG 4x5 Pen stylus id=11 [slave pointer (2)]
⎜ ↳ Wacom Bamboo 16FG 4x5 Pen eraser id=12 [slave pointer (2)]
⎜ ↳ Wacom Bamboo 16FG 4x5 Pad pad id=13 [slave pointer (2)]
⎜ ↳ Wacom Bamboo 16FG 4x5 Finger touch id=21 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ Power Button id=6 [slave keyboard (3)]
↳ Video Bus id=7 [slave keyboard (3)]
↳ Power Button id=8 [slave keyboard (3)]
↳ Sleep Button id=9 [slave keyboard (3)]
↳ Intel Virtual Button driver id=16 [slave keyboard (3)]
↳ Intel HID events id=17 [slave keyboard (3)]
↳ Intel HID 5 button array id=18 [slave keyboard (3)]
↳ Dell WMI hotkeys id=19 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=20 [slave keyboard (3)]
[11:18] manager:~% lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 014: ID 056a:00de Wacom Co., Ltd CTH-470 [Bamboo Fun Pen & Touch]
Bus 001 Device 003: ID 8087:0a2b Intel Corp.
Bus 001 Device 011: ID 044e:1218 Alps Electric Co., Ltd
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
[11:18] manager:~%

[Jason Gerecke]

I've repeatedly replayed the logs for about an hour to try and reproduce this issue locally, but for some reason I don't get any of the characteristic "hooks" in the GIMP output. This is with xf86-input-wacom 0.35.0, but there don't appear to be any relevant driver changes from version 0.34.2 that Denis is using.

No luck finding a MSC_SERIAL culprit yet, and especially odd that Nico reports his Bamboo performs similarly. That device uses EMR tech rather than AES and is much more stable. It also doesn't report a serial number which casts some doubt as to whether MSC_SERIAL is really the cause or something else.

[Nico Schottelius]

Jason,

I have to say that there is a huge difference in number of occurences: with the bamboo it happens maybe once every couple of minutes (let's say 5-30), while with the active pen it happens every few seconds. So with the bamboo I have to cleanup the dots or incorrect strokes rarely, while the the stylus it is sometimes hard remove the dots, as they occur during some setting almost every time I near the pen the display.

That said, I am still waiting for a stylus replacement (ETA 18.09!) and will be able to test then again with it. If I can do anything else in the meantime, let me know!

[Jason Gerecke]

Nico, would you mind trying the patch described in bug 339? I don't suspect it will significantly impact this bug, but it does make a change to how the driver handles the transition between a missing and present MSC_SERIAL.

I've been doing some more work and am now able to see the hooks, but no spurious dots yet. It looks like the hooks, at least, are a result of the poor accuracy-at-a-distance of AES pens combined with the averaging done within the driver. If I set RawSample to "1" as suggested by Denis (which disables averaging) the hooks disappear. Since disabling averaging can lead to noisy lines, perhaps we could "reset" the averaging once the pen contacts the screen. Another option would be to use a weighted average to reduce the influence of "old" points or switch from using a proper predictor instead of just averaging.

[Denis Auroux]

I agree that the patch for #339 seems to fix the problem of the spurious dots, at first glance at least (I've only tried it for a few minutes so far) -- apparently no more buttonpress events when the pen is still hovering. The hooks are still there; even if RawSample is 1, when drawing fast I see some inaccuracy (but it's much less problematic). If I have time I will look into the code and try something quick and dirty about this, at least for my own purposes.

Denis

[Denis Auroux]

Besides the patch for #339 to avoid spurious button events, I'd like to suggest something along the lines of:

--- wcmFilter.c 2017-08-31 11:10:16.764050053 -0700
+++ /home/auroux/DL/xf86-input-wacom/src/wcmFilter.c    2017-08-31 11:00:51.793441138 -0700
@@ -288,6 +288,10 @@

    DBG(10, common, "common->wcmRawSample = %d \n", common->wcmRawSample);


+        // DA: attempt: disable averaging when pressure goes from zero to nonzero
+        if ((ds->device_type == STYLUS_ID || ds->device_type == ERASER_ID) && pChannel->valid.states[0].pressure==0) 
+           wcmResetSampleCounter(pChannel);
+
    storeRawSample(common, pChannel, ds);

    state = &pChannel->rawFilter;

The correct test might be something other than previous pressure being 0, maybe rather the pressure being below some threshold or the pressure button (#1) being marked off in the previous valid state. Also I have no idea whether testing for stylus or eraser is necessary.

Only resetting the sample counter before storing the raw sample when ds->pressure == 0 does not work, because if one does that the first nonzero pressure state still gets averaged with the last zero pressure state, and on my laptop at least the last zero pressure state can have xy coordinates that are very far off -- strongly biased towards the location where the previous stroke ended (this is not averaging at work, I'm looking at the raw state).

I suspect something worse than parallax at this point, or perhaps the pen just doesn't sample often enough when it's hovering between the two strokes. Either way, the reported pen position while hovering seems to not only be affected by parallax but also lag very much behind the actual hand motion, and sometimes the reported position when hitting the screen still lags a little bit though not nearly as dramatically.

By the way, these AES pens seem to me to have much worse local accuracy than the pens found in older Thinkpads -- fast handwriting looks a lot sloppier, even after fixing these averaging issues. Either they sample much less often than the older generation tablet PC pens, perhaps particularly when the pen is hovering, or something is not working as advertised. Is there a setting that would affect how often a raw sample gets sent by the pen / tablet to the driver for processing?

[Jason Gerecke]

I made a sample patch myself, taking the alternate route of changing how averaging is performed. I completely forgot to post a note here though. Sorry! Its good to know that resetting works (and especially good to know about that gotcha you stumbled across) -- it might end up being the chosen path afterall since changing the averaging is more invasive.

I'd be interested in knowing how the modified averaging works for you: if you're still working from the git repository I had you clone, run git remote update; git reset --hard; git checkout experiment and then rebuild/reinstall the driver. The fix for bug 339 is included, and it adds prediction to the averaging code. In my testing, it does a much a better job of determining the correct value, regardless of how high or low RawSample is set. You could also try git checkout experiment2 for an even more extensive modification which also uses weighted instead of simple averaging in addition to prediction, though I'm not really sure if it actually performs any better than just the "experiment" branch...

The steady-state sampling rate of AES pens is pretty high (my Dell system is ~200Hz, compared to most EMR sensors at 133Hz). The hover height, though, is much poorer than EMR (don't have a ruler handy, but I can easily feel the difference). If you get too far away, the sensor has a hard time seeing the pen and ends up getting a "choppy" feeling. I wouldn't be surprised if the strength of the signal varies between units and pens, but without trying others out for comparison its hard to say if your unit is outside of that normal variation.

Unfortunately, as far as I'm aware there aren't any commands that we can send to the hardware that would affect its performance. The best we can do is improve the driver's coordinate filtering so that bad data isn't as noticable.

[Denis Auroux]

Hi Jason,

"experiment" wasn't too impressive on my system -- strokes felt noticeably less smooth than with standard averaging and there were still some noticeable 'tails' at the start of strokes. "experiment2" seems better to me, but there are still 'tails' because presumably the weighted average still includes some of the coordinates from before the pen hit the screen. To be honest, I think my simple-minded 'reset the queue if the pen wasn't on the screen' does the best job of cleaning up the starting portion of strokes for my purposes (but I realize it might be brutally doing the wrong thing for some other possible scenarios).

Need to get back to work, so for the time being I'll return to the standard driver (need touchscreen to be able to work reliably)... sorry!

Denis

[Jason Gerecke]

Thanks for testing that. Agreed that resetting is probably the way to go given your experiences. I'll try to find the best place to do the reset to minimize issues with other scenarios.

[Denis Auroux]

So... I think my real problem is that, when the pen is hovering above the screen, the reported position of the pen lags far behind the actual position, despite the refresh rate being of the order of at least 50 to 100 Hz while hovering -- the trajectory is followed fairly accurately but if the pen is moving quickly, the pointer can be quite a bit behind the motion, by a half-second or so; and pressing on the screen causes the position to be suddenly caught up in time. (So, with averaging, a lot of what happened above-screen a half-second or so before hitting the screen gets into the average). Which brings the question of why the lag when hovering.

See attached log from xinput test [device], showing xy coordinates -- this was with RawSample=1 so in principle no averaging, data should be equivalent to the raw xy values getting into the driver. I was repeatedly drawing strokes from lower-left to upper-right, with the pen returning back to lower-left very close above the screen in between. So a[0] increases smoothly from button press to button release, then keeps varying smoothly while the pen is off, but during the decreasing portion it's far behind the actual motion, and when the next button press happens the value of a[0] teleports by about 3000 or 4000 to catch up with the actual position. (If instead I hadn't pressed but just stayed hovering, it would eventually have caught up after a second or so).

Which brings the question: is there some averaging / prediction / ... taking place on the reported coordinates while hovering, independently of RawSample, ahead of the enqueuing/averaging code I've seen in wcmFilter.c ? and is it specific to AES vs. EMR? (or is it something introduced in the driver recently that I didn't notice because the distribution on my older laptops with EMR pens isn't as up to date?)

Sorry for the weird problem and the not-so-precise reports. I will ask another Yoga X1 owner if they have the same lag above-screen, just in case it's a hardware issue on mine.

Denis

[Hrvoje Zeba]

Hi,

I'm just chipping in. I have the same system (X1 2nd gen Yoga; 20JD) and the pen exibits the same behavior described in the first post. I can help out with the testing, if you guys need me to. Just a note... By thinking the pen was broken, I bought another one but the problem presisted. Then I installed windows just to see if it's a hardware issue and both pens work just fine there.

[Nico Schottelius]

Hello,

I also have access again to my pen, so if there is anything that I can help with testing, I can help, too.

Best,

Nico

[Jason Gerecke]

EMR and AES pens treated largely the same by our driver; the difference is mostly in decoding the reports sent by hardware. There's no averaging or prediction done in the kernel, and xf86-input-wacom should only use "RawSample" events in its own averaging. If you're seeing laggy behavior with RawSample=1, that strongly suggests some kind of averaging being done by the hardware itself. The EMR and AES devices I've used in the past don't behave like what you've described: the pointer reliably follows the pen with the same aount of latency regardless of if the pen is up or down and never "teleports" at the moment the pen touches the screen.

EDIT: Looks like I linked to the capture.sh script before... Were you experiencing variable lag at the time you recorded that capture?

[Jason Gerecke]

I've been able to temporarily get my hands on a 2nd-gen Lenovo X1 Yoga and see the lag that you're talking about. Indeed, the amount of lag changes whether the pen is touching the screen or not. I can make strokes at identical speed both hovering and dragging and the former has noticibly more pointer lag. Looking at the raw data from the tablet, I also notice the "teleport" effect that you've mentioned: the reported position can jump a significant distance the moment the pen comes into contact with the screen.

I'm not sure if there's anything that can be done inside the driver since we're being fed bad data... I'll do a little more thinking though...

[Denis Auroux]

Thanks for confirming yourself -- I was trying to look into recording
with your script + filming things, but busy life got in the way; it's
even better if you've witnessed it directly.

I can't tell completely for sure, and lack the expertise to record any
kind of usable data, but my impression is that the pointer lag while
hovering is also present in Windows, so it's probably indeed a hardware
issue and not a linux driver issue.

Do you happen to know if it's likely to be a general issue with AES,
versus a specific Lenovo issue? Should I think of the relevant part of
the hardware as being the pen or the screen? Are there other fairly
common laptops, or standalone Wacom tablets, that use the same AES
technology? (I'd buy a non-lenovo pen if I knew it would solve the issue...)

Denis

[Jason Gerecke]

Yeah, I also noticed what appeared to be some lag in Windows, though I wanted to double-check in Linux since I couldn't perform debugging under Windows and had no assurance about what the Windows driver was doing.

I'm not aware of this being a general issue with AES. Certainly yours is the first report we've heard of this kind of lag, and I haven't seen anything like it before. It's possible that the lag is specific to the hardware/firmware used on this particular device, or that other devices are also affected. It could be some kind of hardware issue (e.g. noisy electrical components), a firmware bug, some kind of tuning to reduce power consumption, etc. My own tests were performed with a Wacom-branded AES pen rather than the pen the Lenovo came with, so about the only thing that I can say with some degree of certainty is that its a sensor-side issue rather than a pen-side issue.

Most "active pen" devices I'm aware of use Wacom's AES tech -- we've seen reports from other Lenovo products as well as HP, Dell, and others. The notable outlier is Microsoft's Surface line of devices which use nTrig tech instead.

[Jason Gerecke]

While doing my testing I noticed that my fix-bug-339 branch wasn't resetting the filter when expected. It seems that the location I added my fix at didn't have the state of the tipswitch calculated yet (we don't use the tipswitch state from the kernel or a non-zero pressure since some worn-out pens may report a small pressure value even when not touching the screen). I've pushed an update to my github which moves this after our calculation of actual tipswitch state. Please give this update a test and verify that it works for you. If it does, I'll be sending the necessary fixes out to the mailinglist soon.

[Denis Auroux]

Yes, the new patch works properly! (unlike the previous one which didn't quite reset in the right way). I'm now using fix-bug-339 branch as is, and it seems to work fine. I'll report if I notice anything bad, but it seems that it should be good to go.

Thanks a lot for all your work on these issues. (And shame on Lenovo for the quality of their new digitizers, seems like there's something they still need to figure out).

Denis