Re: [Patientos-users] usability evaluation of PatientOS

[Greg C. <cau...@gm...>]

Hi Stephanie,

What a fantastic report!  It is a great feeling knowing someone has
taken the time to evaluate the system and perform usability testing.
I wish I had taken the full HCI course in university as there is no
substitute for that experience and usability is such an exciting field
of study.

I agree with all of your observations of course and you have given me
some fresh new ideas on how to improve the system (the perceived view
only access of the demographics made me laugh as it is obvious now
that the button to edit the demographics is not well placed).

Please thank your team for me.

thanks!

Greg



On 3/19/08, ace <ac...@ta...> wrote:
> Dear PatientOS Developers:
>
> I am a student at the University of Amsterdam in the Netherlands, in a
> medical informatics Master's program.  As one of our weekly projects we
> were to pick a medical application and perform a mini usability study on
> it. My group chose your project, PatientOS, to evaluate, because a couple
> of us are particularly interested in Open Source software.  We wanted to
> share our results with you, in the hope that you can use them to enhance
> the usability of your product.
>
> However, you should know that we are far from usability experts, so we
> probably did not catch all potential usability problems.  We took our
> techniques mostly from Dix's book, "Human Computer Interaction."
> Specifically, we performed a Neilson's heuristic evaluation, a cognitive
> walkthrough with one task (create a new patient and enter information
> about them), and a think-aloud session with the same task.  We only
> evaluated the Demo system, as we don't have permissions to install
> client/server software on the school's computers.  We used two of our
> classmates as our "users," one from our team and one who had not seen the
> system before to act as a "novice user."
>
> At first glance, the system looks pretty good.  Our novice user even said,
> "Oh, yeah, it reminds me of Outlook."  Since a lot of users are going to
> be familiar with those kind of calendaring systems, that's a good thing.
>
> As our users tried to perform tasks, though, one problem kept coming up in
> various forms:  the users either did not get feedback from the system,
> didn't see the feedback when it was given, or didn't understand the
> feedback when they saw it.  I'll give you some examples:
> - Neither of our users could tell whether an appointment slot was selected
> or not in the Schedule.
> - Both of our users took at least a minute to notice the error messages at
> the bottom of the Patient Registration screen.  Even after they noticed
> them, they continued to miss new error messages that appeared there.  To
> prevent users from missing feedback messages, you could move these closer
> to the user's field of view:  for example, an error message about an
> incorrect date could appear somewhere near the date field.
> - Neither user understood that the "disappearing" date meant that they had
> entered the date incorrectly.  Both users entered the same date twice
> before they understood that the system was indicating an error.
> - After the patient info has been correctly entered and the user clicks
> OK, the system returns the user to the scheduling screen without
> indicating that the record has been saved.  Both of our users weren't sure
> if they had completed the task correctly.  The newly-created patient is
> not "visible" for the user to interact with further:  for example, create
> an appointment for the patient, start a medical record, order tests, etc.
>
> All of these individual things are symptoms of the same usability problem:
> the system does not provide enough feedback to make it easy for users to
> understand what they have done, what the system is doing, or what they
> need to do next.
>
> One of the users tried to click OK too early, before completing the
> patient info required by our task.  He had completed enough information
> for the OK button to work, though.  After he clicked OK, the user realized
> he needed to enter more info about the same patient but found it literally
> impossible to get back into the record and finish entering the
> information.  He tried using the "search" function which allowed him to
> view it but not edit it, which frustrated him enough that he pounded on
> the keyboard with his fist.  He ended up clicking "new patient" again and
> re-entering the information, and in fact he did this several times before
> finishing our tasks, resulting in several duplicate records.
>
> Right now, it seems like the system isn't very forgiving:  it's not easy
> to undo mistakes or back up if you get lost.  Adding an "undo" function
> such as ctrl-z could be useful, but for our particular user's problem, the
> user will need to see where their record "went" and how to get back into
> it.  Another approach would be changing the "OK" button to say something
> like "Finished" or "Done" to make those kind of mistakes less likely.
>
> To summarize, the big problems noticed in our usability analysis were that
> the system has problems with providing feedback to the user, and that the
> system does not (yet) provide a good mechanism for recovery after an
> error.  Fixing those things is (we think) going to be very important for
> improving the usability of the system.
>
> There are some other things that we noticed that were not specifically
> problems, but that we felt we could possibly offer some suggestions for
> improvement:
>
> One thing we talked about was the choice of items to place on the front
> screen versus things which are in menus (in the default configuration).
> We feel the best approach is to think about your target groups for this
> software.  Having the schedule be the first thing you see might be great
> for the Receptionist role, but when the Doctor logs into the system,
> what's the first thing he needs to see?   You want the things that need to
> be done fast to be easy, and you want the things that need to be done
> commonly to be easy.  A thorough requirements analysis might help you to
> tune your software to each user group's information needs and normal
> workflow.
>
> As an example, I practiced veterinary medicine for six years before coming
> to school here at the UvA.  The most common thing I wanted to do was to
> pull up a patient record.  I am terrible at remembering people's names, so
> I preferred to search by other cues - diagnosis, approximate date of last
> visit, and memorable characteristics such as approximate age and (in the
> case of animals) breed and species.  One of the things I noticed in your
> software is that it's a bit of an exercise to add an "emergency" patient,
> which is the kind of function that needs to be done quickly. If you have
> some medical professionals that you can to talk with about what kind of
> functions need to be done commonly and what kind of things need to be done
> quickly, I think you will be able to tailor an interface that is targeted
> specifically at your users' needs.
>
> Another thing we discussed is consistency within the system.  For example,
> the system handles errors in a couple of different ways (sometimes it
> turns the field red, sometimes it doesn't).  Consistency helps users know
> what to expect from the system and makes them feel comfortable using it.
>
> There are a handful of minor "bugs" and glitches that we noticed as well.
> These are the ones I didn't see included in your list of known bugs, my
> apologies if they are there and I missed them.
> - The system for correcting the date doesn't work well.  If the user
> enters the birth date "12-12-27" the system fills in the date "12-12-2027"
> which is probably not what the user meant.  It accepts dates such as
> "12-12-1066" and even converts the display in the Date field to say
> "12-12-66," which might cause the user to miss the error, since the full
> date and age appear at the top of the screen and thus out of the user's
> immediate field of vision.
> - If the user enters "garbage" into the SSN field, the system will attempt
> to make a valid SSN number out of it.  If the garbage happens to include 9
> digits it will not give any indication that there was an error.  To see an
> example of this enter something like "a3w4fgo8i9uawe3rlkjfd8iesdolijw3e37"
> into the field.
> - The billing address does not update when the home address is changed,
> except for the first time a home address is entered.  Thus if the user
> makes a typo in the home address and then fixes it, the typo will remain
> in the billing address.  Since we don't want it to *always* autoupdate you
> may need a checkbox or button to indicate when the billing address should
> or should not be synched to the home address.
> - The MRN does not autoassign.
> - The user can click and drag the days of the week and reorder them.  I
> can imagine this being confusing if the user does it on accident.
>
> Our studies necessarily focused on usability flaws, but we did make note
> of some good things too:  We liked your mouseover help, especially that it
> waits a moment before displaying the help text.  We liked that the
> functions are logically grouped, and that there are very few screens to go
> through to perform any one task.  "Too many screens" is a common problem
> in complex applications, and one that you have successfully avoided thus
> far.  The layout is good in that it is generally easy to find the things
> that are on the first layer and our users recognized the functions when
> they saw them.  Generally you do a good job of "speaking the user's
> language" rather than using computer lingo.
>
> Thanks for putting in the hard work to write this system!  I agree
> completely that the future of hospital information systems is (or at least
> should be) in open source.  I hope that you will find our little study
> interesting, and that you will be able to use it to make the PatientOS
> even better.
>
> Sincerely,
> Dr. Stephanie Medlock, DVM
>


-- 
Greg Caulton
Principal
PatientOS Inc.
Cell: (857) 869-0394
Fax: (857) 241-3022