First Experience with Usability Testing
As part of our LMS selection process, we’re conducting some small-scale usability testing. We really want to see how much people can figure out in each of these systems without any instructions at all to help determine which system will be most intuitive. We also want to identify specific places where we definitely will need to do training and get a better idea of the trouble spots for users.
Although I’ve read Jakob Nielsen’s Designing Web Usability and other books, this is the first time I’ve had an opportunity to design and conduct any usability testing. It’s definitely been an enlightening experience, so I want to reflect on the process and what I’ve learned in this first round of testing.
Test Design & Process
We are testing about a dozen people total, with everyone on the team conducting some tests. The testers are divided into 4 groups:
| LMS A | LMS B | |
|---|---|---|
| Student | Student: LMS A | Student: LMS B |
| Facilitator | Facilitator: LMS A | Facilitator: LMS B |
We also have 2 people who are testing both systems; one in the student role (thanks Mom!) and one as a facilitator. The testers have a pretty wide technical range, although we aren’t including anyone with extremely low technical experience. Partly, that’s because we’re using Adobe Connect to record the actions of most testers, and we need people who can manage multiple programs at once.
For each set of tasks, we have a script with a set of prompts for each task. We tried to have a somewhat logical flow for the tasks. The facilitator testers they grade the assignments right after viewing each type of student work. The tasks are grouped, and every few tasks we’re also asking the testers to rate the overall ease of use for those features. We could have asked it for every task, but we knew we had a lot of tasks to test and wanted to reduce the time.
We used partial mock courses for the tests with dummy accounts and student work provided.
Lessons Learned
In writing the prompts for the facilitator tasks, I realized that this took me much longer than I originally anticipated. Part of the problem was just not having the actual site completed when I was writing the directions; we hadn’t made all the layout and design decisions when I started writing. I also caught myself repeatedly giving too many hints in the prompts. Sometimes I did too much to describe the process of how to do a task, instead of just describing the desired end result and letting the tester figure it out. For example, I might write “Go to Joe Smith’s profile to view his blog and read the post for Activity 6-A-4” instead of simply “Find and read Joe Smith’s blog post for Activity 6-A-4.”
Although I had tested all of the tasks individually on my own, I hadn’t actually run through the entire test from start to finish. I should have. Today I discovered that I needed to reverse the order of 2 tasks; archiving a discussion prevents the facilitator from being able to grade it. Oops. Fortunately I think I was the only person who conducted tests with that order of directions, so no one else was affected. For the next round of testing though, I definitely need to complete a full walk-through myself.
During the actual testing, I had planned to keep track of the timing for each task. With juggling the task prompts, keeping notes on the tester actions and thought processes, and orally guiding the tester as needed, I just wasn’t able to do that too. Fortunately, I have the recordings to review and determine the timing. With more practice, maybe I’d be able to manage. In the next round of testing, I’ll have one in-person tester (my dear husband was coerced volunteered to help). Since I won’t have a backup recording for that, I’ll probably just have to make him wait between tasks so I can finish my notes.
We’re still a ways from having all the data collected and analyzed, but the testing has already given me ideas about training and places we could improve the interface. If you’re selecting a new LMS, I would highly recommend doing this kind of testing. When you spend so much time working with these systems, it’s easy to be blinded to some of the tricky spots for new users. Even just watching one or two newbies walk through your system for the first time can be really beneficial.
If you’ve done usability testing for an LMS or any e-learning before, I’d love to hear about your experiences. What did you learn about the process?
I should mention the way one of our schools uses the Melete Modules tool which I think you might find of interest — they actually use iFrames to embed existing content they’ve authored (they have a lightweight Flash & JS framework that stores most of their content) and they use the modules tool as a kind of super course syllabus to generate the larger navigation for the structure of their courses — mostly weekly lessons. It sounds like a reasonable analog for what you’re looking to do.
So their HTML/Flash pages form the core of the material for their course, and they use the additional Sakai tools as something of a supplement. It fits their model well though, in that it lets individual instructors add on the tools they want to their particular sites.
I’d been trying so hard to not specifically identify which LMSs we were testing. Ah well–I should have realized people would be able to guess based on my prior bookmarks and research.
I like a lot of things in Moodle better than Blackboard; just having discussions that can link directly to the gradebook is a huge improvement.
Our testing in Moodle found several areas that were confusing to users:
* Most people struggled with the wiki. This feature would definitely require specific training if used. The current wiki also doesn’t link to the gradebook, so you have to do a workaround to use it as a graded assignment.
* The calendar isn’t particularly intuitive; the legend really throws people off.
* The gradebook is hard to see, especially from the student perspective.
* As a whole, there’s just a lot of links and a lot to look at. You can make choices in what you include and how you lay things out to alleviate that, but we’ve struggled to find a good balance.
I like a lot of the features in Moodle, and the user community is very active and supportive. If you build your content within the LMS, Moodle has a lot of power.
hi Christy
Did you do a usability test Moodle as well?
At this moment my institute is using Blackboard, but we are not to happy with its usability. So know we are exploring Moodle. If you would have compared these two I would be very interested to hear your experience.
I do wish that co.mments had an automatic detection tool for threads where you’re active like cocomments, but I’ve gotten in the habit of remembering to track the comments now. I have tried the Firefox extension Commentful in the past too; it’s reliable and picks up threads well, but the RSS feed doesn’t include the full text of the comments.
Excellent information on how to adjust those tools. Because it requires developer time, we probably won’t mess with it for purposes of our testing and pilot. However, it’s good to know that those customizations are available if we do choose to go with Sakai.
After talking to several vendors, we have found a company to work with for our pilot that has been very responsive. But, I promise I’m keeping all your contact information in case something changes.
Right now, we’re piloting 2 systems and gathering all the feedback. Once we get done with that, we’ll make our decision and start planning our conversion process.
We don’t actually use the LMS the way most organizations do; almost all of our content is developed as html pages in Dreamweaver. That gives us much more control over the layout, images, and internal navigation. We have breadcrumbs and CSS dropdown menus to get from page to page in the content. So, an LMS that does a great job letting people with no technical expertise build content might not be as good for the rather peculiar way we use the system.
Creating the dropdown menus and breadcrumbs can be time consuming though, so we’ve been exploring options for using the navigation within an LMS for our html pages. That can end up being a “link farm” though, with just a long list of links to various pages. Is that tradeoff worth it? I’m not sure; that’s part of what we’re testing in the pilot.
We have a lot of things to play with and test out. I’m really glad that we have the opportunity to play with both systems for several months and really tinker with them before deciding.
Thanks again for the info on the synoptic tools and the tool list–great to know what’s possible.
P.S. http://co.mments.com was a good call so far, you’re right they do seem to do a better job of picking up the comment threads, though I do miss the Firefox extension from cocomment…
The recent “synoptic” tools on the home page *are* orderable, but it’s not easy — if you go into the admin workspace, you can hand edit the order using the Sites tool, there was a mailing list post [1] on this recently. If you have programmers, and want to change the default order, it’s a source-code edit. Also, Stanford did some work on an alternate home page tool, which is interesting if not quite as accessible for demonstration.
It is possible to categorize the tool list, though not in the default presentation. If you go to either /osp-portal or /xsl-portal (instead of the default portal) I believe both of those portals support categories (and it’s a common request). There is a sakai.properties entry to specify the default portal, but I don’t remember what it is off the top of my head.
[Begin Shameless Plug]
So given that I am part of an organization that looks to help others roll out Sakai… can I ask how you’re looking to use the tools & what your thoughts are going forward? If you were interested, and wanted to forward some contact information I could ask someone to setup a phone call to do some discovery and see if we’d be able to assist in your endeavors.
[End Shameless Plug]
[1] http://www.nabble.com/ordering-synoptic-tools-in-home-page.-td15704440.html#a15704440
The comment feed is kind of hidden. It’s http://christytuckerlearning.com/comments/feed/
I used to use Cocomment, but it was so flaky for me that I switched to http://co.mments.com/ instead. I’ve had much better luck with co.mments, especially on Blogger sites.
Right now we are all stumped on how to change the order of the items on the right side of the course home page (Calendar, Recent News, Forums & Messages, etc.). None of us have been able to find a place to adjust that like we can in the left navigation. Are we missing something, or is that not something we can change?
I wish there was a way to visually group the items in the left navigation by adding a static divider or something. When you get a lot of tools there, it’s hard to see them. In Blackboard, we actually cheat by adding an empty content area with a title “—–” to create a divider; I might just do something similar with a web link in the navigation that just goes to the current page. Is there a less clunky solution though?
Well, trying to use cocomment, anyway…
Any particular shiny widgets or thingys you’re interested in? I’ll admit the contrib modules aren’t always as visible for Sakai, so some of the latest and greatest code which hasn’t had as much time to bake in production deployments yet gets passed on if you’re new to the community.
P.S. Any way to subscribe to comments? Trying cocomment, but didn’t see like a comment feed or email option.
Excellent deduction–Sakai is one of the two choices we’re comparing. Sakai is definitely an improvement over Blackboard; I have no questions about that. I wish that the user community were bigger and more established. The community is clearly growing, and it has this core of really passionate people, which is great. It just hasn’t been around long enough to build up a huge community or lots of resources the way other open source projects have.
Sakai also doesn’t seem to be as powerful or as flexible. The widgets in D2L and Moodle add a lot of features and possibilities that I don’t see in Sakai. However, as I’m spending more time, I’m not sure how much we actually need all those widgets for what we’re doing. I wonder if a simpler interface with fewer bells and whistles might actually be more effective. Of course, that’s part of the point of testing–in the scheme of things, it matters much less what I personally think about the interface than what is easiest for the users.
Hi,
I noticed a bunch of bookmarks about Sakai and notes about doing testing for LMS selection, so…
I’ve been involved with Sakai for a while now (and actually recently moved jobs to build out a division to support Sakai for instituitons) but am commenting since I’m really interested in your experiences with Sakai.
Do share…