As part of our preparations for bringing Gutenberg to NC State, we invited some of our campus WordPress power users to have a sneak preview of the new editor. We asked them to work through a 30-minute content creation exercise, while we watched and recorded both their screen and their reactions. (We reminded them to talk out loud as they worked, and got some great quotes.)
In this post, I’m going to share what we’ve seen so far, plus lots of thoughts from me.
As a disclaimer, this is all very non-scientific (small sample size, inconsistent testing environments, etc.) and very preliminary. Some UI elements will certainly change over the next few months, and we haven’t gotten far enough into building our own custom blocks for those to be included in this test. Even so, we’ve found the feedback we received extremely helpful. I’m hoping others will find this useful as well.
Key Takeaways
Here’s the tl;dr version:
If users are given a little guidance and a sandbox where they can learn the new editor without breaking anything, they should be able to make the switch without too much drama.
There were definitely some bumps, but I think they’re manageable bumps. None of our users were able to complete their exercises without at least one hint from us, but the hints they needed didn’t need to be very specific. (Example: We occasionally had to say, “Try using a different block.”)
In general, these were the big pain points we observed:
- Not knowing what to call different UI elements or where to find them.
- Recognizing/finding the advanced block settings panel.
- Identifying tasks that could be completed through block settings versus tasks that required a different type of block.
- Anticipating whether an action would affect an entire block, or whether an action would format selected text within a block.
All of those issues should sort themselves out with a little experience and a lot of trial-and-error. One of our team’s big tasks between now and WordPress v5.0 will be to come up with ways of reducing how much of that trial-and-error turns into support tickets.
About the Test
Testing took place in a clean WordPress environment, where Gutenberg was the only plugin that would change the editing experience from vanilla WordPress. Most testing took place with Gutenberg v2.0.0, but since tests were scheduled across a couple of weeks, we did have a few users on v1.9.1 and v2.1.0. I don’t think the UI differences between those versions were especially significant—but again, this definitely wasn’t rigorous or scientific user testing.
The goal of this round of testing was to see how users who are already very familiar with WordPress adapt to the new editing experience. We want to get a sense for what kinds of support questions we may see, and to try to anticipate those questions in training materials and documentation.
Who We Tested
The folks who participated in this round are generally what I’d call power users. Even if they didn’t know WordPress very well (which most of them did), they’re comfortable working with HTML, using CMS’s, etc. Coming into the test, only two users had seen Gutenberg before, and neither of them had spent any significant time using it.
Most of these users had experience using at least one of three existing tools built to deal with shortcomings in the Classic Editor, including:
- ACF-heavy page templates
- Shortcake-powered shortcodes
- Divi
This is a very different perspective than a brand-new user or an infrequent WordPress user—which is exactly what we wanted for this test! We have a lot of content creators who are going to be making a big transition to Gutenberg. They have preconceived ideas about what WordPress should be capable of, and how changes to the editor might impact their content workflow.
Test Mechanics
Our content exercises were inspired by the official Gutenberg tests, but there were some specific things we wanted to test for our users.
In each user test:
Step 1. Users were asked to re-create a template “About NC State” page using the Classic Editor. The template page contained rich text, headings, an embedded YouTube video, and a right-aligned image with a caption.
Step 2. Users were asked to re-create the same template page using the Gutenberg Editor. The result should be a page containing paragraph, heading, YouTube embed, and image blocks.
These first two steps established a baseline for us to compare against. We have a direct comparison of them building the same page using the old editor and the new editor. Is it easier or harder for this user to embed a YouTube video in Gutenberg?
Step 3. Users were given a new template page to match, built using some new Gutenberg blocks and block customization options. They were asked to update the page they created in Step 2, including paragraph block formatting (drop cap, font size, background color), adding a cover image block, adding a button block, adding a gallery block, and adding a recent posts block.
Step 4. Users were given free exploration time. This gave them time to try out other Gutenblocks and other features of the new editor.
We would introduce each step and ask them to re-create a given template, but we wouldn’t give any other initial instruction besides that. Users were told that they could ask for hints, but we tried to make most hints open-ended, eg. “Try another block” instead of “Use the Cover Image block.”
Notable Moments
Below, I’m going to highlight several notable moments. These are either experiences that were repeated across multiple user tests, or they stand out as potential trouble spots that we didn’t expect.
A lot of these may seem negative, so I’ll reiterate what I said above: I think the challenges of transitioning to Gutenberg are very manageable. But it’s worth being aware of these points of friction.
“Did it publish? … Is it working on it?”
With the move to JavaScript, clicking the Publish or Update button doesn’t trigger a page reload. Without that visual cue—and with the new two-click publish workflow—just about all of the users we tested were initially unsure of whether their work was really published.
Everyone figured it out, and by the end of our tests I think everyone was comfortable with how publishing worked. But that moment of hesitation is something worth noting, and is representative of the Gutenberg transition as a whole: I think what I just did was what I wanted to do… but I’m not sure. (At least that’s how it’s felt learning to write custom Gutenblocks!)
I think that kind of hesitation will shake itself out for most users after a few days in the new editor. But it’s still a fundamental feature of WordPress not behaving as expected from past experience.
Buttons without visual labels
Multiple users commented on the lack of visible text labels except on :hover, on controls like the preview button or the “add block” button. Some of them liked the clean interface, while others said they didn’t initially understand what the icons meant. A few also totally missed the trailing “add block” button that appears between blocks when you hover over the right target area (which I think may be related to not having visible labels).
In one case, a user didn’t seem to recognize that the “add block” button was a button at all until relatively late in the test—and therefore didn’t realize how many blocks were available until Step 3. This user spent much of the exercise adding only the blocks whose names were visible at the bottom of the editor. On first run-through, that’s the Paragraph and Image blocks. Since that’s all that was visible, and because those are two important things you can do with the Classic Editor, for a while I think he thought those were the only block types available.
I didn’t expect that to happen, but in hindsight I think it makes sense. After 12 years of the Classic Editor providing visible labels, some users might struggle with this design change. It was only one of our eight testers, but (if I can pretend statistics works this way) 1/8 of our campus users having trouble recognizing basic controls would still be a lot of people. An icon glossary seems like an easy support win.
Video block vs. YouTube embed block
By the time they were adding a YouTube embed in Gutenberg, most of our users had a rough intuition about blocks representing different kinds of content in their post. Rather than pasting the YouTube URL into a paragraph block and expecting it to be processed by oEmbed (which would also have worked), these users went looking for the right type of block.
With only one exception, these users tried the Video block (which embeds a video hosted in the Media Library or from a direct URL to the video file) rather than the YouTube embed block. They pasted the YouTube URL in the Video block’s URL field. An empty video player appeared in the editor, and when they published, an empty video player appeared on the page.
A few things stand out to me after watching that sequence play out several times:
- Users don’t initially make the connection that “where my video is hosted” has anything to do with “what block I should choose.” Some had to be told to try a different block after multiple attempts to get the Video block to work.
- The in-block helper text says “Enter URL of video file here…”—referring to a video file rather than a URL from some other video hosting service. But that’s a subtle distinction, especially if you’re reading quickly.
- The first indication that something went wrong should have been the empty video player displayed in the editor. But WordPress users have been trained on years of “what you see isn’t necessarily what you get” (shortcodes, weird pagebuilders, and just the limitations of TinyMCE). One user said, “I can’t see it, but I assume it’s done” and moved on to the next task.
We saw something similar happen in the free exploration part of Step 4, with users trying out social media embed blocks. It’s not clear what those blocks are looking for: a single post or a whole feed? Users tried them out and weren’t sure whether it worked when it failed.
Each of those points above translates into something worth remembering when we start building our own custom blocks:
- Block names are important. Ambiguous or incomplete names are confusing.
- Make the in-block UI clear and explicit. Different helper text could have clarified what kind of video source was expected.
- Blocks should tell you when they break. Rather than displaying an empty video player in the editor (that could be interpreted as a placeholder for the real thing!), URL validation could have detected that the URL provided by the user wouldn’t work and generated a message saying something went wrong.
Since we’ve always discouraged our users from hosting video in their WP Media Library (our web hosting just isn’t optimized for that), the easy solution for our team is to just disable the Video block on sites we manage. Users will have to go looking for something else, and should find the YouTube block pretty quickly. This may be a strategy we use elsewhere: more isn’t always better and there are certainly some superfluous blocks, especially in the “embeds” section.
But beyond this one example, there’s a good lesson here in block design. WordPress developers aren’t used to designing in-editor UIs, but now that’s something we have to think about whenever we create a new Gutenblock.
Headings as separate blocks
At the bottom of the template in Step 3, we had an H2 heading that said “Latest News,” followed by a Recent Posts block. Most of our testers added the Recent Posts block without any hesitation, then went looking for some setting in the block that would let them add the heading there. Some realized on their own that “Latest Posts” was a separate heading block, while others had to be told to that it was a separate block.
That’s definitely consistent with what they’re used to elsewhere in WordPress, like when adding widgets to widget areas, and some shortcodes and ACF modules. But Gutenberg blocks don’t work that way. That’s a good thing, too! Unbundling the heading from a block or block-y thing like a widget gives you more design options and lets you choose the heading type that’s appropriate for where it falls in the content. But it’s a subtle difference that tripped up nearly everyone who ran through this exercise.
“Hidden” features
Our users took a while to discover that blocks had advanced settings in the right-hand panel—and once they discovered it for one block, most didn’t think to revisit that panel for other blocks without some prompting by us.
(Incidentally, some did click on a block’s vertical three dots to open the dropdown, but they were much more interested in block transformations and generally didn’t click “Show Advanced Settings.” My colleague Jen has a guess as to why that was happening: The :focus state stylings on the text “Show Advanced Settings” makes it look at first glance like a heading for the menu items that appear beneath it, rather than something that can itself be clicked.)
In the same vein, only one user we tested tried using the block search tool to find a block. I’m not sure that anyone else even noticed that search tool was there. And I’m 100% certain it never occurred to any of them that they could add a block via /slash command.
These features are all very useful, but they’re apparently not easily discovered. This is something I’m struggling with on Gutenberg: How do we highlight “hidden” features for new users? How do we help them understand what everything does without overwhelming them with too much information?
Block settings vs. different block types
One task in Step 3 of our exercise involved changing the font size and adding a drop cap to a paragraph block, and another involved changing the background color of another paragraph block. Both of these can be done using advanced block settings. But the first step for most of our users was to go looking for another block type to use.
On many of the campus websites we support, these effects (minus the drop cap) are possible using Shortcake-powered shortcodes called “Lead Paragraph” and “Callout.” On some ACF-heavy sites, some similar effects are possible using different ACF modules. In either case, our users have been trained to look for a something to insert into their posts to modify how text is displayed beyond TinyMCE’s default rich text.
This might be a situation where someone new to WordPress has an advantage over a long-time user who has experience with these other tools.
Block settings vs. text formatting
As noted above, one task involved changing the font size and adding a drop cap to a paragraph block. Once users found the advanced block settings, almost all of them highlighted the text of the paragraph before adjusting the font size. Several also highlighted the first letter of the paragraph and tried using the font size tool to make the drop cap, before trying the drop cap toggle.
Two things stand out:
- “Drop cap” is print jargon that some users won’t recognize. Some additional helper text and/or a visual cue showing what it means might be useful.
- It’s not obvious that the advanced block settings apply to the entire block, not just highlighted text.
Highlighting text before using a formatting tool makes a lot of sense after 12 years of the Classic Editor. But that experience is going to lead to some user frustration, eg. I highlighted the first letter and made the font size bigger, but then everything got bigger!
Importantly, that expected interaction still works with other formatting controls. The bold, italics, and link options in the top toolbar only apply to highlighted text. For a new Guten-user, the distinction between controls that affect only to highlighted text and controls that affect the entire block is unclear and arbitrary.
Selected Quotes
First impressions
“WHAT?”
“Ooooh, wow, okay, that’s different.”
10 seconds of silence, followed by: “This is, like, SO different.”
“Urrgh! Oh gosh. Alright. Well this is very different.”
Thoughts on the UI (early in the exercise)
“That’s the preview button? Uh, fine.”
“Did it publish? … Is it working on it?”
“What are these things? Why would they do this to us?” – Discovering UI elements that appear when hovering over certain target areas.
“So I clicked on the little… mini-hamburger options thing looking over here, looking to see what it is that we can do with it. I’m gonna start with preforma– nope, that’s not it.” – First time clicking three dots next to a block.
“The plus thing, to add content, that’s going to take people a while to get used to.”
“Uh… image? Add a thingy? Add embed thingy?”
“I can’t see it, but I assume it’s done.” – Incorrectly adding a YouTube URL to a Video block.
“This is almost harder to use. You have to, like, know what you want to do, and how they look, and since I’ve not done them ever, it’s like, no idea if they’re magic, magical things… You could do cool stuff if you know all of your headings.”
“Don’t know why it did that, but that’s good to know.”
Thoughts on the UI (late in the exercise)
“The locations of these things is not particularly intuitive to me, but having all this at your fingertips is pretty neat.” – Finding blocks organized into Recent, Blocks, Embeds, and Saved.
“The language around drop-cap and doing this kind of stuff is going to be new to some people.”
“This video thing here [Video block], there should be more explanation here.”
“Just knowing when to come over here isn’t clear.” – Advanced block settings, right-hand panel in general.
“This is what I keep forgetting about, the right-hand sidebar.”
“Understanding that every block has more things that you can show or hide, I think is going to be a big step.”
“Yes… we… are… happy!” – Trying a four-across Text Column block, saying the words while typing each one in a different column (which is not how to use columns, in case that has to be said).
Final verdicts
“I don’t think this is going to be overly difficult for folks. You have to poke around, you have to have someone point you to a couple of things, but for the folks who get in here once a week or so, they’re gonna be fine. And people who don’t get in that often, we have to help them anyway.”
“Probably not that long [to adjust], because I’ve got this little preview here. If you just dropped this on me, I would have issues. But just kind of getting to have this structured time to look at it has me pretty far ahead.”
“I’m comfortable, I like it… It was certainly easier to use than I was expecting… Editing [name of site] now is painful. Not because of anything wrong with WordPress or the old versions or anything like that. It’s just, it’s not something I think about wanting to do regularly. That editor, Gutenberg, I might be willing to do that more frequently because I can manipulate the blocks a lot more easily, I can add content blocks quickly and get them on the page where I might want them on the page.”
What’s Next?
Heading into user testing, we weren’t sure what we would see. Panic? Drama? “I don’t want this on my website”-type comments?
With this round of users, we didn’t really get any of that. What we did hear, over and over, was something along the lines of: “Now that I’ve had a chance to see it and you’ve been able to walk me through it, I feel good about using Gutenberg.” A 30-minute guided tour seems to be enough to get these folks moving in the right direction!
We’re going to be doing that kind of hands-on Guten-troduction for a lot of folks in our training workshops for content creators scheduled this spring. (Sign up for workshops now in REPORTER!) We can also start publicizing our Gutenberg sandbox as a learning environment, with self-guided content exercises similar to the user testing we’ve just completed.
Even so, we’re a big, decentralized campus with thousands of WordPress users. There’s a limit to how many people we can reasonably Guten-troduce. One day, everything will look different—and many users won’t know it’s coming.
For a user who hasn’t had any training and hasn’t checked out the sandbox, what does onboarding look like? (Guten-boarding? Okay, I’m sorry, I’ll stop.) What help resources will we make available to guide them into the new interface?
I don’t really have answers to those questions, but we do have a few ideas we’ve talked about internally. So… stay tuned for more!