How did we experiment with a UX copy assessment to get the right name for a new resource in our platform Photo by Surendran MP on Unsplash At commercetools we love experimenting and trying out innovative ideas that let us get the best possible solution for our users. This time we used an existing framework for an […]
At commercetools we love experimenting and trying out innovative ideas that let us get the best possible solution for our users. This time we used an existing framework for an unusual purpose: to come up with a name for a new feature and its resource in our platform.
But before spoiling all the fun, let’s go back to the initial problem we were trying to solve. Based on previous research, we were aware that some customers selling products in different storefronts are often creating one project per storefront needed. As a consequence they end up duplicating data and/or synchronising it between projects.
One of the reasons for that setup was that each storefront had to sell slightly different product catalogs. Imagine having a product catalog of 500K products, all with their product data and pictures, where the 95% of it is available in all storefronts the customer has. We needed to improve our customer experience so they could avoid the multi-project setup scenario and just send the right data to each storefront, or in case of B2B, to each customer.
During the discovery phase of the project, when the team was gathering use cases to define the problem better and model the domain, we conducted multiple interviews with targeted users of our APIs. It’s during these initial steps when the team usually establishes the Ubiquitous Language, a common language between team members and our users.
However, we realised that different participants were using a variety of terms to designate the same type of functionality: a group of products available on a particular storefront. The names were: assortments, catalogs or collections to name a few. This often resulted in the interviewer, one of our commercetools folks using one term and the interviewee, an end-user (usually an engineer), answering by using a completely different word.
Based on that experience, we realised we could just not ignore it and go for the initial name we had in mind. Looking at the official definition of each term and the names already used in the industry for similar solutions made the list of options even longer, adding more terms to take into consideration.
Instead of just talking to people in the business as we were already doing, we decided to empirically find out what users would call the functionality based on the capabilities we were going to offer.
That’s when we decided to follow the Domain-Driven Design approach suggested by Eric Evans in his book “Domain-Driven Design: Tackling Complexity in the Heart of Software”, but instead of just talking to people in the business as we were already doing, we decided to empirically find out what users would call the functionality based on the capabilities we were going to offer. In that way we could make sure that it is easy to discover when using our platform and unlike Elio, the protagonist of the novel Call me by your name, we were going to “have the courage to ask a question like that.”
If you have ever learnt a second language in school, you are definitely going to be familiar with the test we conducted. Remember those exercises in your workbook where you had to fill in the gaps in a paragraph? That’s what’s officially called a cloze test.
The cloze test was created by Wilson L. Taylor in a workshop in the 50’s as a way to measure readability. However, the framework has been used for multiple purposes throughout the time:
The theory behind it says that when we are in front of a cloze test, we combine our background knowledge, previous experiences and our reasoning in order to fill in the gaps.
Instead of verifying that “the content suits the audience you are targeting”, we were interested in finding the most meaningful name for the new functionality and resource in our platform for all types of users. That was the trigger for our experiment and the reason why we considered using an altered version of the cloze test as the means for reaching our goal.
Given the purpose of it and to distinguish it from previous Cloze tests, we are going to call it the Denominative Cloze Test.
You don’t have to be a highly skilled researcher to conduct this study. In fact, if you are a developer building your own API and have good writing skills, you can easily run the experiment. Here are the steps to follow:
The following paragraph is the beginning of our Denominative cloze test:
With the new resource called _____________, merchants are now able to group a specific subset of products to make them available in their separate storefronts or output channels. Once you create a ___________ in your project, there are two ways to…
If you are a researcher or a UX writer reading this article you might consider this usage of the cloze test controversial, as on a regular test where you are testing readability, the paragraphs tend to be longer (the longer the texts are, the better results you get) and words behind the gaps will refer to different terms. But remember we are using this framework for a different purpose: to find out the best resource name.
By being truthful to our intentions, users were more likely to give an open response, making them reasoning on their answers.
Before conducting any session, I encourage you to test the test with a colleague. There are always last minute tweaks on the paragraph that you might want to do. Changing the paragraph in the middle of the research study may lead to inconclusive results as knowing how much the change you did has an impact on the participant’s suggestions is hard to identify.
Denominative Cloze Tests can be conducted in two ways:
Option 1: On a video call or in person
The session happens synchronously as you, the tester, will be present and can bring a note taker. Stay away from the Hawthorne effect by not inviting too many people to the call.
The session is divided in the following way:
Unlike typical cloze tests, Denominative cloze tests don’t have wrong answers.
Option 2: Through a survey
If you cannot conduct the session simultaneously or have a time constraint, you can set up a survey and collect feedback asynchronously. The survey should contain both the introduction and the paragraph with blanks. Then add two field for users to provide both their preferred term and the explanation why they went for that option.
As an optional step, you could provide a second part in your form, including some of the terms you considered and asking participants what they evoke for them when they are used.
Once you have conducted all the sessions, gather all the terms you were given and list all the reasons stated by participants. Include the reasons against the term as well if you got any. In our case, they drove our decision.
Then share the results with your team and with all that evidence open up for discussion.
Discussing and analysing the user’s reasons should guide the team on finally deciding on a term which may or may not be one of the options given by participants.
Coming back to our initial task, in our case, demographics and industry where participants worked definitely played a relevant role in their chosen options. Having a large target audience was the reason why we had a handful of different terms even before launching our experiment, so the bigger your target audience is, the more meaningful running a Nominative Cloze Test could be for you.
In our case, we did not go for the most chosen term, catalogs, as the reasons against using it were strong enough to dismiss it: too traditional, less innovative and very tied to the retail industry. A similar case happened with assortments and other options that were too connected to a certain user group but sounded clearly foreign to others. Once we had reviewed them all, we decided to go for a variation of one of the options a participant gave us: Product Selections.
Product Selections was generic and neutral enough to be understandable by all audiences and did not have any particular meaning previously tied to it as terms more specialised and linked to a certain industry had.
Even though we chose a term that was not suggested multiple times, the Denominative Cloze Test has guided us throughout the process. Without investing the time in conducting this little research experiment, we would have never come up with a term that, from then on, software developers, domain experts and even end-users use.