Ux research and user analytics leading to $5.1 million in 2 months

All names and trademarks are intellectual property of Caterpillar Inc. All information pertaining to this study are my own and do not reflect the views of Caterpillar Inc.

This case study will highlight the work I did as the Interim UX Research Lead for Caterpillar Inc as we launched a software called Caterpillar Turbo, a web based tool that allowed sales teams to pick the right sized machines for customers and therefore make more sales, for 3 product lines; Wheel Loaders, Dozers, and Motor Graders. Turbo was launched sequentially for each product line, and I will discuss the same process that I followed for all three.

I was hired by Caterpillar Inc to restructure the team’s digital product design process by creating a UX Research program to gain user feedback from a roster of 50+ highly specialized business users. I also created an analytics strategy to gain insights on exactly how the product is being used post-launch, leading to data-driven team discussions, faster iterations, and less guess-work, contributing to $5.1 million in revenue in the first two months of product launch.

This case study will start by highlighting how I learned about this industry through a Listening Tour with subject matter experts, it will then highlight how by using User Discovery Interviews I learned the status quo of the sales aspect of the industry and familiarized myself with the end users and their existing sales techniques, and the problems and friction points they face with these techniques. Then I will talk about how I did User Testing with clickable prototype that I had designed, and how I trained the digital innovation team to do these tests as well as the User Discovery Interviews by using my scripts. I will also share some of the insights that came out of the UX research that shaped the product roadmap and product strategy.

Lastly I will cover how I created a User Analytics Strategy by utilizing an events based analytics tool, highlighting what events to track and how to categorize and label them so that we can digest the data, and what insights this captured.

LEARNING ABOUT THE INDUSTRY AND USERS

Listening Tour

As with any new industry that I’m designing software for, I always kick-off by conducting a listening tour. This involved talking to all relevant subject matter experts that work closely with end users within Caterpillar and can therefore represent their needs, struggles and existing processes with detail. Another part of it was speaking with the individuals within the Caterpillar innovation segment in order to be caught up to speed on what they’ve done so far for Turbo, what they see as the barrier to success for Turbo, what has already succeeded or failed, and what the problems we are trying to solve are within the industry. Here are some sample questions that were discussed in the listening tour.

1. What are the different user types relevant to Turbo?  What are their responsibilities and goals?

2. What is a typical sale cycle and sales process like for them before any variation of Turbo existed? 

3. What are the biggest barriers to these individuals completing their sales cycle?

4. Which users work out of the office vs in field? Are there any other context relevant information about their environment in which they’re hypothesized to use Turbo, such as mobile phones or tablets at the site or lacking internet access, that I should be aware of?

5. How often are they hypothesized to use Turbo?

6. What industry jargon should I be aware of that’s relevant to the sales cycle or Turbo, such as “matching the right bucket to the right truck, to the right loader”?

7. Are there any existing tools I should be aware of that directly or indirectly solve the problem Turbo is trying to solve?

8. Can I get access to the product roadmap, any existing UX research and user analytics?

I had several key takeaways from the listening tour:

• #1 goal of Turbo is selling machines, as Caterpillar is a premium brand and price differences are becoming harder to justify

• Caterpillar an alpha version of Turbo which was based on a spreadsheet of the selection algorithm. Dealers didn’t like using the spreadsheet. Alpha version didn’t have any feedback mechanism with end users.

• End user demographics: older, less tech savvy and less interested in tech

• The status quo sales methodology that dealers used was nomenclature selling. The general workflow was that a customer would go to a dealer and specify various parameters of what they need the Caterpillar machine for, such as what type of material they’re moving with the machine. Then the dealer has to customize and configure the machine in order to quote it.

• By knowing the size of the trucks to be loaded, the densities of the materials the customer is handling, and the desired production rate, we can tell them almost the perfect machine and bucket combination, which minimizes their total machinery price and gets the job done

User Discovery Interviews

The next phase involved doing User Discovery Interviews with potential end users of Turbo, in this case sales people who were Caterpillar dealers. Here is a sample script from the questions I did for the bulldozer product line. Note that during my Listening Tour, there was a hypothesis thrown around that the Turbo tool, in addition to being a tool that gives the salesperson the right machine to sell based on the needs of the user, could potentially be a CRM tool as well. As a result, some questions are designed to validate this hypothesis for a need of another CRM.

Brief: We’re part of Caterpillar’s innovation segment building a tool for you to sell more dozers. We want to understand what your day to day is like so we can build this tool better for you.

1. What is your position?  Do you manage other team members?

2. How long have you been selling dozers for Cat?

3. Can you walk me through a typical sale, and share your screen? [Ideally, this covers a lot of the below questions]

4. When a customer is considering a product, how do you know which one to point them towards?

1. How do you justify your recommendation?

2. When there’s technical information such as parameters that’s needed to make a recommendation, where do you get that information from?

3. Do you record the customer or sale’s information? 

1. How do you track the status of the sale ie quote sent, project in bid etc

2. How do you collect the customer’s information? 

3. Where do you store this information?

4. How do you know when to follow up with a sale?

5. How do you do the follow up (phone/email), and what do you say/include in follow-up?

6. What information do you send to customers considering a purchase? (i.e. Reference materials, screenshots, direct link to product page in Cat website). Can you send me a copy of this?

5. How do you gain new customers?

6. What do you like about the sales process?

7. What frustrates you about the sales process?

8. What do you think are the biggest hurdles to making more sales?

9. What is the most challenging step in the sales process?

10. How often do you discuss custom configurations with your customers? (ie Blade configuration)

1. What are the hurdles to building a custom machine?

11. Where are you communicating with the customer [or sales team depending on job title] (in field, phone, store front)?

12. What method of communication do you use to reach Caterpillar?

1. When do you typically try to reach Caterpillar?

2. How often do you go to Cat to get the stock?

3. How long does it take?

13. Is there anything else you’d like to add?

Usability Testing rapid prototypes

Using all of my findings from the Listening Tour and User Discovery Interviews, I created designs that were iterated upon with the feedback of the subject matter experts and stakeholders. Once an iteration was reached where we felt like all of the hypotheses were being tested, I created a clickable prototype, which I then started doing remote usability testing studies on with the end users. I also led another designer to create variations of designs and iterations based on my research and product strategy.

User Testing Script

A lot of the focus with the user testing was to find out if they can easily use the clickable prototype, if they understand the value in the questions being asked of them (or if they want to skip it), and if the industry jargon used in the app is in fact nomenclature that’s understood. The best way to answer the latter was to ask them to describe certain elements of the page in their own words, as we were playing around with a design that hid/showed descriptions. One way see if things were intuitive, was by asking them about their expectation of what would happen if they clicked x, before having them actually click it. As with any usability study, I ended up questions that would allow us to segment the users, and understand how tech-savvy and tech-friendly they are, as well as asking for a Net Promoter Score (NPS), in order to compare the NPS’s of different prototypes against each other.

Brief: 

• We’ve built a tool called Turbo that gives you recommendations on what is the best machine for what your customer needs.

• The purpose of this user test is to find out how to make Turbo more user friendly. I’ll be giving you several tasks. Please think out loud as you do them, the more you talk and express your thought process the better. There is no right or wrong.

• Consider a deal you’ve done recently (or do we give them dummy data?) In what form would they have the data they’re entering for a real sale? Is it just off the top of their head? Is it written down somewhere? Either way we should replicate this as closely as possible with the user test.

Task:

• I want you to go ahead and click Start Now

Task related questions:

• Why do you think you need to enter the customer information as the first step here? 

• Do you always know these details when you’re quoting a machine to a potential customer?

• [If user skips notes]

• What is an example of something you might enter into the notes section [knowing that it will get stored in a database for you to re-access]?

• How would you describe what was the point of this step?

Task: 

• Proceed with Step 2

Task related questions:

• What do you expect to see if you click on each question?

• Which of these questions stand out to you, if any?

• Which of these questions do you think are mandatory?

• How would you describe what was the point of this step?

Task:

• Proceed to Step 3 Applications

• [Once user is at bottom of page]: How would you change the hourly production target to rail cars per hour

Task related question:

• In your own words, what is an ‘Application’?

Task:

• Proceed with the time calculations

Task related question:

• How would you describe idle time and miscellaneous work without reading the descriptions?

Questions [on results page]:

• What do you expect will happen if you adjust these bars?

• What do you expect ‘save record’ to do?

• What do you think will be printed when you hit ‘print’?

Tasks [on results page]:

• What would you do if you wanted to change the density details of the first application?

• How would you add a new application?

• [Observe for time calculation useage] 

• How would you describe each of your choices here without reading the full subtext (daily/weekly/monthly v hourly)?

Other questions:

• On a scale of 0 to 10 how likely are you to recommend Turbo to a colleague?

• What is the biggest barrier to you using Turbo?

Process Questions:

• What does your sales process look like [in-field, vs in the office, vs remotely]? 

• Today, how do you decide what the customer needs? 

• In simple terms, how do you recommend a wheel loader to a customer? 

• What questions do you ask? 

• Where do you get the quantitative information from (ie Lightest Density Material Type)?

Segment Info:

• How many years have you been selling selling wheel loaders? 

• How many years have you been selling machinery (including wheel loaders)?

• What smartphone do you own for personal use?

• Do you have a favorite app?

scaling usability tests

I went on-site to the Caterpillar headquarters and regional offices and trained the innovation team on how to do usability tests of my clickable prototypes, and how to do user discovery interviews. I watched as the guys heading all of the bulldozer division, as well as the other ones leading all of Wheel Loaders and Motorgraders did these tests by reading my scripts and demo-ing my clickable prototypes on iPads in the Arizona desert in trailers. A large part of the education was to stop them from asking leading questions, interrupting the respondent, or hinting towards an answer that they were hoping for.

Insights

With every round of user tests and user discovery interviews, came many insights that shaped the next iteration of the clickable prototype.

Here are some examples of insights that came out of the research with dealers/sales reps for Track Type Tractors:

• Justifying the price premium over competitors is a challenge

• Lack easy access to competitor information, whereas customers are very well informed and expect a conversation about competing products

• Competitors provide financing options that we don’t

• Majority of sales have no custom configurations

  • customers are buying new machines for a project that’s 12 weeks away, and we can’t ship them custom machines until 16-20 weeks

  • 80-90% of sales are what’s in inventory

• CRM tools are busy work and don’t help

• Sales are on a first-name-basis and personal relationships— there is a risk with retiring sales representatives

• Provide access to the questions ahead of time in a printable format

• Some mandatory fields are data we don’t have or are not comfortable answering as we don’t know if we have the right data

Recommendations based on insights

• Have a ‘short path’ to getting quick results with only absolutely-necessary-information, without intimidation of complciated mandatory fields

• Provide easy access to competitor information that’s usaeable to justify price premiums

• Test the demand for configuring custom machines by having a “continue to configurator” button at the end of the flow, and tracking the analytics data of the click through. The page would just redirect with a ‘coming soon’ feature

User Analytics Tracking Plan

I created a list of questions that we wanted to answer through user analytics, before tracking them.

1. “Who are the most active dealers?”

2. “Of the most active dealers, which features are they engaging with?”

3. “Where are users falling out of the funnel?”

4. “What is the most recommended machine?”

5. “Are users downloading the PDF questionnaire before going on-site?”

6. “Is there a demand to build out custom configurations?”

I implemented this by using Google Tag Manager and Google Analytics. Any event tracking analytics tool works for this. I had to create an Analytics Tracking plan, which identifies what would trigger an event to be captured, how it should be labelled, and what parameters should be captured from this click. I had to use acronyms such as WL, to define Wheel Loaders, so that this product line could be tracked separately from Bull Dozers and others.

Project success

When we launched the Beta version after all of the above UX research, it generated $5.1 million in revenue in the first two months. Another measure of project success was the Beta Tester feedback we gathered after launch.