9

I just started working for an enterprise software company, and I'm having a hard time integrating validation to see "what works" for the user.

I'm used to working for more consumer-based software, and they aren't "forced" to use the software. However, how do you validate it when your users are forced to use it?

For example, let's say I have 2 sets of tool buttons to A/B Test. One with just icons, one with just words. How would I validate what is the best? In a b-to-c solution, you can often tell by how often the buttons are clicked, how long they stay with the software, how much they use the software -- but what about at an enterprise solution when the users are forced to use the software? How do you design validation in this situation?

Improve this question

4 Answers 4

Reset to default
2

(most) Enterprise A/B testing tools will give you a margin of error calculation given your desired confidence index.

What this means, (assuming you want 90% confidence)

<confidence %> of the time my conversion rate will fall within +/-<margin of error> of <conversion rate>

90% of the time my conversion rate will be +/- 10% of a 5% conversion rate. Which means 10% of the time my conversion rate will be below 4.5% and above 5.5%

There are plenty of posts that explain this:

I will also mention that the longer you run your test (2 months, 6 months, etc.) the more likely your data will converge. Just like the more you flip a coin the closer you will get to 50/50 but may never become close. That is the reason for longer test durations, and also the reason running A/A tests is a smart idea from time to time.

Improve this answer
1
  • 1
    thank you for the links. I'd been looking for something like this. :)
    – CleverNode
    Nov 26, 2015 at 14:30
3

Test in Context

First, I understand your explanation of the difference between having the choice to use software and not having the choice, but I don't really see how that's going to affect your testing. A user is either using the app to complete a task, or they're not, and you can't really control for all the reasons they might not be using it. Sure, in a "free to choose" system, your design should be as "good" as your best competitor (benchmarking), but that shouldn't impede testing Option A vs Option B in your app.

Meaningful Goals - Potential Affect

If the options you're testing have no potential affect on performance, then you're talking about aesthetics. If there's a potential affect (e.g. will increase/decrease time to completion of task, number of clicks required), then you have the basis for a meaningful goal.

With Lots of Users

A/B test, measure success in the context of your goal (time actions/count clicks), statistically analyze for significant differences.

With a Few Users

User test at least 5 users, identify major issues, or not.

Improve this answer
4
  • It just yields better result statistically, with it being a yes or no. True, the time it takes to complete a click of a button may have a time difference, for example, but it may not be statistically a big enough difference for me to say what is better. 1.3 seconds is obviously better than 1.5, but if that's the difference, because they're forced to have to do the action, it's not a very good indicator, in my opinion. But if no button is clicked at all, that's obviously a clear indicator.
    – CleverNode
    Jul 10, 2015 at 16:23
  • I guess my problem is, finding statistically quantifiable measurement, in a setting where i cannot control the task (what it is, or how it's done and whether it's done at all), and have a very difficult time witnessing the user.
    – CleverNode
    Jul 10, 2015 at 16:24
  • I was reading up on the t test as indicated by @David , which is pretty much a performance test as you suggested, with clear equation of the measurement and an output number of the difference... however, because it is a statistical test, it actually has a clear indicator of what's consider statistically acceptable difference between A and B (using terminology of A/B testing). Of course, I can always design a design so bad that I know will make a statistical difference, but if it's so bad I can already tell will make the difference, why even test it?
    – CleverNode
    Jul 10, 2015 at 16:52
  • I considered adding a paragraph to my answer with the subheading "In your case, it might not matter that much". That is, it might be difficult to get meaningful results from timing interactions and significance testing. Not just because you might struggle to generate enough numbers to get valuable results, but because you have access to users and can answer questions more directly.
    – dennislees
    Jul 10, 2015 at 18:13
1

If you have a measure of performance that you can gather, from use of the software to perform a set task, then you can set up an experiment and analyse the results with a "t-test". (some details can be found here.

Improve this answer
1
  • This is a very good idea. I think I'll have to talk to the development team to see how this can be integrated into it. They do have performance test, but never on the user end, only on the technical end (how many seconds it takes to perform the search, etc)
    – CleverNode
    Jul 10, 2015 at 13:09
1

I am assuming that by Enterprise software, you mean software that is deployed on site and not accessible by the cloud.

In cases where you can't A/B test with a live audience, you can use a choice test to determine the favourable icon. The choice test is like a simple A/B test but you can use it with your own team or recruit some users from the enterprise in which your software is deployed.

Speaking simply, print out the two buttons and show them to a couple of people. This would be a rudimentary test, but will give you some idea on whether it works or not.

To get more testing ideas, you can observe people using your software (where it is deployed) and then form hypothesis of testing ideas.

Improve this answer
1
  • Thank you. By enterprise, I mean it's used within a large enterprise. Some VP at some bank bought it, and force it down to everyone throughout the company. So the buyer is not the actual user, and the users are forced to use the software (it is actually on the cloud). You're right, simply printing it out and asking will work, it helps and i often do it, but i'm just wondering if there's a way to validate tests not on site (since this cannot be done easily because of the hierarchy between our buyer and our user... also, the buyer will factor the cost of providing paid employee for us)
    – CleverNode
    Jul 10, 2015 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.