4

Our client is in the process of doing regression testing on an app for which we did the UX part. Obviously, we took the previous version in consideration, so we have a lot of insight about it and I think we could help with this since usability changes were quite a few (previous one was a mess).

However, the client says that the entire process of regression testing belongs to development area, including usability, which sounds really weird to me since most of the testing will be performed with automated tools and the manual parts mostly fall under our domain

In general, I'd agree with the client, but based on my experience with those developers (which, by the way, designed the UX for the previous version) and the fact that there are many changes, I think we should at least overview the regression testing.

Am I wrong about this? Should regression testing include UX or not? And as we're at it, what's the name for regression testing in UX?

  • As I see it, then, the goal of UX regression testing would be to expose things like a design pattern change which was not uniformly implemented across an application? – msanford Oct 31 '16 at 16:54
  • 1
    @msanford more or less that. Or for example, if a change of UX paradigm that includes technical changes affects usability for previous users – Devin Oct 31 '16 at 17:26
  • 1
    I agree with you Devin: automatic tools might find something has changed, but they aren't going to understand WHY you changed it. – PhillipW Oct 31 '16 at 19:21
2

http://www.usabilityfirst.com/glossary/regression-testing/

after repairing a problem identified in a previous test, re-testing to confirm not only that the problem has been fixed but that no other problems have been introduced in the process. Regression testing is common when fixing software bugs but is even more important when fixing usability problems which may not have any obvious fix and whose repair may often affect other aspects of the design.

  • With all due respect to usabilityfirst.com, I'm not sure how much using a term so strongly associated with system-centred development and applying it to UX is wise given the terminology mess UX is already in. Nor is this definition comprehensive or super clear. Am I the only one troubled by the informal scent of this definition? "May often affect"? – Izhaki Oct 31 '16 at 23:17
  • @Izhaki Regression testing is predominantly a development thing, where the whole system is retested after all the code changes have been applied. However, there is no reason why you cannot redo all your usability testing during the regression testing phase too, especially if your initial usability identified issues, which led to code changes. After all, you want to ensure the usability issues went away, and you want to ensure the code changes didn't introduce new usability issues. – SteveD Nov 1 '16 at 9:53
4

The 'formal' definition

IEEE 610.12 defines regression testing like so:

Selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements.

The definition caters for both hard (bugs, FR) and soft (performance, NFR) goals. No mention of automation.

In everyday language

My own experience alright, but for developers regression testing typically means a suite of tests that runs against the system to make sure that:

  • Changes don't break existing behaviour (often unit tests that are executed with every code change).
  • All requirements are met (comprehensive, component, integration or e2e tests that run pre-deployment).

Regression tests are not necessarily automatic. The manual work of a QA person often constitutes regression testing.

Regardless, I've never seen a single 'regression' test that tests for performance - the system taking 17 seconds to load from Hong-Kong is good to know but not on par with 20% of users getting javascript errors that render the system useless without refresh.

Sure, the performance of a system is often monitored 24/7, with alerts being sent if a threshold is undershoot, but these are not termed regression tests.

To the best of my knowledge, in UX the type of tests you are talking about are called Summative evaluation (against a set criteria or metrics). You can argue that summative evaluation is a form of regression testing, but mind you that this is only valid if there is a set criteria or previous metrics to test against.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.