qualitative, quantitative or both questions
You have to use both kinds of questions. Quantitative only is never enough for this kind of questionnaire. It may give you an answer whether users are happy with something or not, but it gives you no clues as to why they didn't like something. And without those, the information is mostly useless. You know that your application has weak points, but not what these weak points are, or how you could improve them. And a purely quantitative questionnaire makes no sense either. It would require the users to write small essays to each questions. Both answering it and evaluating the answers is so time-consuming that you lose all the advantages of a questionnaire and could go in and do interviews instead for this kind of information.
What I have found to work well is to mix both.
Divide your software into features. For example, if you have created a simple Timer/Countdown application, you could have features like
- You can have a countdown for a preset time
- You can start a timer with one click, it will run until you stop it
- You can pause a countdown and then start it running again
- You can pause a timer and start it running again
- You can only have one countdown or timer running at once.
For each feature, ask the users what you want to know, giving them a Likert scale to answer. For example, ask them "Do you like this feature" for satisfaction, or "Are you aware that our software has this feature" for discoverability (which is a part of learnability), or "do you have trouble using this feature" for general usability. Use 5 or 6 items per scale, depending on whether you are in the even or odd Likert camp (I am personally in the odd). 7 will just about work too, more than that gives you a false sense of precision, because users cannot distinguish their feelings at such granularity in this kind of question. Less than 5 is unnecessary limiting, unless you have lots of features and want to reduce cognitive load per feature so you can include more features in the questionnaire.
For each question, add an optional space for "why you gave the answers above" or however you want to word it, plus optional space for "how would you like us to improve this feature". This is where the qualitative answers will come in.
The number of valid questionnaires returned will differ depending on who the target group is and how you recruited them. But expect that in a typical valid questionnaire, all or almost all Likert questions will be answered, plus maybe 15 to 20% of the text opportunities to be answered. You can do some interesting analysis on such answers. For example, if there are lots of people who give full-text answers to feature number 3 than to others, you can assume that it is somehow more interesting to users. If users repeatedly ask for an improvement which already exists in the system (and they will do so), you know that you have to improve discoverability at that place. And so on, there is lots of stuff you can do with this kind of data.
I'm having difficulties on deciding the type and number of questions for each attributes
You have to keep the questionnaire manageable. The longest you can inflict on the users is a 90 min questionnaire, but you have to motivate them really well for this. Up to 60 minutes will be a better time to aim for. Test the questionnaire on co-workers first so you know how long it takes them. If you have a very large software, this will mean that you can't go onto such a detailed level of features as I described above. In this case, you have to cluster them well and ask a question per cluster instead of per single finegrained feature.
Wording the questions is an art in itself. A few things to keep in mind:
- keep the questions simple. Users can't very well distinguish between different aspects of a single attribute (see also an older answer of mine specifically about product quality questionnaires). Choose one wording per attribute (or sub-attribute, if they are easily distinguishable for the user). For satisfaction, don't ask "did you like it", "is it useful", "is it nice", etc. Pick one and stick with it. Similarly for the other attributes.
Only ask a question a user can really answer. Don't ask them, "Do you think this feature has high learnability", they don't think in these terms. Ask something like "Was it easy for you to start using the feature", or "How long did it take until you were comfortable using that feature".
Look at the type of questions normally used for measuring such attributes. There is lots of research out there. The usual generic questionnaires are SUS, QUIS, TAM/UTAUT, and the venerable user satisfaction model started with Bailey in the 80s, whose latest incarnation is the revised Delone & McLean model. You can find the publications about these on the ACM portal, may need paywall access to some of them.
For several types of information system, there are field-specific ways to measure satisfaction and usability. The largest is probably PARADISE for dialogue systems, but there are also specific approaches for other fields, you may want to search for them.
A version of the approach I describe was published by Joerg Doerr around 2007, I think it was in the IEEE RE conference proceedings, but not 100% sure. He uses something spreadsheet-like instead of a questionnaire, this may be more convenient for a number of repeated questions per feature. Look up the paper, it is interesting.
Beside the questions on per-feature basis, ask for some basic data qualifying the user, and later make an analysis to see if the answer distribution is the same across all types of users (which is unlikely), or if it differs, and how it does differ. Important variables tend to be the user's tech affinity, their domain knowledge, and how much experience they have with your system. For a really good analysis of the results, you have to look a bit deeper into multivariate statistics. But if you just want to know a few trends, even visual comparison of the histograms for different groups can be enlightening.
A note on the analysis: you will see a ceiling effect. The typical histogram on the Likert scale will be a Gauss curve with the median shifted towards the favorable answers, sometimes so far that you only see one slope cut off before the actual cusp. This is not a problem if you account for it.
And another note: don't forget to treat the data as ordinally scaled! Don't try to calculate a mean, use a Spearman correlation if needed, etc. If you don't know how to do this, pick an introductory statistics text. I have seen lots of statistics newbies merrily treating Likert data as cardinal data, which looks very good on the surface, but in fact the results are dangerously misleading.
Good luck with your analysis!
Jakob; others cannot correct it because there is a minimum requirement for 6 changed characters for an edit to be considered valid. It's up to you to fix it.