What is the best way of display a list of options nested within two options?I placed a mockup for reference to my question. If the user click the top radio button he should pick just one among all the options available. Space is limited to display those options but I believe the mockup is strange and would like to know how others approached this type of task.
You should introduce a 2-level hierarchy of options, for example you can add a nested list of suboptions as indicated below. However, nested lists tend to be confusing. You should try to make the main-options clearly distinguishable ("make choices distinguishable" is one of the guidelines in The UX Book).
Keep the suboptions greyed out until the first main option is selected
Furthermore, I would prever radio buttons over the drop down list in the second level, if there are only about 4 suboptions. An explanation you can find here.
In order to reduce the vertical height I have arranged the suboptions horizontally - its a suggestion.
If I look at the sketch you provided, I wonder why you don't use one single combobox for everything or just radio buttons from options 1 to 5.
So if you really want to gather the first 4 options in one sub-group, then I would suggest to put a name to this group and use it as text in the first radio button. For example, something like this:
Alternatively, you could put the combobox below the associated radio button and use progressive disclosure, to make the combobox appear only if its radio button is selected. This is often seen in web forms.
Depending on your use case, for example, if your "Option 2" radio button is representing all the options from the combobox, or maybe none of them, you may go for something like in Google Chrome synchronization settings:
If you select the "Sync everything" from the combobox (replacing your radio buttons), all the checkboxes below will be insensitive but still checked (you may use radio buttons instead of your combobox). If you select the "Choose what to sync" option, then all the checkboxes are enabled and you can (un-)check the one(s) you want individually (in your case, just one sub-option).
What kind of option do you have? and how many in the sub-group? some context would help us to help you :)
Edit after first comment:
Giving the fact that you have 7 items for the sub-group, I would suggest to use a combobox instead of radio-buttons. Here is an example:
You may replace the terms by whatever is appropriate in your context ("active/inactive" instead of "activated/deactivated", "objects", etc.).
Alternatively, you may still keep the combobox enabled even if the second radio button is selected. And you may select the first radio button automatically as soon as the user select something in the combobox. I'm not sure if there is any usability study on this... but I've seen both cases in several applications.