It's about ROI
Whether it is the project manager or yourself that sets UX priorities, some cost/benefit analysis has to be made.
Nearly all systems will involve many more end-users than admins, so it makes sense to cater for end-users more than to admins. There's nothing wrong with that - it's a wise economical choice.
Remember that UX and development time cost the business money, so it is in the business interest to maximise return on investment.
If we narrow UX to usability only (a measure of performance load) than you'd generally prioritise the task with most impact; that is, the task that will reduce performance load the most, all users taken into account.

Ways to combat the low-priority neglect
It is common for low priority tasks to stay in the queue forever and never get done. While perhaps justified, these do bother some stakeholders, for whom a particular task may be highly important.

Another important point, sadly often overlooked, is that project managers tend to prioritise high-value tasks, ignoring the fact that low-value tasks, once these accumulate, can offer more value combined than a single high-value task. A lot of small UX issues can result in performance load penalty that is higher than that of a single UX issue. That's another argument of why you should not neglect lower-value improvements.

While largely a matter of project management, companies do employ various strategies to combat this. Here are a few selected ones:
Petition the king
Based on this metaphor, Petition the king is a session (say weekly) where various stakeholders (employees) come to the king (typically the project manager) to ask for certain things to get done.
The king may reject their request, explaining the grounds. But if someone comes week after week with the same request, the king may grant that person his/her wishes.
Club system
Based on another metaphor, consider the 3 queues to a fashionable club, in order of business priority:
- VIPs
- Ticket holders
- Non-ticket holders
VIPs go in straight away, ticket holders get priority over non-ticket holders. But the club still has to let people in from each queue (albeit with different throughput) or ticket holders will get upset or non-ticket holders will give up.
Similarly, some companies allocate:
- 80% for high-priority tasks
- 15% for medium-priority ones
- 5% for low-priority ones
Outside UX, this is how low-priority bugs may ever get fixed.
Independence days
Some companies (eg, Google) allow employees to spend some of their time on whatever they wish, so long it has some benefit for the business. A 10% independence allocation may be taken as 1 day a week, or one week every 10 weeks.
It is during these independence days that employees can do work that is important to them, increasing both motivation and job-satisfaction; it also allows employees to break from management constraints, which may lead to valuable research, skunkworks, or other tasks which would otherwise never get done.
To get your wishes over the line as a UXer, you'll have to get a buy-in from one or more developers, but help works both ways.