I know that we should answer based on your requirements, but in this case, there is a chance your requirements are wrong.
Let me explain.
there are 20 priority levels.
You can not have 20 priority levels!
Here are some reasons.
1) Categories
Prioritize means to give categories so humans can detect and compare, and then do some actions based on that value of rank, urgency, etc.
Humans struggle if you have more than 6-8 categories. Of course, you can have subcategories but for special purposes like searches and classifications.
Imagine the priorities translated to words.
- Extremely high.
- Very high.
- High
- Not that high
- Medium-high
- Medium
- Medium-low
- Low-High
- Low-medium
- Lower than the previous one...
I just have 10 and it is not logical at all.
2) Adjacent Colors
Yes, humans can perceive a lot of colors... But when they compare them together.
You can have two shades of green next to each other and see if one is "lighter" than others.

But let us assume you need to scroll down a bit. Now you have another similar green... But I'm pretty sure you will have no idea if it is lighter, darker or equal than the previous one.
3) Color names
Similar case as "1". I can barely name some colors
- Red
- Orange
- Yellow
- Green
- Turquoise
- Cyan
- Blue
- Purple
- Magenta
- Brown... do you know brown is a dark orange? How dark?
4) Colors - Lighter
"Lighter" is a tricky adjective.
Here are some circles with two lighter greens on each side... But different lighter. One is more yellowish, In painting, this shift is used to represent a more "perceptually correct" lighter.

Points 3 and 4 will render some classification complicated because the color is a three-dimensional model. And priorities is a unidimensional one.
using shades of a single color
Let us assume you use one single shade of red.
Here is a uniform gradient bar with a small 10% width bar on each side. One pure red and the other pure white.

You can barely see the bar on the left but you can see a lot better the bar on the right.
Human perception of light is logarithmic, not linear. You notice better the difference between #FFF vs #FEE than #F00 vs #E00.
But combine that with point 2 and you can not see a lot of these differences.
- Light light red
- Just light red
- Red
Conclusion:
1) See Dave Haigh proposal of just using 4-6 colors.
2) You have alternatives for the other sub-levels of "priorities" like position. The top ones are more important than the lower ones.
Is this square lighter, darker or is the same as the one on the top?

Edited:
Coxy commented something interesting. Red-orange-yellow is somehow a well-established sequence. But if you have for some reason just green and blue color probably it is difficult to say which has lower priority.
Probably making a yellowish green can help instead of dark green.

Here are some other options for color schemes.


This last one is probably better for some types of daltonism.
Edited sometime later: One interesting case on how colors are not the clearest example of priority is that inclusive DEFCON has priority 4 and 5 colors mixed. Sometimes 4 is blue and 5 green and sometimes is the other way.
https://en.wikipedia.org/wiki/DEFCON