Using blue would be a good choice. The reason the IRS uses dark blue is likely (though of course this is speculation) because it's what most pens write with. So it's a form of skeuomorphism, if you will.
You can follow suit, or you can try coming up with your own color, if you want to stick to your companies' branding colors or something. Things to keep in mind when designing your own annotations:
What color (hue) to use?
Some colors already have connotations. Red for example is often used for correction. Blue is a common pen color, but you can tweak it to suit your liking a bit more. For example:
The first is a standard MS Paint blue, the second is based on the blue from the site logo but darkened a bit to still be visible over that blue background. So you can tie in branding a bit but you shouldn't use the exact same colors.
Contrast between text, notes, and background.
You already mention that blue will have less contrast with the white. This is true, but it needn't be a problem. Contrast needn't be THAT high; these color combinations pass through WCAG (web content accessibility guidelines):
And those are quite medium grays. So yeah, you can use an annotation color that is slightly less contrasting than pure black and white.
Contrast when photocopied
Do you want your notes to be visible on a photocopy, or don't you? Many copiers won't copy red pen (or some shades of red anyway) which makes red pen excellent for notes that you don't want copied, but terrible if you want to annotate and then file a copy in archives.
It's a bit of an edge case since many photocopiers do gray scale or even color nowadays but hey - still good to keep in mind.
I've personally used this in design work where I wanted to add a splash of color to documents but wanted to keep them safe for photocopiers. Example:
When photocopied should end up looking something like:
You didn't ask but I'm gonna tell you anyway; using a different font will make your notes stand out much better:
That's using a gray which doesn't stand out much in the first case, but in the second example we can also differentiate by the shape of the letters. For handwritten notes and signatures this is a non-issue of course, but for computer annotations it's good to have a handwriting-mimicking font. Use Comic Sans MS if nothing else is around.