I think what you're looking for would fall under the "No Keyboard Trap" (SC 2.1.2) of WCAG 2.0:
"The intent of this Success Criterion is to ensure that that content
does not "trap" keyboard focus within subsections of content on a Web
page. This is a common problem when multiple formats are combined
within a page and rendered using plug-ins or embedded applications.
There may be times when the functionality of the Web page restricts
the focus to a subsection of the content, as long as the user knows
how to leave that state and "untrap" the focus."
Basically, if a user can ignore the dialog and interact with background items using a mouse, they should have the same freedoms using keyboard commands as well.
As with a lot of usability issues, it depends on the case. It would help to understand what your use of a non-modal dialog is in order to provide a more comprehensive answer.
Source: http://www.w3.org/TR/UNDERSTANDING-WCAG20/keyboard-operation-trapping.html