Persuasive Design is a part of Persuasive Technology and can be defined as follows:
Persuasion design is the design of messages by analyzing and
evaluating their content, using established psychological research
theories and methods.
Behavioral Design is quite similar, but its process is a bit more oriented to the user itself rather than all elements of the communication process.
To be more clear: Persuasion design might analyze the media that transmit the message, whether the user interacts or not with it. Behavioral design analyzes the interactions an user has with such media
Finally, Captology comes from the acronym Computers As Persuasive Technologies (CAPT) and is defined as such:
Captology is the study of computers as persuasive technologies. This
includes the design, research, and analysis of interactive computing
products (computers, mobile phones, websites, wireless technologies,
mobile applications, video games, etc.) created for the purpose of
changing people’s attitudes or behaviors.
How do they relate to each other?
Well, Captology and Persuasive Design are a part of Persuasive Technology (as a matter of fact, Captology is usually mentioned as a branch of Persuasive Design), while Behavioral Design may or may not be a part of Persuasive Technology since it has a different array of application (psychology, research, experimental and so on)
PS: if you're interested in Captology, I strongly recommend Stanford University Resource Library on Captology