I would omit the declarative yes/no step before address entry.
But unless such constraints apply, simply present the fields necessary to enter an address voluntarily, and identify those unambiguously as optional. This way, if your users do wish to provide an address - consider their privacy concerns - they can do so by simply filling in the necessary information. Otherwise they may just skip the address step - a choice your application should noticeably tolerate.
That clears up the conundrum of needing to cache address information (and then repopulating it) through potentially multiple changes of mind by the user (Yes... No... Yes... er, actually, no!)
Consider presenting a 'Clear all' button in context with the address field block, which allows your users to erase collectively any address information they may have started to provide but then decided against disclosing.
The following may be off topic, and whether you choose to implement it should depend on the specific business case your site seeks to satisfy, but it is a hallmark of good attention to UX detail: When addressing a global audience you may want to consider that address formatting is not uniform across the world, and with that, user habits of writing an address may differ internationally.
You can accommodate that characteristic and make them feel at home.
A good pattern to follow is iOS's (and likely also Android OS's) 'New Contact...' workflow, which decides whether a postal code appears as suffix to the location information, as is common in most English speaking countries (e.g. London WC1A 2XY / New York NY 10016), or as a prefix, (e.g. 75016 Paris / 10159 Berlin etc.) as is customary in mainland Europe and parts of South America.
It is easy to do that when the country choice - even though located at the bottom of the field block - is presented as a drop list, with all preceding fields left void, so the country choice attracts the user's attention first. The same goes for the order in which house number and street name are written, and so forth.
Selection of a specific country causes the postal code - location, and house number / street name fields to trade places as applicable. This should also work if those fields have already been filled. It may seem like a mere nice luxury at first, but to my mind steadfast attention to such details truly distinguishes great UX.
I hope that helps. Best of luck!