Considering both these issues can be solved by the browser, or host operating system, it seems like overkill and potentially a lot of extra work, depending on how complex your site design is.
Assuming you're not using tiny fonts and really low contrast colours, I would say it's safe to assume that if a user with vision problems is having trouble reading your site, they're probably also suffering from the same issues on most other sites as well. To that end, they would probably already be using accessibility tools that would improve their situation such as magnifiers, large fonts and high contrast overrides.
What I would recommend instead is that you make sure your HTML markup is as clean as possible with proper headings, paragraphs, annotations, etc. This will not only make it easier for people to override your styles, but also make things far more accessibly to screen readers for the blind.
Finally, if you're still concerned and want to add this functionality, do so using some accessibility tools, rather than trying to implement them yourself. They're usually written in Javascript and can be dropped on top of any site. They're often go far beyond simply changing contrast or font size and you don't have to waste time re-inventing the wheel.