You're lucky to have ability to deal with this right from the start. Once I had to rework complex news website to get (WCAG) 2.0 accreditation and it took me close to 9 month to complete this project! It was incredibly challenging when done at such a late stage after the release.
slugolicious, had very good suggestion. Try testing your users or yourself doing complex tasks using calendar just with keyboard.
I'll try to suggest you few improvements when it comes to dealing with screen readers or accessibility in general.
One of the most important features for keyboard navigation and screen readers is the ability to skim through content
- Ability to have a data summary in the header of each day, and perhaps allow user choose if he would like to skip this without going thru every single item inside the table.
- I didn't see ability to navigate forward or backwards within calendar using arrows left or right. Perhaps you didn't want to include this, but if it's been forgotten think how frustrating would it be to navigate to the next month even with ability to skip thru content.
- Scenario when no data present in the calendar, think about what information should be displayed to cater user needs.
- Scenario when user reaches the end of the calendar, or no more data loaded, provide ability to display a link and allow user to choose what he would like to do. Invisible Content Just for Screen Reader Users
Note: content should only be hidden from sighted users and made
available to screen reader users when content is apparent visually,
but not apparent to screen reader users.
You also didn't mention if this would be used for touch devises? :) NN group have some useful tips in this article "Screen Readers on Touchscreen Devices"
Summary: People who are blind or have low vision must rely on their
memory and on a rich vocabulary of gestures to interact with
touchscreen phones and tablets. Designers should strive to minimize
the cognitive load for users of screen readers.
More info when designing for screen reader compatibility
Links
One way is to use the Tab key to jump from link to link. This gives the user an idea of where the page links to, and can be a useful way to run through the content if the user is looking for a specific link. A related technique is to obtain a list of the links on the page, arranged alphabetically. The drawback of these methods is that the user does not hear any of the non-link content, and may miss important information this way.
Implication: Links should make sense when read out of context. Also,
the distinguishing information of the link should be at the beginning
of the link.
Headings
Another way to skim the page to get an overall impression of a page's content is to jump from heading to heading. Users can hear an outline of the page's main ideas, then backtrack to read the parts they are most interested in. The main drawback to this technique is that too many pages lack headings. Without headings, this method of skimming through content is completely useless.
Implication: Authors should organize content with headings. To the
extent possible, the headings should represent an accurate outline of
the content.
Paragraphs and Page elements
Users can jump from paragraph to paragraph, listening to the first sentence or two before moving on to the next paragraph. This technique is most like the visual skimming technique used by some sighted people. Users can also jump from element to element, such as tags, links, form elements, list items, or other units of content.
Implication: When possible, place the distinguishing information of a
paragraph in the first sentence.
"Skip navigation" links
Skip links at the top of the page which allow users to skip over the navigation links aren't exactly a method of skimming the content, but they are a method of getting straight to the main content of the page. Such links speed up the reading process and help users distinguish between the main navigation and the main content.
Implication: Where appropriate, allow users to skip over repetitive
navigation links.
http://webaim.org/techniques/screenreader/
This is just a sketch of how the navigation is supposed to look like. More info about each appointment will of course be available. 
