I have a vague recollection of giving similar feedback but was unable to find it, though perhaps it was on the beta calendar forum...
My suggestion is to be able to click the month name in the small calendar and have a drop down appear there, as well as a drop down for the year, similar to the small calendar you get when picking an end date for a recurring event. Then you can click on a date in that calendar to jump to it on the large calendar.