You've heard about the reading room at the New York Public Library, haven't you? Then I need not go on. Just look at it. And you're allowed to hang around in there!
Rudy's Pastry Shop is a little place my boyfriend and I like to go to on Sunday mornings. The place is super small and cozy and full of all the sweetest treats you can find. They have such a large selection of pastries and the coffee is just as delicious. The location is cute as well and a nice place to stroll afterward. There is a cute record store right around the corner.
Another Gramercy all-star, I really like this place, you need a reservation or early arrival, sitting in the bar is always fun and the food is amazing!