When I first visited NY in 2010 the New Museum left a permanent impression. I love the architecture and the concept of the piled-white-shoe-box look-a-like building at the edge of the Lower East Side neighborhood. The exhibition program is appealingly different, and the bookstore in the foyer is not to be missed!
I live above TriBeCa and take walks into and around this area often. It's quiet, usually on the emptier side, and hosts a few gems of restaurants including Tiny's, Takahachi, and Odeon - all on the same block of West Broadway.