I'm also a travel literature fan, and besides the shelf of Lonely Planet books, I've got a shelf of travel/philosophy/diary books. Long ago and far away I read (and loved) Mathiesson's books that are heavy on the contemplation. I tried to read the diaries of an ex-school headmaster from the UK that sat on a bike and circled the world, but the style just didn't suit me, so I quit trying to read it. Che Guevara's diaries are also a big hit lately, and are actually fun reading. When researching for a translation project, I read Benedict Allen's 'Into the Crocodile's Nest', which is ok if you like adventure travel stories. How much of it is true, I'll leave to the experts (and the topic of nonfiction that proves to be fiction).
Lately, though, most travel books that I read are originally written in Croatian, so I can't really help you with suggestions. It's interesting how such a small country could have such a rich travel and travel-writing culture. Hopefully a few of these authors that I highly esteem will make it in the English speaking world one day (but that's for the literary translation part of this forum...)