As for fleece jackets and the like, there's absolutely no point in carrying a spare. The stuff dries quickly, and it's overkill to have an extra when you're backpacking.
Still, for summer use there is one thing I like to do, and it doesn't add much weight to the kit: I pack in a Patagonia Silkweight T-shirt ($30; www.patagonia.com), plus bottoms made from the company's same Capilene fabric ($32). I try to reserve this setup for nighttime, although of course it can come in handy during the day too. But having clean sleepwear does help keep the sleeping bag cleaner, plus it's just a little more comfortable. And the light Patagonia stuff weighs just ounces and takes up as much space as a pair of socks.