At face value, a pack is a good place to look for excess weight. Some big packs weight six or seven pounds-as much as most tents. But, keep in mind that the pack isn't simply weight, it's active weight that does a great deal of work in supporting the main load. If it fails to do so, then you'll have to deal with poor weight transfer to the hips, less lung capacity as the pack collapses and constricts your upper body, and all-around discomfort.
That said, if the load is light, then the pack has much less work to do. These days, manufacturers have taken two approaches to developing lightweight packs. Mountainsmith, for instance, starts with a "regular" pack then does all it can to reduce the weight. The result: A pack such as the Auspex ($259), which has 4,200 cubic inches of capacity and weighs three pounds, seven ounces. It carries extremely well for a light pack, and can handle even 40-plus pounds if need be.
Other makers have tried to completely re-think the pack, starting from a blank page then adding the fewest things possible. GoLite takes that tack with the Breeze ($85), which, when you get right down to it, isn't much more than a bag with two shoulder straps attached. It's super light (just 15 ounces!), but there's a penalty: even the GoLite folks don't recommend much more than 20 pounds in the Breeze.
So there you go. My own preferences are for the pared-down big pack approach. But plenty of ultralight travelers swear that shaving a few pounds from the pack itself is the way to go.