There are two likely culprits. Number one is the belt itself. It may not fit, either because it's just the wrong size or because it isn't contoured properly. You don't say what type of pack you have, but often the belt is interchangeable. You can swap it out for a different size from the pack's manufacturer, or retrofit another maker's belt to it. So if the belt comes off, maybe try that.
Number two is your clothing. Particularly if you're wearing cotton shorts or pants, moisture may be getting trapped under the belt, and wet cotton clothes can chafe badly. So wear synthetic briefs made from Coolmax and shorts or pants from nylon or one of the newer performance fabrics such as Schoeller Dryskin. Some companies make shorts specifically designed for wear under a pack belt. An example: Mountain Hardwear's Synergy Short ($78). For one thing, it's made of wicking, fast-drying fabric so you're skin will stay dry. For another, it's cut specifically to avoid pinching under a pack belt, with what Mountain Hardwear calls a "conical cut" waist. So you might give something like that a try, too.
Support Outside Online
Our mission to inspire readers to get outside has never been more critical. In recent years, Outside Online has reported on groundbreaking research linking time in nature to improved mental and physical health, and we’ve kept you informed about the unprecedented threats to America’s public lands. Our rigorous coverage helps spark important debates about wellness and travel and adventure, and it provides readers an accessible gateway to new outdoor passions. Time outside is essential—and we can help you make the most of it. Making a financial contribution to Outside Online only takes a few minutes and will ensure we can continue supplying the trailblazing, informative journalism that readers like you depend on. We hope you’ll support us. Thank you.