Ask around for local clubs in your area. Any local kayak dealer will know of one. Also ask how welcoming the clubs are to newcomers, and gravitate to the one with the best rep. Then, go to a meeting and volunteer to do something for the club. Every club needs volunteer help at something, and you'll instantly make new friends.
As for gear, rent what you can for now (boat-wise) to get a sense of what works best for you. Try different boats to see how the varying fits and hull designs affect the performance of each and to see what suits you best. Helmet, PFD (personal flotation device), and stuff like that you'll want to buy; REI and L.L. Bean, among other places, have lots of good, affordable gear. You can also find stuff on eBay, of course. And those new friends of yours in the local kayak club are bound to have gear they're selling. In time, you'll likely even find a boat through club members who are "upgrading." So keep your eyes peeled and ears open.