Troy, you really should give mini racing a consideration. Remember, minis and pocket bikes are 2 different things. I have more room on our CR85 than I do on any sport bike, and I'm only an inch shorter than you. You'll have just as much fun, if not more, at a mini track day or race than you will on a big bike. You can push the limits to their extreme and not worry about shredding the bike up or yourself. If you want, you can come along with me next time we go. If we do a SEMRA event, you can ride our bike in the adult novice class. Ask Kaneman, as soon as you get on one you'll contemplate selling your other bikes. LOL Our current setup has 17" wheels, uses the same slicks as the 125GP bikes do. Though, I'm getting ready to sell/trade them off for a set of 12s. The 17s will make you feel just like you're on a "big" bike...but with the flickability of a 180lb bike.
Our bike is fully setup by Traxxion Dynamics and I've done a lot of work on getting the geometry setup to perfection. All you need though is an 85 and the wheels. Total investment would start at around 2k, and you can get a lot more conservative than that! I think there are a few WELL SETUP bikes out there now for about 2500. Heck, right here is one of the best bikes around...2200 dollars.
http://www.supermotojunkie.com/showthread.php?t=83310
All I can say is...try it! I think we might be doing a mini race/trackday coming up Oct 3&4th.