Got it out. That was ridiculous. Went through about 6 more drill bits. Even after the remaining piece of the ball joint was out, one of the rivets still wouldn't budge!
Had to drill all of them out from the bottom, which sucked; barely enough room for the drill under it. Once I found a decent 1/8" bit in the bulk pack I bought, it drilled through nicely.
For anyone in the future referencing this, load up on the 1/8", 3/16", and 1/4" bits. If you think 1 pack of 5-7 each will be enough, buy 3!

 

Thanks all for the advice!! Looking forward to doing the driver's side. (NOT!!)

 

Well, at least from this point on, they'll be bolted in, and a piece of cake to change!

 

Hopefully that was just a isolated case - I have never had that much trouble and have always been able to pop the rivets out with a punch after the rivet head was removed - However I don't air chisel the heads off I zip them off with my 4.5'' hand grinder.
Best of luck

 

Did the driver side last night. Still had to drill the lower ball joint rivets, but only from the top, maybe 1/4" deep; just to get enough out of the control arm mounting holes. Three of them began rotating slightly as I was chiseling them, so I figured it would be an easier go than the passenger side. Took about 2.5hrs total to replace the ball joints and shock.
Hopefully no issue with the upper ball joint. I load them up with some grease prior to putting a load on them. The upper was leaking grease around the top flange. Hopefully once I get a load on it, it will seal so I can fully lube it.