It's hard to tell from the picture but would you be able to get your drill inside the pot to ream out the rough spot? If not a regular drill an angle drill should work.
If you don't have an angle drill check with Home Depot about renting one for half a day.