How are you drilling the hole? What type of drill bit are you using? I always use a Forstner type bit.
Are you drilling the hole while the blank is mounted on the lathe in a 4 jaw chuck? i would think that it would be very difficult to drill an off center hole this way.