Yeah a short screw chuck/mandrel might well be the easiest thing. Pour in some low temp pewter or lead shot in epoxy afterwards and it's good to go. I might two part step drill it so that you have a (very) slight recess for the felt so its partially hidden..
Turning a piece just to size is imho a lot harder than just turning a piece and doubly so when the piece of wood is exactly the right length already. Certainly an interesting challenge