I'd give a bit of heat on the inside of the drum, no need to go berserk it should come relatively easily..don't think there's much other choice than to tap the inner race...should be ok if you hold the drift square and firm and don't hit the cage? Obviously the harder you(have to) hit it gives a greater chance of potential damage. I assume you've removed the circlip?
Good luck