Richard, I had exactly that issue too. Also the outer rim of the pulley interfered with the bottom of the casing, but won't talk about that here.
From memory, I considered realigning the idler gear/shaft but it would've affected the alignment of the pulleys, so just carefully sculpted some off with the Dremel, and also bevelled the corresponding inside corner of the pulley a bit.
Also from memory, I ended up with~ 0.020" side-float on the idler which I decided wouldn't matter too much as, and be 'self centred' by the belt/pulleys, having decided too risky to use a shim that thin on something spinning.
Felt/ cork washers were too thick, so left them out.
Have only done about 3K miles since, but all seems ok so far.......next oil change will try remember to have a look-see.
Have photos buried somewhere, I think I posted them in a different thread at the time .....!
Edited- To free the taper, I slipped a couple of strips of flat bar behind the pulley to pull it against the case, before tapping the loose nut on the shaft...works a treat- no heat needed