Sorry to jump in, but I can't really see to advantage of what you are trying to achieve. If the correct shims are fitted to the inner race, the whole assembly is clamped up solid to the crank by the retaining nut.
If you delete these shims and shim the outer race, the crank assembly will still be clamped up solid, but you will need the same shim you deleted, fitted between the crank seal spacer and the shock absorber sleeve, in order to keep your sprockets in line.
It seem to me you are trying to fix a none problem.