While the wear happens where parts actually rub they end up leaving little shoulders on either side, these break off to produce visible swarf as will little bits of threads. Hard faceing can come loose and flake off and the spigot on the bottom of the cylinders can crumble a little at the edges. Slightly larger bits can break off the bearing cages.
Then there is all the flotsum & jetsum that seems to be sucked into the engine every time you open it up. It is amazing just how long little bits of crud can take to eventually end up in the sump.
I had a cage let go at one time and despite having it apart several times including flushing out the cases a couple of times with a few gallons of kerrosene ( parrafin to some) rivets "magically" continued to appear in the sump for several years.
So a small amount of visible metal in the sump is not a big deal.
What you need to look out for is this volume of swarf stadily increasing which is a sure sign that something is likely to let go or has already gone and is slowly crumbleing away.
An A65 that had the main spin in the cases yielded enough metal to cover an Australian shilling ( about the same size as a US dime ) with every oil change and it did this for three years before I finally decided that I was pushing my luck too far and retired the bike .
A WDB40 that I owned had a cage on the big end roller break up and slowly fall out over a period of 18 months so the weekly oil changes contained a lot of visible metal although this engine did sound like a shaking can of bolts at low revs because the now uncaged rollers would bunch up at low speeds allowing nearly 1/8 " of free travel of the con rod , yet another of the same model did the same thing but this time the bits did not fall out and got hammer welded to the journal and it locked up solid in no time flat.
The 4 way oil filtering system that BSA designed was not an accident. The gauze in the sump will only pull out lumps of metal that are too big to travel through the scavenge side of the pump. The smaller bits get pumped back into the oil tank, but the outlet screen is 80% smaller than the scavenge screen so these bigger bits, which would damage the delivery side of the pump get trapped in the oil tank as designed. The finer still bits of crap get centrifuged out into the sludge trap and the metal particles in there are usually so small that you need a strong glass the see them.