Minimal difference, period.
In the old days, we had steel guns and wooden grips/stocks. Plastic grips became a thing, and people argued about it. When aluminum came around, it was like the Hatfields and McCoys. When Glock started flooding the markets with polymers, that's when the stink really hit the fan. Don't even get me started with the machined vs MIM controversy. It almost seems as if gun guys are just looking for something to argue about.
But you know what? Whenever I hear someone whinging about how they don't shoot this type of gun or that type well, I ask myself how Jerry Miculek would do. If the answer is Jerry would blow through a magazine in two seconds and every shot would be right on target, then we can only conclude it's not the gun. The complainer is just a bad shooter.
The fact is, steel, aluminum, and polymer are all great materials for making guns. If the gun meets all your criteria, don't get hung up on what it's made of.