Sorry to harsh your “the explanation must be fundamental” vibe, but I have some new information.
My students have been working on inelastic collisions and I was reminded of something. An elastic collision between two objects is a system of equations:
mv + mv = mv + mv and
.5mv^2 + .5mv^2 = .5mv^2 + .5mv^2
Where there are 4 different(maybe) v’s and 2 different(maybe) m’s, and we know the starting values for all.
This system is a circle and a line which has at most two solutions. They are typically the starting velocities and the ending velocities. For two objects of equal mass, the objects will always trade velocities, it is the only possible solution.
From there, we can consider Newton’s cradle to be a system of collisions that happen one at a time. This is true because maybe there is a small space between the balls and definitely because they are not perfectly rigid, so it takes time for them to begin moving once struck. In any case, you end up with a series of moving balls striking still balls and becoming still. If you trace this motion until it reaches the other side, you arrive at all the results that we actually see.
I think it is possible that in a perfect system, some of the weird solutions that we found could happen? And that the reason they do not, is because our imperfect system does not allow all 5 balls to be colliding together at the exact same time.