It's like people didn't get the hint. The longbow, then the pike formation, then the firing line, it finally took the goddamn machine gun to convince people that charging on horseback had finally become untenable.
They didn't get the hint because, until the first world war, cavalry were still indispensable. Take a look at the American Civil War, for example. Both mounted and dismounted cavalry not only managed to change the results of infantry battles, but also to have strategic effects on campaigns - for example, Grant's first attempt to take Vicksburg was frustrated by a cavalry raid on his supply base.
One of the mistakes people make is in thinking of older methods of warfare in modern terms. The reason cavalry continued to be used so much isn't that it was so romantic, but that, until the advent of the WW2 tank, truck, and half-track, (and I do mean World War 2) there simply wasn't any way to move fast around the battlefield. Even in World War 2, the Germans often relied on horse transport. The Russians would have had to as well, save for the industrial might of the United States, which shipped them thousands of vehicles. And the Japanese just marched everywhere, for the most part.
If you want to understand why people stuck to cavalry, study the American Civil War. Using Cavalry, mounted or dismounted, provided a singular benefit: that of getting there first, with the most. In days before our own, transit times were measured in weeks, and your estimate of when you might get there could be off by as much as a week. In that situation, four-legged transportation could ride around an army, cut supply lines, charge a vulnerable flank, or hold a position until reinforcements arrive - as Buford did at Gettysburg, holding the Confederates until Meade brought his army up.
And don't knock medieval cavalry too much. Yes, pikemen could hold a charge. If they stood firm. But I've been around the kind of animals that knights rode. They are not the sleek thoroughbreds that everyone thinks of when they think of a horse. They are more like draft horses, huge, bulky animals - and when one of them gallops by, the earth shakes, from fifty feet away. I should know, I've been around them and my family owned horses - still does, although I don't. It is one thing to talk about pikemen in a comfortable chair in front of a screen - it is another thing to stand, shivering on a field, as you hold a thin pole between your hands, as the charge in front of you grows to fill the horizon, and the earth itself betrays your feet.
The reason Swiss Pikemen were so remarked upon is because they wouldn't run. Most did. And most knights were not fighting Swiss Pikemen.
People in the Middle Ages and the Victorian Era were not stupid. They used cavalry because, on the whole, it
worked. The reason we remember things like Crecy is because battles like that were the
exception, not the rule. Two thousand years from now, historians may be chuckling over how the idiotic Chinese Republic thought tanks could defeat Korean mech infantry armed with railguns. But the reality will have been that the reason those hypothetical future Chinese Republicans used tanks is that, until the advent of the infantry-portable railgun, tanks
worked.