Hi Fallback. I too got exasperated at the pirouetting of my units. I have recently begun a conscious tactic of placing my units further apart and not crowding them. An ACW battlefield had big gaps in it where there were no units. I have found since doing this that my regiments tend to get confused far less, so as Con20or says, spaace your regiments out more.
Also when manouvering them about use road column or column of divisions and only order them into line when they are in the location you intend them to fight.
Yes, a line that is shaky and looking over its shoulder can collapse and retreat when it is ordered to fall back. I find this feature one of the most realistic in the game as a number of accounts of men in battle from various wars will support this. Men tend to stay put for a long time but any uncertainly by their officers, confused orders or a sudden order to fall back can make shaken units collapse. You should try to rotate your units out of the firing line and replace them with fresh ones when you get the opportunity. My method is always to keep a reserve, often just 1 unit in a brigade but 2 unengaged regts if I can.
I rarely see the extremes of response to fire that you describe. The AI routines for calculating morale and combat effectiveness seem reasonably subtle in the game but I suppose you can get extreme results as you describe but I should think what you are seeing there is the far ends of the spectrum. I once saw the 20th Maine stand in the open and get bravely (but idiotically) shredded by two enemy brigades but such foolhardy performances are rare. I don't ever recall seeing a unit defending a wall run so easily, unless they were low quality troops (conscripts).