It doesn't interfere with the drivability of the vehicle at all but it's annoying cuz I like things to be where they are supposed to be. Any thoughts on how I can get it to stay? Also, what is this piece called? I've been trying to Google it with no luck.

 

The FSM calls it the Knee Blocker. It looks like you are missing the ears that set atop it where you'd screw it to the dash frame. They are at the outer edge on each side and that tab near the steering column goes into a slot.

 

it appears as if there are four of those little tabs, but one is missing. What's the FSM?

 

Factory Service manual.
There are a couple in the FAQ section you can download or you can look online. Rock Auto sells them from the original printer but I think its on CD

 

#5DL83PD5 COVER, Steering Column

Mopar # and description above

 

Those Tabs on the top slide into the corresponding holes in the dash. Unbolt the screws, insert the tabs on top, then screw in the panel.