I tried a flat head and a little hammer and the thing isn't budging. Do I have to get a special tool?

 

Drift punch and big hammer (4lb sledge here). I don't call either a "special tool". OTOH, they do make a special tool to help; https://www.summitracing.com/parts/skg-21835/overview/

I don't remember if I had to grind one end off or not first.

If so, then you'll need an angle grinder or a Dremel-type tool also.

RwP