Does the special weapon rule for hammers also apply on a 2 handed hammer (like thrain)?

Yes it does, but you still get the -1 on the duel roll, and +1 on the wound roll.

So if you win te fight and knock someone up you get a +3 with the 2 handed weapon?

No, all hammers do is knock someone down, you do not get a chance to wound them (that turn) at all. A two handed weapon confers a +1 to the roll off between the two combatants.

I.E: A strength 3 model with a two handed hammer wins the duel roll and bashes his opponent, the model with the two handed hammer adds 1 to his dice roll in the 'roll off' between the two models.

The only way you could get +3 on the roll off is if your model has three strength higher than the enemy, or has two strength more than the enemy and a two handed weapon.