I wouldn't replace the pitman arm just because the idler arm is worn. I would replace the pitman arm only if it was worn.
The manual says that on a 4wd you need to remove the steering gear to replace the pitman arm.
I may be wrong, but I seem to recall unbolting the steering gear from the frame to adjust the position of the steering gear enough to get a puller on to the pitman arm and remove it without taking the steering gear completely out of the truck.