Once a nail is hammered in, it is friction that holds it in place. Friction is the force that arises when two surfaces are sliding, or trying to slide, against each other. If you try to pull apart two blocks of wood that have been nailed together, the wood fibers grip the shaft of the nail. The nail feels a force trying to rip it apart along its length, and we call that force tension. Your experiment can now fail in one of two ways ─ either the nail stretches and splits in half because the tension force is too large for the nail, or the nail comes loose because the friction force is overcome. The force it would take to stretch the nail is much larger than the friction forces on the surface, so we don't have to worry too much about the former. It's the friction with which we need to concern ourselves.