Now, let's say you lose your job before the money plus interest is paid back. You still have to pay back whatever the remaining balance is on that loan. And if you don't have the money to pay it back, it will be taxed to you as ordinary income. That's in addition to the 10% penalty if you were under the typical retirement age of 59 1/2.