That's wrong, because you can (if you wish) buy a $10,000 Treasury bill that will bring all your interest payments right back to you--thereby, in effect, taxing yourself to pay off the debt right away, and limiting your damage to the initial $10,000.
Most people do not buy Treasury bills.