Your problem is now you have an illegal IRA account. You can take the chance that IRS will not notice and go on pretending its a legal account, but the problem is at any time, the IRS can discover that its illegal and essentially claim all the funds or a major portion of them in taxes, penalties, and interest. That could happen years from now when you are close to retirement. Is it worth it?

Better to remove the funds, pay the taxes and penalties so you don't have to worry about it. That is especially the case if the amount involved is small. If its large, then you can hire the pros you need to get it resolved. But it may still not be worth it.