Don't confuse home ownership with investing. A home you and your family live in. Its not an investment to flip or sell in the future. Its not a bank.

When the mortgage is paid, all you have to worry about is taxes and upkeep. You can't lose it otherwise. You can sleep at night.

Think of the worst case scenario. You take out money or refinance. You then lose your job or get disabled. Then you find out your investments have lost 50%. Will you be able to make the mortgage now? If the house was paid for, you wouldn't be quit as worried.

JLC