The remaining money would go to the mortgage with a small amount left to play with.
...
I could probably pay the mortgage off in about 4 years.

As has been pointed out here before, making double or triple mortgage payments does NOT relieve you of the obligation to make the next month's payment. If something happened you could be evicted with only a few months payments left.

Methinks it would be better to put those extra payments into your E-fund where it's available for whatever you decide to do with it later.