Don't drill into the cavity wall and fill it with rock fibre.
Don't line your inner walls with polystyrene tiles.
Just crochet and knit yourself a house cosy.
It might not be quite as efficient but it is certainly more attractive to the eye.