Preheat the oven to 170*C/330*F/ gas mark 3. Butter a 9 X 4 inch loaf tin and line with paper. Butter the paper.
Rub the lemon zest into the sugar until very fragrant. Cream the butter in a bowl along with the lemon sugar until light and fluffy. Beat in the vanilla and then eggs, one at a time. Stir in the caraway seeds. Sift the flour over top and then add the ground almonds, folding all together well. Stir in the milk. Spoon into the prepared loaf tin.
Bake in the preheated oven for 50 minutes. It should be well risen and golden brown. A toothpick inserted in the centre should come out clean. Let sit in the pan for 10 minutes then turn out onto a rack to cool completely.