Preheat the oven to 350°F and lightly grease a 9-inch square loaf pan.
Beat the butter with the white sugar until smooth.
Mix the brown sugar with the cinnamon in another bowl, and set aside.
Add the eggs to the butter mixture, one at a time, beating after each one.
Add the vanilla.
Sift the flour and baking powder together, then add this bit by bit to the butter mixture.
Stir in the milk until smooth.
Pour half of this mixture into the loaf pan.
Arrange half the apples on top.
Sprinkle with half the cinnamon mixture.
Press the apples into the batter gently.
Add the rest of the batter then the rest of the apples and finally the rest of the cinnamon sugar.
Swirl the cinnamon sugar using a knife.
Bake for about 45 minutes or until a toothpick inserted into the center comes out clean.
Let the bread sit for 10 minutes then turn on to a wire rack.
Combine the milk and powdered sugar and drizzle this over the apple fritter bread before serving.