In small bowl, whisk together flour, salt baking powder and baking soda; set aside.
Using a food processor, blend together granulated sugar, lemon peel and lavender buds until it comes together in a fine lavender sugar blend.
In a mixer, cream together butter and lavender sugar mixture for 5-6 minutes, scraping bowl occasionally. Add eggs one at a time, blending between additions.
In small bowl, stir together buttermilk, vanilla and lemon juice. Set aside.
On low speed, alternate adding half the flour mixture then slowly pour in buttermilk mixture. Add remaining flour mixture until just combined. Don't overmix.
Pour into greased mini loaf pans, dividing batter evenly. Bake for 30-35 minutes or until toothpick inserted in center comes out clean. Let cool on cooling rack in pans for 15 minutes before turning out loaves and cooling completely.
In a medium bowl, whisk powdered sugar, lemon juice, lemon zest and buttermilk (or milk) together. Drizzle or pour over cooled lavender lemon loaves. Stir in a pinch of lavender buds. Let sit until glaze is set.