Your yard--if you are lucky enough to have space for a garden, or even potted plants, many of your herbal supplies become next to free. If you don't garden, basic garden herbs like rosemary, mint, sage and thyme, tend to grow like crazy when happy and their owners are usually more than happy to let you come take their trimmings away.
Walmart--While I dislike going to Walmart on principle, most of your basic supplies can be found here, and if you are doing candle magic on a budget, they have a wide variety of inexpensive colored candles.
Dollar Stores--Sometimes I get lucky with candles and seasonal items here.
Local grocery--Obviously you can get salt here--for crafting I prefer uniodized salt, but that's about as fancy as I get. For basic spell-work, an all-purpose incense like sandalwood will do. You can find this at most grocery stores these days. I have to admit though, I tend to splurge on my incense since I have to smell it for the rest of the day after use.
Seconds stores-- if you are lucky enough to have a good seconds store nearby (like TJ Maxx, DFO or Tuesday Morning) you can usually find almost everything you need to accessorize your altar. Many of my favorite pagan decorating purchases came from Marshalls. Especially after Samhain and Yule, they practically give cool stuff away!
Yard sales--I love the second-hand cauldron I rescued a few months back. Yard sales involve a lot more time and effort, but they are generally worth it since the items you get tend to have more personality.