I do this always, and even for my spliff ends (we call the cardboard bit a roach here) in a pocket ash tray.
It really bugs me when I find a rather nice smoke spot, lovely nature area, and upon my third or fourth visit there is just rubbish everywhere.
Folks have cleaned the areas, only to find them in such a state in practically no time at all.
I wish people would take more responsibility with their rubbish. It literally takes only one pocket, or a small otherwise unused compartment in my bag. It's ridiculously easy to not litter!!