Could be a couple different things. A lot of times to get a good even bur going you're going to have to polish the edge and then the face of your card by wet sanding with a fine grit. If you skip this, you might be skipping across the high spots with your burnisher.
Another problem could be that your current burnishing tool might not have enough texture. You can always grab a honing steel from the kitchen and give that a try.
On a separate note..... How do you like the diamond plates for honing? I feel like the diamond plates I've used in the past take off too much material for frequent use.