I've done it before twice, it's fine as long as you apply your normal safety practices for being out in a city at night but there are two caveats: you will be very bored and you will feel like a steaming heap of dog shit the next day
(I have not been to Madrid and do not know what it is like as a city, so this is meant generally)