Super-gentle swabbing with a Magic Eraser? You could control the paint break-through that way and not rub too much off (maybe).