I'm not sure how many people actually want this, but since it was a little finnicky to figure out what exactly I had to change, I figure I may as well share it and save a potential headache for someone:

a#mark_all_link.updater_link {
display : none;
}

If you wanted to make any other changes that are specifically to the "Mark All As Read" button (and no other buttons!) then a#mark_all_link.updater_link is what you'll want to refer to.