An -ism is a bias against someone based on their membership in a group they didn't choose to belong to. Sexism is a bias against someone because of their sex (because of the way our society developed, the majority of sexism is bias against females). Racism is a bias against someone because of their race. Classism is a bias against someone because of the socio-economic class they were born into (or were put into due to circumstances outside their control). And so on.
There is, sadly, still some pretty horrible and deliberate -ism in our society. The KKK still exists. There are people who think women shouldn't be allowed to work outside the home. There are people who think if you're gay you should be killed.
Fortunately, though, we've mostly moved beyond such overt bigotry. What largely remains are two things -- systemic or structural -isms, and casual -isms.
Systemic or structural -isms are biases that were long ago "built into" how certain components of our society works. In most cases, they aren't deliberately being perpetuated; rather, change is hard and slow and requires significant effort by people who care strongly about equality. By way of example, there are drastically fewer women in technology fields, not because there are a large number of people trying to keep them out, but because our social and educational systems tend to be built in a way that discourages girls and women from developing and pursuing the interest. And a host of other reasons, too.
For another example, there still are fewer people of color graduating from college than you'd expect based on the proportion of the general population that are people of color. A big part of this is that people of color are far more likely not to have access to the resources that encourage them to attend and prepare them to be successful in college, in part because the 50 years after the Civil Rights Movement isn't long enough to fix the historical disadvantages (e.g. people of color are more likely to be poor, because the most common way to be wealthy is to inherit some significant portion of your wealth -- it's a vicious circle).
These kinds of -isms are typically vestiges of more overt bigotry in the past, that haven't yet been eliminated. That doesn't make them "ok" or "not a big deal"; being on the short end of systemic -isms sucks. But part of why such -isms take so long to dismantle is the other kind of subtle -ism.
Casual -isms are unconscious biases, often fed by the structural and systemic -isms that still exist in our society. (This sort of feedback loop, where systemic bias feeds casual bias feeds systemic bias, is why both are so difficult to solve.) Most people will perpetuate some casual -ism at some point; it doesn't make you a bad person, it isn't deliberate. But it's still hurtful.
For example, when a company lists an engineering position and says of the qualified applicant "he will be able to think on his feet", the author and approvers of that language reveal an unconscious belief that engineering candidates -- at least the good ones -- will be male. Now, if you ask the author, he or she will almost certainly say "no, that's not what I meant at all! Of course a woman can be a good engineer, and we would hire a qualified woman if she was the best candidate." And he or she will really believe that! They aren't a bad person at all!
But it's still sexism. It reveals that there is a bias. Whoever wrote that tends to think that engineers are men. Given our social history, it's not an unreasonable bias. It doesn't make you a bad person if your default mental image of an engineer is a man. But it does make you an unwitting accomplice to the very systemic sexism that gave you that casual bias in the first place.
And of course, other -isms operate on similar principles.
So while letting slip a casual -ism doesn't make you a bad person, it's still something you did that was harmful in some small way. It's still something you did to contribute to the problem, in much the same way accidentally dropping a candy wrapper and not realizing it contributes to a litter problem. But I'd hope that you take steps to try to avoid dropping wrappers. And I'd certainly expect that if someone calls your attention to it your reaction would be "whoops, sorry about that!" rather than outrage.
And that's really all I'm asking of people who get called out on casual -isms -- acknowledge that you made a mistake, and work to correct it.