In the world of the Victorian Club, it does not do for a gentleman to be seen to take an interest in any member of the working classes. How dreadfully inconvenient then, when the narrator's waiter, begins to display alarming tendencies towards having a personality, a home life, emotions and fears of his own.
The last thing any respectable gentleman should do is permit himself to be drawn into this other dubious world...let alone entertain emotions such as interest or pity for the underclass. So our gentleman is at great pains to avoid all of this. Great pains.