Why did the author of Sai Satcharitra initially refuse to visit the saint in Bandra?

In Chapter 21, the author, Hemadpant, shares a personal story to illustrate a spiritual principle. When he was a Magistrate in Bandra, a famous Siddha named Pir Maulana lived there. Despite being repeatedly urged by the saint's servant, Inus, to come for a visit, Hemadpant refused. He questioned why he should go just because thousands of others did and worried about losing his own prestige by being pressured. He attributes this reluctance to his own ego and a period of "ill-luck" that stood in his way, preventing him from gaining the company of a saint, even one who was so close. He reflects that many years passed before his fortune changed and he was drawn to Shirdi.