Based on the text, can we conclude that most pilgrims who visited Sai Baba were part of the Ramdasi sect from Madras?

No, that conclusion cannot be drawn from the provided text. Chapter 29 describes one specific "group of devotees" from Madras who were on a pilgrimage to Kashi and stopped at Shirdi. This group of four people followed the Ramdasi sect tradition. While their fame had spread in Madras, the text presents them as just one example of the many types of people who came for Baba's darshan. Verse 7 mentions that "crowds came," including various performers and devotees, but it does not provide information to suggest that any single group or sect formed a majority of the pilgrims.