Got kicked out of line for having my 3-year-old and wife rejoin me

I was in line today and my wife had to take our 3-year-old to the restroom.

After about 20 minutes they came back, I lifted the chain so they could rejoin me, and a few people behind us walked ahead totally fine with me. About 10 minutes later the woman now behind us told a CM that we “cut.” CM pulled us out of line and I had to explain to another cast member what happened.

Karma kicked in because the ride broke down for 15 minutes after that. The second CM apologized and gave us a walk-on pass for any ride and said we never should’ve been removed from the line.

Does this happen often? Am I overreacting or was that first CM way out of line?