Plot twist: the baseballs were all full of drugs.

You very likely were more watched, and understood, than you think. Each of those balls was probably checked, and it was likely that there were strict penalties for inmates who grabbed one, assuming any inmates would be able to. There was probably a disposal point for them as to keep employees from bringing outside contraband in. This is very different to walking up to the prison wall, and throwing some random object over it, especially doing it multiple times, showing you are trying to get it over the walls. There are thousands of videos online of guys standing on the closes public road to prisons, and jails, and they get attention in minutes, when they are just taking pictures, and some video.

Even for facilities in densely populated areas, like downtowns in metro areas, similar to the one I used to work in, who have to be a lot more tolerant of people on their grounds, and at the periphery, if someone went into some place uncommon, or was acting strangely, there would be guards there in minutes to question them, and drive them away. The people on the walking/biking path right next to the building, who would often linger around, lean on the fence, etc., rarely caused a reaction though.