Well, not quite the only person.
For one thing, there was Harry Potter, though he wasn't actually outside, unless you wanted to count his head, which was poked out the window, staring down at the younger of his two arch-nemeses with a good amount of amusement.
Well, Draco Malfoy had been an arch-nemesis of his last week. But so many odd things had happened over the last week that Harry wasn't entirely sure where they stood. But that didn't mean he couldn't be very amused when the boy received comeuppance, which he just had, of a sort.
And indeed, he wasn't the only person entirely outside after all, because so was another boy his age, who did not consider himself outside, because he considered all the property his, despite the fact all the money for it had been paid by his father, and "outside" was beyound the fence.
And Dudley Dursley was breaking the silence, having knocked Draco down, by yelling, "Admit my hair's prettier! Come on, admit it!"