Yes, most definately.
Buggy doesn't stand a shot against Caribou. Buggy is fucked if he faces Kidd. He can't even dream of touching Kaidou. Akainu could, as I said earlier, melt him and his crew in seconds, Smoker would choke Buggy would with his smoke, and Mihawk would scewer his crew.
But what you're thinking of is the WORST of the worst. Most pirates aren't like that.
You forgot about the crew that Kidd slaughtered upon entering the New World. You forgot about "Brownbeard", who got destroyed by Hawkins. You forgot about the 55 million beri bounty pirate who got taken down with ONE SHOT from a pistol by Demalo Black (fake Luffy).
Most pirates aren't very strong. There are beasts, and monsters, but Buggy can easily pull a "Oh noes, I died!" like he did with Zoro, and then slaughter them from behind.
You're thinking in FAR too large terms. There is a reason that Luffy is so feared. There is a reason the rookies were called "The Eleven Super Nova's", and were feared across the first part of the Grand Line.
That is, they are rare. REALLY rare.
Most pirates are pathetic brawlers, who Buggy will even be able to handle without a problem. And who knows, maybe Oda will even throw a power boost in Buggy's general direction? x)