It can be annoying when it continues for more then a minute, but overall, i don't get the impression anyone is actually mad beyond the annoyance of not having one yet. The legendaries can make or break fights, especially for dps, so continually not having one while others with them are passing you can get irritating.
So yes, it can be annoying, but i personally dont see it as toxic. Others may disagree.