You gotta be worried about what I used to call the 'Raider Effect'. Let me use an example: Nnamdi Asomugha was regarded as one of the best corners in the league and teams would avoid him anytime they played against the Raiders. Unfortunately when he became an Eagle teams realized he only sucked a bit less than the rest of the Raiders, so he gave the appearance of a good player while still sucking.
I also call it my 'Special Olympics Effect', but it seemed to work really well with Nnamdi and the Raiders.