Yes, it's known as the Hawthorne Effect and it's one of many cognitive bias' to be aware of when running usability testing sessions.
The Hawthorne effect (also referred to as the observer effect) is a type of reactivity in which individuals modify an aspect of their behavior in response to their awareness of being observed.
Does it make the research flawed? Well yes, in a way. But that doesn't mean it's not worth doing. You can mitigate it to some extent - making the testing facilities as natural and comfortable as you can, not standing there in a lab-coat with a clipboard, generally being friendly and human and not like a machine. And (importantly) telling the user up-front that it's not them that you're testing, it's the system.
But bias will always be there, you need to be aware of it, try testing with multiple methods and users and don't assume after the testing that how people behaved during the test is exactly how they'll behave in 'the real world'.
But you still get useful test data from people, even if they know they're being monitored. People will try even harder to solve problems in a lab than they would at home, so if they can't figure out how to get to the checkout on your eCommerce store when you're testing them in a lab then you really know you've got a problem - because people at home would've bailed out long ago.