I'm talking about the space between the browser window and content. I've found that it's uncomfortable to read when the text is too close to the edges and I was wondering if there is any rule that applies to this.
Narrow margins could mean a reduction in white space (depending on the specific design).
A List Apart links micro-whitespace to legibility. However this is referring to the space between characters and words. Not exactly margins. That would be macro-whitespace.
Macro-whitespace can lend a feeling of elegance and calmness to a design. It can also help lead the eye to important features.
According to The Paradox of Whitespace on I'd Rather Be Writing
Generous margins and clear simple layouts make it easier for the eye to work. Cluttered layouts tire the eye quickly and hinder clarity.
With regards to fluid layouts there can be issues with larger resolutions creating too much white space. This can be flipped when viewing on smaller screens where the margins become squashed (they are a percentage of the overall size). Thus it is harder to control and inevitably there will be scenarios where the lack of whitespace might impact legibility.
To mitigate this you could try for a simpler design. A design that fills the space on a big monitor could look messy on a small one. A simpler design often scales better to smaller resolutions. Many responsive designs drop content or features on smaller screens.
There are probably other technical approaches to take; however it depends on how small you are intending to go. Consider the Pros and Cons of Elastic vs Fixed vs Fluid.
For extreme changes you might consider using a different layout. Designs will only be flexible to a point. For the sake of usability, I would set a maximum and minimum size for margins.
Controlling line length is very important for legibility and there are many methods of doing this, margins being just one of them. You can google "line length for readability" to find suggestions, research and arguments for an optimal line length, I like to limit mine to 60 characters.
Now beyond the issue of line length, margins are important for legibility but should be sacrificed when the screen is very narrow as on a phone. This becomes a balancing act - you want some margin but you don't want to reduce text line length too much.