Low-fidelity is for testing the concept: information architecture, navigation design & logical flow and so on.
High-fidelity is for testing the design: visual design, interactions, animations, UI element states (e.g. is it clear that it is disabled or enabled).
So it simply depends on what your question or goal is. When you're in the beginning, you might want to find out if your concept even makes sense to potential users, so you create a rough, low-fidelity prototype. When you're in a later stage and already know the concept makes sense, you might want to find out if the details make sense, so you can't have just a rough sketch, you need a close representation of the real thing.
I found an article that summarized it well:
Low-fidelity prototypes
are often paper-based and do not allow user interactions. They range from a series of hand-drawn mock-ups to printouts. In theory, low-fidelity sketches are quicker to create. Low-fidelity prototypes are helpful in enabling early visualization of alternative design solutions, which helps provoke innovation and improvement. An additional advantage to this approach is that when using rough sketches, users may feel more comfortable suggesting changes.
High-fidelity prototypes
are computer-based, and usually allow realistic (mouse-keyboard) user interactions. High-fidelity prototypes take you as close as possible to a true representation of the user interface. High-fidelity prototypes are assumed to be much more effective in collecting true human performance data (e.g., time to complete a task), and in demonstrating actual products to clients, management, and others.
Source: usability.gov