The summary on MDN describes 404 Error as;
The server can not find requested resource. In the browser, this means
the URL is not recognized. In an API, this can also mean that the
endpoint is valid but the resource itself does not exist. Servers may
also send this response instead of 403 to hide the existence of a
resource from an unauthorized client.
Since we're dealing with browsers, it means the URL is not recognized
by target server or server's hiding the existence of a resource from an unauthorized client
. Either way, it's not a desirable situation since it gives user trying to reach a source deprecated or somehow something happened.
In a world of web and/or software development, it's practically almost unavoidable to build and maintain something unbreakable as an experience. Because things are evolving as user needs, so the products eventually. To make a bit clarification on this, either firms are seeking growth in order to grasp more amount of money or the products are evolving with the personalization of users needs via technology, there's always a room for improvement.
Even I was perfectionist before I started to work as a developer, it's not possible to command a server's reaction or internet connection at least. One should take into consideration every improvement and bug fixes to be covered within documentation to be able to get a smoothest UX and development within UI, while backend, DB, and system part should provide these details in a different manner somehow disciplined as the same above.
Like the evolution of the democracy wasn't in perfect form, the need for this redirected error page also wasn't the idea of UX-origin people and not represents the clearest point of view appearantly. It's more clearer from the text below with the provided link above,
Many web sites customize the look of a 404 page to be more helpful to
the user and provide guidance on what to do next. Apache servers can
be configured using an .htaccess file and a code snippet like the
following example.
It was started with developers or server management firms to handle unsatisfaction of their customers but using a general error page instead all of the other errors would make it more appealing for developers and firms creating values in that sense. So rather than solely for 404, it's used as an error page for all kind of HTTP Status Codes.
Well, I don't know of any research on this topic but I'm also on the side of using it, instead of a blank page in a point whenever it's inevitable. It's better rather the page is fed with the information according to the user's need like re-routing, re-direction, why it has happened if possible, go back or vice versa rather than fancy visualizations.