What is the market economy and why does it work?
The market economy is an abstraction created based on the pattern of reproduction, so it just works as a computation that creates technology to shape the environment in relationship to whatever the mechanism of reproduction for humans is, so the network of human beings (collective consciousness) computes in real time what is going to be necessary to shape a certain environment based on whatever the fundamental parts of human nature were necessary to survive in the past. The limbic system is the most fundamental part of the brain that creates these desires, which are more commonly known as the meslo hierarchy of needs.
This is the reason that if the market economy is unregulated it converges towards a pareto distribution over time, in general woman prefer men in the top of hierarchies because that creates a funnel of information, genetic distribution is a consequence of mutation that is going to define a geometrical structure of how DNA is related to spacetime, the pressure of natural selection defines the funnel I just described, and it is shown in human beings through reproduction. So to simplify the purpose of the market economy is to compute in real time how does the limbic system (of the collective consciousness) wants to change the environment so it is more beneficial to it. Money is a database, markets are computation, technology is a consequence of how the limbic system shapes the environment to fulfill it. The main benefit of the market economy is that it's not necessary to know all of the complexity of the environment before hand, it just computes the technology on real time, this is why it works well to a certain extent and is scalable.
Why is the tendency for technology to destroy humans over time?
The limbic system overfits the environment, through the computation of the market economy the limbic system of the collective consciousness creates technology that shapes the environment, this is a cycle that is directly correlated with the frequency of computation of the market economy, the problem is that the limbic system doesn't adjust itself to the new environment it creates over time at the same rate that it changes the environment, politics are supposed to be the regulating force that recalibrates the limbic system to work on the new environment created by the market economy, the issue is that the human hardware (the biochemistry of our biology) is not scalable on spacetime, in order to maximize energy efficiency (Occam's razor) humans limit themselves to tribes, so when humans scale to numbers that go way beyond a tribe (around 150 people) the natural tendency is for different groups/tribes to be created, and as a result there will be conflict between them, the boundaries between different tribes show one of the core esseences of the multifractal geometrical structure of how societes shape themselves in relationship to spacetime (cities, states, countries...).
So how is this related to search engines?
Search engines are an extention of how the brain consumes information from spacetime, the computation the limbic system does is directly correlated with how it perceives information, the brain filters the complexity of the environment based on what the subconscious (body) thinks is important, which is a consequence of how that body existed through spacetime, because it is directly correlated with gravity. That means that search engines will be a consequence of how this mechanism work on the collective consciousness level, the results of searches will be based on whatever the gravitational force of the network that creates the hierarchies of importance (relative to the current state of spacetime) is. This means search engines are a byproduct of how the limbic system of the collective consciousness filters the complexity of the environment and also a direct consequence of the computation of the market economy, which creates the incentives for the results the search engine will show.
Computational boundary of search
The way search engines are going to display results is then a consequence of the frequency of computation of the market economy and also of how data of the environment becomes information through computation on a network of nodes (society). Because of the nature of markets the expansion of knowledge is going to follow a tree data structure (the deeper into the unknown the more specialized you have to become), because of this and the fact the limbic system is not consciously aware of the context of the environment in which it exists, the results shown are always going to be biased towards what currently satisfies the needs of the limbic system, which lacks balance with the environment it creates, in simple terms, the search engine becomes gamefied through SEO and the production of results is biased because nodes are incentivized to only produce content for the masses, since that's where there is most money to be gained, this generates a vicious cycle where the gravitational force of the network becomes unscalable on spacetime.
Conclusion
In order to make better search engines it's necessary to understand the biases the computation that a network create over spacetime, and be aware of how the search mechanism is going to become bias overtime because of the paradox of search, you can only search what you already know to a certain extent, so the results a search engine displays are always going to be a consequence of how the geometrical structure of a network of nodes gains information on spacetime, what is considered to be true for the network is just a consequence of the gravitational force of how the information was computed by the network in relationship to spacetime, if there isn't any mechanism to recalibrate this overtime there will be a high chance that the network will overfit the environment at some point, because there will always be variance on spacetime (entropy), because that is the essence of time and space itself.