Book Review: The Black Box Society: The Secret Algorithms That Control Money and Information by Frank Pasquale
Did someone nudge you in to this website from your previous searches? Who is going to see what articles you looked at here? Will reading this article affect your credit rating?
University of Maryland law professor Frank Pasquale’s book The Black Box Society: The Secret Algorithms That Control Money and Information, examines the dangers of data creep and hidden algorithms. The book’s title refers to a term in computing where a system can be viewed in terms of its inputs and outputs, without any knowledge of its internal workings. According to Pasquale, systematic secrecy is present online, in politics, the legal system and in finance with the explicit purpose to obfuscate. His purpose is to once again make society intelligible.
The big web platforms like Google, Facebook, and Amazon changed the internet experience quite a lot the last decade without the average user being fully aware of it. When data turned into a valuable commodity, openness was discarded. The current web is about gathering users for a service and acquire data about them to be sold to data brokers, advertisers and other collectors. Many users have no idea of the data trails they are leave behind, nor do they understand how this data is used, making it difficult for them to exercise their rights. Regular users are more transparent than ever, but it is not a two-way relationship as companies and government agencies like the NSA aggregate data.
Online recommendations and interactions turned into filter bubbles. Are the algorithms behind search results as neutral as purported? Pasquale indicates bias through some examples, both on the grounds of prejudice but also self-serving to the interests of the tech giants.
Your online reputation affects your career, medical services, insurances, credit rating etc. but it is difficult to understand precisely how this reputation is created, disseminated, and could lead to cascading disadvantages. Some users get weblined, namely offered preferential prices and services based on their online behavior while others might get discriminated. There could be a valid explanation, but Pasquale points out that this is being kept secret.
Intelligence agencies have put up ingenious systems for exchanging information through back room deals between government and corporations, bending the law by mission creep. Surveillance measures previously directed at terrorism are now used to detect ordinary crimes.
Are the algorithms behind search results as neutral as purported?
The current state of finance is perhaps the most known secretive part of society known to the public. The increased complexity enacted by the quants, financial quantitative analysts, do not serve to make markets more efficient but less so, in order to hide the failures leading up to the crash of 2008.
The Black Box Society makes for a passionate and ambitious read about secrecy, but its anecdotes sometimes make up for a lack of depth. Pasquale overestimates the degree of control the digital giants have over the effects of their algorithms. Algorithms often depend on fudged data, blanks and misinterpretations, which is an issue in itself. How will these bugs in the systems affect us?
Pasquale’s “New Deal” of internet commissions and regulatory agencies do run counter to his complaints, as he argues for even further surveillance. The voucher system proposed for culture sounds like a feeble substitute for the copyright system disappearing on the platforms.
The questions raised will become ever more important, as the current adoption of the Internet of Things will raise a variety of privacy and security-related concerns, but now in physical space.
The drawing of “the creepy line” is not same in every area, but the general direction of drawing it ever further is damaging to individuals and society. The book does point out the necessity of digital sustainability, of keeping reciprocal trust between users and service providers rather than relying on opacity and lack of knowledge.
