Pause the psy­cho­log­i­cal tricks. Con­sumers should not be hackable.

As machine learn­ing and Arti­fi­cial Intel­li­gence become more sophis­ti­cat­ed, these com­plex tech­nolo­gies will large­ly be turned toward the gen­er­a­tion of per­son­al pro­files for each indi­vid­ual. The aim of such an appli­ca­tion of tech­nol­o­gy to the nuances and demands of every­day life, is to cre­ate entire­ly per­son­al­ized, tai­lor-made con­sumer expe­ri­ences. This essen­tial­ly means reduc­ing peo­ple to mar­ket seg­ments of a mas­sive, aggre­gat­ed user experience.

Addi­tion­al per­son­al data—including bio­met­rics har­vest­ed from wearables—will make this process even eas­i­er. Our psy­cho­log­i­cal pro­files will be built out with pre­dict­ed emo­tion­al respons­es to cer­tain stim­uli, allow­ing the right com­bi­na­tion of words or images to be intro­duced to us to com­pel us into a pur­chase. And, sad­ly, we will be com­plic­it in this exchange. After all, shop­ping releas­es dopamine, a “reward” chem­i­cal in our brains, and these new algo­rithms will become ever more adept at trig­ger­ing this phys­i­o­log­i­cal reac­tion. So what’s the prob­lem here? If our desires are being met more reg­u­lar­ly and accu­rate­ly, isn’t that a good thing? The issue lies in the manip­u­la­tive aspect. This is not just a pas­sive­ly respon­sive process. Tech­no­log­i­cal com­pa­nies aren’t just ful­fill­ing our cur­rent needs and demands, more nefar­i­ous­ly, they’re man­u­fac­tur­ing new desires. Com­pa­nies then prof­it off the feed­back loop of telling poten­tial cus­tomers what they should want and then sup­ply­ing those fab­ri­cat­ed demands right back to them.

In 2018, hun­dreds of bil­lions of dol­lars was spent glob­al­ly on adver­tis­ing across tele­vi­sion and inter­net plat­forms.¹⁴ The fact that so much mon­ey is required to sell a range of con­sumer prod­ucts shows that demand for con­sumer goods is large­ly man­u­fac­tured. As the trove of our per­son­al and inti­mate data becomes more plen­ti­ful and our emo­tion­al respons­es become more pre­dictable, it will become eas­i­er for com­pa­nies to make it look like they’re offer­ing increas­ing­ly inti­mate expe­ri­ences when, in fact, buy­ing into that mar­ket will effec­tive­ly mean that we’re giv­ing up our auton­o­my. Mean­while, the creep­ing monop­o­liza­tion of online ser­vices only means that a scant four or five com­pa­nies will soon own all the lat­est psy­cho­log­i­cal manip­u­la­tion tricks. If this comes to fruition, soon our urges and men­tal states may end up exist­ing at the var­i­ous whims of a small selec­tion of busi­ness­es. We vehe­ment­ly oppose this sce­nario and encour­age stead­fast resis­tance against giv­ing into such bla­tant monop­o­liza­tion over human agency.

Ulti­mate­ly, we all have a choice to make. Sure­ly, some would make the argu­ment that as long as our desires are being met, then it hard­ly mat­ters what the process is that led to such sati­a­tion.  This argu­ment for­wards a reduc­tive view of the human: our sole pur­pose being to expe­ri­ence the indul­gence of pass­ing impuls­es. But we think oth­er­wise. We think this is only one part of us. We are more than just our plea­sure cen­ters and our cred­it cards. We are com­plex, often con­tra­dic­to­ry, cre­ative crea­tures.

There’s much to be learned from the ten­sion we expe­ri­ence with­in our­selves, with­in our rela­tion­ships, and with­in our soci­ety. Pop­u­lar sci­ence fic­tion offers an espe­cial­ly cre­ative space for explor­ing how cer­tain ten­sions might play out, and has con­sis­tent­ly guid­ed thought around what the future might offer. Two major dystopi­an works of the 20th cen­tu­ry, George Orwell’s 1984 and Aldous Huxley’s Brave New World, pre­sent­ed two almost dia­met­ric oppo­site views of our future.

The nov­el 1984 is set in an extrav­a­gant­ly oppres­sive, and relent­less­ly mis­er­able, author­i­tar­i­an regime. The antag­o­nist of the book sum­ma­rizes the mood, “If you want a pic­ture of the future, imag­ine a boot stamp­ing on a human face—forever.” In this sce­nario, the human spir­it is crushed into a state of sub­servience and sub­mis­sion, with no space for indi­vid­ual desires.

In con­trast, Brave New World envi­sions a soci­ety awash with sen­su­al indul­gences. All dis­sat­is­fac­tion is root­ed out by the pres­ence of high-tech recre­ation­al activ­i­ties, sex­u­al free­dom, and the reg­u­lar use of “soma”, a drug that imparts total bliss. Here, the com­plex­i­ty of the human being is reject­ed. All neg­a­tive emo­tions are dis­card­ed in favor of con­tin­u­al stim­u­la­tion of our base-reward mech­a­nisms. Soci­ety is made com­plete­ly pre­dictable and unchang­ing, each class of peo­ple made com­fort­able in their own sta­tion through con­di­tion­ing in infan­cy. This vision of the future is dystopi­an because of its reduc­tion of the whole human. We believe that it is only through access to the full tex­ture of our emo­tions that we man­age to learn, grow, cre­ate, and evolve.

Con­sid­er­ing the mount­ing evi­dence to sup­port the idea that we are more in con­trol of our evo­lu­tion than we might have pre­vi­ous­ly believed, it’s incum­bent upon each of us to choose the ways in which we want to evolve. The ques­tion of what future we wish to inhab­it hinges upon, not only how we regard events tran­spir­ing around us, but more impor­tant­ly, what we believe we can be a part of cre­at­ing… To this end, inno­va­tion must not be about find­ing more effi­cient ways to con­trol and pre­dict human behav­ior. Inno­va­tion is bet­ter turned toward the empow­er­ment of peo­ple, allow­ing each of us to choose the ways we intu­it will enhance life. We do not want to be treat­ed like organ­ic machines with ful­ly-decod­ed inputs and out­puts. As we become more capa­ble of affect­ing our sen­sa­tions and moods at will, we must be cau­tious that we do not become mar­i­onettes to an algo­rith­mic pup­pet mas­ter. Rather, it is crit­i­cal we con­struct nar­ra­tives in which life is sup­port­ed to grow organ­i­cal­ly, with­out manip­u­la­tive inter­fer­ence, and to the full poten­tial of our innate gifts.