The Problem With Connection Theory

(For those unfamiliar with Connection Theory, or ‘CT’, check out the Leverage Research website. My link points to the older version, which is somewhat out of date but much more detailed than the current website, which only spends a few paragraphs describing it.)

Science is a difficult project, and even many professional scientists get it wrong. One can make lots of mistakes, while still driving a field of knowledge forward. However, some problems are so serious that they call into question the credibility of the authors as researchers. Much to my regret, Connection Theory appears to have several problems in this class.

Several years ago, an experiment was performed to test Connection Theory. This experiment was done using an “RP test”, a new form of experiment devised by Leverage Research. RP tests are conducted as follows:

“To run an RP test, select a test participant. Make a comprehensive and accurate CT chart of the participant’s mind. Ask the participant which elements of her mind she would like to change. Then use CT, the participant’s chart and facts about the participant’s environment to derive a set of predictions. Each prediction should be of the form “if the participant performs actions A1, A2, A3, etc., in that order, then at time T the participant’s mind will change in way W”, where way W is one of the ways the participant would like her mind to change.

Next, have the participant perform the actions in the order prescribed. Keep track of the results, either by questioning the participant or by having the participant keep a diary. Once the participant stops performing the actions stated in the antecedents of the predictions, either because all of the prescribed actions have been taken or for some other reason, the RP test is complete. Collect all of the results. Be sure to determine which actions were taken at which times and which results occurred or failed to occur at which times. For results that occurred, determine how long results lasted.”

This experimental protocol lacks many common features of scientific research, like randomization, blinding, and control groups. It even lacks a ‘self-control’, common in self-experimentation, where before an experiment one tries something similar without the “key element” (in this case, CT) that one is testing. In fact, it ranks at the bottom of the usual hierarchy of study credibility:

- Randomized controlled double-blind study
– Randomized controlled study, not double-blind
– Case-control study (including twin or sibling)
– Longitudinal or correlational study
– Case series
- Case study (emphasis added)

Of course, in the real world, it isn’t practical to do a fully randomized, controlled, double-blind study every time one does research. There are many hypotheses to test, limited resources, and sometimes one should take ‘shortcuts’ that reduce a study’s power but make it easier to run. However, the paper then goes on to say:

“All of the preceding applies to RP tests conducted with the highest standard of care and rigor. For RP tests conducted less carefully, the resultant evidence for or against CT will be proportionally less strong.”

(Emphasis added.)

So, we are not describing a ‘preliminary study’, to be followed up by a more rigorous test, but an experiment conducted with the “highest standard of care and rigor”.

The paper then continues:

“Determine how unusual it was that the results occurred. If a result, for instance, was the production of a particular action, it is important to note how frequently the participant performed that action prior to the RP test. Was this the first time in five years the participant got herself to perform the action in question? Or did the participant perform the action sporadically in the past?”

ie, instead of a control group, the test must rely on the experimenter’s subjective impression of ‘unusualness’. Again, this is a test to be ‘conducted with the highest standard of care and rigor’.

The paper continues:

“For each prediction recorded as coming true, one should look to see which of these explanations are more elegant and which are less so. The more unusual the effect, the less plausible it will be that it was merely a result of random chance. The more concrete the predicted effect, the less plausible it will be that there was an error in recording the results. Any prediction that actually came true provides some evidence for CT. The more clearly the prediction came true, the better. The less plausible it is that some other factor caused the result, the better. Results that clearly occur, just as predicted, and are not plausibly explained by other factors provide very strong evidence for CT or for a very similar theory.”

(Emphasis added.) Note that this does not (and indeed the full paper does not) describe what criteria a prediction must meet, other than that it must describe a participant’s mind changing in some way. Furthermore, no protocol, method, or control is provided for differentiating a “CT prediction” from a “non-CT prediction”; any prediction made by a “CT researcher” is assumed to be a “CT prediction”. Hence, even an obvious prediction, such as “if Alice spends a day describing the mistakes her co-worker Bob has made during his employment, then Alice will have fewer positive emotions towards Bob”, would be treated as “very strong evidence for CT” under the proposed protocol, despite not being related to CT at all.

The next section describes the real experiment which was conducted. The participants were chosen as follows:

“I have conducted two full RP tests. The first was performed by me on myself. The second was performed by me on a friend and colleague of mine.”

ie., the experiment was conducted with N = 2, and the test subjects were not only not randomly selected, but were active proponents of the theory in question.

Self-experimentation has a long history in medicine, but in such cases, the prediction is usually an objective physical measurement over which the subject has no control (eg. cancer, or stomach inflammation). Attempting to predict one’s own behavior suffers from the obvious confounder that one might act differently in response to predictions. For example, a Muslim trying to prove the existence of God might predict that he works harder on days when he conducts his daily prayers. Of course, he can then make the prediction come true simply by changing how hard he works.

The subjectiveness of the predictions in question, and the assessment thereof, is well demonstrated in the following example:

- “Background: Before doing the RP test, the participant would switch back and forth between feeling like he could be himself while still being accepted and feeling like he needed to be someone he wasn’t. This was a feature of his personality that had been present for ten or more years.
– The required action: The participant must determine a way to achieve nearly universal world virtue that is more effective and feasible than any of the participant’s current ways and does not involve the participant being an approachable example for others to emulate. The participant must also determine a feasible way to search for friends the participant won’t need to change to be friends with. The participant needs to do this while still being virtuous and while retaining the ability to walk away from people.
– The predicted effect: As soon as the participant completes the actions just stated, the participant will at all times feel like he can be himself while still being accepted.
– The result: At some point after the participant completed the above tasks, the participant started to feel at all times like he could be himself while still being accepted. The time the tasks were completed was not recorded carefully. Some time after the tasks were completed, the participant started to feel at all times like he could be himself and still be accepted. The time this effect occurred was also not recorded carefully. Based on the participant’s guesses, the effect occurred less than one month after the above tasks were completed. As of nine weeks later, it still appeared that the effect was persisting. After that week, the participant noticed that the effect has stopped. A CT-compliant explanation of the effect ending was later determined.”

Even given these remarkably fluid and vague methods of prediction assessment, one notes that the test still essentially failed. The participant, by his own admission, did not “at all times feel like he can be himself”, and there is no indication that his behavior after the intervention differed from his prior behavior of “switch[ing] back and forth”, which necessarily implies significant time spent in both states. However, this test was still recorded as a success, as an (unspecified) “CT-compliant explanation” was “later determined”.

The paper goes through another, similar case:

“The participant had been smoking and drinking for approximately 6 years. The participant tried to quit smoking in little pockets here and there, but was not typically not successful. The most successful times were two times when the participant quit smoking for a period of two months each time. During those times, the participant did not want to smoke and did not have cravings. (…)

The predicted effect: As soon as the participant completes all three of these actions, the participant will stop drinking and stop smoking.

The result: The participant did not note when exactly the effects began to occur, but at some point after the completion of the required actions, the participant stopped drinking and smoking. After the participant stopped drinking and smoking, the participant also stopped having a desire to drink or smoke. The participant did not have to struggle with cravings at all and did not have to struggle with the difficulty of getting out of an old routine. The effect persisted for eight weeks. The effects abruptly came to an end when a particular negative event occurred and substantially altered the participant’s life plans. The day after this event, the participant began smoking and drinking again.”

Once again, there is no indication that the CT intervention affected behavior at all. Nevertheless, the paper counts this as a success, justifying:

“One might think that the fact that the participant in the third case started drinking and smoking again weighs against the predictions having been correct. This is not true. The changes caused by RP plans are not supposed to be permanent. Whether they persist or not depends on what other things happen in the person’s mind. Effects are frequently long-lasting since people’s minds do not change so frequently. But it is perfectly possible that some external event would change a person’s mind and reintroduce a mental phenomena we had eliminated.”

Of course, when the phenomenon in question (smoking) was known to be intermittent to begin with, it is not very impressive to predict that it will at some point stop and then at some later point start again. The paper continues:

“In fact, the negative event that immediately preceded the relapse was exactly the sort of event that CT predicts would cause a relapse – it was an event which cut off the new paths to the participant’s IGs that the participant had created through the actions recommended by the recommendation plan. Had I had the prescience to write down the conditions under which the participant would start smoking and drinking again, there would be an additional successful prediction from that RP test.”

Indeed, making predictions is difficult, especially about the future. However, a theory must be “prescient” in order to be scientifically valid. One cannot claim the validity of a theory based on, after a prediction being falsified, a post-hoc claim that the theory ‘would’ have predicted the true result, given some ill-defined counterfactual. Such behavior is the domain of numerology and horoscopes, not science.

Again, I am in favor of “seat-of-the-pants” experiments, and even of using data from such experiments when no better option is available. However, any reasonable researcher must be very mindful of the weaknesses of such methods, and be careful to spell them out.

The paper’s assessment finishes (emphasis added):

“The RP tests provide very strong evidence in favor of CT or a variant of CT. I successfully predicted 16 conceptually independent effects beforehand. In each case, I specified a narrow window of time in which the effect was supposed to occur. In many cases, the predicted effect would break a trend of months or years. And yet I observed the predictions coming true and did not observe any predictions failing. No other existing psychological theory has anything close to this degree of predictive power.

No citations, data, reasoning, or examples of alternative theories are used to justify this extraordinarily sweeping claim, which is founded on two case studies evidently lacking in even the most basic rigor, such as systematic recording of data. To quote Cosma Shalizi:

“Normally, scientific work is full of references to previous works, if only to say things like “the outmoded theory of Jones [1], unable to accommodate stubborn experimental facts [2--25], has generally fallen out of favor”. This is how you indicate what’s new, what you’re relying on, how you let readers immerse themselves in the web of ideas that is an particular field of research. [The author] has deliberately omitted references. Now, this is sometimes done: Darwin did it in The Origin of Species, for instance, to try to get it to press quickly. But [the author] has written 1100 pages over about a decade; what would it have hurt to have included citations? (…) To acknowledge that he had predecessors who were not universally blinkered fools would, however, conflict with the persona he tries to project to others, and perhaps to himself.”

Connection Theory is not quite as well-researched as the book Shalizi discusses. Nevertheless, in the 80-page essay “Connection Theory: Theory and Practice“, the author neglects to include any citations or bibliography; despite being an experimental paper on a theory of psychology, the paper discussed displays no understanding of the existing psychological literature or of established research methodology. As many psychologists will admit, these are often highly flawed, but (as Shalizi points out) the correct response is to explain why they are flawed and how they can be improved. A middle school science student is not excused from citing Newton in a paper on gravity, merely because the ‘wrong’ Newton was later overturned by Einstein.

If Leverage Research had merely reported what they had done, without any commentary, it would just have been another mildly interesting self-report on exploring psychology, as the tens of thousands of others which have been collected online. And of course, any good scientist will spend some time thinking over speculative theories with little evidence. But to make the extraordinary claims described above, and to use Connection Theory as one of the primary foundations of an organization with twenty-six team members as of 2012 (edit: Leverage has clarified that they have 15 full-time employees, and that the majority of the 26 mentioned here were volunteers), is to hold yourself to a higher, not lower, standard than commonly employed in sixth grade science fairs.

Science is a difficult project. Even when conducted in university laboratories with large grants, 80% of non-randomized medical studies turn out to be false, and even 25% of “gold-standard randomized trials” turn out to be false. And this is in the field of ‘hard’ medical studies, with concrete observables such as cancer and heart disease. In psychology, one often finds that:

“Social psychology experiments in the laboratory tend to throw up spectacular mind-boggling effects. Many of these fail to replicate and are later discredited. The ones that do replicate are not always generalizable – sometimes an even slightly different situation will remove the effect or create exactly the opposite effect. The effects that remain robust in the laboratory may be too short-lasting or too specific to have any importance in real life. And the ones that do matter in real life may respond unpredictably or even paradoxically to attempts to control them.”

Some of this difficulty is unavoidable. But we can at least be aware of these problems, and do what we can to not make them worse.

Glasses Can Be Cheap

Glasses are actually much cheaper than many people think. Some people think glasses are expensive, costing $300 or more, because the big brands (Oakley, Ray-Ban, Persol, etc.) are all really the same company, Luxottica, which uses their monopoly power to distort the market. The major retail glasses stores (Sunglass Hut, LensCrafters, Pearle, etc.) are all owned by Luxottica, as are “vision insurance” companies.

Fortunately, we can now use the Internet to fight back against monopolists. Below, I have posted Tilia Bell’s guide to getting high-quality glasses cheaply.


The pair of glasses I currently wear cost me $42.33. My prescription sunglasses cost me just a bit more at $44.86 (+$6 for shipping). Both of these sites would allow me to return my (completely custom made) glasses for a full refund within something like 7-10 days, had I not been overwhelmingly satisfied with my new glasses.

Here is all the information I believe will lead to a successful online-glasses-shopping experience.

1. You will need your vision prescription.

This means your full prescription, including Sphere, Cylinder, Axis and (Addition?) for each eye, as well as your Pupillary Distance. I had all these things except my PD, which I was able to measure with a ruler. I did not need to verify with either of these sites that this was the correct prescription for me; I just plugged in the numbers.

2. You will need the size of your best fitting glasses.

We have all tried on each other’s glasses and found that some fit our faces better than others. This is because, like shoes, glasses come in different sizes. Too small, and they will pinch. Too big, and they will fall off. Also, large glasses will look ridiculous on your face. I have a small face, and so I look for glasses whose frame size across the front is 120-125mm ideally. My current glasses are 126mm across and fit really well, though I prefer a closer face hug sort of feeling.

To find out the size glasses you should wear, look for take your best fitting glasses and measure across the front of the frame in millimeters.

3. You will need some idea of the kind of glasses that will look good on you.

This is really subjective. I personally think that most people look good in rectangular frames, and that only really fashion savvy people can pull off the more visually intense looks of wayfarers (typical large hipster glasses) or oval, or horn-rimmed (librarian glasses). What I did was I would go into eye-glasses shops whenever I walked past one and just try things on. I did this many times till I had a solid idea of what I wanted.

Once you have your prescription, and size, and preferred style in hand, you can start looking for a pair of frames that suits you! Most sites allow you to search by many metrics including color, shape, material and style. They also have a “try on” feature, where you can upload a photo of yourself and put the frames on that way, but I found this to be virtually useless.

That’s it! Please share this article, and update your glasses!

Money Doesn’t Matter In Politics

Many have warned about the grave dangers ofmoney in politicsin the US. However, does this theory fit the data? Let’s test it out.

Since this is a test case, we should make it really really clear-cut. For example, even if money is super-important in politics, any one billionaire won’t get everything he wants. There could be another billionaire wanting the opposite. But if a whole bunch of billionaires got together, and said they all wanted the same thing, then you’d expect them to win if money really mattered.

Likewise, if a billionaire wants something, but he’s quiet and keeps it to himself, nothing will happen. So we should choose a case where billionaires are being really loud and actively lobbying for something. That way, we know that Congress knows what they want.

And similarly, we should pick something that’s at least moderately popular, and isn’t too big of a change. Even if Congress were in the pockets of Charles Koch or George Soros, they still wouldn’t vote for something really extreme, like 99% tax rates or abolishing the military. There are about 200,000 pages in the US Code, so for our test case, let’s also pick a proposal that changes, say, four or five of them.

Fortunately, the last year has given us a perfect ‘experiment’. Nine of the richest companies in the US, with a total valuation of $1.4 trillion, all got together and said exactly what they wanted. To make sure everyone knew, they even took out a full-page ad in the New York Times. And their idea was even pretty popular, with 60% of Americans in favor.

Result: nothing.

Okay, so maybe powerful national politicians can’t be pushed around by wealthy corporations. But let’s take an even more extreme example. Let’s take a big, powerful, multi-national corporation, with piles of money and powerful friends, against a little local government that nobody knows about. And sometimes the media or the courts will “fight for the little guy”, so let’s take a case where they aren’t involved.

Last year, Google wanted to build a bridge over a creek, between two Google office buildings. Nothing special, just an ordinary twenty-foot bridge, to save people from making a long, polluting car trip on congested Highway 101. But, to build it, they needed the permission of the Mountain View City Council. Result: denied.

The denial was ostensibly for environmental reasons, even though the bridge would save thousands of miles of polluting car travel. So Google offered to conduct an environmental impact review, to prove the bridge was green-friendly. But even that was denied.

To illustrate how extreme the disparity is here, let’s examine a few key statistics:

Google budget: $45,860,000,000

Mountain View budget: $87,000,000

Google employees: 49,430

Mountain View employees: 378

Google is also Mountain View’s largest employer, and owns 11% of Mountain View’s real estate by valuation. According to the money-in-politics theory, the City of Mountain View should be bending over backwards to please them. Yet, this isn’t what we see at all.

The money-in-politics theory has been formally tested, by famed economist Steven Levitt. Political campaigns are normally hard to experiment with, since there are many “uncontrolled variables” – did Barack Obama win because of all his donations, or did people donate because he was a popular candidate? And so Levitt looked at were races where the same two candidates ran against each other multiple times. He found that, in Congressional races where candidates spent about $250K (1990 dollars), every $100K spent got another 0.3% of the vote, a tiny amount.

Reading the newspapers, one hears about powerful people making political donations. One might surmise that the donations cause the power – first you donate, then you become powerful. However, looking more closely, this seems like a cause-and-effect error. People, for the most part, first become powerful, through some as-yet-unknown process. Then, after they have power, they start donating to campaigns. Attempts to do it the other way around have tended to not work so well.

San Francisco’s Tech Problems

There seems to be a bit of a problem with tech companies in San Francisco.

If tech companies pay their workers generously, then it’s their fault for rising rents. But if they are misers and pay little, then it’s their fault for mistreating working people.

If tech companies all move to San Francisco, then it’s their fault for changing the fabric of local communities. But if they left San Francisco, it’d be their fault for crashing the local economy.

If tech companies try to establish a libertarian utopia, this shows they don’t care about average people. But if they stay in SF and try to work within the current system, that’s the nefarious influence of money in politics.

If tech companies send buses to pick up their workers, this is terrible because it drives up housing costs. But if they don’t send buses, traffic would be so bad that I-280 and 101 would grind to a halt, and clearly that’s the tech companies’ fault.

“Friedrich Spee von Langenfeld, a priest who heard the confessions of condemned witches, wrote in 1631 the Cautio Criminalis (‘prudence in criminal cases’) in which he bitingly described the decision tree for condemning accused witches: If the witch had led an evil and improper life, she was guilty; if she had led a good and proper life, this too was a proof, for witches dissemble and try to appear especially virtuous. After the woman was put in prison: if she was afraid, this proved her guilt; if she was not afraid, this proved her guilt, for witches characteristically pretend innocence and wear a bold front. Or on hearing of a denunciation of witchcraft against her, she might seek flight or remain; if she ran, that proved her guilt; if she remained, the devil had detained her so she could not get away.” – Conservation of Expected Evidence

The Charlie Shrem Files, Part 3

The Bitcoin company BitInstant is now facing a class-action lawsuit, as well as federal felony charges against their CEO Charlie Shrem. Included below is a BitInstant presentation describing the company’s business operations as of 2012. Of particular interest is this quote:

“BitInstant is not a money transmitter and does not interact directly with regulated financial institutions, instead using intermediary bodies to process fund transfers and acts as a third party to accelerate transactions. All of BitInstant’s transactions are Closed Loop, meaning all monies cannot be withdrawn without proper AML and KYC measures being taken as outlined in our ALM compliance policy. Customers cannot simply cash out of Bitcoins as all of the exchanges with which BitInstant works have strict limits and AML procedures. Additionally, BitInstant does not store any customer funds and does not engage in currency exchange.”

Compare to BitInstant’s FinCEN registration data below, which was supplied to FinCEN by BitInstant, and clearly lists it as a “money transmitter”. The Brooklyn registration address is listed under Charlie’s father Alan Shrem; according to Reuters, he will be confined there under house arrest until his trial.

The Charlie Shrem Files, Part 2

More recent documents from the class action lawsuit against BitInstant show concerning things about the company. In particular, from the plaintiff’s attorneys:

“We represent Plaintiffs in the above referenced action. We write to request a status conference because this case has come to a halt; Defendant BitInstant is not engaging in discovery.

Plaintiffs filed their complaint on July 8, 2013. On September 9, 2013, Your Honor entered a case management plan, which required the Parties to exchange Fed. R. Civ. P. 26(a) discovery by November 4, 2013. To date, Defendant has not made its initial disclosures. Plaintiffs served document requests on November 14, 2013. To date, Defendant [BitInstant] has not responded. The Parties also have yet to discuss electronic discovery protocols. Defendant’s counsel has represented that she intends to file a motion to be relieved as counsel. However, that intention does not discharge Defendant of the responsibility to engage in this litigation. As such, Plaintiffs respectfully request a conference with the Court.”

Response from BitInstant’s counsel:

“On December 20, 2013, after Plaintiffs’ counsel filed his letter via ECF, we had another telephonic meet and confer to discuss the possibility of settlement given our client’s [BitInstant's] financial condition. At that time, we confirmed that an insurance policy does not exist and that for several reasons, we intend to seek to withdraw as counsel in this litigation. However, in a final attempt to resolve this dispute in the interest of all parties, we agreed to assist in another settlement offer before seeking to withdraw as counsel. On December 30, 2013, we received that settlement offer from Plaintiffs’ counsel and conveyed it to our client [BitInstant]. We have informed our client [BitInstant] of the urgency of settlement in light of the discovery deadlines in this case, and we have advised our client that we intend to file a motion to withdraw.”

Letter to Judge McMahon

Withdrawal of Counsel

The Charlie Shrem Files, Part 1

This morning, Charlie Shrem, CEO of BitInstant, was arrested on money laundering charges. As someone who knew Charlie and who used to run a Bitcoin company, I am interested in the case, and will over the coming days post a number of documents related to BitInstant and the Bitcoin business.

First up: the class action lawsuit filed against BitInstant by angry customers in July of 2013. By their own admission, BitInstant received 17,300 customer complaints in their last month of operation, with many saying they never received the Bitcoins they paid for. Below are copies of the original complaint, BitInstant’s motion for dismissal, and the plaintiffs’ response.


Get every new post delivered to your Inbox.

Join 56 other followers