Evidence Soup

Fun-with-Evidence Friday: Limericks about wine swirling and scams about shoe toning.

Happy Fun-with-Evidence Friday. I have two things for you.

First, from the always-delightful Annals of Improbable Research. Scientists are commended for contributing valuable evidence about wine swirling. Poets chose co-winners in the Wine Swirling Limerick Competition, which honored the study 'Oenodynamic': Hydrodynamic of Wine Swirling, by Martino Reclari et al. (Video here.)

Winning limericks came from investigator Patrick McKeon:

In a lab during end of year crunch,
Thought a researcher needing a hunch:
 'This wine I do cherish,
  So (publish or perish!)
I'll send in a film of my lunch.'

And from limerick laureate Matrin Eiger:

They use several metrics to show
The circular motion and flow
  Of wine as it swirls,
  As it whirls and it twirls,
Although why, I am too drunk to know.

Shame on you, Joe Montana. And of course, there's the settlement of charges against Skechers for false claims about their so-called toning shoes. The U.S. Federal Trade Commission announced the Skechers shoe company will pay $40 million for deceiving consumers with cherry-picked evidence (provided by celebrity spokespeople). Among other things, the FTC alleged unsupported claims that Shape-ups would provide more weight loss, and more muscle toning and strengthening than regular fitness shoes.

Gotta love their can-do attitude, though: According to the San Francisco Chronicle, Skechers CFO David Weinberg denied the allegations, saying the company only settled to avoid a lengthy legal battle. "While we believe we could have prevailed in each of these cases, to do so would have imposed an unreasonable burden on the company, regardless of the outcome."

Oops. As The Consumerist points out, one chiropractor who testified in support of the shoes said "After performing a six week clinical trial testing the benefits of SKECHERS Shape-ups, I am confident in recommending them to patients to increase their low back endurance and improve gluteal strength. Patients also benefited from weight loss and improved body composition.” But turns out he is married to a Skechers marketing exec.

Have a great weekend, everybody.

Hard evidence teaches hard lessons: Rapture billboard in the rearview mirror.

Happy Fun-with-Evidence Friday. I've been traveling for family stuff. Driving I-80 through Wyoming, I spotted this orphaned billboard. Remember that Rapture many were predicting for May 21 last year (according to Harold Camping's biblical numerology)? I got this quick snapshot from the car (somewhere between Laramie and Little America). The yellow circle to the left reads "The Bible Guarantees It!".

This is why you can't get a straight answer out of an attorney or a statistician. I've got to admire people who show such courage in their convictions. After a Judgment-free May 21 came and went, Camping said he'd had a "really tough weekend." But you can't fake reality. Past performance is not a reliable indicator of future anything, etc etc. And while some made pretty funny jokes about punking the Rapture, others were seriously hurt by their beliefs, which isn't so funny.

Now folks are predicting Doomsday on December 21 - all that Mayan calendar, 12/21/12 stuff. Dan Piraro, creator of the Bizarro comic (and most awesome member of my high school graduating class), offers the best explanation I've seen: That's all the room they had on their ancient calendar.

Happy weekend, everybody. I look forward to dishing up more Evidence Soup as we approach the End of Days.

U.S. agencies aren't supposed to allow political interference with scientific evidence. Good luck with that.

Evidence Soup is back in business. These past 3 months, I've been distracted by a number of things, including a move from Denver, Colorado to the San Francisco Bay Area. So, where were we?

Will "4.37 Degrees of Separation" play at the multiplex? Awhile back I wrote about recent research to test the Six Degrees of Separation theory. New evidence suggests that people are separated by an average of 4.74 degrees (only 4.37 in the U.S.). Doesn't really roll off the tongue, and I wouldn't expect Will Smith to star in a sequel. But this latest research applies only to people on Facebook; a New York Times piece reminds us "the cohort was a self-selected group, in this case people with online access who use a particular Web site".

Desperately seeking scientific integrity. Early on, President Obama launched an effort to ensure that the public can "trust the science and scientific process informing public policy decisions". His March 9, 2009 memorandum made agencies responsible for "the highest level of integrity in all aspects of the executive branch's involvement". Great idea to de-politicize science, though it's much easier said than done -- as evidenced by the nearly-two-year delay in providing "guidelines" for agencies issuing this policy. Really. Those guidelines were published December 17, 2010 [pdf here].

Oh, and meanwhile the Obama Administration was widely criticized for using sloppy science to support its moratorium on offshore drilling after the BP disaster on the Gulf Coast. It's never simple: In real life it requires weighing risks, factoring in economic impacts, and making political tradeoffs (consider this analysis of the economic impact of that ban - [pdf here]). A scientific integrity rule won't give you all the answers you need when making such complex decisions: It's surely not as simple as determining whether scientific findings were manipulated for political purposes.

As explained in a recent National Public Radio story, now we're seeing the first challenge to federal evidence-gathering under this new regime: It's directed at the Bureau of Land Management (a branch of the Department of the Interior). More about that in a moment.

Why is this so hard? It's been rough going for agencies issuing scientific integrity policies. The basics are straightforward enough: Preventing people from twisting or quashing scientific evidence. But here are some reasons why the process is so problematic.

1. Scope. There's no bright line identifying what "science" should be included in an assessment, and therefore subjected to integrity requirements. This complicates things enough when you're working with "objective" evidence. It's even trickier when you bring in fuzzier stuff, like the dismal science (economics), or risk assessment. How can we say for sure what should be considered? Our assumptions about what's important determine what evidence we recognize - whether consciously or subconsciously. Our choices are influenced by our values - but we may not be fully aware of our values, or we may not want to articulate them in a transparent way.
2. Dissemination. Each agency's integrity policy is supposed to provide for open communication, and guide how evidence is presented to the public (see the 2010 guidelines mentioned earlier). It wouldn't serve anyone to have a free-for-all; consistent, controlled dissemination can improve usefulness and understanding. But not everyone agrees on the rights and responsibilities of scientists who want to discuss their findings with the public.
3. Whistle blowing. Among the substantial hurdles is the handling of whistle blowers. (I suppose if such a policy is going to have teeth, the people who want to blow whistles need to feel they can do so without losing their heads.)
4. Transparency. Scientific groups - such as the Union of Concerned Scientists - say they still want to see external accountability under these policies. So far, investigations of misconduct are internal.

12291 all over again? Thirty years ago, President Reagan signed Executive Order 12291, requiring cost-benefit analysis for 'major' federal regulations (those expected to impact the U.S. economy by $100 million or more). Clinton issued a similar order in 1993. In theory, this should have de-politicized some agency decision-making processes. The results (or lack thereof) were the subject of my doctoral dissertation.

As with the new mandate for scientific integrity, a requirement to weigh regulatory costs against benefits leaves lots of room for interpretation, and requires value judgments. When EPA issues a rule under the Clean Air Act, it's difficult enough to estimate how many hospital visits or early deaths are caused by a particular type of airborne particulate matter. But figuring out the social costs is harder still: Requiring businesses & governments to reduce those emissions can lead to job cuts and economic loss, which themselves cause poverty and negative health impacts.

Challenging BLM's process. Citing the new scientific integrity policy, a group called Public Employees for Environmental Responsibility (PEER) has filed a complaint against the BLM, saying "The U.S. Bureau of Land Management is carrying out an ambitious plan to map ecological trends throughout the Western U.S. but has directed scientists to exclude livestock grazing as a possible factor in changing landscapes....  [O]ne of the biggest scientific studies ever undertaken by BLM was fatally skewed from its inception by political pressure.... As a result, the assessments do not consider massive grazing impacts even though trivial disturbance factors such as rock hounding are included, [and they] limit consideration of grazing-related information only when combined in an undifferentiated lump with other native and introduced ungulates (such as deer, elk, wild horses and feral donkeys)." I didn't know we had a feral ungulate problem. But I digress.

This is a good example of how choices about collecting evidence can strongly influence the results. NPR explains that the Dept. of Interior has a scientific integrity officer who is responsible for investigating allegations of political interference. I wish him Godspeed.

 

Is 'six degrees of separation' fact or fiction? Social scientists collect evidence to find out.

We've always heard about Six Degrees of Separation. Now let's see if evidence backs it up. As explained in the Mercury News, "The world's population has almost doubled since social psychologist Stanley Milgram's famous but flawed 'Small World' experiment gave people a new way to visualize their interconnectedness with the rest of humanity. Something else has also changed - the advent of online social networks, particularly Facebook's 750 million members, and that's what researchers plan to use."

You can join in. Social scientists at Yahoo! and Facebook have launched the Small World Experiment, "designed to test the hypothesis that anyone in the world can get a message to anyone else in just 'six degrees of separation' by passing it from friend to friend. Sociologists have tried to prove (or disprove) this claim for decades, but it is still unresolved.

"Now, using Facebook we finally have the technology to put the hypothesis to a proper scientific test. By participating in this experiment, you'll not only get to see how you're connected to people you might never otherwise encounter, you will also be helping to advance the science of social networks."

Photo credit: Film-Buff Movie Reviews, where they play Six Degrees of Kevin Bacon. I chose this picture of young Kevin because this weekend I saw a trailer for the Footloose remake. Some things should probably stay un-remade. (I was there to see Crazy, Stupid, Love, which I highly recommend. Kevin's in that one, too.)

Big Ideas may not have supporting evidence, but they sell books by the boatload.

Haven't we had enough of authors pitching an oversimplified analysis of something important? It's great to boil things down into plain language, but when an entire book is based on A Big Idea, complicated things are glossed over, evidence is cherry-picked, we get bamboozled. (And books are sold. Maybe I'm just jealous.)

Writing in New York Magazine, Kathryn Schulz takes Tina Rosenberg to task for touting a "social cure", formerly known as peer pressure, "a panacea so powerful that it can help you quit smoking, lose weight, escape poverty, ace calculus, combat Islamic terrorism, overthrow a dictator". (Thanks to Elizabeth Lusk, @gestaltKT, for the link. Here's her Evidence Soup interview.)

Group Think praises Rosenberg's earlier work (she is a MacArthur Foundation grant recipient). But Schulz has a bone to pick with her latest book, Join the Club: How Peer Pressure Can Transform the World. [Photo courtesy of New York Magazine.] She explains:

Big Idea books have been around for a long time; see The Communist Manifesto. But the Big Idea Book Club (I mean “club” as Rosenberg defines it: an identifiable in-group with enough status to influence the behavior of others) is a recent phenomenon. Its accidental founder and president in apparent perpetuity is Malcolm Gladwell. Its membership, like the membership of most powerful groups, is largely male. Its combined sales are stratospheric; whatever these books are hawking, we can’t stop buying it.

As for the books themselves, I’ll generalize (as, often, do they). Big Idea tomes typically pull promiscuously from behavioral economics, cognitive science, and evolutionary psychology. They coin phrases the way Zimbabwe prints bills. They relish upending conventional wisdom: Not thinking becomes thinking, everything bad turns out to be good, and the world is—go figure—flat.

(With Gladwell’s Blink, this mania for the counterintuitive runs top-speed into a wall, crumples to the ground, and stares dizzily at the little birds circling overhead. This is, let me remind you, a best-selling book about the counterintuitive importance of thinking intuitively.) What troubles me about the Big Idea Book Club is the way ideas often slide toward ideologies - grand unifying theories of culture, cognition, happiness, talent, the Internet, the future, you name it.

Well said, Ms. Schulz.

Granular smart grid evidence creates 'aha' moment: Data every 15 seconds vs. 15 minutes.

Peak load not at 5:00pm after all? Typical communications from the local utility ask us to minimize applicance use around 5:00pm, because that's when peak load occurs. That's a busy time, no doubt. But our beliefs may be based on incomplete evidence.

Last week's Science Friday was recorded in San Antonio, Texas. The city's mayor, Julian Castro, and several others spoke about efforts to modernize the energy grid there and across the state. Turns out, their peak load happens closer to 10:30pm. This discovery was made after they began collecting very granular meter data during a pilot project.

Aha moment in Texas. Brewster McCracken, executive director of the Pecan Street Project in Austin, explained that "We kind of almost by accident have ended up with the world's deepest database on how people use energy and now gas and water. It's in 15-second increments.... The most granular data before that was 15 minutes."

"... that's obviously going to be pretty impractical and actually not necessary for utilities, but it is very important for product development to understand, you know, if you're going to try to create a product that is of value to a customer, it's really important to understand what the customer wants. And so one of the ways to do that is to find out their data. And we're finding out, for instance, in the summertime, surprisingly, that the peak time of electricity usage in terms of draw on the grid is 10:30 at night. It's not 5:00 in the afternoon, which was a huge surprise."

McCracken continued: "... here's why, actually. And it doesn't show up in 15-minute data. It only shows up when you take it down to a finer level. We do this in my family, a lot of folks do. You turn down your AC when you go to bed at night to make it a little bit cooler. And the AC has such a huge influence on your home energy usage that it's about - the draw, peak draw can be up to 20 percent higher at like 10:30 at night, 10:00 at night, as it is at 5:00 in the afternoon."

Anecdotal evidence from Colorado. It's 10:16pm as I finish this post. Next thing I'm doing? Cranking up the AC and turning in for the night.

The 'Baby Einstein' evidence debate continues. University sponsoring flawed study pays $175K in legal fees.

There's still a fight over the evidence on Baby Einstein videos, and it's being fought in the press. Today the Denver Post ran the story 'Baby Einstein' DVD creators find redemption in documents suggesting negative study was flawed. (Baby Einstein is a local company.)

I first wrote about the kerfuffle in 2007. It all began when the University of Washington announced a study finding that for each hour-per-day spent watching baby DVDs/videos, infants understood on average six to eight fewer words than those who didn't watch, and recommended that parents limit their use. "This analysis reveals a large negative association between viewing of baby DVDs/videos and vocabulary acquisition," they claimed. (There's a recap of events on Wikipedia. The mainstream press piled on in a big way.)

Much ado about nothing? Initially, I thought this seemed to be a case of a researcher making alarmist statements to the press, not because the evidence was significant, but to promote themselves. And that's still how it seems. I asked back then "Even if this is true, where is the evidence that 6-8 fewer infant words will matter in the long run?" Using this limited research to claim that Baby Einstein (and other) videos are harmful seemed over the top. Another reason I doubted the importance of the study: "Only 17 percent of 384 babies in the survey were put in front of videos for an hour or more each day. The average baby watched only about 9 minutes a day." One has to wonder what factors influenced the children's development during the, um, other 23 hours and 51 minutes of their day.

Show me all the evidence. The creators of Baby Einstein have been fighting back ever since. Bill Clark and Julie Aigner-Clark have obtained internal documents related to the study, and say they've succeeded in having $175,000 of their legal bills paid by the University of Washington. They also say the documents "confirm what they always suspected: The study was deeply flawed and unfairly characterized Baby Einstein products."

According to today's story, the Clarks found "correspondence from one researcher concerned about how certain results were analyzed. While children 8 months to 16 months who watched baby videos fell behind in vocabulary, the study also found that in children 17 months to 24 months, vocabulary increased and the negative effects evaporated." (Apparently that vocabulary rebound was part of the published report but was downplayed in news releases.)

No follow-up. An internal email, between a reviewer and one of the researchers, asked  "What's the notion about how (we're) reconciling the fact that there was an effect on the young kids but it washed out by the time they were 17-24, and we now will be wanting to follow the young kids when they're older?" Apparently there was supposed to be follow-up research with the same parents, but that was scrapped due to cost.

What a waste of resources. Disney eventually was pressured to offer refunds to Baby Einstein customers. The U.S. Federal Trade Commission (FTC) even wasted time on this. Eventually, as explained in the Washington Post, a separate group of researchers found that although "an hour a day of television viewing won't make your kid a genius, it won't harm his development either.... At first glance, the study's authors found decreased language and visual motor skills in kids who watched more television. But then, the researchers adjusted the results to account for the mother's age, income, education, marital status and vocabulary. And you know what? They found that mom's education level and vocabulary rather than the television greatly impacted baby's."

 

Evidence reveals least effective medical treatments: Field-tested do's and don'ts for primary care physicians.

The Good Stewardship Working Group has developed The "Top 5" Lists in Primary Care: Meeting the Responsibility of Professionalism. Their work was published May 23 in the Archives of Internal Medicine (paywall); free abstract here. Here's how they describe the project:

Physicians can adhere to the principles of professionalism by practicing high-quality, evidence-based care and advocating for just and cost-effective distribution of finite clinical resources. To promote these principles, the National Physicians Alliance (NPA) initiated a project titled "Promoting Good Stewardship in Clinical Practice" that aimed to develop a list of the top 5 activities in family medicine, internal medicine, and pediatrics where the quality of care could be improved.

NPA members sought to identify "ideal" medical activities, meaning they would be common in primary care practice, would be strongly supported by evidence, would lead to significant health benefits, and would reduce risks, harms, and costs. A number of primary care activites were field tested in family medicine, internal medicine, and pediatrics: Participants identified common clinical activities that could lead to higher quality care and better use of finite clinical resources. Based on the findings, the Good Stewardship group is recommending that its "Top 5" lists of activities be implemented in primary care practice across the United States.

Medscape recapped the findings in Most Wasteful Treatments, Procedures in Primary Care (free registration required). "A study commissioned by a reform-minded medical society has come up with specialty-specific lists of procedures in primary care that do little if anything to improve outcomes but excel at wasting limited healthcare dollars."

Medscape concludes that many of the Top 5 recommendations consist of "don'ts." For instance: "A prime example is routinely ordering diagnostic imaging for patients with low back pain — but with no warning flags, such as severe or progressive neurologic deficits — within the first 6 weeks, according to the study."

To put the findings into practice, NPA plans to "produce training videos to help physicians discuss the guidelines with patients and gain their understanding and trust, lest patients think they are being denied needed care (a common scenario when they ask for antibiotics to treat a viral infection)."

 

Three people to follow. So you can follow the evidence.

Three people I recommend following: John D. Cook, Carnival of Math, and William M. Briggs. Each of these guys uses math/statistics to explore the evidence in helpful ways. Thankfully, they bring the smarts *and* the funny.

1. John D. Cook is into programming, research, math, and statistics. He tweets via an impressive list of accounts (entire list here), including @AnalysisFact and @StatFact.

Responding to the common criticism of evidence that's not grounded in classic RCT (randomized, controlled trial) methodology, he writes: "As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence-based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence-based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."

2. Carnival of Math (oh, how I love that name). By day he goes by Mike Croucher, making insightful and witty observations, such as this one about the 95% confidence interval: "Last year around 27,000 psychology papers used a 95% confidence interval. So we can say with 95% confidence that at least 1,292 of them were wrong."

Mr. Carnival also writes technical stuff at Walking Randomly "because it's more fun than getting there in a straight line."

3. William M. Briggs bills himself as the Statistician to the Stars! He consults on statistical problems, and is an adjunct professor of statistical science at Cornell University (Ithaca, New York, USA). Awhile back, Briggs wrote an excellent piece Classical statistics is like a criminal trial: "Some thing has happened, and a hypothesis is put forth which, based on probative background information, the investigator thinks likely caused the thing. Evidence -- mostly in the form of “data”, but non-quantitative information too -- for this hypothesis is put forth.... The jury consists of mathematical formulae whose duty is to report solely on the likelihood of the evidence given the hypothesis is false. And since the evidence nearly always appears rare assuming the hypothesis is false, most trials result in a conviction -- meaning, the investigator’s viewpoint is confirmed."

Briggs explains how he thinks we should look at things, using global warming as an example: "Trying to fit carbon dioxide into the frame as the culprit for a historically observed increase in temperature is vastly less interesting and useful than in being able to accurately predict what will happen. It is easy enough to find enough evidence to convict our poor gas classically. But if it were part of a gang of gases, the others would go free. Meanwhile, in celebrating our conviction, we would remain ignorant of what will happen, or will issue poor predictions because we were so intent on assigning blame."

I've got my own evidence the world soon will end. Not because of the Rapture. It's the lawsuits and the whining.

It's an especially delicious Fun-with-Evidence Friday. After all, someone is loudly proclaiming to have evidence that the world is coming to an end (like, tomorrow). More about that in a minute. But first, I have my own evidence that the End of Days is fast approaching. Consider these items:

SMS/TCPA lawsuit against Twitter. Some folks unsubscribed from Twitter SMS messages. So Twitter sent them SMS messages confirming that they were unsubscribed. Now these folks are suing (possibly in a class action), claiming violation of privacy under the Telephone Consumer Protection Act. Details on TechCrunch.

Supreme Court allows cheap legal shot at Chipotle. My admiration for Chipotle's human resources policy is well documented. So it disturbs me that someone who's basically a litigious, gold-digging whiner is being allowed to use up their resources in court. Here's the complaint: People in wheel chairs can't see the food being displayed at Chipotle because the counters are a bit too high. Evidently, the individual bringing the suit has sued dozens of establishments for access violations, and taken cash settlements. Yet the U.S. Supreme Court is allowing the case to go on.

Wacky Warning Label competition. Sadly, U.S. companies often feel the need to place extreme warning labels on consumer products. So much so that there's a contest to determine the silliest cautionary statement. NPR interviewed Bob Dorigo Jones, one of the competition organizers, who recently announced finalists for 2011 (the contest is in its 14th year). They were particularly astounded by a brochure accompanying a child's bicycle. "There are pictures of little children riding bikes, most of them on training wheels, but the warning label says: 'The action depicted in this brochure is potentially dangerous. The riders seen are experienced experts or professionals.'" *sigh*

Robert Siegel of NPR reminds us that "If you turn up your radio dial, the volume will get louder."

Okay. About that other evidence. You may have heard that God's Elect are being called home on May 21. I was curious what evidence they have: Carl Sagan reminded us that "Extraordinary claims demand extraordinary evidence". I happened across an interview with Harold Camping, who calculated the doomsday date. In the NPR story Divining Doomsday,  Camping says "the Bible is written in a code, and for those who are able to decipher it, it's clear as daylight. With all the proofs that God has given us, and all the signs, I am absolutely certain [that Judgment Day will arrive on May 21]. It is going to happen. There is no Plan B."

NPR reminds us "even Jesus said he didn't know when Judgment Day would come. But Camping is not bothered by that, nor by the fact that he wrongly predicted Judgment Day once before, in 1994. 'It was based on incomplete research and I was quite aware that the research was incomplete,' he says."

Camping is confident he's correctly calculated the date this time. Soon we'll all know whether or not his evidence was valid.