Challenging Peer Review (on several fronts); Consumerism in Health Survey 2007; Accepting suggestions from Customers using Web 2.0 at Starbucks

March 28th through March 29th:

  • WordPress ? Search and Replace « WordPress Plugins – WordPress 2.5 is out. I have a feeling this plugin will be useful to have handy
  • JAMA — Preserving Confidentiality in the Peer Review Process, March 24, 2008, DeAngelis and Thornton 0 (2008): 299.16.jed80000 – With tremendous respect for Catherine DeAngelis’ leadership during a tough situation. I am left wondering if the best place to hide is out in the open – if peer review became more Web2.0 like. What would happen in a situation like this?
  • Findings From the 2007 EBRI/Commonwealth Fund Consumerism in Health Survey – EBRI – About 2 percent of the population is enrolled in a consumer directed health plans. Significant points for me: (1) almost half of the population with a chronic condition reports not filling medications or skipping doses or delaying care due to cost. Sobering reminder that patients can and do choose to do what we doctors prescribe. (2) “There have been no significant gains int he provision of information on provider cost and quality by any health plan type over the three years of the survey. There has been no increase in the share of CDHP or HDHP enrollees who say their health plans provide them with quality and cost information about their providers, and they remain no more likely to receive such information than enrollees in more comprehensive plans.” Okay, one more point – they did not ask about the impact of involvement in care in choosing a health plan – no mention of medical records access or involvement in information sharing at the level of the encounter.
  • My Starbucks Idea – How about doing this for a health care org?
  • Bronson Beta – Mail.appetizer – Nice Mail notification tool, Leopard

10 Comments

As a Salesforce.com ("SF") administrator at my day job, I've been aware of how Starbucks is taking the salesforce.com "Ideas" feature and using it to collect comments from its own customers.

Normally the IdeaExchange feature is used within a company for SF users to complain or request features, and it's used by SF itself to collect complaints and requests from its customers. (Amusingly, when you want to report a bug, they tell you to report it on the IdeaExchange, which has led some users to refer to it as the BugExchange. Ah, spin. "That's not a bug – it's an idea!" Anyone notice Monty Python's "That parrot's not dead" in there?)

Anyway it's a splendid idea. The big challenge will be whether the organization (Starbucks or anyone else) has the stones to *listen* if customers (aka patients) tell them they're full of crap on a particular point!

I'm a little stunned by the JAMA editorial. Mind you, I speak as a lay reader, both the beneficiary and the victim of how the process works and doesn't work.

> We firmly believe that ensured confidentiality of reviews

> allows reviewers to provide professional critiques of manuscripts

> without fearing potential repercussions from authors.

What's up with that??? Repercussions? Who's in a position of punitive power here, able to punish someone who doesn't say the truth?

I'm not trying to be naive, but if in fact there's a power structure that people fear, then this could turn out to be an industry that's in danger of collapsing very rapidly.

So, like, what was she talking about?

And, btw, is anyone involved in this process working vigorously to root out the causes of *misleading* peer-reviewed conclusions?

From the consumer's point of view (i.e. the one whose family holds a funeral if the answer was wrong), I'll tell you what I want: I want somebody to take gatekeeper responsibility for warning the reading public (including MDs) when all the best peer-reviewed data is dangerously out of date.

How many years is it (I read it recently) for a research idea to go from "gleam in the eye" to landing in someone's in-box? It was either 7 or 14 years.

Anyway that's a side issue. I just can't tolerate the idea that the peer review process happens in secret, and I'm *livid* at the idea that it happens because of fear of repercussions.

So, enlighten me – who are the feared repercussers?

Dave,

I didn't realize this is where that Starbucks app came from. I love the behind the scenes on these things – it demystifies them and makes it easier to envision doing them ourselves.

You're right about having the fortitude to listen. After just having come back from Medical College of Georgia (see my 2 part series on this institution), I know it can be done for sure,

Ted

Dave,

I think it is interesting me likewise that you picked up on this point, one that I think a lot of us in the profession take for granted.

I don't know the answer for sure, as I don't travel in peer reviewed circles. The professional community I work in is in operations and implementation in a field where there is still a lag in quality research to guide our efforts. I have been postulating that maybe the peer review research community will never catch up to us due to the rapid rise of Web2.0 as a means for disseminating and rating information (the "healing/feeling ecosystem").

My guess is that a prominent author in a field might submit an article that would be reviewed by a junior investigator working to get established, and a negative review might impact that junior investigator's ability to get grants. If anyone else reading this has more information, please comment.

I do think it's correct to say that there are problems with this system, given the evidence presented in studies like this, and that many people in the peer reviewed world are interested in correcting these.

Last thing, you almost got the number right about years it takes: it's allegedly 17, and it's years from research to use in your care. This number came from an article published in 2000 (Balas E, SA B. Managing Clinical knowledge for health care improvement. In: Yearbook of Medical Informatics; 2000). I only recently tracked down the reference and I have never read the article; I suspect others are in the same boat.

Re 17 years from conception to "use in your care": I have full respect for a methodical process and for the scientific method. I was raised an engineer, and I know if people kid themselves about whether they've got reliable facts, bridges and buildings collapse, software doesn't work, etc.

Having said that, it's REALLY IMPORTANT (yes I'm shouting) for people to realize the peril of outdated information!

I've said before, when I was a pup of a product manager, we could study a situation and design a product and engineer it and build it and launch it and the situation would be identical to when we conceived it. Companies who work that way don't exist anymore.

The metaphor isn't perfect but the underlying issue is the same: the rate of introduction of important new information means it's VERY important for patients and their doctors to get up-to-date information, or they could be missing a life-altering or -saving fact.

When my own cancer hit last year, ALL the misleading information I found was from well-respected peer-reviewed sources. It turned out that the only peers worth a damn were my own peers – other patients in the same boat as me.

If you're now telling me that the whole peer review process is essentially corrupted by fear of repercussions, to the point where secrecy is the only solution, then that undermines the fundamental belief I cited at the top.

What's the deal here – does no institution truly put the patient's needs at the top of the priority stack?

The medical world has been a fine example of a strong hierarchical system. Very strongly top-down. Ted, I suppose, can confirm that in hospital settings, most developments of new "improvements" in the various parts designed to treat, care and track patients are designed from the top down, requiring long data collection period, and an even longer analysis period.

Just look at how long it took to get the concept of "patient-centered care" moved from a few academic publications to the clinical settings where the term finally got embraced while its content got rejected 🙂

BTW, a big thanks to Ted for fining the Balas article. It's good to know that the 17 years theory has some basis!

Going back to the peer-review system and the delay in availability of state-of-art medical knowledge, it is quite evident that we are going to live in a very unsettled world for a while. Publishers-owned Per-reviewed medical publications have evolved over a long period of time and do represent the best system for the pre-internet world. Open Access (OA) peer-reviewed web-based journals such as JMIR (Journal of Medical Internet Research) or the various PLOS journals or any of the OA journals published on BioMed Central offer a much faster publication cycle. Still, they cannot compete, in terms of delivery speed with medical online groups.

e-patients have a responsibility in figuring what must be learned from classical peer-reviewed publications and what can then be gathered from discussions with their own peers. I suppose that, as usual, that equation will vary depending on the type of disease. The rarer the disease, the more e-patients should rely on their peers for accurate information. Dave's personal experience is typical. The power of participatory medicine can only be understood if you experience it directly. Almost the antithesis of peer-reviewed publications!

With regard to anonymity, I think Ted and Gilles have made the key point pretty clearly, but perhaps not bluntly. That leaves me the opportunity to be blunt.

If you peer review people openly and honestly, you will run into people (LOTS of academic people) who have personalized the process, and will look for revenge. Frankly, a lot of these folks went into science thinking it was fun, and would afford them a fun career, allow them to raise a family and live the American dream. Like most of us, they've watched while 1% of the population, including the directory of their hospital/institute/university, gets rewarded way out of proportion to contribution. Thus, their entire being, worth as individuals and sense of self gets invested in that research project. When I give it a thumbs down, there will be hell to pay, sooner or later, if they know who I am.

I typically identify myself as a review if the option exists. I have no idea if that has come back to haunt me. But i suspect it's only a matter of time.

AND, yes Dave, it should be different. But, it's not unique to science and research. web 2.0 is also filled with ego, flare ups and irrational behavior. Imagine if every time you got into an argument on line, you wondered if the guy you were ranting at was also the guy who writes your paycheck? Would he be secure enough as a person to compartmentalize that "discussion" and not seek retribution in the real world?

Gentlemen,

Great commentary and vision from a different side of the back yard.

I think that in general the human race is a top down organization. My basic premise, without finding fault with any person or institution, is that more involvement of patients, families, and communities is better.

In Toyota fashion, I tend to be interested in learning where we came from and why things are the way they are, discovering the good intentions behind them, and amplifying those. Glass 1/2 full, or as I say, 3/4 full,

Ted

There may be the beginnings of some tectonic shifts, though. The peer-reviewed system fits beautifully into the medical system before the evidence-based folks started thinking about the real nature of evidence. Medical authority tended to rest with the weightiest expert (aka the loudest shouter). Now, at least some folks are clamoring for data.

This is also reflected in the way the FDA collects adverse events reports. In times past, the agency was interested only in reports from health care professionals, preferably doctors, and patients were discouraged from submitting their experience. Now, the agency is trying to think about ways to tap into patients' experience and their willingness to report it without becoming overwhelmed.

Dan,

> it's not unique to science and research …

> Imagine if every time you got into an argument on line,

> you wondered if the guy you were ranting at

> was also the guy who writes your paycheck?

Splendid example, valid point. And I humbly suggest that there's an important difference between a ranter and someone (or some body of people) who claim to be the keeper of truth, the protector of people who otherwise might hurt themselves.

That's exactly what we keep seeing, as people warn patients about "sometimes fatal" mistakes, even though no such case has ever been documented, and MANY cases have been documented of deaths from well-credentialed people conducting well-peer-reviewed procedures.

Now, if you're saying that Dr. X might be peer-reviewing his/her *boss's* own work, that's a different sticky problem. I think in that case I'd recuse myself.

Ted Eytan, MD