Critique of Meta-Analysis

Sir David Goldberg introduced me to the concept of meta-analysis. He was skeptical and expressed concern that it was difficult to build a sturdy wall as a conglomerate of miscellaneous pebbles, cobble, and boulders.

(I was reminded of a critique of Emmanuel Kant's system of philosophical thought as a sausage machine from which a homogeneous output flows from heterogeneous sources.)

Read what Senn (University of Glasgow Department of Statistics) says about meta-analysis, and try to avoid the errors:

Senn, 2009


p.s. You can learn a lot about a lot if you read Senn's paper carefully. Here is an illustration:

A meta-analysis by Hackshaw et al [16] in the BMJ considered passive smoking. The method involved weighting reported log-odds ratios using reported (or calculated from confidence intervals) standard errors. However, Peter Lee, in an extremely important but sadly neglected article [17] in Statistics in Medicine has pointed out that the fact that the standard error for a log-odds ratio is approximately equal to the square root of the sum of the reciprocals of the frequencies in the corresponding four-fold table provides various lower bounds on the standard error. Conversely, a given standard error implies a minimum sample size. In fact for a given total sample size N, the split of cases and non-cases in exposed and unexposed groups that gives rise to the minimum standard error is an equal split of N/4 subjects in each cell. It follows, for example, that for any reported variance, V [,] the total sample size, N [,] must satisfy the requirement that N ≥ 16/V. Similar inequalities exist for the total of any two cells and for the numbers in any given cell. 


Re-read this sentence fragment with care, and you will learn (or re-learn) a basic principle about 2x2 contingency table analysis that seems to be lost by many in the dense biostatistics curriculum [but here I would caution that Senn uses 'N' where an epidemiologist generally should be using 'n' so that big N denotes the population size and little n denotes the observed sample  size. [Statistics and epidemiology don't always have the same conventions here, but within epidemiology we can keep ourselves straight by paying attention to the population N versus the sample n.]

......the standard error for a log-odds ratio is approximately equal to the square root of the sum of the reciprocals of the frequencies in the corresponding four-fold table....


The rest of the same paragraph is worth reading and committing to memory. Doing so, you might get a sense of a rationale for senior epidemiologists always asking about the unweighted cell counts used to derive an X-Y estimate via odds ratios or log-odds-ratios. As  the cell count goes toward or to zero, the reciprocal of that frequency responds accordingly, and the size of the sum grows (as does its square root). What seems to be a large estimated association can be rendered essentially null by a small cell count in just one cell of the 2x2 table.

Of course, this postscript is not a critique of meta-analysis. It has to do with practical numeracy in epidemiology at a quite basic level that we could teach in primary school once simple division and square roots are grasped (even before logarithms are mastered). Epidemiologists worth their salt know and apply these principles of our field.

Child Psychiatry Diagnostic Assessments

Reliability and validity of child psychiatric interview, as of 1979, when I started paying attention.


More recent:
Kiddie-SADS and more

Egger-1

Egger-2


Click here for Potential futures

While chasing down these citations, I ran across the most amazing multiple co-authors work:

https://www.ncbi.nlm.nih.gov/pubmed/26799652



#child
#lapouse
#authorship

Farr on the concept of between-census population registers

#Farr
#tools
(Picture of Farr): click here 👍🏻

See the prior note on Humphreys's 1885 edited collection of Willam Farr's works.
The following image is from page 33 of that volume, and it is from an 1871 census report.

Farr is writing about the concept of a country's population.

A potentially useful quotation is shown below. (You might evaluate his 19th century noun choice as you might evaluate the 18th century Declaration of Independence's "We hold these truths....")

"Statistics is in some respects, in the present day, dealing with men like trigonometry dealing with lines and angles, able to deduce from certain given data others of which there is no trace; from a basis of observed facts other facts can be determined; ...."

Kratom in the news

NIDA:

https://www.drugabuse.gov/publications/drugfacts/kratom

DEA:
https://www.federalregister.gov/documents/2016/10/13/2016-24659/schedules-of-controlled-substances-temporary-placement-of-mitragynine-and-7-hydroxymitragynine-into

https://www.dea.gov/divisions/hq/2016/hq083016.shtml

https://www.dea.gov/pr/multimedia-library/publications/drug_of_abuse.pdf#page=84



NPR:
http://www.npr.org/sections/health-shots/2016/10/12/497697627/kratom-gets-reprieve-from-drug-enforcement-administration

Drug Policy Alliance:

http://www.drugpolicy.org/news/2016/10/unprecedented-move-drug-enforcement-administration-withdraws-emergency-kratom-ban

DAD:

http://www.drugandalcoholdependence.com/article/S0376-8716(14)00793-5/abstract

Continuing notes on William Farr

A series of notes on William Farr.
#Farr
#Susser
#tools
#Condorcet
#Switzerland

Back in 1975, I read this essay in my effort to understand epidemiology. It is in Am J Epidemiology, 1975.

Susser, M. and Adelstein, A. (1975)  "An Introduction to the work of William Farr," American Journal of Epidemiology 101(6): 469-476.

Link to M. Susser & A. Adelstein (1975)

(This is not a stable URL. Perhaps JSTOR has a stable URL and either Jacob or Adnan will do us the favor of writing a comment to this blog, with the stable URL in the note, either via JSTOR or the publisher, NCBI PMC, or other repository.).

Down below are two versions of URL link to Humphreys's edited version of Farr's collected works, possibly not a stable URL:
https://archive.org/stream/vitalstatisticsm00farruoft/vitalstatisticsm00farruoft_djvu.txt

Humphreys, N.A. (ed.) (1885) Vital Statistics: A Memorial Volume of Selections from the Reports and Writings of William Farr, London:  Office of the London Sanitary Institute, 74A Margaret Street, West Edward Stanford, 55, Charing Cross, S.W.

There is a google books version, where it is easier to read the tables via page image, but requires google sign-in:

https://play.google.com/store/books/details?id=uvKToo__d5QC

[Down at bottom of page there is a page number tool. Might need to use it to work from page to page. I sometimes get a blank page.]

In 1978, Morton Kramer showed me a table in his bound copy of this book, and allowed me to sit in his office to read the book. Many years later, while we both were professors at Johns Hopkins Schhol of Hygiene and Public Health, Mort inscribed his copy of the book for me as a gift. I've loaned it to JHU Professor Alvaro Munoz, but few others. There now is the online version, from which screenshots can be made, and perhaps the online version is the best place to start reading Farr in the original.

If time is short, I recommend these two starting passages:

During his two years' residence in Paris, Dr. Farr attended the lectures of Orfila, Louis, Dupuytreu, and Lisfranc on various branches 
of medical science ; of Andralon hygiene ; of Gay Lussac and Thenard 
on chemistry ; of Pouillet on natural philosophy ; of Geoffery St. 
Hilaire, Dumeril, and Blainville, on comparative anatomy and 
physiology ; of Cuvier on the history of natural sciences ; and of Guizot 
and Villemain on history and literature. It was during the course of 
his studies in Paris that the subject of hygiene, and of medical statistics 
bearing thereon, began to attract his special attention, and to engross 
his interest. 

AND

On leaving Paris, Dr. Farr and Dr. Bain [physician W.P Bain, Farr’s friend] travelled in Switzerland, and the latter writes : "I had many opportunities of studying and admiring my friend's character. In a diary which I kept of our tour, I find recorded that <Mr. Farr>, while of a simple disposition, is endowed with a vastness of ideas and a philosophic mind.” He gave evidence then of observation and research. “I well remember the scene at our inn at Martigny, when, after a walk to see the celebrated waterfall about four miles away, I returned weary and hungry to dinner. To my surprise, I found the entrance blocked up by at least a hundred of those miserable beings, the Cretins, who inhabited the Valais in great numbers. On inquiry of the landlord he told me that the gentleman inside had commissioned him to get together as many Cretins as he could, so that he might examine them. After some difficulty, I wedged my way into a room, where Mr. Farr was standing ....”

AND search for Farr's notes of admiration for Condorcet, French revolutionary and early contributor to sociology's emergence as a science, who shared a vision of universal education for all people, irrespective of sex, skin color, religion, or other personal or familial attributes used to constrain freedom of non-royals at that time.

Finally, let me draw you attention to archives.org as a bibliographic treasure house, with a handy search tool in the upper right hand corner at the top of each text file: 

https://flic.kr/p/Mb8mZz search tool link image

Be skeptical about weighted survey estimates and inspect the alternatives

Why do I insist that scientific articles must show unweighted numbers as well as estimates based on these two weights: (1) weighted by probability of selection only, and (2) weighted by the sampling weight and the post-stratification factors as well. If you study only (2), but not (1) and the unweighted numbers, you've started to drink some of the Jonestown kool-aid.

Read this article for some insights:

http://www.nytimes.com/2016/10/13/upshot/how-one-19-year-old-illinois-man-is-distorting-national-polling-averages.html?_r=0

If no information is available beyond (2), as in R-DAS, at least ask for an approximation of unweighted numbers, whether subgroup estimates have been reported based on surpassing a minimum unweighted observation count, and ask for a distribution of estimates stratified by quantiles of the available analysis weights. Look for anomalies.

#survey
#tools

Instructions to Reviewers Who Critique NIH K-type Applications (updated July 2016)

Definitions of Criteria and Considerations for K Critiques

1. Study them carefully.

2. Use them as a checklist when you are writing or critiquing a K application.

3. Mentors, co-mentors, consultants, and referees should be alerted to the specific questions, and their letters of support should highlight the criteria. I try to write a few sentences or paragraph on each item when I am writing letters of recommendation.

4. Recall that these are instructions. Some reviewers do not read them and some reviewers do not follow them.