A Higher Level of Evidence Synthesis
In medical research, different study designs provide different levels of evidence. Randomized controlled trials sit near the top. Meta-analyses that combine multiple trials sit above them. But there is an even higher level: the umbrella review.
An umbrella review does not look at individual studies. Instead, it collects all existing systematic reviews and meta-analyses on a topic, evaluates their quality using standardized tools, and summarizes what the entire body of summarized evidence actually tells us. It is a “review of reviews” the highest vantage point from which to assess a field.
Ma and colleagues (2026) conducted an umbrella review of treatments for erectile dysfunction. Their goal was not to discover new treatment effects, but to evaluate how much trust we can place in the existing evidence. The full text is available here:umbrella review by Ma and colleagues (2026) on ED treatments
The findings are sobering. The vast majority of meta-analyses on ED treatments are of low or critically low quality. This does not mean the treatments are ineffective. It means the evidence supporting them is weaker than most clinicians and patients realize.
What Is an Umbrella Review?
A standard meta-analysis identifies individual randomized controlled trials, extracts their data, and combines them statistically to produce a pooled estimate of treatment effect. The quality of that meta-analysis depends entirely on the quality of the individual trials it includes. If those trials are small, poorly designed, or biased, the meta-analysis will share those flaws.
An umbrella review moves one level higher. It searches for meta-analyses themselves, not primary studies. It applies quality assessment tools designed specifically for systematic reviews most commonly AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews). Each included meta-analysis is rated as high, moderate, low, or critically low quality based on explicit criteria: Did the authors register a protocol? Did they search multiple databases? Did they assess publication bias? Did they account for risk of bias in the primary studies?
The value of an umbrella review is not in reporting new treatment effects. The value is in revealing the methodological health or illness of an entire research field.
The Methods Used by Ma and Colleagues
The authors systematically searched for meta-analyses on any treatment for erectile dysfunction. They included both pharmacological interventions (PDE5 inhibitors, testosterone, herbal products) and non-pharmacological interventions (shockwave therapy, stem cells, platelet-rich plasma, lifestyle modification, psychotherapy).
Each identified meta-analysis was evaluated using AMSTAR-2. This tool has 16 items covering critical domains: protocol registration, search strategy completeness, justification for study selection, data extraction with duplicate review, assessment of risk of bias, appropriate statistical methods, investigation of publication bias, and declaration of funding sources.
The authors also extracted the pooled effect estimates from each meta-analysis typically mean differences in IIEF scores or risk ratios for treatment response and noted which interventions showed statistically significant benefits.
What They Found
The search identified 23 meta-analyses encompassing 1,191 individual primary studies. These meta-analyses reported 42 distinct pooled effect estimates across 36 different interventions for erectile dysfunction.
The range of treatments was broad. Oral medications (various PDE5 inhibitors) were the most studied. Testosterone therapy for hypogonadism appeared in several meta-analyses. Shockwave therapy was well represented. Stem cell therapy, platelet-rich plasma, herbal supplements, vacuum devices, pelvic floor exercises, and psychosexual counseling were also covered.
Then came the quality assessment using AMSTAR-2.
- Critically low quality – 13 of 23 meta-analyses (56.5%).
- Low quality – 7 of 23 (30.5%).
- Moderate quality – 2 of 23 (8.7%).
- High quality – 1 of 23 (4.3%).
In other words, 87% of the published meta-analyses on ED treatments were of low or critically low quality. Only one meta-analysis representing just 4.3% of the total met high-quality standards.
Why Are Most ED Meta-Analyses Low Quality?
The authors did not speculate extensively, but the common deficiencies can be identified from the AMSTAR-2 criteria.
- Lack of protocol registration. A preregistered protocol (on PROSPERO or a similar registry) is a basic requirement for a high-quality systematic review. It prevents outcome switching and post-hoc changes to inclusion criteria. Many meta-analyses skip this step.
- Incomplete search strategies. High-quality reviews search multiple databases (PubMed, Embase, Cochrane, Scopus, trial registries) with no language restrictions. Many meta-analyses use only one or two databases, missing relevant studies.
- No assessment of publication bias. Small positive studies are more likely to be published than small negative studies. A proper meta-analysis should assess this statistically (funnel plots, Egger’s test). Many do not.
- Inadequate risk of bias assessment. The primary studies in an ED trial may have serious flaws lack of blinding, high dropout rates, selective outcome reporting. A good meta-analysis evaluates these flaws and considers them in the interpretation. Many meta-analyses check boxes without meaningful integration.
- Heterogeneity ignored or mishandled. ED treatments vary widely in protocols. Shockwave studies use different energy levels, different numbers of sessions, different patient populations. Pooling such heterogeneous data without adequate subgroup analysis or meta-regression produces misleading estimates.
- Industry funding without disclosure. Some meta-analyses are funded by device manufacturers or pharmaceutical companies. This is not automatically disqualifying, but it should be disclosed and the potential for bias should be discussed. Many meta-analyses omit this information.
What This Means for Clinical Practice
The umbrella review does not conclude that ED treatments are ineffective. It concludes that the evidence base as represented by published meta-analyses is largely low quality. This distinction matters.
For treatments with very strong evidence from large, well-designed trials, the low quality of some meta-analyses is not a concern. PDE5 inhibitors have been studied in trials with thousands of patients, double-blind designs, and consistent results. Even a poorly conducted meta-analysis cannot make these drugs ineffective.
For newer treatments with smaller evidence bases shockwave therapy, stem cells, platelet-rich plasma the low quality of meta-analyses is more troubling. These treatments may still work. But the current evidence is not as robust as a glance at the literature might suggest. A statistically significant p-value in a low-quality meta-analysis does not provide the same confidence as a similar finding in a high-quality meta-analysis.
For patients, the message is to ask better questions. “What is the quality of the evidence for this treatment?” “How many high-quality randomized trials have been done?” “What are the chances this will work for me based on the best available data?”
For clinicians, the message is to exercise methodological caution when interpreting meta-analyses. Citing a meta-analysis is not sufficient. One must also evaluate that meta-analysis. The AMSTAR-2 tool, though designed for researchers, offers a framework that clinicians can use informally: Was there a protocol? Was the search comprehensive? Was bias assessed? Are the authors transparent about funding?
The One High-Quality Meta-Analysis
The abstract does not specify which single meta-analysis received a high-quality rating. The identity is less important than the proportion: only one out of twenty-three. This is a signal that the field of ED research needs methodological improvement.
Future systematic reviewers should register protocols, search thoroughly, assess publication bias, handle heterogeneity appropriately, and disclose conflicts. Primary trialists should design larger, longer, better-blinded studies. Only then will an umbrella review a decade from now show a different distribution.
Clinical Application
At Adult & Pediatric Urology (APUMN), we incorporate the findings of this umbrella review into our clinical reasoning. When evaluating evidence for an ED treatment, we do not simply accept the conclusion of a meta-analysis. We ask about its quality. Was AMSTAR-2 applied? Was the protocol registered? Were the primary studies of adequate quality? For first-line treatments like PDE5 inhibitors and testosterone replacement, the evidence is robust regardless of meta-analysis quality. For second-line or experimental treatments, we discuss the uncertainty explicitly with patients. We also track our own outcomes through clinical registries (similar to the AQUA registry discussed in the AUA adherence study), recognizing that real-world data from high-quality clinical practices can complement published meta-analyses when those meta-analyses are of low quality.
Conclusion
The umbrella review by Ma and colleagues is an important methodological audit of erectile dysfunction research. After evaluating 23 meta-analyses covering 1,191 primary studies, the authors found that 87% were of low or critically low quality. Only one meta-analysis met high-quality standards.
This does not invalidate ED treatments. PDE5 inhibitors remain highly effective. Testosterone helps appropriate patients. Lifestyle changes improve both erectile function and general health. But the umbrella review reminds us that published meta-analyses vary widely in trustworthiness. A statistically significant result in a low-quality meta-analysis should not carry the same weight as a similar finding from a rigorous, preregistered, transparently reported review.
The solution is not to ignore evidence but to demand better evidence. Researchers must design higher-quality primary studies and higher-quality meta-analyses. Clinicians must critically appraise what they read. And patients must be partners in this process, understanding that medicine is an evidence-based profession but evidence exists on a spectrum from weak to strong.
Author
Author: Gregory S. Parries, M.D., PhD
Author
Jerome P. Keating, M.D
