We're unable to sign you in at this time. Please try again in a few minutes.
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Research Letter |

Optical Biopsy of Bladder Cancer Using Crowd-Sourced Assessment

Stephanie P. Chen, BS1; Sarah Kirsch, BS2; Dimitar V. Zlatev, MD1; Timothy Chang, MD1; Bryan Comstock, MS2; Thomas S. Lendvay, MD2; Joseph C. Liao, MD1,3
[+] Author Affiliations
1Department of Urology, Stanford University School of Medicine, Stanford, California
2Department of Urology, University of Washington, Seattle
3VA Palo Alto Health Care System, Palo Alto, California
JAMA Surg. 2016;151(1):90-93. doi:10.1001/jamasurg.2015.3121.
Text Size: A A A
Published online


This study assesses the use of confocal laser endomicroscopy to diagnose and grade bladder cancer and applied crowdsourcing to determine the barriers to learning how to diagnose cancer using confocal laser endomicroscopy.

Crowdsourcing and optical biopsy are emerging technologies with broad applications in clinical medicine and research. Crowdsourcing, an interactive digital platform that uses multiple individual contributions to efficiently perform a complex task, has been successfully used in diverse disciplines ranging from performance assessment in surgery to optimization of tertiary protein conformations.1,2 Optical biopsy technologies provide real-time tissue imaging with histology-like resolution and the potential to guide intraoperative decision making.35 An example is confocal laser endomicroscopy (CLE), which can be used for the diagnosis and grading of bladder cancer.6 To further assess the adoptability of optical biopsy as a diagnostic tool, we applied crowdsourcing to determine the barriers to learning how to diagnose cancer using CLE. We hypothesized that a nonmedically trained crowd could learn to rapidly and accurately distinguish between cancer and benign tissue.

Figures in this Article

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal
• Rent this article ?

First Page Preview

View Large
First page PDF preview


Place holder to copy figure label and caption
Figure 1.
Representative Screenshots of Online Modules for Crowd-Sourced Assessment of Bladder Cancer Using Confocal Laser Endomicroscopy (CLE)

A, Each crowd worker was presented with a computer-based CLE training module that included previously validated diagnostic criteria of a cancerous urothelium and a benign urothelium. B, Crowd workers were then asked a test question. An incorrect answer excluded the crowd worker’s responses from subsequent analysis. C, Crowd workers were randomly assigned to evaluate 1 of 12 video sequences. The video sequences consisted of 3 benign urothelia and 9 cancerous urothelia (4 low-grade carcinomas and 5 high-grade carcinomas). Crowd workers were asked to designate the video image as cancer or benign, as well as evaluate 6 microscopic features (flat vs papillary, organization, morphology, cellular cohesiveness, cellular borders, and vascularity). Crowd workers could elaborate on their observations with free text responses. Additional CLE videos could be reviewed by reentering the system. Each CLE video received a minimum of 100 responses.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.
Diagnostic Accuracy of Crowd-Sourced Assessment of Confocal Laser Endomicroscopy Imaging for Bladder Cancer

To classify a video image as a cancerous urothelium, a threshold of 70% agreement by the crowd was used on the basis that this represented the lowest percentage with a 1-sided 90% CI that excluded a random classification for cancerous vs benign by the crowd. The crowd was able to accurately distinguish between a cancerous and a benign urothelium in 11 of 12 video sequences (92%), with 1 video sequence of (low-grade) cancer incorrectly classified as benign. Diagnostic accuracy was lowest for papillary structure; this provides a presumptive explanation for the single erroneous video classification for which the majority of crowd workers missed the presence of neoplastic papillary features.

Graphic Jump Location




Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.


Some tools below are only available to our subscribers or users with an online account.

0 Citations

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal
• Rent this article ?

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections