The uptake of next generation sequencing (NGS) technology in clinical practice is happening at a rapid pace. The immense power of NGS has enabled a high throughput method for the efficient identification of sequence variants associated with heritable diseases. However, NGS, like any other laboratory methodology, has its limitations, especially within GC-rich regions and for large insertion/ deletion (INDEL) variants. In order to overcome these limitations, multiple independent methods may be required to achieve a test with appropriate sensitivity and diagnostic utility.

As our understanding of the human genome evolves, so does the number of genes associated with hereditary conditions. In the case of colorectal cancer (CRC), approximately 20-30% is due to familial inheritance, with 5% of CRCs due to highly penetrant single-gene mutations [1,2]. The three major types of inherited CRC are Lynch syndrome (formerly known as hereditary non-polyposis CRC), familial adenomatous polyposis (FAP), and MUTYH-associated polyposis (MAP). While there are six primary genes (MLH1, MSH2, MSH6, PMS2, APC, and MUTYH) that contribute to these three major types of inherited CRC, there are multiple additional genes which are also known to cause hereditary CRC or contribute to an increased risk for colorectal cancer. Attempting to simultaneously sequence each of these genes using the standard Sanger sequencing approach would yield a test far too expensive for routine clinical application.

In order to provide an inexpensive and comprehensive genetic testing for hereditary CRC, we developed and validated a clinical Hereditary Colorectal Cancer Panel (HCCP), enabling the screening of numerous hereditary colon cancer genes in a timely and cost effective manner. The complete HCCP assay is a complex integration of complementary technologies, including NGS. While most of the technologies utilized for this test are well established in molecular testing, NGS is a relatively new technology in the clinical setting. Therefore, we have focused the majority of this article on the development and validation strategies employed for NGS test implementation.

Gene selection and probe design

Candidate genes for the panel were selected based on feedback provided by Mayo Clinic clinicians specializing in cancer genetics, Gene Reviews and other reputable sources [1,3,4]. The original list included 22 genes, all of which corresponded to entries in Online Mendelian Inheritance in Man (OMIM) [McKusick-Nathans Institute of Genetic Medicine, Johns Hopkins University (Baltimore, MD), {July, 2013}. A set of evidence-based criteria was then used to rule out genes with insufficient evidence to demonstrate cancer risk, thus ensuring the clinical utility of the genes included on the panel. These criteria were internally devised by the Mayo Molecular Genetics Laboratory independent of the gene list generation (Table 1) and applied to the initial list of candidate genes to aid in the final selection of our gene panel. Only genes with well-established disease risk, as demonstrated by a substantial body of research (i.e., those satisfying criterion 3, 4 or 5 listed in Table 1), were included. Additional evidence such as frequency of germline mutations in affected versus control populations and segregation with colorectal cancer in families was also considered. The final list of 16 genes, in order of established risk, included: APC, MLH1, MSH2, MSH6, PMS2, MUTYH, BMPR1A, SMAD4, STK11, PTEN, EPCAM, SCG5, AXIN2, TP53, CHEK2, and MLH3. Once the final gene list was determined, a SureSelect XT Custom Target Enrichment System (Agilent Technologies, Inc., Santa Clara, CA) reagent was designed to reference sequence GRCh37/hg19 (February 2009) using the Agilent eArray design software.

Table 1: Scoring criteria for including or excluding genes for the Hereditary Colorectal Cancer NGS Panel. *Sufficient meaning a) repeated occurrence of germline mutations in “X” gene with greater frequency in individuals with the desired phenotype† than in controls b) germline mutations in “X” gene segregating with disease in multiple families c) the identification of germline mutations in unrelated individuals with tumors demonstrating absence of protein expression for that gene. †Personal or family history of colon cancer.

Test design and workflow

For the 16 gene panel, a combination of NGS (13 genes), multiplex ligation-dependent probe amplification (MLPA) and long-range PCR (LR-PCR) with Sanger sequencing (one gene, PMS2), and array comparative genomic hybridization (aCGH) (16 genes) was utilized (Table 2). Reportable NGS variants or low coverage regions were identified for confirmatory or supplementary sequencing.

Table 2: NGS Hereditary Colorectal Cancer Panel (HCCP) genes and testing methods.* UCSC Genome Browser, GRCh37/hg19 (February 2009) primary reference assembly. †CHEK2 and MLH3 are not associated with a known hereditary cancer syndrome defined by a distinct spectrum of tumors. However, literature suggests that mutations in these genes may confer an increased risk of colon cancer and therefore, are predicted to contribute to cancer risk in patients and families.

Processing of samples and data was managed utilizing the services of three clinical core facilities (NGS, Sanger, and aCGH), a bioinformatics pipeline and a reporting laboratory, each with their own role. The overall workflow is shown in Figure 1.

Figure 1: HCCP test work flow.

Library preparation

The SureSelect XT Custom Target Enrichment System [5] was utilized for library preparation and involved mechanical shearing (ultra-sonication) of 3 μg of genomic DNA to 150 base pair fragments with the LE220 (Covaris, Inc., Woburn, MA). Library preparation was automated on the Biomek FXp Laboratory Workstation (Beckman Coulter, Inc., Brea, CA) to include: enzymatic mediated end repair, adenine addition (a-tailing), adapter oligonucleotide ligation, and enrichment of adapter ligated fragments via a six cycle PCR. Following each of these process steps, the libraries were purified with the AMPure XP Purification system (Agencourt, Beckman Coulter, Inc.). Custom biotinylated RNA oligonucleotides complimentary to the target sequences were used for in-solution hybridization. After hybridization, Dynabead MyOne Streptavidin T1 beads (Life Technologies, Inc., Carlsbad, CA) were used to isolate (capture) the target regions and a 14 cycle PCR amplification added sample specific index sequences. A final quantitation of libraries was performed to ensure sufficient product (≥ 2.0 nM) for sample normalization and pooling. Library quantity and quality were evaluated on the Qubit 2.0 Fluorometer (Life Technologies, Inc.) and the 2100 Bioanlayzer (Agilent Technologies, Inc.).

Sequencing

Following library preparation, samples were normalized to a concentration of 2.0 nM, pooled, and sequenced on the Illumina GAIIx instrument [6] to produce 101 base single reads. Sixteen samples were pooled across each of four lanes of the Illumina GAIIx flow cell, with a total of 32 samples included on each instrument run.

Bioinformatics

The raw sequence output files were processed using CASAVA (Illumina, Inc.) and demultiplexed into individual sample FASTQ [7] sequence files. Initial read quality was assessed using the FastQC algorithm. Secondary analysis was performed using the CLC Genomics Server v4.1 (CLC Bio, Boston, MA), with read alignment to generate a BAM [8] file completed using the standard aligner and variant calling using the quality-score based caller. Variants require a minimum base quality of 20 and a neighbourhood (5 bases to the left and right of variant position) mean quality value of 15.

Alignment was done against the complete hg19 version of the human reference genome and subsequent variant calls reported back for the assay target region only. Post-secondary analysis was carried out using custom scripts to annotate the variants and load them into a custom-built web application for quality review and results interpretation. Validation of the variant calling was carried out using both synthetic and Sanger confirmed samples. INDEL detection was profiled using synthetic data and shown to be accurate up to insertion and deletion lengths of eight base pairs. The complete bioinformatics workflow was further validated through input/output testing by the Mayo Software Quality Assurance team. Target sequences with 100% redundant sequence found in off-target genomic regions were identified by a systematic analysis of all read-length fractions within the assay target, using the CLC aligner.

Data consolidation and confirmatory testing

Data from the testing performed by the three clinical core facilities were consolidated and reviewed, following completion of NGS, aCGH, and PMS2 sequencing. NGS variants or low coverage regions were identified and exported to a custom data consolidation software program, which transferred information to the Sanger sequencing core where confirmatory or supplementary sequencing was performed.

NGS assay validation

Samples were selected for end-to-end process validation with additional samples being run by NGS alone to satisfy NGS assay specific validation requirements [9-12].

In total, 115 individual samples were run: 80 cases known to be genotype positive, 20 phenotypically normal, expected to be genotype negative, ten analytical measurement range (AMR) to test the limit of detection at given variant frequencies, and five analytical range (AR) samples to test the limit of detection at various input DNA quantities.

Included within the 20 phenotypic negatives were six samples collected in either acid-citrate-dextrose (n = 3) or sodium heparin (n = 3), to test for interfering substances. All other samples were initially collected as whole blood samples with EDTA as the anticoagulant.

The assay validation was completed in five instrument runs, each with 32 individually indexed samples multiplexed together. Assay parameters tested were accuracy, precision, reportable range, reference range, analytical sensitivity (limit of detection), analytical specificity, and interfering substances/events.

The results described below are those obtained from the NGS portion of the overall assay validation.

Accuracy

Of the 100 samples sequenced to assess accuracy, (80 phenotypic positive and 20 phenotypic negative) two phenotypic positive samples failed due to technical error. Within the remaining 78 phenotypic positive samples, six samples contained multi-exon deletions or duplications that were not expected to be detected due to the limitations of NGS technology at the time of testing, and were excluded from the NGS accuracy calculations. Two additional samples contained INDELs greater than eight base pair.

While these INDELS were not expected to be detected by our NGS informatics process, we included these two samples in the accuracy calculations, leaving 92 total samples evaluated for accuracy. Accuracy was considered acceptable if the assay demonstrated greater than 90% concordance with the previously reported results obtained by Sanger sequencing or restriction fragment length polymorphism (RFLP) methods. Out of 92 samples, 89 were accurately called by the NGS assay for an overall accuracy of 96.7%.

Of the 72 phenotypic positive samples included in the accuracy calculations, 71 samples (98.6%) had variations accurately called by NGS. Single nucleotide variant (SNV) detection accuracy was 100% with 43/43 called correctly by NGS. INDEL accuracy was 100% for INDELs of one to eight nucleotides in length (n = 27/27) and 97% overall (range 1-14 nucleotides, n = 28/29), with only one of two INDELs greater than eight nucleotides detected by this assay.

A 14 base pair insertion was correctly called and confirmed by Sanger sequencing, however an insertion of 11 base pairs was not detected. Detection of INDELs greater than eight base pairs was a known limitation of this technology at the time of initial validation.

The 20 phenotypic negative samples were anonymous laboratory waste specimens referred for other non-related testing and had not been evaluated for hereditary colorectal cancer. Eighteen out of these twenty were identified as negative.

One sample had an alteration in MUTYH, consistent with carrier status for the autosomal recessive CRC, MUTYH (or MYH)-associated polyposis syndrome and thus not expected to significantly increase the risk for CRC. Another clinically normal sample harbored a c.1100delC variant in CHEK2. Mutations in this gene are known to be of lower penetrance, increasing a patient’s risk of breast cancer or Li-Fraumeni like syndrome, but at a lower penetrance where it’s not entirely unusual that we could find a mutation in a normal donor sample [3,13].

Nine of these samples each had a likely benign variant identified by NGS and confirmed with Sanger sequencing. Therefore, of the 20 phenotypic negative samples, two of them harbored a variant that would be reported out as significant, but not necessarily indicative of imminent disease.

Precision

Assay precision was assessed for both inter-assay precision and intra-assay precision through the use of 3×3×3 experiments.

These experiments included three phenotypic positive samples across three separate instrument runs (inter-assay precision) or three samples run in triplicate within the same instrument run (intra-assay precision).

In both cases, the stated acceptance criteria (100% concordance of variant calls within the regions of interest and a repeat rate of <10%) were met. Precision data is shown in Table 3.

Table 3: Results for Intra-Assay and Inter-Assay Precision Experiments *UCSC Genome Browser, GRCh37/hg19 (February 2009) primary reference assembly. †These mutations are the same. The Sanger call is correct and is called most 3'. ‡The Molecular Genetics Laboratory uses the historic nomenclature for the APC gene. See included genomic location for confirmation of mutation calling.

Analytical sensitivity was measured using a three-fold approach: conducting a false negative analysis for samples having full Sanger sequence data available, a study of AMR (limit of detection at given variant frequencies), and a study of analytical range (limit of detection at given input DNA quantities). There were a total of 26 samples utilized to measure analytical sensitivity, 20 phenotypic negative and six phenotypic positive samples. These samples were Sanger sequenced for all genes and exons on this panel. We treated all variants called by Sanger as an event for which either a correct variant call (true positive) or a variant not called (false negative) was made by NGS. The final calculation was made by dividing the number of true positives by the sum of true positives and false negatives.

Sanger sequencing identified 619 variants within the targeted region, of which eight were a common 16 base insertion not expected to be detected by NGS [13]. Of the remaining 611 Sanger identified variants; there were no false negatives within our data set, resulting in 100% sensitivity.

AMR and analytical range were set up to test the limits of detection (LOD). The AMR tested decreasing amounts of library preparation input material, starting at the manufacturer’s recommended 3.0 μg of DNA and decreasing in amounts to 2.0, 1.0, 0.5 and 0.25 μg. All assay quality metrics for library preparation, instrument run and NGS data analysis were met for the 3 μg and 2 μg inputs. The 1.0 μg input sample met the minimal library preparation quality requirements, but failed to meet instrument run and NGS data analysis quality requirements. The samples below 1.0 μg input failed to meet any of the quality requirements. Therefore, the determined AMR LOD for this assay was 2.0-3.0 μg.

The analytical range of the NGS assay was tested by creating synthetic genotypes of decreasing frequency by mixing samples with known homozygous variants with wild type DNA. We tested the ability of the assay to detect variants at frequencies of 50, 35, 30, 25, 20, 15, 10, 5, 2, and 1 percent. Due to bioinformatics filtering and the reduced reliability of Sanger sequencing to confirm reportable alterations below a variant frequency of 20%, we did not review variants ≤10% (even though they were often observed in the raw data), therefore the NGS analytical range LOD was set to ≥15%. Table 4 shows the NGS determined frequency for variants tested in the analytical range LOD experiment, down to 15%.

Table 4: Determined Variant Allelic Frequency (Analytical Range LOD) – artificial constructs homozygous for the listed polymorphism created at 50%, 35%, 30%, 25%, 20%, and 15%. *This alteration was detected at a lower than expected frequency within a region of low coverage (<100x). The sample was confirmed heterozygous by Sanger sequencing. †Both samples used for creation of the artificial constructs are homozygous for this polymorphism. The expected result would be 100% variant frequency across all dilutions. ‡Excludes MSH6 c.260+22C>G which did not meet minimum coverage requirements.

Analytical specificity

The analytic specificity of the NGS assay was measured in two ways. First, the reads mapped to the human genome and reads mapped to our target regions were evaluated. Read data from regions outside the reportable range were masked and not included in the assay validation. Table 5 shows read mapping data observed in one of the five instrument runs of this validation.

Table 5: Read mapping data from one of the five instrument validation runs.

The second measure of analytical specificity was a false positive analysis for those samples which had full Sanger sequencing data. Each nucleotide to be evaluated (all nucleotides within the region of interest, specifically- exons plus 30 bases flanking each exon) was treated as an event for which either a correct, wild type sequence, call (true negative) or an incorrect variant call (false positive) was made. The final calculation was made by dividing the number of true negatives by the sum of true negatives and false positives. The number of true negatives becomes quite large due to the fact that a single wild type sequence would contain 48,513 true negative calls and thus the reported value will be approaching one. The analytical specificity as calculated with our false positive analysis was (1260737/1260737) 100% specificity. While this number is quite impressive, one must take note of the raw false positive numbers as they do occur, only at a very low frequency as compared to all the bases interrogated in the design. In our analysis, we detected 18 false positives in total in nine total samples, averaging two per sample. These DNA alterations of undetermined significance were identified by NGS and confirmed by Sanger sequencing would eventually be classified as benign and therefore, not reported.

The reference range of our assay is the sequence noted in the UCSC Genome Browser, GRCh37/hg19 (February 2009) primary reference assembly. The acceptance criteria were met since ≥ 99% of the NGS reads generated on a run aligned to the reference sequence (Table 5). The reportable range for this assay is any reportable alteration within exons ±30 base pair of those genes listed in Table 2.

Bioinformatics

The bioinformatics pipeline was extensively validated through our Software Quality Assurance (SQA) team to be certain that all scripts and file transfer operations occurred with high fidelity. In addition, the ability of the bioinformatics pipeline to detect known variants and minimize the calling of false positives was assessed as part of our global assay validation previously discussed. In addition, specific operations were performed to assess the reproducibility of the CLC Bio and FASTQC software. These three samples were run through the bioinformatics pipeline three separate times and sample quality (Table 6), mapping reports (Table 7), variant information (Table 8), and coverage plot reproducibility (Figure 2) information collected.

Table 6: Sample quality - CLC Bio & FASTQC software.

Table 7: Mapping report - CLC Bio & FASTQC software.

Table 8: Variant information - CLC Bio & FASTQC software.

Figure 2: Sample VAL450 coverage plot* - 14 CLC Bio & FASTQC software reproducibility.

The sample quality results (Table 6) indicate that FASTQC is not biased when run on the same sample multiple times, and the sequence quality and GC content remain unchanged. The mapping report (Table 7) shows that the total number of reads, reads mapped to reference, and reads mapped to the bed file (target regions) remain unchanged for all three sample. The number of variant calls (Table 8) indicates the number of single nucleotide polymorphism (SNP) and insertion/ deletion polymorphism (DIP) calls remains unchanged across the three samples. Figure 2 includes an example from one of the samples and shows uniformity of coverage across the target regions for each of the three replicates of this sample. This data verifies the reproducibility of the CLC Bio software is 100% concordant for all three samples.

The continued maturation of NGS technology is allowing clinical diagnostic laboratories to implement these methods in pursuit of comprehensive routine testing. Through the work described here, we were able to develop, optimize, and successfully implement a comprehensive panel utilizing multiple molecular technologies useful for diagnosing patients with suspected hereditary colorectal cancer.

Gene panel based testing is rapidly gaining traction since it offers a more comprehensive and cost effective approach for diagnostic analysis of patients suspected to have a certain disease. NGS technology allows this type of testing to be performed less expensively than investigating just two or three individual genes by traditional Sanger sequencing methods. More importantly, given the diagnostic overlap with disorders that have different genetic involvement, broad panel based tests can be more diagnostically efficient than traditional single gene serial analysis approaches. Though payer reimbursement has been challenging in this space, careful analysis should prove NGS panel tests to be of a higher value for both patients and payers.

Another key advantage inherent to the NGS process is the sensitivity of the assay to detect specific mutations like single nucleotide variants and small INDELS. Our work here demonstrated that we could easily detect mutations down to 15% variant/normal allele ratios. This is a self-imposed limit within our laboratory, chosen since beyond this point we began to see false positive rates rising. In reality though, it was quite obvious to manually inspect our traces and see evidence of the variant alleles below 5%. In production, however, it is not practical for manual review of all loci, so our cut off was set at a level that allows for reliable calling of mosaic alterations automatically. As a point of reference, Sanger sequencing has a commonly recognized limit of detection in the 20% range [14]. Improved sensitivity utilizing NGS, as we have demonstrated here, is important since many hereditary diseases can be the result of new (de novo) mutations in the index cases. For example, in FAP the index de novo mutation rate is as high as 25%. Using traditional, less sensitive methods might not detect the causative mutation. Like any other technology, expert knowledge must first be gained by the user prior to routine clinical use. A welldesigned validation can be of great service in this capacity, and is especially important in the application of NGS as it, like any other test, has limitations, though often the excitement around the technology makes it easy for laboratorians to under-discuss the topic.

One of the most important limitations of clinical NGS applications is the decrease in detection efficiency for larger insertions and deletions (i.e., 8 bp). The chemistry does not seem to be the limiting factor as manual inspection often shows that there are reads confirming the large event, yet the alignment software is unable to place them properly and thus call them out in a variant file. While the size of a detectable event is getting larger all the time, it still cannot rival that of Sanger sequencing. Furthermore, detecting complete exon deletions and duplications is just becoming available.

This limitation in the bioinformatics capabilities highlights another barrier to accessing the technology. Currently, it is still preferable to have a team of savvy bioinformaticians and information technology specialists at your disposal. There is a paucity of end to end bioinformatics solutions that one could purchase off the shelf and that can be run by a standard laboratory technician. Given the nascent nature of NGS testing, using an off the shelf solution without expert understanding and testing of the complex analytics used could put a laboratory in jeopardy. What is more commonly deployed, as was done in this case, is to create a hybrid of purchased and custom created software. The consequence of this is that any internally developed solution requires expert bioinformatics support to fix bugs, implement new/improved industry capabilities, and validate the systems as required by the laboratory.

Another limitation needing careful consideration is the potential for false positive results by NGS as compared to Sanger sequencing. This topic has been addressed by nearly every publication on NGS methodologies. However, digging beyond the surface, the need to confirm all results seems to be an overly-Draconian approach to the real, yet limited risk of a false positive result. A recent publication suggests that within controlled parameters, specific alterations like SNVs might not need confirmation [15,16]. We agree and feel that our zero false positive rate in our validation data suggest this is a more realistic and evidence based position to take.

In summary, we have described the validation and implementation of a multigene NGS panel useful for diagnosing patients suspected of a hereditary colorectal cancer syndrome. In our pursuit of this validation we have learned the strengths and weaknesses of the current NGS methodologies, as tested here. Considering both, we feel that the time is right for laboratories to begin gathering experience in this field and panels offer an attractive and limited way to begin this education. NGS is a complicated method, yet not too dissimilar from many new technologies that get introduced into the laboratory. They must be studied in a carefully designed validation to fully know the correct application and limitations of the assay. Once that has been completed, the laboratorian has another strong diagnostic tool in their arsenal of methods to help our patients.

We thank the staff of the Clinical Genome Sequencing Laboratory (CGSL) at Mayo Clinic for their ongoing support of the institution’s clinical NGS needs. We would also like to thank the Mayo Clinic Department of Laboratory Medicine and Pathology and the Center for Individualized Medicine for their generous support of this work.