Live ClickHouse latency check — 2026-09-29
None of these PRs is merged or deployed. #151, #154 and #160 are open. All numbers come from the current public ClickHouse database (cbioportal_public_librechat_blue), with each PR's SQL run inline, next to what is deployed today. They describe the database side only, not what users would see end to end.
#160 · IN-subquery rewrite
saved per eligible view call; median 0.29 s over 8 gene × alteration cases. Results identical in all 8.
#154 · precomputed lookups
live fallback SQL today, vs an estimated ~3–4 ms keyed lookup once the tables exist.
#151 · schema cache
server time of each SHOW TABLES / DESCRIBE a cache hit would avoid.
DB share of answer latency
the 09-23 run artifacts record tool names and timings but not the SQL each call ran, so no answer could be replayed.
Summary
- #160 (rewrite the per-gene frequency views to filter through an
INsubquery) saves 0.24–0.60 s per call, median 0.29 s, on TP53 and ERBB2 for mutation, amplification, deep deletion and structural variant. Every case returns the same rows as the deployed view (identical SHA-256 over the sorted result). Rows read don't change: the saving is join/filter work, not I/O. - #154 (domain tools backed by precomputed aggregate tables): the live fallback SQL those tools run when the tables are missing takes 0.08–1.35 s. A keyed lookup on a small table takes about 3–4 ms today; that is an estimate for the future tables, which don't exist yet. Its bigger potential win, replacing several exploratory SQL calls and model turns with one tool call, can't be timed from the database.
- #151 (cache SHOW TABLES / DESCRIBE TABLE / study guides for an hour) avoids 1.4–2.6 ms of server time per hit. The ~0.55 s laptop round trip per call is mostly client start-up and connection set-up and says nothing about in-cluster MCP calls.
- The database's share of answer latency was not measured. The 2026-09-23 run records 292 answers and 1,467 SQL tool calls, but not the SQL each call executed, so no historical answer could be replayed. That share is unknown, not zero.
- What this suggests: the measured per-call DB savings (fractions of a second, up to ~1.3 s) are small next to the 09-23 median answer times of 29.6 s (Haiku) and 71.9 s (Sonnet), which took a median of 9–10 LLM calls per answer. A rough bound, estimated, not measured: Haiku made 708 SQL calls over 146 answers, ≈ 4.8 calls per answer; at 0.1–1.3 s each that is on the order of 0.5–6 s of DB time in a Haiku answer. That makes reducing model/tool rounds (e.g. purpose-built tools like #154) a plausible larger lever than DB tuning, but it is a hypothesis: testing it needs a run that captures the SQL each call executes, such as the Claude Code runner, which saves full transcripts.
For scale: the 2026-09-23 run
| Model | Answers | SQL tool calls | Median answer s | p90 answer s | Median LLM calls |
|---|---|---|---|---|---|
| Haiku | 146 | 708 | 29.57 | 74.85 | 9 |
| Sonnet | 146 | 759 | 71.93 | 157.28 | 10 |
Method and safety
- Read-only, one query at a time. Every query ran from a fresh
clickhouse clientwithreadonly=1,max_threads=4,max_execution_time=120s and an 8 GB memory cap, strictly sequentially with a 2 s pause before each. No writes,CREATE,INSERTor setting changes; no unfiltered scans of the coverage views; the #154 table-build SELECTs were restricted to one study each. - Warm-up + 3 runs, median. Each of the 43 queries ran once as a discarded warm-up, then three timed runs (129 timed executions, all successful, all under the caps; slowest single run 2.03 s). Each column below is its own median over the three runs.
- Server time is ClickHouse's own elapsed time from the response statistics (with rows and bytes read; bytes are uncompressed logical bytes, not network bytes).
system.query_logwas not accessible (permission denied, error 497), so it wasn't used. - Laptop round trip is the whole fresh client process: start-up, TLS connection, query and response, from a laptop. It is not representative of the MCP server's in-cluster connection and is shown only for completeness.
- Result parity: SHA-256 over the sorted canonical result rows, computed outside the timed query and checked stable across all four executions. Only aggregate results were handled.
- These are warm-query timings on today's data and load; the database, its load and query caching may have differed on 09-23.
Every measured query
| Query | Server ms | Rows read | Bytes read | Laptop round trip ms |
|---|---|---|---|---|
| Schema and metadata | ||||
metadata-show-tables | 2.58 | 82 | 5,543 | 568.46 |
metadata-describe-cancer_study | 1.72 | 24 | 3,736 | 566.20 |
metadata-describe-genomic_event_derived | 1.38 | 19 | 2,563 | 548.27 |
metadata-describe-sample_to_gene_panel_derived | 1.37 | 5 | 459 | 539.48 |
metadata-describe-clinical_data_derived | 1.38 | 7 | 1,586 | 541.05 |
metadata-describe-sample_derived | 1.55 | 12 | 969 | 590.97 |
metadata-describe-gene | 1.37 | 4 | 307 | 543.38 |
metadata-describe-genetic_profile | 1.40 | 12 | 963 | 555.84 |
metadata-version-timezone | 1.90 | 1 | 1 | 546.55 |
metadata-settings | 42.75 | 1,609 | 603,095 | 712.86 |
inventory | 3.49 | 82 | 7,019 | 556.85 |
| Keyed-lookup estimates | ||||
lookup-estimate-gene | 4.00 | 44,896 | 510,560 | 547.76 |
lookup-estimate-study | 3.06 | 548 | 13,834 | 552.48 |
| #160 before/after | ||||
160-before-TP53-mutation | 991.79 | 30,599,882 | 567,050,014 | 1,704.56 |
160-after-TP53-mutation | 394.83 | 30,599,882 | 567,050,014 | 992.66 |
160-before-TP53-amplification | 875.87 | 43,829,849 | 681,771,092 | 1,420.19 |
160-after-TP53-amplification | 489.16 | 43,829,849 | 681,771,092 | 1,058.41 |
160-before-TP53-deep_deletion | 753.27 | 43,829,849 | 702,416,428 | 1,444.88 |
160-after-TP53-deep_deletion | 468.67 | 43,829,849 | 702,416,428 | 1,017.18 |
160-before-TP53-structural_variant | 485.29 | 14,117,465 | 294,297,667 | 1,107.88 |
160-after-TP53-structural_variant | 211.08 | 14,117,465 | 294,297,667 | 1,033.98 |
160-before-ERBB2-mutation | 744.86 | 30,599,882 | 538,585,958 | 1,340.93 |
160-after-ERBB2-mutation | 361.58 | 30,599,882 | 538,585,958 | 1,035.03 |
160-before-ERBB2-amplification | 765.85 | 43,829,849 | 701,038,072 | 1,394.65 |
160-after-ERBB2-amplification | 496.49 | 43,829,849 | 701,038,072 | 1,087.27 |
160-before-ERBB2-deep_deletion | 732.77 | 43,829,849 | 680,061,820 | 1,295.10 |
160-after-ERBB2-deep_deletion | 487.97 | 43,829,849 | 680,061,820 | 1,093.79 |
160-before-ERBB2-structural_variant | 572.37 | 14,117,465 | 290,145,240 | 1,216.84 |
160-after-ERBB2-structural_variant | 270.05 | 14,117,465 | 290,145,240 | 859.12 |
| #154 live fallback | ||||
154-frequency-msk_chord_2024 | 186.83 | 6,162,615 | 22,762,255 | 875.77 |
154-top-mutation-msk_chord_2024 | 282.01 | 10,281,781 | 46,600,333 | 838.16 |
154-profiled-msk_chord_2024 | 138.48 | 2,916,228 | 32,829,525 | 781.63 |
154-frequency-brca_tcga_pan_can_atlas_2018 | 82.09 | 6,432,951 | 16,190,499 | 648.44 |
154-top-mutation-brca_tcga_pan_can_atlas_2018 | 310.15 | 10,019,637 | 32,958,606 | 889.98 |
154-profiled-brca_tcga_pan_can_atlas_2018 | 78.31 | 2,850,692 | 9,002,888 | 641.21 |
154-cancer-type-TP53 | 1,348.46 | 30,599,882 | 567,050,014 | 1,902.63 |
154-cancer-type-TP53-pan-cancer-tcga | 852.74 | 30,599,882 | 567,050,014 | 1,408.48 |
| #154 table-build SELECTs (one study each) | ||||
154-build-study_gene_alteration_counts-msk_chord_2024 | 558.55 | 13,498,040 | 84,986,876 | 1,231.98 |
154-build-study_gene_alteration_counts-brca_tcga_pan_can_atlas_2018 | 547.78 | 14,038,712 | 87,346,296 | 1,753.96 |
154-build-cancer_type_gene_alteration_counts-msk_chord_2024 | 997.13 | 13,718,830 | 91,693,482 | 1,895.59 |
154-build-cancer_type_gene_alteration_counts-brca_tcga_pan_can_atlas_2018 | 1,036.70 | 14,161,198 | 93,593,018 | 2,821.80 |
154-build-study_profiled_counts-msk_chord_2024 | 69.33 | 2,916,228 | 32,829,525 | 698.10 |
154-build-study_profiled_counts-brca_tcga_pan_can_atlas_2018 | 35.99 | 2,850,692 | 9,002,888 | 686.03 |
SQL for all 43 queries
metadata-show-tables
SHOW TABLES
metadata-describe-cancer_study
DESCRIBE TABLE cancer_study
metadata-describe-genomic_event_derived
DESCRIBE TABLE genomic_event_derived
metadata-describe-sample_to_gene_panel_derived
DESCRIBE TABLE sample_to_gene_panel_derived
metadata-describe-clinical_data_derived
DESCRIBE TABLE clinical_data_derived
metadata-describe-sample_derived
DESCRIBE TABLE sample_derived
metadata-describe-gene
DESCRIBE TABLE gene
metadata-describe-genetic_profile
DESCRIBE TABLE genetic_profile
metadata-version-timezone
SELECT version(), timezone()
metadata-settings
SELECT name, value FROM system.settings LIMIT 1000
inventory
SELECT name, total_rows, total_bytes FROM system.tables WHERE database=currentDatabase() ORDER BY name
lookup-estimate-gene
SELECT hugo_gene_symbol, entrez_gene_id FROM gene WHERE hugo_gene_symbol='TP53'
lookup-estimate-study
SELECT cancer_study_identifier FROM cancer_study WHERE cancer_study_identifier='msk_chord_2024'
160-before-TP53-mutation
SELECT * FROM gene_alteration_frequency_by_cancer_type(preference='all_studies_non_redundant', gene='TP53', alteration='mutation')
160-after-TP53-mutation
WITH cohort AS (
SELECT cancer_study_identifier
FROM cancer_study_query_preferences
WHERE preference_name = 'all_studies_non_redundant'
),
sample_cancer_type AS (
SELECT cd.sample_unique_id, cd.attribute_value AS cancer_type
FROM clinical_data_derived cd
JOIN cohort c USING (cancer_study_identifier)
WHERE cd.attribute_name = 'CANCER_TYPE'
),
altered AS (
SELECT sct.cancer_type,
COUNT(DISTINCT ged.sample_unique_id) AS altered_samples
FROM genomic_event_derived ged
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
WHERE ged.hugo_gene_symbol = 'TP53'
AND ged.off_panel = 0
AND (
('mutation' = 'mutation'
AND ged.variant_type = 'mutation'
AND ged.mutation_status != 'UNCALLED')
OR ('mutation' = 'amplification'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = 2)
OR ('mutation' = 'deep_deletion'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = -2)
OR ('mutation' = 'structural_variant'
AND ged.variant_type = 'structural_variant')
)
GROUP BY sct.cancer_type
),
profiled_samples_for_gene AS (
-- Map the user-facing alteration token to the alteration_type stored
-- on sample_to_gene_panel_derived. Same gene-in-panel-or-WES branch
-- as the mutation view, but with the matching alteration_type filter.
SELECT stgp.sample_unique_id, stgp.cancer_study_identifier
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
-- IN-subquery instead of JOIN gene: resolves the symbol to its entrez
-- id(s) once, rather than joining every panel row against the gene
-- table before filtering. COUNT(DISTINCT) downstream makes the result
-- identical even for symbols with several (or duplicated) gene rows.
-- IS NOT NULL keeps NULL keys unmatched (as the equality JOIN did) even
-- under transform_null_in=1.
WHERE gpl.gene_id IN (
SELECT entrez_gene_id FROM gene
WHERE hugo_gene_symbol = 'TP53' AND entrez_gene_id IS NOT NULL)
AND stgp.alteration_type = multiIf(
'mutation' = 'mutation', 'MUTATION_EXTENDED',
'mutation' = 'amplification', 'COPY_NUMBER_ALTERATION',
'mutation' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'mutation' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
UNION ALL
SELECT sample_unique_id, cancer_study_identifier
FROM sample_to_gene_panel_derived
WHERE gene_panel_id = 'WES'
AND alteration_type = multiIf(
'mutation' = 'mutation', 'MUTATION_EXTENDED',
'mutation' = 'amplification', 'COPY_NUMBER_ALTERATION',
'mutation' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'mutation' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
),
profiled AS (
SELECT sct.cancer_type,
COUNT(DISTINCT p.sample_unique_id) AS profiled_samples
FROM profiled_samples_for_gene p
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
GROUP BY sct.cancer_type
)
SELECT a.cancer_type,
a.altered_samples,
p.profiled_samples,
ROUND(a.altered_samples * 100.0 / NULLIF(p.profiled_samples, 0), 1) AS frequency_pct
FROM altered a
JOIN profiled p USING (cancer_type)
WHERE p.profiled_samples >= 50160-before-TP53-amplification
SELECT * FROM gene_alteration_frequency_by_cancer_type(preference='all_studies_non_redundant', gene='TP53', alteration='amplification')
160-after-TP53-amplification
WITH cohort AS (
SELECT cancer_study_identifier
FROM cancer_study_query_preferences
WHERE preference_name = 'all_studies_non_redundant'
),
sample_cancer_type AS (
SELECT cd.sample_unique_id, cd.attribute_value AS cancer_type
FROM clinical_data_derived cd
JOIN cohort c USING (cancer_study_identifier)
WHERE cd.attribute_name = 'CANCER_TYPE'
),
altered AS (
SELECT sct.cancer_type,
COUNT(DISTINCT ged.sample_unique_id) AS altered_samples
FROM genomic_event_derived ged
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
WHERE ged.hugo_gene_symbol = 'TP53'
AND ged.off_panel = 0
AND (
('amplification' = 'mutation'
AND ged.variant_type = 'mutation'
AND ged.mutation_status != 'UNCALLED')
OR ('amplification' = 'amplification'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = 2)
OR ('amplification' = 'deep_deletion'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = -2)
OR ('amplification' = 'structural_variant'
AND ged.variant_type = 'structural_variant')
)
GROUP BY sct.cancer_type
),
profiled_samples_for_gene AS (
-- Map the user-facing alteration token to the alteration_type stored
-- on sample_to_gene_panel_derived. Same gene-in-panel-or-WES branch
-- as the mutation view, but with the matching alteration_type filter.
SELECT stgp.sample_unique_id, stgp.cancer_study_identifier
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
-- IN-subquery instead of JOIN gene: resolves the symbol to its entrez
-- id(s) once, rather than joining every panel row against the gene
-- table before filtering. COUNT(DISTINCT) downstream makes the result
-- identical even for symbols with several (or duplicated) gene rows.
-- IS NOT NULL keeps NULL keys unmatched (as the equality JOIN did) even
-- under transform_null_in=1.
WHERE gpl.gene_id IN (
SELECT entrez_gene_id FROM gene
WHERE hugo_gene_symbol = 'TP53' AND entrez_gene_id IS NOT NULL)
AND stgp.alteration_type = multiIf(
'amplification' = 'mutation', 'MUTATION_EXTENDED',
'amplification' = 'amplification', 'COPY_NUMBER_ALTERATION',
'amplification' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'amplification' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
UNION ALL
SELECT sample_unique_id, cancer_study_identifier
FROM sample_to_gene_panel_derived
WHERE gene_panel_id = 'WES'
AND alteration_type = multiIf(
'amplification' = 'mutation', 'MUTATION_EXTENDED',
'amplification' = 'amplification', 'COPY_NUMBER_ALTERATION',
'amplification' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'amplification' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
),
profiled AS (
SELECT sct.cancer_type,
COUNT(DISTINCT p.sample_unique_id) AS profiled_samples
FROM profiled_samples_for_gene p
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
GROUP BY sct.cancer_type
)
SELECT a.cancer_type,
a.altered_samples,
p.profiled_samples,
ROUND(a.altered_samples * 100.0 / NULLIF(p.profiled_samples, 0), 1) AS frequency_pct
FROM altered a
JOIN profiled p USING (cancer_type)
WHERE p.profiled_samples >= 50160-before-TP53-deep_deletion
SELECT * FROM gene_alteration_frequency_by_cancer_type(preference='all_studies_non_redundant', gene='TP53', alteration='deep_deletion')
160-after-TP53-deep_deletion
WITH cohort AS (
SELECT cancer_study_identifier
FROM cancer_study_query_preferences
WHERE preference_name = 'all_studies_non_redundant'
),
sample_cancer_type AS (
SELECT cd.sample_unique_id, cd.attribute_value AS cancer_type
FROM clinical_data_derived cd
JOIN cohort c USING (cancer_study_identifier)
WHERE cd.attribute_name = 'CANCER_TYPE'
),
altered AS (
SELECT sct.cancer_type,
COUNT(DISTINCT ged.sample_unique_id) AS altered_samples
FROM genomic_event_derived ged
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
WHERE ged.hugo_gene_symbol = 'TP53'
AND ged.off_panel = 0
AND (
('deep_deletion' = 'mutation'
AND ged.variant_type = 'mutation'
AND ged.mutation_status != 'UNCALLED')
OR ('deep_deletion' = 'amplification'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = 2)
OR ('deep_deletion' = 'deep_deletion'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = -2)
OR ('deep_deletion' = 'structural_variant'
AND ged.variant_type = 'structural_variant')
)
GROUP BY sct.cancer_type
),
profiled_samples_for_gene AS (
-- Map the user-facing alteration token to the alteration_type stored
-- on sample_to_gene_panel_derived. Same gene-in-panel-or-WES branch
-- as the mutation view, but with the matching alteration_type filter.
SELECT stgp.sample_unique_id, stgp.cancer_study_identifier
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
-- IN-subquery instead of JOIN gene: resolves the symbol to its entrez
-- id(s) once, rather than joining every panel row against the gene
-- table before filtering. COUNT(DISTINCT) downstream makes the result
-- identical even for symbols with several (or duplicated) gene rows.
-- IS NOT NULL keeps NULL keys unmatched (as the equality JOIN did) even
-- under transform_null_in=1.
WHERE gpl.gene_id IN (
SELECT entrez_gene_id FROM gene
WHERE hugo_gene_symbol = 'TP53' AND entrez_gene_id IS NOT NULL)
AND stgp.alteration_type = multiIf(
'deep_deletion' = 'mutation', 'MUTATION_EXTENDED',
'deep_deletion' = 'amplification', 'COPY_NUMBER_ALTERATION',
'deep_deletion' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'deep_deletion' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
UNION ALL
SELECT sample_unique_id, cancer_study_identifier
FROM sample_to_gene_panel_derived
WHERE gene_panel_id = 'WES'
AND alteration_type = multiIf(
'deep_deletion' = 'mutation', 'MUTATION_EXTENDED',
'deep_deletion' = 'amplification', 'COPY_NUMBER_ALTERATION',
'deep_deletion' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'deep_deletion' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
),
profiled AS (
SELECT sct.cancer_type,
COUNT(DISTINCT p.sample_unique_id) AS profiled_samples
FROM profiled_samples_for_gene p
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
GROUP BY sct.cancer_type
)
SELECT a.cancer_type,
a.altered_samples,
p.profiled_samples,
ROUND(a.altered_samples * 100.0 / NULLIF(p.profiled_samples, 0), 1) AS frequency_pct
FROM altered a
JOIN profiled p USING (cancer_type)
WHERE p.profiled_samples >= 50160-before-TP53-structural_variant
SELECT * FROM gene_alteration_frequency_by_cancer_type(preference='all_studies_non_redundant', gene='TP53', alteration='structural_variant')
160-after-TP53-structural_variant
WITH cohort AS (
SELECT cancer_study_identifier
FROM cancer_study_query_preferences
WHERE preference_name = 'all_studies_non_redundant'
),
sample_cancer_type AS (
SELECT cd.sample_unique_id, cd.attribute_value AS cancer_type
FROM clinical_data_derived cd
JOIN cohort c USING (cancer_study_identifier)
WHERE cd.attribute_name = 'CANCER_TYPE'
),
altered AS (
SELECT sct.cancer_type,
COUNT(DISTINCT ged.sample_unique_id) AS altered_samples
FROM genomic_event_derived ged
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
WHERE ged.hugo_gene_symbol = 'TP53'
AND ged.off_panel = 0
AND (
('structural_variant' = 'mutation'
AND ged.variant_type = 'mutation'
AND ged.mutation_status != 'UNCALLED')
OR ('structural_variant' = 'amplification'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = 2)
OR ('structural_variant' = 'deep_deletion'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = -2)
OR ('structural_variant' = 'structural_variant'
AND ged.variant_type = 'structural_variant')
)
GROUP BY sct.cancer_type
),
profiled_samples_for_gene AS (
-- Map the user-facing alteration token to the alteration_type stored
-- on sample_to_gene_panel_derived. Same gene-in-panel-or-WES branch
-- as the mutation view, but with the matching alteration_type filter.
SELECT stgp.sample_unique_id, stgp.cancer_study_identifier
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
-- IN-subquery instead of JOIN gene: resolves the symbol to its entrez
-- id(s) once, rather than joining every panel row against the gene
-- table before filtering. COUNT(DISTINCT) downstream makes the result
-- identical even for symbols with several (or duplicated) gene rows.
-- IS NOT NULL keeps NULL keys unmatched (as the equality JOIN did) even
-- under transform_null_in=1.
WHERE gpl.gene_id IN (
SELECT entrez_gene_id FROM gene
WHERE hugo_gene_symbol = 'TP53' AND entrez_gene_id IS NOT NULL)
AND stgp.alteration_type = multiIf(
'structural_variant' = 'mutation', 'MUTATION_EXTENDED',
'structural_variant' = 'amplification', 'COPY_NUMBER_ALTERATION',
'structural_variant' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'structural_variant' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
UNION ALL
SELECT sample_unique_id, cancer_study_identifier
FROM sample_to_gene_panel_derived
WHERE gene_panel_id = 'WES'
AND alteration_type = multiIf(
'structural_variant' = 'mutation', 'MUTATION_EXTENDED',
'structural_variant' = 'amplification', 'COPY_NUMBER_ALTERATION',
'structural_variant' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'structural_variant' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
),
profiled AS (
SELECT sct.cancer_type,
COUNT(DISTINCT p.sample_unique_id) AS profiled_samples
FROM profiled_samples_for_gene p
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
GROUP BY sct.cancer_type
)
SELECT a.cancer_type,
a.altered_samples,
p.profiled_samples,
ROUND(a.altered_samples * 100.0 / NULLIF(p.profiled_samples, 0), 1) AS frequency_pct
FROM altered a
JOIN profiled p USING (cancer_type)
WHERE p.profiled_samples >= 50160-before-ERBB2-mutation
SELECT * FROM gene_alteration_frequency_by_cancer_type(preference='all_studies_non_redundant', gene='ERBB2', alteration='mutation')
160-after-ERBB2-mutation
WITH cohort AS (
SELECT cancer_study_identifier
FROM cancer_study_query_preferences
WHERE preference_name = 'all_studies_non_redundant'
),
sample_cancer_type AS (
SELECT cd.sample_unique_id, cd.attribute_value AS cancer_type
FROM clinical_data_derived cd
JOIN cohort c USING (cancer_study_identifier)
WHERE cd.attribute_name = 'CANCER_TYPE'
),
altered AS (
SELECT sct.cancer_type,
COUNT(DISTINCT ged.sample_unique_id) AS altered_samples
FROM genomic_event_derived ged
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
WHERE ged.hugo_gene_symbol = 'ERBB2'
AND ged.off_panel = 0
AND (
('mutation' = 'mutation'
AND ged.variant_type = 'mutation'
AND ged.mutation_status != 'UNCALLED')
OR ('mutation' = 'amplification'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = 2)
OR ('mutation' = 'deep_deletion'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = -2)
OR ('mutation' = 'structural_variant'
AND ged.variant_type = 'structural_variant')
)
GROUP BY sct.cancer_type
),
profiled_samples_for_gene AS (
-- Map the user-facing alteration token to the alteration_type stored
-- on sample_to_gene_panel_derived. Same gene-in-panel-or-WES branch
-- as the mutation view, but with the matching alteration_type filter.
SELECT stgp.sample_unique_id, stgp.cancer_study_identifier
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
-- IN-subquery instead of JOIN gene: resolves the symbol to its entrez
-- id(s) once, rather than joining every panel row against the gene
-- table before filtering. COUNT(DISTINCT) downstream makes the result
-- identical even for symbols with several (or duplicated) gene rows.
-- IS NOT NULL keeps NULL keys unmatched (as the equality JOIN did) even
-- under transform_null_in=1.
WHERE gpl.gene_id IN (
SELECT entrez_gene_id FROM gene
WHERE hugo_gene_symbol = 'ERBB2' AND entrez_gene_id IS NOT NULL)
AND stgp.alteration_type = multiIf(
'mutation' = 'mutation', 'MUTATION_EXTENDED',
'mutation' = 'amplification', 'COPY_NUMBER_ALTERATION',
'mutation' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'mutation' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
UNION ALL
SELECT sample_unique_id, cancer_study_identifier
FROM sample_to_gene_panel_derived
WHERE gene_panel_id = 'WES'
AND alteration_type = multiIf(
'mutation' = 'mutation', 'MUTATION_EXTENDED',
'mutation' = 'amplification', 'COPY_NUMBER_ALTERATION',
'mutation' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'mutation' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
),
profiled AS (
SELECT sct.cancer_type,
COUNT(DISTINCT p.sample_unique_id) AS profiled_samples
FROM profiled_samples_for_gene p
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
GROUP BY sct.cancer_type
)
SELECT a.cancer_type,
a.altered_samples,
p.profiled_samples,
ROUND(a.altered_samples * 100.0 / NULLIF(p.profiled_samples, 0), 1) AS frequency_pct
FROM altered a
JOIN profiled p USING (cancer_type)
WHERE p.profiled_samples >= 50160-before-ERBB2-amplification
SELECT * FROM gene_alteration_frequency_by_cancer_type(preference='all_studies_non_redundant', gene='ERBB2', alteration='amplification')
160-after-ERBB2-amplification
WITH cohort AS (
SELECT cancer_study_identifier
FROM cancer_study_query_preferences
WHERE preference_name = 'all_studies_non_redundant'
),
sample_cancer_type AS (
SELECT cd.sample_unique_id, cd.attribute_value AS cancer_type
FROM clinical_data_derived cd
JOIN cohort c USING (cancer_study_identifier)
WHERE cd.attribute_name = 'CANCER_TYPE'
),
altered AS (
SELECT sct.cancer_type,
COUNT(DISTINCT ged.sample_unique_id) AS altered_samples
FROM genomic_event_derived ged
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
WHERE ged.hugo_gene_symbol = 'ERBB2'
AND ged.off_panel = 0
AND (
('amplification' = 'mutation'
AND ged.variant_type = 'mutation'
AND ged.mutation_status != 'UNCALLED')
OR ('amplification' = 'amplification'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = 2)
OR ('amplification' = 'deep_deletion'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = -2)
OR ('amplification' = 'structural_variant'
AND ged.variant_type = 'structural_variant')
)
GROUP BY sct.cancer_type
),
profiled_samples_for_gene AS (
-- Map the user-facing alteration token to the alteration_type stored
-- on sample_to_gene_panel_derived. Same gene-in-panel-or-WES branch
-- as the mutation view, but with the matching alteration_type filter.
SELECT stgp.sample_unique_id, stgp.cancer_study_identifier
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
-- IN-subquery instead of JOIN gene: resolves the symbol to its entrez
-- id(s) once, rather than joining every panel row against the gene
-- table before filtering. COUNT(DISTINCT) downstream makes the result
-- identical even for symbols with several (or duplicated) gene rows.
-- IS NOT NULL keeps NULL keys unmatched (as the equality JOIN did) even
-- under transform_null_in=1.
WHERE gpl.gene_id IN (
SELECT entrez_gene_id FROM gene
WHERE hugo_gene_symbol = 'ERBB2' AND entrez_gene_id IS NOT NULL)
AND stgp.alteration_type = multiIf(
'amplification' = 'mutation', 'MUTATION_EXTENDED',
'amplification' = 'amplification', 'COPY_NUMBER_ALTERATION',
'amplification' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'amplification' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
UNION ALL
SELECT sample_unique_id, cancer_study_identifier
FROM sample_to_gene_panel_derived
WHERE gene_panel_id = 'WES'
AND alteration_type = multiIf(
'amplification' = 'mutation', 'MUTATION_EXTENDED',
'amplification' = 'amplification', 'COPY_NUMBER_ALTERATION',
'amplification' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'amplification' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
),
profiled AS (
SELECT sct.cancer_type,
COUNT(DISTINCT p.sample_unique_id) AS profiled_samples
FROM profiled_samples_for_gene p
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
GROUP BY sct.cancer_type
)
SELECT a.cancer_type,
a.altered_samples,
p.profiled_samples,
ROUND(a.altered_samples * 100.0 / NULLIF(p.profiled_samples, 0), 1) AS frequency_pct
FROM altered a
JOIN profiled p USING (cancer_type)
WHERE p.profiled_samples >= 50160-before-ERBB2-deep_deletion
SELECT * FROM gene_alteration_frequency_by_cancer_type(preference='all_studies_non_redundant', gene='ERBB2', alteration='deep_deletion')
160-after-ERBB2-deep_deletion
WITH cohort AS (
SELECT cancer_study_identifier
FROM cancer_study_query_preferences
WHERE preference_name = 'all_studies_non_redundant'
),
sample_cancer_type AS (
SELECT cd.sample_unique_id, cd.attribute_value AS cancer_type
FROM clinical_data_derived cd
JOIN cohort c USING (cancer_study_identifier)
WHERE cd.attribute_name = 'CANCER_TYPE'
),
altered AS (
SELECT sct.cancer_type,
COUNT(DISTINCT ged.sample_unique_id) AS altered_samples
FROM genomic_event_derived ged
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
WHERE ged.hugo_gene_symbol = 'ERBB2'
AND ged.off_panel = 0
AND (
('deep_deletion' = 'mutation'
AND ged.variant_type = 'mutation'
AND ged.mutation_status != 'UNCALLED')
OR ('deep_deletion' = 'amplification'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = 2)
OR ('deep_deletion' = 'deep_deletion'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = -2)
OR ('deep_deletion' = 'structural_variant'
AND ged.variant_type = 'structural_variant')
)
GROUP BY sct.cancer_type
),
profiled_samples_for_gene AS (
-- Map the user-facing alteration token to the alteration_type stored
-- on sample_to_gene_panel_derived. Same gene-in-panel-or-WES branch
-- as the mutation view, but with the matching alteration_type filter.
SELECT stgp.sample_unique_id, stgp.cancer_study_identifier
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
-- IN-subquery instead of JOIN gene: resolves the symbol to its entrez
-- id(s) once, rather than joining every panel row against the gene
-- table before filtering. COUNT(DISTINCT) downstream makes the result
-- identical even for symbols with several (or duplicated) gene rows.
-- IS NOT NULL keeps NULL keys unmatched (as the equality JOIN did) even
-- under transform_null_in=1.
WHERE gpl.gene_id IN (
SELECT entrez_gene_id FROM gene
WHERE hugo_gene_symbol = 'ERBB2' AND entrez_gene_id IS NOT NULL)
AND stgp.alteration_type = multiIf(
'deep_deletion' = 'mutation', 'MUTATION_EXTENDED',
'deep_deletion' = 'amplification', 'COPY_NUMBER_ALTERATION',
'deep_deletion' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'deep_deletion' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
UNION ALL
SELECT sample_unique_id, cancer_study_identifier
FROM sample_to_gene_panel_derived
WHERE gene_panel_id = 'WES'
AND alteration_type = multiIf(
'deep_deletion' = 'mutation', 'MUTATION_EXTENDED',
'deep_deletion' = 'amplification', 'COPY_NUMBER_ALTERATION',
'deep_deletion' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'deep_deletion' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
),
profiled AS (
SELECT sct.cancer_type,
COUNT(DISTINCT p.sample_unique_id) AS profiled_samples
FROM profiled_samples_for_gene p
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
GROUP BY sct.cancer_type
)
SELECT a.cancer_type,
a.altered_samples,
p.profiled_samples,
ROUND(a.altered_samples * 100.0 / NULLIF(p.profiled_samples, 0), 1) AS frequency_pct
FROM altered a
JOIN profiled p USING (cancer_type)
WHERE p.profiled_samples >= 50160-before-ERBB2-structural_variant
SELECT * FROM gene_alteration_frequency_by_cancer_type(preference='all_studies_non_redundant', gene='ERBB2', alteration='structural_variant')
160-after-ERBB2-structural_variant
WITH cohort AS (
SELECT cancer_study_identifier
FROM cancer_study_query_preferences
WHERE preference_name = 'all_studies_non_redundant'
),
sample_cancer_type AS (
SELECT cd.sample_unique_id, cd.attribute_value AS cancer_type
FROM clinical_data_derived cd
JOIN cohort c USING (cancer_study_identifier)
WHERE cd.attribute_name = 'CANCER_TYPE'
),
altered AS (
SELECT sct.cancer_type,
COUNT(DISTINCT ged.sample_unique_id) AS altered_samples
FROM genomic_event_derived ged
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
WHERE ged.hugo_gene_symbol = 'ERBB2'
AND ged.off_panel = 0
AND (
('structural_variant' = 'mutation'
AND ged.variant_type = 'mutation'
AND ged.mutation_status != 'UNCALLED')
OR ('structural_variant' = 'amplification'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = 2)
OR ('structural_variant' = 'deep_deletion'
AND ged.variant_type = 'cna'
AND ged.cna_alteration = -2)
OR ('structural_variant' = 'structural_variant'
AND ged.variant_type = 'structural_variant')
)
GROUP BY sct.cancer_type
),
profiled_samples_for_gene AS (
-- Map the user-facing alteration token to the alteration_type stored
-- on sample_to_gene_panel_derived. Same gene-in-panel-or-WES branch
-- as the mutation view, but with the matching alteration_type filter.
SELECT stgp.sample_unique_id, stgp.cancer_study_identifier
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
-- IN-subquery instead of JOIN gene: resolves the symbol to its entrez
-- id(s) once, rather than joining every panel row against the gene
-- table before filtering. COUNT(DISTINCT) downstream makes the result
-- identical even for symbols with several (or duplicated) gene rows.
-- IS NOT NULL keeps NULL keys unmatched (as the equality JOIN did) even
-- under transform_null_in=1.
WHERE gpl.gene_id IN (
SELECT entrez_gene_id FROM gene
WHERE hugo_gene_symbol = 'ERBB2' AND entrez_gene_id IS NOT NULL)
AND stgp.alteration_type = multiIf(
'structural_variant' = 'mutation', 'MUTATION_EXTENDED',
'structural_variant' = 'amplification', 'COPY_NUMBER_ALTERATION',
'structural_variant' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'structural_variant' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
UNION ALL
SELECT sample_unique_id, cancer_study_identifier
FROM sample_to_gene_panel_derived
WHERE gene_panel_id = 'WES'
AND alteration_type = multiIf(
'structural_variant' = 'mutation', 'MUTATION_EXTENDED',
'structural_variant' = 'amplification', 'COPY_NUMBER_ALTERATION',
'structural_variant' = 'deep_deletion', 'COPY_NUMBER_ALTERATION',
'structural_variant' = 'structural_variant', 'STRUCTURAL_VARIANT',
'')
),
profiled AS (
SELECT sct.cancer_type,
COUNT(DISTINCT p.sample_unique_id) AS profiled_samples
FROM profiled_samples_for_gene p
JOIN cohort c USING (cancer_study_identifier)
JOIN sample_cancer_type sct USING (sample_unique_id)
GROUP BY sct.cancer_type
)
SELECT a.cancer_type,
a.altered_samples,
p.profiled_samples,
ROUND(a.altered_samples * 100.0 / NULLIF(p.profiled_samples, 0), 1) AS frequency_pct
FROM altered a
JOIN profiled p USING (cancer_type)
WHERE p.profiled_samples >= 50154-frequency-msk_chord_2024
WITH
events AS (
SELECT sample_unique_id,
(variant_type = 'mutation' AND mutation_status != 'UNCALLED') AS is_mutation,
(variant_type = 'cna' AND cna_alteration = 2) AS is_amplification,
(variant_type = 'cna' AND cna_alteration = -2) AS is_deep_deletion,
(variant_type = 'structural_variant' AND mutation_status != 'UNCALLED') AS is_structural_variant
FROM genomic_event_derived
WHERE cancer_study_identifier IN ('msk_chord_2024')
AND hugo_gene_symbol IN ('TP53')
AND off_panel = 0
),
profiled AS (
SELECT stgp.sample_unique_id AS sample_unique_id,
stgp.alteration_type AS alteration_type
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
JOIN gene ge ON gpl.gene_id = ge.entrez_gene_id
WHERE stgp.cancer_study_identifier IN ('msk_chord_2024')
AND ge.hugo_gene_symbol IN ('TP53')
AND (stgp.alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT') OR (stgp.alteration_type = 'COPY_NUMBER_ALTERATION' AND stgp.genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE')))
UNION ALL
SELECT sample_unique_id, alteration_type
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('msk_chord_2024')
AND gene_panel_id = 'WES'
AND (alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT') OR (alteration_type = 'COPY_NUMBER_ALTERATION' AND genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE')))
)
SELECT *
FROM (
SELECT groupUniqArray(hugo_gene_symbol) AS matched_genes
FROM gene WHERE hugo_gene_symbol IN ('TP53')
) AS gene_match
CROSS JOIN (
SELECT groupUniqArray(cancer_study_identifier) AS matched_studies
FROM cancer_study WHERE cancer_study_identifier IN ('msk_chord_2024')
) AS study_match
CROSS JOIN (
SELECT uniqExactIf(sample_unique_id, is_mutation) AS altered_mutation,
uniqExactIf(sample_unique_id, is_amplification) AS altered_amplification,
uniqExactIf(sample_unique_id, is_deep_deletion) AS altered_deep_deletion,
uniqExactIf(sample_unique_id, is_structural_variant)
AS altered_structural_variant,
uniqExactIf(sample_unique_id, is_mutation OR is_amplification
OR is_deep_deletion OR is_structural_variant) AS altered_any
FROM events
) AS altered
CROSS JOIN (
SELECT uniqExactIf(sample_unique_id, alteration_type = 'MUTATION_EXTENDED')
AS profiled_MUTATION_EXTENDED,
uniqExactIf(sample_unique_id, alteration_type = 'COPY_NUMBER_ALTERATION')
AS profiled_COPY_NUMBER_ALTERATION,
uniqExactIf(sample_unique_id, alteration_type = 'STRUCTURAL_VARIANT')
AS profiled_STRUCTURAL_VARIANT,
uniqExact(sample_unique_id) AS profiled_ANY
FROM profiled
) AS profiled_counts154-top-mutation-msk_chord_2024
WITH
altered AS (
SELECT hugo_gene_symbol, COUNT(DISTINCT sample_unique_id) AS altered_samples
FROM genomic_event_derived
WHERE cancer_study_identifier IN ('msk_chord_2024')
AND off_panel = 0
AND (variant_type = 'mutation' AND mutation_status != 'UNCALLED')
GROUP BY hugo_gene_symbol
ORDER BY altered_samples DESC, hugo_gene_symbol ASC
LIMIT 20
),
wes_samples AS (
SELECT DISTINCT sample_unique_id
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('msk_chord_2024')
AND gene_panel_id = 'WES'
AND (alteration_type IN ('MUTATION_EXTENDED'))
),
panel_profiled AS (
SELECT ge.hugo_gene_symbol AS hugo_gene_symbol,
COUNT(DISTINCT stgp.sample_unique_id) AS n
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
JOIN gene ge ON gpl.gene_id = ge.entrez_gene_id
WHERE stgp.cancer_study_identifier IN ('msk_chord_2024')
AND (stgp.alteration_type IN ('MUTATION_EXTENDED'))
AND stgp.sample_unique_id NOT IN (SELECT sample_unique_id FROM wes_samples)
AND ge.hugo_gene_symbol IN (SELECT hugo_gene_symbol FROM altered)
GROUP BY ge.hugo_gene_symbol
)
SELECT a.hugo_gene_symbol AS hugo_gene_symbol,
a.altered_samples AS altered_samples,
(SELECT count() FROM wes_samples) + COALESCE(p.n, 0) AS profiled_samples
FROM altered a
LEFT JOIN panel_profiled p ON a.hugo_gene_symbol = p.hugo_gene_symbol
ORDER BY altered_samples DESC, hugo_gene_symbol ASC154-profiled-msk_chord_2024
WITH
profiled AS (
SELECT cancer_study_identifier, alteration_type AS profile_type,
sample_unique_id, gene_panel_id
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('msk_chord_2024')
UNION ALL
SELECT cancer_study_identifier, 'COPY_NUMBER_ALTERATION_DISCRETE' AS profile_type,
sample_unique_id, gene_panel_id
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('msk_chord_2024')
AND ((alteration_type = 'COPY_NUMBER_ALTERATION' AND genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE')))
UNION ALL
SELECT cancer_study_identifier, 'ANY_MUT_CNA_SV' AS profile_type,
sample_unique_id, gene_panel_id
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('msk_chord_2024')
AND (alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT') OR (alteration_type = 'COPY_NUMBER_ALTERATION' AND genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE')))
),
sample_patient AS (
SELECT sample_unique_id, patient_unique_id
FROM sample_derived
WHERE cancer_study_identifier IN ('msk_chord_2024')
)
SELECT * FROM (
SELECT cancer_study_identifier, 'ALL_SAMPLES' AS profile_type,
COUNT(DISTINCT sample_unique_id) AS samples,
COUNT(DISTINCT patient_unique_id) AS patients,
toUInt64(0) AS wes_samples
FROM sample_derived
WHERE cancer_study_identifier IN ('msk_chord_2024')
GROUP BY cancer_study_identifier
UNION ALL
SELECT p.cancer_study_identifier, p.profile_type,
COUNT(DISTINCT p.sample_unique_id),
COUNT(DISTINCT nullIf(sp.patient_unique_id, '')),
COUNT(DISTINCT if(p.gene_panel_id = 'WES', p.sample_unique_id, NULL))
FROM profiled p
LEFT JOIN sample_patient sp ON p.sample_unique_id = sp.sample_unique_id
GROUP BY p.cancer_study_identifier, p.profile_type
)
ORDER BY cancer_study_identifier, profile_type = 'ALL_SAMPLES' DESC,
profile_type = 'ANY_MUT_CNA_SV' DESC, samples DESC, profile_type154-frequency-brca_tcga_pan_can_atlas_2018
WITH
events AS (
SELECT sample_unique_id,
(variant_type = 'mutation' AND mutation_status != 'UNCALLED') AS is_mutation,
(variant_type = 'cna' AND cna_alteration = 2) AS is_amplification,
(variant_type = 'cna' AND cna_alteration = -2) AS is_deep_deletion,
(variant_type = 'structural_variant' AND mutation_status != 'UNCALLED') AS is_structural_variant
FROM genomic_event_derived
WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
AND hugo_gene_symbol IN ('TP53')
AND off_panel = 0
),
profiled AS (
SELECT stgp.sample_unique_id AS sample_unique_id,
stgp.alteration_type AS alteration_type
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
JOIN gene ge ON gpl.gene_id = ge.entrez_gene_id
WHERE stgp.cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
AND ge.hugo_gene_symbol IN ('TP53')
AND (stgp.alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT') OR (stgp.alteration_type = 'COPY_NUMBER_ALTERATION' AND stgp.genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE')))
UNION ALL
SELECT sample_unique_id, alteration_type
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
AND gene_panel_id = 'WES'
AND (alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT') OR (alteration_type = 'COPY_NUMBER_ALTERATION' AND genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE')))
)
SELECT *
FROM (
SELECT groupUniqArray(hugo_gene_symbol) AS matched_genes
FROM gene WHERE hugo_gene_symbol IN ('TP53')
) AS gene_match
CROSS JOIN (
SELECT groupUniqArray(cancer_study_identifier) AS matched_studies
FROM cancer_study WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
) AS study_match
CROSS JOIN (
SELECT uniqExactIf(sample_unique_id, is_mutation) AS altered_mutation,
uniqExactIf(sample_unique_id, is_amplification) AS altered_amplification,
uniqExactIf(sample_unique_id, is_deep_deletion) AS altered_deep_deletion,
uniqExactIf(sample_unique_id, is_structural_variant)
AS altered_structural_variant,
uniqExactIf(sample_unique_id, is_mutation OR is_amplification
OR is_deep_deletion OR is_structural_variant) AS altered_any
FROM events
) AS altered
CROSS JOIN (
SELECT uniqExactIf(sample_unique_id, alteration_type = 'MUTATION_EXTENDED')
AS profiled_MUTATION_EXTENDED,
uniqExactIf(sample_unique_id, alteration_type = 'COPY_NUMBER_ALTERATION')
AS profiled_COPY_NUMBER_ALTERATION,
uniqExactIf(sample_unique_id, alteration_type = 'STRUCTURAL_VARIANT')
AS profiled_STRUCTURAL_VARIANT,
uniqExact(sample_unique_id) AS profiled_ANY
FROM profiled
) AS profiled_counts154-top-mutation-brca_tcga_pan_can_atlas_2018
WITH
altered AS (
SELECT hugo_gene_symbol, COUNT(DISTINCT sample_unique_id) AS altered_samples
FROM genomic_event_derived
WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
AND off_panel = 0
AND (variant_type = 'mutation' AND mutation_status != 'UNCALLED')
GROUP BY hugo_gene_symbol
ORDER BY altered_samples DESC, hugo_gene_symbol ASC
LIMIT 20
),
wes_samples AS (
SELECT DISTINCT sample_unique_id
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
AND gene_panel_id = 'WES'
AND (alteration_type IN ('MUTATION_EXTENDED'))
),
panel_profiled AS (
SELECT ge.hugo_gene_symbol AS hugo_gene_symbol,
COUNT(DISTINCT stgp.sample_unique_id) AS n
FROM sample_to_gene_panel_derived stgp
JOIN gene_panel gp ON stgp.gene_panel_id = gp.stable_id
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
JOIN gene ge ON gpl.gene_id = ge.entrez_gene_id
WHERE stgp.cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
AND (stgp.alteration_type IN ('MUTATION_EXTENDED'))
AND stgp.sample_unique_id NOT IN (SELECT sample_unique_id FROM wes_samples)
AND ge.hugo_gene_symbol IN (SELECT hugo_gene_symbol FROM altered)
GROUP BY ge.hugo_gene_symbol
)
SELECT a.hugo_gene_symbol AS hugo_gene_symbol,
a.altered_samples AS altered_samples,
(SELECT count() FROM wes_samples) + COALESCE(p.n, 0) AS profiled_samples
FROM altered a
LEFT JOIN panel_profiled p ON a.hugo_gene_symbol = p.hugo_gene_symbol
ORDER BY altered_samples DESC, hugo_gene_symbol ASC154-profiled-brca_tcga_pan_can_atlas_2018
WITH
profiled AS (
SELECT cancer_study_identifier, alteration_type AS profile_type,
sample_unique_id, gene_panel_id
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
UNION ALL
SELECT cancer_study_identifier, 'COPY_NUMBER_ALTERATION_DISCRETE' AS profile_type,
sample_unique_id, gene_panel_id
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
AND ((alteration_type = 'COPY_NUMBER_ALTERATION' AND genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE')))
UNION ALL
SELECT cancer_study_identifier, 'ANY_MUT_CNA_SV' AS profile_type,
sample_unique_id, gene_panel_id
FROM sample_to_gene_panel_derived
WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
AND (alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT') OR (alteration_type = 'COPY_NUMBER_ALTERATION' AND genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE')))
),
sample_patient AS (
SELECT sample_unique_id, patient_unique_id
FROM sample_derived
WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
)
SELECT * FROM (
SELECT cancer_study_identifier, 'ALL_SAMPLES' AS profile_type,
COUNT(DISTINCT sample_unique_id) AS samples,
COUNT(DISTINCT patient_unique_id) AS patients,
toUInt64(0) AS wes_samples
FROM sample_derived
WHERE cancer_study_identifier IN ('brca_tcga_pan_can_atlas_2018')
GROUP BY cancer_study_identifier
UNION ALL
SELECT p.cancer_study_identifier, p.profile_type,
COUNT(DISTINCT p.sample_unique_id),
COUNT(DISTINCT nullIf(sp.patient_unique_id, '')),
COUNT(DISTINCT if(p.gene_panel_id = 'WES', p.sample_unique_id, NULL))
FROM profiled p
LEFT JOIN sample_patient sp ON p.sample_unique_id = sp.sample_unique_id
GROUP BY p.cancer_study_identifier, p.profile_type
)
ORDER BY cancer_study_identifier, profile_type = 'ALL_SAMPLES' DESC,
profile_type = 'ANY_MUT_CNA_SV' DESC, samples DESC, profile_type154-cancer-type-TP53
SELECT * FROM (
SELECT cancer_type, 'TP53' AS hugo_gene_symbol,
altered_samples, profiled_samples
FROM gene_alteration_frequency_by_cancer_type(
preference='all_studies_non_redundant', gene='TP53',
alteration='mutation')
)
ORDER BY altered_samples / profiled_samples DESC, altered_samples DESC, cancer_type ASC
LIMIT 20154-cancer-type-TP53-pan-cancer-tcga
SELECT * FROM (
SELECT cancer_type, 'TP53' AS hugo_gene_symbol,
altered_samples, profiled_samples
FROM gene_alteration_frequency_by_cancer_type(
preference='pan_cancer_tcga', gene='TP53',
alteration='mutation')
)
ORDER BY altered_samples / profiled_samples DESC, altered_samples DESC, cancer_type ASC
LIMIT 20154-build-study_gene_alteration_counts-msk_chord_2024
WITH
alteration_to_profile AS (
SELECT t.1 AS alteration_type, t.2 AS profile_type
FROM (
SELECT arrayJoin([
('mutation', 'MUTATION_EXTENDED'),
('amplification', 'COPY_NUMBER_ALTERATION'),
('deep_deletion', 'COPY_NUMBER_ALTERATION'),
('structural_variant', 'STRUCTURAL_VARIANT'),
('any', 'ANY')
]) AS t
)
),
events AS (
SELECT cancer_study_identifier,
hugo_gene_symbol,
sample_unique_id,
multiIf(
variant_type = 'mutation' AND mutation_status != 'UNCALLED', 'mutation',
variant_type = 'cna' AND cna_alteration = 2, 'amplification',
variant_type = 'cna' AND cna_alteration = -2, 'deep_deletion',
variant_type = 'structural_variant' AND mutation_status != 'UNCALLED',
'structural_variant',
'') AS event_type
FROM (SELECT * FROM genomic_event_derived WHERE cancer_study_identifier='msk_chord_2024')
WHERE off_panel = 0
),
altered AS (
-- event_type, not alteration_type: in ClickHouse a SELECT alias shadows
-- the source column in WHERE, so `'any' AS alteration_type` below would
-- turn a WHERE on alteration_type into a constant.
SELECT cancer_study_identifier, hugo_gene_symbol, event_type AS alteration_type,
COUNT(DISTINCT sample_unique_id) AS altered_samples,
COUNT(*) AS altered_events
FROM events
WHERE event_type != ''
GROUP BY cancer_study_identifier, hugo_gene_symbol, event_type
UNION ALL
SELECT cancer_study_identifier, hugo_gene_symbol, 'any' AS alteration_type,
COUNT(DISTINCT sample_unique_id) AS altered_samples,
COUNT(*) AS altered_events
FROM events
WHERE event_type != ''
GROUP BY cancer_study_identifier, hugo_gene_symbol
),
typed_profile_rows AS (
-- Same profile filters as the {mutation,cna,sv}_*_coverage views:
-- CNA rows only from DISCRETE profiles.
SELECT cancer_study_identifier, alteration_type AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='msk_chord_2024')
WHERE alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT')
OR (alteration_type = 'COPY_NUMBER_ALTERATION'
AND genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE'))
),
profile_rows AS (
SELECT cancer_study_identifier, profile_type, sample_unique_id, gene_panel_id
FROM typed_profile_rows
UNION ALL
SELECT cancer_study_identifier, 'ANY' AS profile_type, sample_unique_id, gene_panel_id
FROM typed_profile_rows
),
sample_buckets AS (
-- One row per sample per profile type: its panel-set signature.
SELECT cancer_study_identifier, profile_type, sample_unique_id,
arraySort(groupUniqArray(gene_panel_id)) AS panels
FROM profile_rows
GROUP BY cancer_study_identifier, profile_type, sample_unique_id
),
wes_profiled AS (
SELECT cancer_study_identifier, profile_type, COUNT(*) AS n
FROM sample_buckets
WHERE has(panels, 'WES')
GROUP BY cancer_study_identifier, profile_type
),
signature_counts AS (
SELECT cancer_study_identifier, profile_type, panels, COUNT(*) AS n
FROM sample_buckets
WHERE NOT has(panels, 'WES')
GROUP BY cancer_study_identifier, profile_type, panels
),
panel_genes AS (
-- Same panel -> gene chain as mutation_panel_gene_coverage.
SELECT gp.stable_id AS gene_panel_id, g.hugo_gene_symbol AS hugo_gene_symbol
FROM gene_panel gp
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
JOIN gene g ON gpl.gene_id = g.entrez_gene_id
),
panel_profiled AS (
SELECT cancer_study_identifier, profile_type, hugo_gene_symbol, SUM(n) AS n
FROM (
-- DISTINCT: a gene listed by two panels of one signature counts once.
SELECT DISTINCT s.cancer_study_identifier, s.profile_type, s.panels, s.n, pg.hugo_gene_symbol
FROM (
SELECT cancer_study_identifier, profile_type, panels, n,
arrayJoin(panels) AS gene_panel_id
FROM signature_counts
) s
JOIN panel_genes pg ON s.gene_panel_id = pg.gene_panel_id
)
GROUP BY cancer_study_identifier, profile_type, hugo_gene_symbol
)
SELECT a.cancer_study_identifier,
a.hugo_gene_symbol,
a.alteration_type,
a.altered_samples,
COALESCE(w.n, 0) + COALESCE(p.n, 0) AS profiled_samples,
a.altered_events,
now() AS built_at
FROM altered a
JOIN alteration_to_profile m ON a.alteration_type = m.alteration_type
LEFT JOIN wes_profiled w
ON w.cancer_study_identifier = a.cancer_study_identifier
AND w.profile_type = m.profile_type
LEFT JOIN panel_profiled p
ON p.cancer_study_identifier = a.cancer_study_identifier
AND p.profile_type = m.profile_type
AND p.hugo_gene_symbol = a.hugo_gene_symbol154-build-study_gene_alteration_counts-brca_tcga_pan_can_atlas_2018
WITH
alteration_to_profile AS (
SELECT t.1 AS alteration_type, t.2 AS profile_type
FROM (
SELECT arrayJoin([
('mutation', 'MUTATION_EXTENDED'),
('amplification', 'COPY_NUMBER_ALTERATION'),
('deep_deletion', 'COPY_NUMBER_ALTERATION'),
('structural_variant', 'STRUCTURAL_VARIANT'),
('any', 'ANY')
]) AS t
)
),
events AS (
SELECT cancer_study_identifier,
hugo_gene_symbol,
sample_unique_id,
multiIf(
variant_type = 'mutation' AND mutation_status != 'UNCALLED', 'mutation',
variant_type = 'cna' AND cna_alteration = 2, 'amplification',
variant_type = 'cna' AND cna_alteration = -2, 'deep_deletion',
variant_type = 'structural_variant' AND mutation_status != 'UNCALLED',
'structural_variant',
'') AS event_type
FROM (SELECT * FROM genomic_event_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
WHERE off_panel = 0
),
altered AS (
-- event_type, not alteration_type: in ClickHouse a SELECT alias shadows
-- the source column in WHERE, so `'any' AS alteration_type` below would
-- turn a WHERE on alteration_type into a constant.
SELECT cancer_study_identifier, hugo_gene_symbol, event_type AS alteration_type,
COUNT(DISTINCT sample_unique_id) AS altered_samples,
COUNT(*) AS altered_events
FROM events
WHERE event_type != ''
GROUP BY cancer_study_identifier, hugo_gene_symbol, event_type
UNION ALL
SELECT cancer_study_identifier, hugo_gene_symbol, 'any' AS alteration_type,
COUNT(DISTINCT sample_unique_id) AS altered_samples,
COUNT(*) AS altered_events
FROM events
WHERE event_type != ''
GROUP BY cancer_study_identifier, hugo_gene_symbol
),
typed_profile_rows AS (
-- Same profile filters as the {mutation,cna,sv}_*_coverage views:
-- CNA rows only from DISCRETE profiles.
SELECT cancer_study_identifier, alteration_type AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
WHERE alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT')
OR (alteration_type = 'COPY_NUMBER_ALTERATION'
AND genetic_profile_id IN (SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE'))
),
profile_rows AS (
SELECT cancer_study_identifier, profile_type, sample_unique_id, gene_panel_id
FROM typed_profile_rows
UNION ALL
SELECT cancer_study_identifier, 'ANY' AS profile_type, sample_unique_id, gene_panel_id
FROM typed_profile_rows
),
sample_buckets AS (
-- One row per sample per profile type: its panel-set signature.
SELECT cancer_study_identifier, profile_type, sample_unique_id,
arraySort(groupUniqArray(gene_panel_id)) AS panels
FROM profile_rows
GROUP BY cancer_study_identifier, profile_type, sample_unique_id
),
wes_profiled AS (
SELECT cancer_study_identifier, profile_type, COUNT(*) AS n
FROM sample_buckets
WHERE has(panels, 'WES')
GROUP BY cancer_study_identifier, profile_type
),
signature_counts AS (
SELECT cancer_study_identifier, profile_type, panels, COUNT(*) AS n
FROM sample_buckets
WHERE NOT has(panels, 'WES')
GROUP BY cancer_study_identifier, profile_type, panels
),
panel_genes AS (
-- Same panel -> gene chain as mutation_panel_gene_coverage.
SELECT gp.stable_id AS gene_panel_id, g.hugo_gene_symbol AS hugo_gene_symbol
FROM gene_panel gp
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
JOIN gene g ON gpl.gene_id = g.entrez_gene_id
),
panel_profiled AS (
SELECT cancer_study_identifier, profile_type, hugo_gene_symbol, SUM(n) AS n
FROM (
-- DISTINCT: a gene listed by two panels of one signature counts once.
SELECT DISTINCT s.cancer_study_identifier, s.profile_type, s.panels, s.n, pg.hugo_gene_symbol
FROM (
SELECT cancer_study_identifier, profile_type, panels, n,
arrayJoin(panels) AS gene_panel_id
FROM signature_counts
) s
JOIN panel_genes pg ON s.gene_panel_id = pg.gene_panel_id
)
GROUP BY cancer_study_identifier, profile_type, hugo_gene_symbol
)
SELECT a.cancer_study_identifier,
a.hugo_gene_symbol,
a.alteration_type,
a.altered_samples,
COALESCE(w.n, 0) + COALESCE(p.n, 0) AS profiled_samples,
a.altered_events,
now() AS built_at
FROM altered a
JOIN alteration_to_profile m ON a.alteration_type = m.alteration_type
LEFT JOIN wes_profiled w
ON w.cancer_study_identifier = a.cancer_study_identifier
AND w.profile_type = m.profile_type
LEFT JOIN panel_profiled p
ON p.cancer_study_identifier = a.cancer_study_identifier
AND p.profile_type = m.profile_type
AND p.hugo_gene_symbol = a.hugo_gene_symbol154-build-cancer_type_gene_alteration_counts-msk_chord_2024
WITH
alteration_to_profile AS (
SELECT t.1 AS alteration_type, t.2 AS profile_type
FROM (
SELECT arrayJoin([
('mutation', 'MUTATION_EXTENDED'),
('amplification', 'COPY_NUMBER_ALTERATION'),
('deep_deletion', 'COPY_NUMBER_ALTERATION'),
('structural_variant', 'STRUCTURAL_VARIANT'),
('any', 'ANY')
]) AS t
)
),
sample_cancer_type AS (
SELECT q.preference_name AS preference_name,
cd.cancer_study_identifier AS cancer_study_identifier,
cd.sample_unique_id AS sample_unique_id,
cd.attribute_value AS cancer_type
FROM (SELECT * FROM clinical_data_derived WHERE cancer_study_identifier='msk_chord_2024') cd
JOIN cancer_study_query_preferences q ON cd.cancer_study_identifier = q.cancer_study_identifier
WHERE cd.attribute_name = 'CANCER_TYPE'
),
events AS (
SELECT cancer_study_identifier,
hugo_gene_symbol,
sample_unique_id,
multiIf(
variant_type = 'mutation' AND mutation_status != 'UNCALLED', 'mutation',
variant_type = 'cna' AND cna_alteration = 2, 'amplification',
variant_type = 'cna' AND cna_alteration = -2, 'deep_deletion',
variant_type = 'structural_variant' AND mutation_status != 'UNCALLED',
'structural_variant',
'') AS event_type
FROM (SELECT * FROM genomic_event_derived WHERE cancer_study_identifier='msk_chord_2024')
WHERE off_panel = 0
AND cancer_study_identifier IN (SELECT cancer_study_identifier FROM cancer_study_query_preferences)
),
typed_events AS (
SELECT sct.preference_name AS preference_name, sct.cancer_type AS cancer_type,
e.hugo_gene_symbol AS hugo_gene_symbol, e.event_type AS event_type,
e.sample_unique_id AS sample_unique_id
FROM events e
JOIN sample_cancer_type sct
ON e.cancer_study_identifier = sct.cancer_study_identifier
AND e.sample_unique_id = sct.sample_unique_id
WHERE e.event_type != ''
),
altered AS (
SELECT preference_name, cancer_type, hugo_gene_symbol, event_type AS alteration_type,
COUNT(DISTINCT sample_unique_id) AS altered_samples
FROM typed_events
GROUP BY preference_name, cancer_type, hugo_gene_symbol, event_type
UNION ALL
SELECT preference_name, cancer_type, hugo_gene_symbol, 'any' AS alteration_type,
COUNT(DISTINCT sample_unique_id) AS altered_samples
FROM typed_events
GROUP BY preference_name, cancer_type, hugo_gene_symbol
),
profile_rows AS (
SELECT cancer_study_identifier, alteration_type AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='msk_chord_2024')
WHERE alteration_type IN ('MUTATION_EXTENDED', 'COPY_NUMBER_ALTERATION', 'STRUCTURAL_VARIANT')
UNION ALL
SELECT cancer_study_identifier, 'ANY' AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='msk_chord_2024')
WHERE alteration_type IN ('MUTATION_EXTENDED', 'COPY_NUMBER_ALTERATION', 'STRUCTURAL_VARIANT')
),
sample_buckets AS (
SELECT sct.preference_name AS preference_name, sct.cancer_type AS cancer_type,
pr.profile_type AS profile_type, pr.sample_unique_id AS sample_unique_id,
arraySort(groupUniqArray(pr.gene_panel_id)) AS panels
FROM profile_rows pr
JOIN sample_cancer_type sct
ON pr.cancer_study_identifier = sct.cancer_study_identifier
AND pr.sample_unique_id = sct.sample_unique_id
GROUP BY preference_name, cancer_type, profile_type, sample_unique_id
),
wes_profiled AS (
SELECT preference_name, cancer_type, profile_type, COUNT(*) AS n
FROM sample_buckets
WHERE has(panels, 'WES')
GROUP BY preference_name, cancer_type, profile_type
),
signature_counts AS (
SELECT preference_name, cancer_type, profile_type, panels, COUNT(*) AS n
FROM sample_buckets
WHERE NOT has(panels, 'WES')
GROUP BY preference_name, cancer_type, profile_type, panels
),
panel_genes AS (
SELECT gp.stable_id AS gene_panel_id, g.hugo_gene_symbol AS hugo_gene_symbol
FROM gene_panel gp
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
JOIN gene g ON gpl.gene_id = g.entrez_gene_id
),
panel_profiled AS (
SELECT preference_name, cancer_type, profile_type, hugo_gene_symbol, SUM(n) AS n
FROM (
SELECT DISTINCT s.preference_name, s.cancer_type, s.profile_type, s.panels, s.n,
pg.hugo_gene_symbol
FROM (
SELECT preference_name, cancer_type, profile_type, panels, n,
arrayJoin(panels) AS gene_panel_id
FROM signature_counts
) s
JOIN panel_genes pg ON s.gene_panel_id = pg.gene_panel_id
)
GROUP BY preference_name, cancer_type, profile_type, hugo_gene_symbol
)
SELECT a.preference_name,
a.cancer_type,
a.hugo_gene_symbol,
a.alteration_type,
a.altered_samples,
COALESCE(w.n, 0) + COALESCE(p.n, 0) AS profiled_samples,
now() AS built_at
FROM altered a
JOIN alteration_to_profile m ON a.alteration_type = m.alteration_type
LEFT JOIN wes_profiled w
ON w.preference_name = a.preference_name
AND w.cancer_type = a.cancer_type
AND w.profile_type = m.profile_type
LEFT JOIN panel_profiled p
ON p.preference_name = a.preference_name
AND p.cancer_type = a.cancer_type
AND p.profile_type = m.profile_type
AND p.hugo_gene_symbol = a.hugo_gene_symbol154-build-cancer_type_gene_alteration_counts-brca_tcga_pan_can_atlas_2018
WITH
alteration_to_profile AS (
SELECT t.1 AS alteration_type, t.2 AS profile_type
FROM (
SELECT arrayJoin([
('mutation', 'MUTATION_EXTENDED'),
('amplification', 'COPY_NUMBER_ALTERATION'),
('deep_deletion', 'COPY_NUMBER_ALTERATION'),
('structural_variant', 'STRUCTURAL_VARIANT'),
('any', 'ANY')
]) AS t
)
),
sample_cancer_type AS (
SELECT q.preference_name AS preference_name,
cd.cancer_study_identifier AS cancer_study_identifier,
cd.sample_unique_id AS sample_unique_id,
cd.attribute_value AS cancer_type
FROM (SELECT * FROM clinical_data_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018') cd
JOIN cancer_study_query_preferences q ON cd.cancer_study_identifier = q.cancer_study_identifier
WHERE cd.attribute_name = 'CANCER_TYPE'
),
events AS (
SELECT cancer_study_identifier,
hugo_gene_symbol,
sample_unique_id,
multiIf(
variant_type = 'mutation' AND mutation_status != 'UNCALLED', 'mutation',
variant_type = 'cna' AND cna_alteration = 2, 'amplification',
variant_type = 'cna' AND cna_alteration = -2, 'deep_deletion',
variant_type = 'structural_variant' AND mutation_status != 'UNCALLED',
'structural_variant',
'') AS event_type
FROM (SELECT * FROM genomic_event_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
WHERE off_panel = 0
AND cancer_study_identifier IN (SELECT cancer_study_identifier FROM cancer_study_query_preferences)
),
typed_events AS (
SELECT sct.preference_name AS preference_name, sct.cancer_type AS cancer_type,
e.hugo_gene_symbol AS hugo_gene_symbol, e.event_type AS event_type,
e.sample_unique_id AS sample_unique_id
FROM events e
JOIN sample_cancer_type sct
ON e.cancer_study_identifier = sct.cancer_study_identifier
AND e.sample_unique_id = sct.sample_unique_id
WHERE e.event_type != ''
),
altered AS (
SELECT preference_name, cancer_type, hugo_gene_symbol, event_type AS alteration_type,
COUNT(DISTINCT sample_unique_id) AS altered_samples
FROM typed_events
GROUP BY preference_name, cancer_type, hugo_gene_symbol, event_type
UNION ALL
SELECT preference_name, cancer_type, hugo_gene_symbol, 'any' AS alteration_type,
COUNT(DISTINCT sample_unique_id) AS altered_samples
FROM typed_events
GROUP BY preference_name, cancer_type, hugo_gene_symbol
),
profile_rows AS (
SELECT cancer_study_identifier, alteration_type AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
WHERE alteration_type IN ('MUTATION_EXTENDED', 'COPY_NUMBER_ALTERATION', 'STRUCTURAL_VARIANT')
UNION ALL
SELECT cancer_study_identifier, 'ANY' AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
WHERE alteration_type IN ('MUTATION_EXTENDED', 'COPY_NUMBER_ALTERATION', 'STRUCTURAL_VARIANT')
),
sample_buckets AS (
SELECT sct.preference_name AS preference_name, sct.cancer_type AS cancer_type,
pr.profile_type AS profile_type, pr.sample_unique_id AS sample_unique_id,
arraySort(groupUniqArray(pr.gene_panel_id)) AS panels
FROM profile_rows pr
JOIN sample_cancer_type sct
ON pr.cancer_study_identifier = sct.cancer_study_identifier
AND pr.sample_unique_id = sct.sample_unique_id
GROUP BY preference_name, cancer_type, profile_type, sample_unique_id
),
wes_profiled AS (
SELECT preference_name, cancer_type, profile_type, COUNT(*) AS n
FROM sample_buckets
WHERE has(panels, 'WES')
GROUP BY preference_name, cancer_type, profile_type
),
signature_counts AS (
SELECT preference_name, cancer_type, profile_type, panels, COUNT(*) AS n
FROM sample_buckets
WHERE NOT has(panels, 'WES')
GROUP BY preference_name, cancer_type, profile_type, panels
),
panel_genes AS (
SELECT gp.stable_id AS gene_panel_id, g.hugo_gene_symbol AS hugo_gene_symbol
FROM gene_panel gp
JOIN gene_panel_list gpl ON gp.internal_id = gpl.internal_id
JOIN gene g ON gpl.gene_id = g.entrez_gene_id
),
panel_profiled AS (
SELECT preference_name, cancer_type, profile_type, hugo_gene_symbol, SUM(n) AS n
FROM (
SELECT DISTINCT s.preference_name, s.cancer_type, s.profile_type, s.panels, s.n,
pg.hugo_gene_symbol
FROM (
SELECT preference_name, cancer_type, profile_type, panels, n,
arrayJoin(panels) AS gene_panel_id
FROM signature_counts
) s
JOIN panel_genes pg ON s.gene_panel_id = pg.gene_panel_id
)
GROUP BY preference_name, cancer_type, profile_type, hugo_gene_symbol
)
SELECT a.preference_name,
a.cancer_type,
a.hugo_gene_symbol,
a.alteration_type,
a.altered_samples,
COALESCE(w.n, 0) + COALESCE(p.n, 0) AS profiled_samples,
now() AS built_at
FROM altered a
JOIN alteration_to_profile m ON a.alteration_type = m.alteration_type
LEFT JOIN wes_profiled w
ON w.preference_name = a.preference_name
AND w.cancer_type = a.cancer_type
AND w.profile_type = m.profile_type
LEFT JOIN panel_profiled p
ON p.preference_name = a.preference_name
AND p.cancer_type = a.cancer_type
AND p.profile_type = m.profile_type
AND p.hugo_gene_symbol = a.hugo_gene_symbol154-build-study_profiled_counts-msk_chord_2024
WITH
sample_patient AS (
SELECT sample_unique_id, patient_unique_id
FROM (SELECT * FROM sample_derived WHERE cancer_study_identifier='msk_chord_2024')
),
discrete_cna_profiles AS (
SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE'
),
profiled AS (
SELECT cancer_study_identifier, alteration_type AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='msk_chord_2024')
UNION ALL
SELECT cancer_study_identifier, 'COPY_NUMBER_ALTERATION_DISCRETE' AS profile_type,
sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='msk_chord_2024')
WHERE alteration_type = 'COPY_NUMBER_ALTERATION'
AND genetic_profile_id IN (SELECT stable_id FROM discrete_cna_profiles)
UNION ALL
SELECT cancer_study_identifier, 'ANY_MUT_CNA_SV' AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='msk_chord_2024')
WHERE alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT')
OR (alteration_type = 'COPY_NUMBER_ALTERATION'
AND genetic_profile_id IN (SELECT stable_id FROM discrete_cna_profiles))
)
SELECT cancer_study_identifier,
'ALL_SAMPLES' AS profile_type,
COUNT(DISTINCT sample_unique_id) AS samples,
COUNT(DISTINCT patient_unique_id) AS patients,
toUInt64(0) AS wes_samples,
now() AS built_at
FROM (SELECT * FROM sample_derived WHERE cancer_study_identifier='msk_chord_2024')
GROUP BY cancer_study_identifier
UNION ALL
SELECT p.cancer_study_identifier,
p.profile_type,
COUNT(DISTINCT p.sample_unique_id) AS samples,
COUNT(DISTINCT nullIf(sp.patient_unique_id, '')) AS patients,
COUNT(DISTINCT if(p.gene_panel_id = 'WES', p.sample_unique_id, NULL)) AS wes_samples,
now() AS built_at
FROM profiled p
LEFT JOIN sample_patient sp ON p.sample_unique_id = sp.sample_unique_id
GROUP BY p.cancer_study_identifier, p.profile_type154-build-study_profiled_counts-brca_tcga_pan_can_atlas_2018
WITH
sample_patient AS (
SELECT sample_unique_id, patient_unique_id
FROM (SELECT * FROM sample_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
),
discrete_cna_profiles AS (
SELECT stable_id FROM genetic_profile WHERE datatype = 'DISCRETE'
),
profiled AS (
SELECT cancer_study_identifier, alteration_type AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
UNION ALL
SELECT cancer_study_identifier, 'COPY_NUMBER_ALTERATION_DISCRETE' AS profile_type,
sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
WHERE alteration_type = 'COPY_NUMBER_ALTERATION'
AND genetic_profile_id IN (SELECT stable_id FROM discrete_cna_profiles)
UNION ALL
SELECT cancer_study_identifier, 'ANY_MUT_CNA_SV' AS profile_type, sample_unique_id, gene_panel_id
FROM (SELECT * FROM sample_to_gene_panel_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
WHERE alteration_type IN ('MUTATION_EXTENDED', 'STRUCTURAL_VARIANT')
OR (alteration_type = 'COPY_NUMBER_ALTERATION'
AND genetic_profile_id IN (SELECT stable_id FROM discrete_cna_profiles))
)
SELECT cancer_study_identifier,
'ALL_SAMPLES' AS profile_type,
COUNT(DISTINCT sample_unique_id) AS samples,
COUNT(DISTINCT patient_unique_id) AS patients,
toUInt64(0) AS wes_samples,
now() AS built_at
FROM (SELECT * FROM sample_derived WHERE cancer_study_identifier='brca_tcga_pan_can_atlas_2018')
GROUP BY cancer_study_identifier
UNION ALL
SELECT p.cancer_study_identifier,
p.profile_type,
COUNT(DISTINCT p.sample_unique_id) AS samples,
COUNT(DISTINCT nullIf(sp.patient_unique_id, '')) AS patients,
COUNT(DISTINCT if(p.gene_panel_id = 'WES', p.sample_unique_id, NULL)) AS wes_samples,
now() AS built_at
FROM profiled p
LEFT JOIN sample_patient sp ON p.sample_unique_id = sp.sample_unique_id
GROUP BY p.cancer_study_identifier, p.profile_type#160 — gene-frequency views through an IN subquery
“Before” is the currently deployed parameterized view gene_alteration_frequency_by_cancer_type(preference='all_studies_non_redundant', …). “After” is the PR's SELECT body inlined with the same parameters substituted. CNA is covered by the two supported values, amplification and deep_deletion; there is no generic cna parameter.
| Gene · alteration | Before ms | After ms | Saving ms | Rows read (both) | Same result |
|---|---|---|---|---|---|
| TP53 · mutation | 991.79 | 394.83 | 596.96 | 30,599,882 | yes |
| TP53 · amplification | 875.87 | 489.16 | 386.70 | 43,829,849 | yes |
| TP53 · deep deletion | 753.27 | 468.67 | 284.60 | 43,829,849 | yes |
| TP53 · structural variant | 485.29 | 211.08 | 274.21 | 14,117,465 | yes |
| ERBB2 · mutation | 744.86 | 361.58 | 383.28 | 30,599,882 | yes |
| ERBB2 · amplification | 765.85 | 496.49 | 269.36 | 43,829,849 | yes |
| ERBB2 · deep deletion | 732.77 | 487.97 | 244.80 | 43,829,849 | yes |
| ERBB2 · structural variant | 572.37 | 270.05 | 302.32 | 14,117,465 | yes |
Saving 0.245–0.597 s per call, median 0.293 s. Rows and bytes read are unchanged: the rewrite cuts join/filter work, not the data read. The per-answer gain is this saving times the number of such calls in an answer, which needs the historical SQL to establish.
#154 — domain tools on precomputed aggregates
The PR's tools read small precomputed tables and fall back to live SQL when those are missing. None of the three tables exists on the public database, so this measures (1) the exact fallback SQL from the PR's query builders, and (2) the table-build SELECTs, each restricted to one study.
- Alteration frequency: TP53, all alteration types, one aggregate row. Top genes: mutation, top 20. Profiled counts: all profile types. All three on
msk_chord_2024andbrca_tcga_pan_can_atlas_2018. - Frequency by cancer type takes a cohort preference instead of a study: TP53 mutation for
all_studies_non_redundantand for its default,pan_cancer_tcga. - Lookup cost is estimated from keyed SELECTs on existing small tables:
cancer_study(548 rows) 3.06 ms andgene(44,896 rows) 4.00 ms, so a lookup is estimated at 3–4 ms. These are proxies, not measurements of the PR's tables.
One fallback call replaced by one lookup
| Fallback query | Fallback s | Est. saving s |
|---|---|---|
154-frequency-msk_chord_2024 | 0.187 | 0.183 |
154-top-mutation-msk_chord_2024 | 0.282 | 0.278 |
154-profiled-msk_chord_2024 | 0.138 | 0.134 |
154-frequency-brca_tcga_pan_can_atlas_2018 | 0.082 | 0.078 |
154-top-mutation-brca_tcga_pan_can_atlas_2018 | 0.310 | 0.306 |
154-profiled-brca_tcga_pan_can_atlas_2018 | 0.078 | 0.074 |
154-cancer-type-TP53 | 1.348 | 1.344 |
154-cancer-type-TP53-pan-cancer-tcga | 0.853 | 0.849 |
Estimated saving = fallback time − 4.00 ms, the upper end of the 3–4 ms lookup estimate. Database side only. In the 09-23 questions, a text screen finds 54 gene-frequency answers (median 5 SQL calls), 26 top-genes answers (median 4) and 6 profiled-count answers (median 2) that such tools could serve (template-turns.json; categories overlap). If a tool replaces N such calls with one lookup it saves roughly N × fallback − lookup, but that's an assumption, not a measured benchmark saving, and SQL calls are not the same as sequential model turns.
Table-build SELECTs (computation only)
| Build SELECT | Server ms | Rows read |
|---|---|---|
154-build-study_gene_alteration_counts-msk_chord_2024 | 558.55 | 13,498,040 |
154-build-study_gene_alteration_counts-brca_tcga_pan_can_atlas_2018 | 547.78 | 14,038,712 |
154-build-cancer_type_gene_alteration_counts-msk_chord_2024 | 997.13 | 13,718,830 |
154-build-cancer_type_gene_alteration_counts-brca_tcga_pan_can_atlas_2018 | 1,036.70 | 14,161,198 |
154-build-study_profiled_counts-msk_chord_2024 | 69.33 | 2,916,228 |
154-build-study_profiled_counts-brca_tcga_pan_can_atlas_2018 | 35.99 | 2,850,692 |
Three build SELECTs back the four tools: study_gene_alteration_counts (frequency, top genes), cancer_type_gene_alteration_counts (by cancer type) and study_profiled_counts (profiled counts). Each source table was replaced by a study-filtered subquery, with all other joins, filters and grouping kept. Timing covers the SELECT only, not inserting into MergeTree; per-study costs can't safely be multiplied into a full-build estimate.
#151 — schema and guide cache
#151 caches SHOW TABLES, DESCRIBE TABLE and the dynamic study guides, one hour by default. The SHOW/DESCRIBE timings are the database work a cache hit skips.
| Cacheable call | Server ms saved per hit | Laptop round trip ms (not in-cluster) |
|---|---|---|
metadata-show-tables | 2.58 | 568.46 |
metadata-describe-cancer_study | 1.72 | 566.20 |
metadata-describe-genomic_event_derived | 1.38 | 548.27 |
metadata-describe-sample_to_gene_panel_derived | 1.37 | 539.48 |
metadata-describe-clinical_data_derived | 1.38 | 541.05 |
metadata-describe-sample_derived | 1.55 | 590.97 |
metadata-describe-gene | 1.37 | 543.38 |
metadata-describe-genetic_profile | 1.40 | 555.84 |
The 09-23 run made 101 list-tables and 327 describe-columns tool calls; the cache hit rate is unknown. SELECT version(), timezone() and system.settings were measured too (main table) but #151 doesn't cache them. Study-guide savings weren't measured. A cache hit still costs the model round trip that asked for the tool.
Limitations
- No DB share per answer. The 09-23 artifacts hold 292 answer latencies and every tool's name and status, but no tool inputs or executed SQL (SQL shown inside answers is an example, not evidence of what ran). No raw trace export was available. So 0 historical answers and 0 historical queries were replayed, and the median/p90 DB share is N/A. Recovering the traces is the outstanding part.
- Laptop, not cluster. Round-trip numbers include a fresh client process and a TLS handshake from outside the cluster and overstate what the MCP server pays per call.
- Different day, warm cache. Measured on 2026-09-29, warm; data, load and query-cache state may differ from 09-23.
- #154 lookups are estimates from small existing tables; the PR's tables were not built. Build timings are per-study SELECTs, not a full build.
- Only three open PRs, on a handful of genes and two studies. Other genes, studies or query shapes may behave differently.
Data and links
- measurements.csv / measurements.jsonl: per-query medians and result hashes. raw.jsonl: every execution including the warm-up (
iteration0). - queries.json: the exact SQL of every measured query. table-inventory.json: row and byte counts of the database's tables.
- turn-inventory.csv: every 09-23 answer, its latency and LLM/SQL call counts, and why its DB share is missing. template-turns.json: the answers the #154 tools could serve.
- Parameterized-view evidence: all 15 deployed parameterized views return rows when given parameters.
- PRs: #151, #154, #160 on cBioPortal/cbioportal-mcp. Benchmark: 2026-09-23 run.