Update databases, libraries

.gitignore

@@ -1,7 +1,7 @@
 *.pyc
 .idea
 .vscode
-venv
+venv/
 bio-api/__pycache__
 bio-api/.pytest_cache
 bio-api/media

DEPLOYING.md

@@ -71,7 +71,7 @@ To import all databases in MongoDB:
 Alternatively (but **not recommended** due to high computational demands) you can run a separate ETL process to download from source, process and import the databases into MongoDB.
 
 1. Install the requirements:  
-    - [R language](https://www.r-project.org/). Version 4.3.2 (Only necessary if you want to update the Gene information database from Ensembl and CiVIC)
+    - [R language](https://www.r-project.org/). Version 4.5.1 (Only necessary if you want to update the Gene information database from Ensembl and CiVIC)
     - Some python packages. They can be installed using:  
         `pip install -r config/genomic_db_conf/requirements.txt`  
 2. The ETL process is programmed in a single bash script for each database. Edit in the bash file of the database that you want to update the **user** and **password** parameters, using the same values that you set in the `docker-compose.yml` file. Bash files can be found in the *'databases'* folder, within the corresponding directories for each database:  

README.md

@@ -30,23 +30,23 @@ This document is focused on the **development** of the system. If you are lookin
 BioAPI obtains information from different bioinformatics databases. These databases were installed locally to reduce data search time. The databases currently integrated to BioAPI are:
 
 1. Gene nomenclature: [HUGO Gene Nomenclature Committee](https://www.genenames.org/).  
-HGNC is the resource for approved human gene nomenclature. Downloaded from its official website in September 2022.  
+HGNC is the resource for approved human gene nomenclature. Downloaded from its official website in May 2026.  
 2. Gene information:  
 
-   - [ENSEMBL](http://www.ensembl.org/biomart/martview): BioMart data mining tool was used to obtain a gene-related dataset from Ensembl. Ensembl is a genome browser for vertebrate genomes that supports comparative genomics, evolution, sequence variation and transcriptional regulation research. Ensembl annotates genes, computes multiple alignments, predicts regulatory function, and collects disease data. Downloaded using *BioMart data mining tool* in September 2022.
-   - [RefSeq](https://www.ncbi.nlm.nih.gov/refseq/): the summary of each human gene was obtained from the RefSeq database. RefSeq (Reference Sequence) is the public database of annotated and curated nucleic acid (DNA and RNA) and protein sequences from the National Center for Biotechnology Information (NCBI). To obtain the summaries, the R package called [GeneSummary](https://bioconductor.org/packages/release/data/annotation/html/GeneSummary.html) was used, which obtains the abstracts from version 214 of RefSeq.
-   - [CIViC](https://civicdb.org/welcome): A description of the genes oriented to clinical interpretation in cancer was obtained from the CIViC database, an open-source platform supporting crowd-sourced and expert-moderated cancer variant curation. The database was downloaded from its official website in April 2023.  
+   - [ENSEMBL](http://www.ensembl.org/biomart/martview): BioMart data mining tool was used to obtain a gene-related dataset from Ensembl. Ensembl is a genome browser for vertebrate genomes that supports comparative genomics, evolution, sequence variation and transcriptional regulation research. Ensembl annotates genes, computes multiple alignments, predicts regulatory function, and collects disease data. Downloaded using *BioMart data mining tool* in May 2026.
+   - [RefSeq](https://www.ncbi.nlm.nih.gov/refseq/): the summary of each human gene was obtained from the RefSeq database. RefSeq (Reference Sequence) is the public database of annotated and curated nucleic acid (DNA and RNA) and protein sequences from the National Center for Biotechnology Information (NCBI). To obtain the summaries, the R package called [GeneSummary](https://bioconductor.org/packages/release/data/annotation/html/GeneSummary.html) was used, which obtains the abstracts from version 220 of RefSeq.
+   - [CIViC](https://civicdb.org/welcome): A description of the genes oriented to clinical interpretation in cancer was obtained from the CIViC database, an open-source platform supporting crowd-sourced and expert-moderated cancer variant curation. The database was downloaded from its official website in May 2026.  
 
 3. Metabolic pathways: [ConsensusPathDB](http://cpdb.molgen.mpg.de/).  
-ConsensusPathDB-human integrates interaction networks in Homo sapiens including binary and complex protein-protein, genetic, metabolic, signaling, gene regulatory, and drug-target interactions, as well as biochemical pathways. Data originate from currently 31 public resources for interactions (listed below) and interactions that we have curated from the literature. The interaction data are integrated in a complementary manner (avoiding redundancies), resulting in a seamless interaction network containing different types of interactions. Downloaded from its official website in September 2022.
+ConsensusPathDB-human integrates interaction networks in Homo sapiens including binary and complex protein-protein, genetic, metabolic, signaling, gene regulatory, and drug-target interactions, as well as biochemical pathways. Data originate from currently 31 public resources for interactions (listed below) and interactions that we have curated from the literature. The interaction data are integrated in a complementary manner (avoiding redundancies), resulting in a seamless interaction network containing different types of interactions. Downloaded from its official website in May 2026.
 4. Gene expression: [Genotype-Tissue Expression (GTEx)](https://gtexportal.org/home/).  
-The Genotype-Tissue Expression (GTEx) project is an ongoing effort to build a comprehensive public resource to study tissue-specific gene expression and regulation. Samples were collected from 54 non-diseased tissue sites across nearly 1000 individuals, primarily for molecular assays including WGS, WES, and RNA-Seq. GTEx is being used in its version [GTEx Analysis V8 (dbGaP Accession phs000424.v8.p2)](https://gtexportal.org/home/datasets#datasetDiv1) and was downloaded from its official website in September 2022.
+The Genotype-Tissue Expression (GTEx) project is an ongoing effort to build a comprehensive public resource to study tissue-specific gene expression and regulation. Samples were collected from 54 non-diseased tissue sites across nearly 1000 individuals, primarily for molecular assays including WGS, WES, and RNA-Seq. GTEx is being used in its version [GTEx Analysis V8 (dbGaP Accession phs000424.v8.p2)](https://gtexportal.org/home/datasets#datasetDiv1).
 5. Therapies and actionable genes in cancer: [OncoKB](https://www.oncokb.org/).  
-OncoKB™ is a precision oncology knowledge base developed at Memorial Sloan Kettering Cancer Center that contains biological and clinical information about cancer-related genomic alterations. Alterations and tumor-type-specific therapeutic implications are classified using the OncoKB™ [Levels of Evidence system](https://www.oncokb.org/levels), which assigns clinical actionability to individual mutational events. Downloaded from its official website in November 2023.  
+OncoKB™ is a precision oncology knowledge base developed at Memorial Sloan Kettering Cancer Center that contains biological and clinical information about cancer-related genomic alterations. Alterations and tumor-type-specific therapeutic implications are classified using the OncoKB™ [Levels of Evidence system](https://www.oncokb.org/levels), which assigns clinical actionability to individual mutational events. Downloaded from its official website in May 2026.  
 6. Gene Ontology [Gene Ontology (GO)](http://geneontology.org/).
 It is a project to develop an up-to-date, comprehensive, computational model of biological systems, from the molecular level to larger pathways, cellular and organism-level systems. It provides structured and standardized annotations of gene products, in a hierarchical system of terms and relationships that describes the molecular functions, biological processes, and cellular components associated with genes and gene products. Downloaded from its official website in June 2023
 7. Cancer-related drugs [Pharmacogenomics Knowledge Base (PharmGKB)](https://www.pharmgkb.org/).
-It is a resource that provides information about how human genetic variation affects response to medications. PharmGKB collects, curates, and disseminates knowledge about clinically actionable gene-drug associations and genotype-phenotype relationships. Downloaded from its official website in February 2024
+It is a resource that provides information about how human genetic variation affects response to medications. PharmGKB collects, curates, and disseminates knowledge about clinically actionable gene-drug associations and genotype-phenotype relationships. Downloaded from its official website in May 2026
 8. Predicted functional associations network [STRING](https://string-db.org/)
 It is a database of known and predicted protein-protein interactions. The interactions include direct (physical) and indirect (functional) associations; they stem from computational prediction, knowledge transfer between organisms, and interactions aggregated from other (primary) databases. String is being used in its version 12.0.
 9. Pharmaco-transcriptomics [DrugBank](https://go.drugbank.com/)
@@ -474,7 +474,7 @@ This service retrieves information on FDA-approved precision oncology therapies,
                 "foundation_one_cdx",
                 "foundation_one_heme",
                 "vogelstein",
-                "sanger_cgc"
+                "cosmic_cgc"
             ],
             "therapeutic":[
                 {

bio-api/bioapi.py

@@ -537,7 +537,7 @@ def get_data_from_oncokb(genes: List[str], query: str) -> Dict[str, Dict[str, An
     """
     collection_actionability_gene = mydb["oncokb_biomarker_drug_associations"]
     collection_cancer_gene = mydb["oncokb_gene_cancer_list"]
-    collection_precision_oncology_therapies = mydb["oncokb_precision_therapies"]
+    collection_precision_oncology_therapies = mydb["oncokb_precision_oncology_therapies"]
     query1 = {'gene': {'$in': genes}}
     query2 = {'hgnc_symbol': {'$in': genes}}
     projection = {'_id': 0}

config/bioapi_conf/requirements.txt

@@ -1,15 +1,15 @@
 pymongo==4.6.1
 Flask==3.1.0
-flask-cors==5.0.0
-gunicorn==21.2.0
-ConfigParser==6.0.0
-pytest==8.3.3
+flask-cors==6.0.2
+gunicorn==26.0.0
+ConfigParser==7.2.0
+pytest==9.0.3
 tqdm==4.66.1
 gprofiler-official==1.0.0
 Werkzeug==3.1.3
-flask_swagger_ui==4.11.1
+flask_swagger_ui==5.32.6
 apispec==6.7.1
 apispec-webframeworks==1.2.0
-marshmallow==3.23.1
-setuptools==75.5.0
+marshmallow==4.3.0
+setuptools==82.0.0
 flasgger==0.9.7.1

config/genomic_db_conf/requirements.txt

@@ -1,8 +1,9 @@
-ConfigParser==5.2.0
-numpy==1.23.1
-pandas==1.4.3
+ConfigParser==7.2.0
+numpy==2.4.6
+pandas==3.0.2
+openpyxl==3.1.5
 pymongo==4.5.0
-pytest==7.1.2
+pytest==9.0.3
 six==1.16.0
 tqdm==4.64.0
-urllib3==1.26.9
+urllib3==2.7.0

databases/gene_info/geneinfo2mongodb.sh

@@ -7,8 +7,7 @@ user=
 password=
 db=bio_api
 #######################################
-CIVIC_URL="https://civicdb.org/downloads/nightly/nightly-GeneSummaries.tsv"
-# Using this URL you always get the latest release of CiVIC Gene Summaries
+
 
 date
 echo "INFO	Downloading genes database..."

databases/gene_info/get_datasets.R

@@ -1,44 +1,48 @@
-if (!require("BiocManager", quietly = TRUE))
+# Install BiocManager if it doesn't exist
+if (!requireNamespace("BiocManager", quietly = TRUE)) {
   install.packages("BiocManager")
-BiocManager::install(version = "3.18", ask = FALSE)
-
-if("biomaRt" %in% rownames(installed.packages()) == FALSE)
-  BiocManager::install("biomaRt", force = TRUE)
-
-if("devtools" %in% rownames(installed.packages()) == FALSE)
-  install.packages("devtools", repos = "http://cran.us.r-project.org")
-
-if("dbplyr" %in% rownames(installed.packages()) == FALSE)
-  devtools::install_version("dbplyr", version = "2.3.4", force = TRUE)
-
-if("GeneSummary" %in% rownames(installed.packages()) == FALSE)
-  BiocManager::install("GeneSummary", update = FALSE, ask = FALSE, force = FALSE)
-
+}
+
+# Auxiliary function to install from CRAN
+install_if_missing <- function(pkg) {
+  if (!requireNamespace(pkg, quietly = TRUE)) {
+    install.packages(pkg, repos = "http://cran.us.r-project.org")
+  }
+}
+
+# Install CRAN packages
+cran_packages <- c("devtools", "dplyr", "stringr")
+invisible(lapply(cran_packages, install_if_missing))
+
+# Install specific version of dbplyr
+if (!requireNamespace("dbplyr", quietly = TRUE) || packageVersion("dbplyr") != "2.3.4") {
+  devtools::install_version("dbplyr", version = "2.3.4", repos = "http://cran.us.r-project.org")
+}
+
+# Install Bioconductor packages
+bioc_packages <- c("biomaRt", "GeneSummary")
+for (pkg in bioc_packages) {
+  if (!requireNamespace(pkg, quietly = TRUE)) {
+    BiocManager::install(pkg, update = FALSE, ask = FALSE)
+  }
+}
+
+# Load libraries
 library(GeneSummary)
 library(dplyr)
 library(biomaRt)
 library(stringr)
-library(BiocManager)
 
-civicUrl="https://civicdb.org/downloads/nightly/nightly-GeneSummaries.tsv"
-try(civicUrl <- Sys.getenv(CIVIC_URL), silent = TRUE)
 
-# GRCh38 ####
-ensembl_grch38 = useEnsembl(biomart = "genes", dataset = "hsapiens_gene_ensembl") 
-atr <- c("ensembl_gene_id", "description", "chromosome_name", "start_position", "end_position", "strand", "band", "percentage_gene_gc_content", "gene_biotype", "hgnc_symbol", "hgnc_id", "entrezgene_id") 
-fil_chrom <- c("MT", "X", "Y", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22")
 
-res_grch38 <- getBM(
-  attributes = atr,
-  filters = c("chromosome_name", "with_hgnc"),
-  values = list(fil_chrom, TRUE),
-  mart = ensembl_grch38 
-)
-res_grch38$description <- data.frame(do.call('rbind', str_split(res_grch38$description, " \\[Source")))$X1
+civicUrl <- "https://civicdb.org/downloads/nightly/nightly-FeatureSummaries.tsv" # URL to download the latest version of CiVIC Gene Summaries
+
 
+atr <- c("ensembl_gene_id", "description", "chromosome_name", "start_position", "end_position", "strand", "band", "percentage_gene_gc_content", "gene_biotype", "hgnc_symbol", "hgnc_id", "entrezgene_id") 
+fil_chrom <- c("MT", "X", "Y", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22")
 
 # GRCh37 ####
-ensembl_grch37 = useEnsembl(biomart="genes", dataset = "hsapiens_gene_ensembl", GRCh = 37)
+ensembl_grch37 = useEnsembl(biomart="ensembl", dataset = "hsapiens_gene_ensembl", GRCh = 37)
 # listFilters(ensembl_grch37)
 # listAttributes(ensembl_grch37)
 atr <- c("ensembl_gene_id", "description", "chromosome_name", "start_position", "end_position", "strand", "band", "percentage_gene_gc_content", "gene_biotype", "hgnc_symbol", "hgnc_id", "entrezgene_id") 
@@ -52,30 +56,39 @@ res_grch37 <- getBM(
 res_grch37$description <- data.frame(do.call('rbind', str_split(res_grch37$description, " \\[Source")))$X1
 
 
+# GRCh38 ####
+ensembl_grch38 = useMart(biomart = "ensembl", dataset = "hsapiens_gene_ensembl") 
+res_grch38 <- getBM(
+  attributes = atr,
+  filters = c("chromosome_name", "with_hgnc"),
+  values = list(fil_chrom, TRUE),
+  mart = ensembl_grch38 
+)
+res_grch38$description <- data.frame(do.call('rbind', str_split(res_grch38$description, " \\[Source")))$X1
+
 # Summary of genes for Humans from RefSeq
 RefSeqSummaryGenes = loadGeneSummary(organism = 9606)[, c(4,6)] # col 4: EntrezID. col 6: Summary of RefSeq gene
 RefSeqSummaryGenesFiltered <- distinct(RefSeqSummaryGenes, Gene_ID, .keep_all = TRUE) # remove duplpicates ids 
 names(RefSeqSummaryGenesFiltered)[names(RefSeqSummaryGenesFiltered) == "Gene_ID"] <- "entrezgene_id"
 names(RefSeqSummaryGenesFiltered)[names(RefSeqSummaryGenesFiltered) == "Gene_summary"] <- "refseq_summary"
 
 # dataset of genes from CiVIC 
-civicGenes = read.csv(civicUrl, sep = "\t")[, c(4,5)]
-civicGenes <- subset(civicGenes, !(description == "")) # remove empty descriptions
-civicGenesFiltered <- distinct(civicGenes, entrez_id, .keep_all = TRUE) # remove duplpicates  
-names(civicGenesFiltered)[names(civicGenesFiltered) == "description"] <- "civic_description"
-names(civicGenesFiltered)[names(civicGenesFiltered) == "entrez_id"] <- "entrezgene_id"
-
-
+civicGenes_raw <- read.csv(civicUrl, sep = "\t")
+civicGenesFiltered <- civicGenes_raw %>%
+  dplyr::select(entrez_id, description) %>%
+    filter(!is.na(description) & description != "") %>%
+      distinct(entrez_id, .keep_all = TRUE) %>%
+      rename(
+        civic_description = description,
+        entrezgene_id = entrez_id
+      )
 
 res_grch37 = merge(res_grch37, civicGenesFiltered, 
                              by = "entrezgene_id", all.x = TRUE)
-
 all_grch37_gene_data = merge(res_grch37, RefSeqSummaryGenesFiltered, 
                              by = "entrezgene_id", all.x = TRUE)
-
 res_grch38 = merge(res_grch38, civicGenesFiltered, 
                    by = "entrezgene_id", all.x = TRUE)
-
 all_grch38_gene_data = merge(res_grch38, RefSeqSummaryGenesFiltered, 
                              by = "entrezgene_id", all.x = TRUE)
 

databases/hgnc/hgnc2mongodb.sh

@@ -7,7 +7,7 @@ user=
 password=
 db=bio_api
 ############# Database URL ############
-hgnc_url="ftp://ftp.ebi.ac.uk/pub/databases/genenames/hgnc/tsv/hgnc_complete_set.txt"
+hgnc_url="https://storage.googleapis.com/public-download-files/hgnc/tsv/tsv/hgnc_complete_set.txt"
 
 date
 echo "INFO	Downloading HGNC database..."