Annual Lexicon Release

This page describes annual release procedures for the SPECIALIST LEXICON. This Lexicon is included in the Metathesaurus DVD release.

Location

• Development: ${LEXICON_DIR}/data/${YEAR}
• Backup: ${BACKUP_DIR}/Releases/UMLS/${YEAR}_pre_release/LEX
• Web: ${WWW_DIR}/Apache/apache/docs/LexSysGroup/Projects/lexicon/${YEAR}/release
• Internal Deploy: /net/lexdev/export/home/lu/Backup/Releases/UMLS/${YEAR}_pre_release

• Internal release: ${BACKUP_DIR}/Releases/UMLS/${YEAR}_pre_release (simplfied after 2023+)
• Official release: ${BACKUP_DIR}/Releases/UMLS/${YEAR}_AA_release

Preparation steps

1. Freeze LEXICON and inflVars from LexBuild

2. Prepare release: ${YEAR}_AA_release/LEX in the following steps
   • LEX:
     • Annual Lexicon: Lexicon LR-Tables, MISC/*, NUMBERS/*
     • Verb Complmentatin Types: VERB_COMPLEMENTS/*
     • Database: LEX_DB/AM_DB,DM_DB,SM_DB, MISC/negCueWords
     • Lvg: DOCS/lvg, LEX_PGMS/lvg${YEAR}.tgz
   • LEX_Extra:
     • Annual Lexicon: ASCII
     • Annual Lexicon: XML
3. LEXICON Repository (LEX):

   File Name	Source	Owner
   DOCS
   DOCS/lvg/index.html	previous year	Chris
   DOCS/lvg/docs	${LVG_DIR}/lvg${YEAR}/docs	Chris
   DOCS/techrpt.pdf	previous year (if no change)	Amanda
   LEX_DB
   LEX_DB/AM.DB	${ANTONYM_DIR}/data/0.Antonym/${YEAR}/output/antonyms.data.${YEAR}.release (after 2022+)	Chris
   LEX_DB/DM.DB	${DERIVATION_DIR}/5.allD/data/${YEAR}/data/derivation.data.${YEAR}.release (after 2012+)	Chris
   LEX_DB/NC.DB	previous year (if no change)	Chris
   LEX_DB/SM.DB	${SYNONYM_DIR}/data/${YEAR}/outData/Results/synonym.data.${YEAR}.release (after 2016+)	Chris
   LEX_PGMS
   LEX_PGMS/lvg${YEAR}.tgz	$WWW_DIR/Apache/apache/docs/LexSysGroup/Projects/lvg/${YEAR}/release/lvg${YEAR}.tgz	Chris
   ./
   LEXICON	${LEXICON_DIR}/data/${YEAR}/data/LEXICON.frozen	Chris
   LRABR	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRABR	Chris
   LRAGR	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRAGR	Chris
   LRCMP	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRCMP	Chris
   LRFIL	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRFIL	Chris
   LRFIL.crlf	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRFIL.crlf	Chris
   LRFLD	Previous year (if no change)	Chris
   LRMOD	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRMOD	Chris
   LRNOM	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRNOM	Chris
   LRPRN	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRPRN	Chris
   LRPRP	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRPRP	Chris
   LRSPL	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRSPL	Chris
   LRTRM	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRTRM	Chris
   LRTYP	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRTYP	Chris
   LRWD	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRWD	Chris
   MISC
   MISC/inflection.table	${LEXICON_DIR}/data/${YEAR}/tables.frozen/inflection.table	Chris
   MISC/inflVars.data	${LEXICON_DIR}/data/${YEAR}/tables.frozen/inflVars.data	Chris
   MISC/LRFIL	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRFIL	Chris
   MISC/LRFIL.crlf	${LEXICON_DIR}/data/${YEAR}/tables.frozen/LRFIL.crlf	Chris
   MISC/negCueWords	${ANTONYM_DIR}/data/0.Antonym/${YEAR}/output/antonyms.data.${YEAR}.release	Chris
   MISC/prevariants	${LEXICON_DIR}/data/${YEAR}/tables.frozen/prevariants	Chris
   NUMBERS
   NUMBERS/NRNUM	previous year (if no change)	Chris
   NUMBERS/NRVAR	previous year (if no change)	Chris
   NUMBERS/number.entries	previous year (if no change)	Chris
   NUMBERS/number.grammar.txt	previous year (if no change)	Chris
   VERB_COMPLEMENTS
   VERB_COMPLEMENTS/lightVerbs.data	${MULTIWORD_DIR}/data/${YEAR}/outData/14.VerbComplements/lightVerbs.data.${YEAR}.release	Chris
   VERB_COMPLEMENTS/lightVerbs.inflVars	${MULTIWORD_DIR}/data/${YEAR}/outData/14.VerbComplements/lightVerbs.inflVars.${YEAR}.release	Chris
   VERB_COMPLEMENTS/verbParticles.data	${MULTIWORD_DIR}/data/${YEAR}/outData/14.VerbComplements/verbParticles.data.${YEAR}.release	Chris
   VERB_COMPLEMENTS/verbParticles.inflVars	${MULTIWORD_DIR}/data/${YEAR}/outData/14.VerbComplements/verbParticles.inflVars.${YEAR}.release	Chris

4. Prepare ${YEAR}_AA_release/LEX_Extra
   • ASCII
   • XML

5. Prepare ${YEAR}_AA_release/LEX.tgz
   • Items that need to wait for the Lexical Tools
     • ./DOCS/lvg/docs
     • ./LEX_PGMS/lvg${YEAR}.tgz

   • shell> cd ${BACKUP_DIR}/Releases/UMLS/${YEAR}_pre_release
   • shell> tar -czvf LEX.tgz LEX
   • send to OCCS for the internal release (after 2008)

6. Programs for preparing ${YEAR}_AA_release/LEX and LEX_Extra
   • cd ${LEXICON_DIR}/bin

   • 6.ReleaseLexicon ${YEAR}

   • 11.ReleaseLexiconExtra ${YEAR}

   • cd ${BACKUP_DIR}/Releases/UMLS/${YEAR}_AA_release
   • create LEX.tgz (tar -czvf LEX.tgz LEX)

7. Copy to ${WWW_LEXICON}
   • cd ${BACKUP_DIR}/Releases/UMLS/${YEAR}_AA_release/LEX

   • cp -rp * ${WWW_LEXICON}/htdocs/release/LEX_DOC

   • cd ${BACKUP_DIR}/Releases/UMLS/${YEAR}_AA_release/LEX_Extra

   • cp -rp * ${WWW_LEXICON}/htdocs/release/LEX_DOC

   • cd ${BACKUP_DIR}/Releases/UMLS/${YEAR}_AA_release

   • cp -rp LEX.tgz ${WWW_LEXICON}/htdocs/release/.

8. zip LEX_DOC
   => no need to do this, it is done automatically in the uploadFiles script.
   • cd ${BACKUP_DIR}/Releases/UMLS/${YEAR}_AA_release

   • tar -czvf LEX_DOC.tgz LEX_DOC

9. Upload to LHC-NEXUS: lhc-lexicon-raw/www/lexicon/${YEAR}
   • cd ${WWW_LEXICON}
   • make uploadRelease
     • upload ./release/LEX.tgz
     • upload ./release/LEX_DOC.tgz
   • make uploadAll (not used after 2024+)
     • upload ./release/LEX.tgz
     • upload ./release/LEX_DOC.tgz
     • upload ./docs/baseOrder.tgz
     • upload ./docs/java8_data.tgz
     • upload ./docs/mwConsumerData.tgz
     • upload ./docs/mwData.tgz
   • Test and validate uploaded files

     • downloadFiles ${YEAR} TRUE

       => downloaded above 6 tgz files to ${HOME}/ci-cd-data/lexicon

     • unzipFiles ${YEAR} TRUE

       => upzip above 5 tgz files (exclude LEX.tgz) to ${HOME}/ci-cd-data/lexicon
       => manually check contents

   • Upload to LHC-download:

     Send an email to Ci-Cd team (NLM LHC_AWS_DevOps) for following upload

     • From: https://lhc-nexus.nlm.nih.gov/#browse/browse:lhc-lexicon-raw > www > lvg > ${YEAR} > lvg${YEAR}.tgz
       To: https://data.lhncbc.nlm.nih.gov/public/lsg/lvg/${YEAR}/release/lvg${YEAR}.tgz
     • From: https://lhc-nexus.nlm.nih.gov/#browse/browse:lhc-lexicon-raw > www > lvg > ${YEAR} > lvg${YEAR}.tgz
       To: https://data.lhncbc.nlm.nih.gov/public/lsg/lvg/${YEAR}/release/lvg${YEAR}.tgz

     • From: https://lhc-nexus.nlm.nih.gov/#browse/browse:lhc-lexicon-raw > www > lexicon > ${YEAR} > release > LEX.tgz
       To: https://data.lhncbc.nlm.nih.gov/public/lsg/lexicon/${YEAR}/release/LEX.tgz
     • From: https://lhc-nexus.nlm.nih.gov/#browse/browse:lhc-lexicon-raw > www > lexicon > ${YEAR} > release > LEX_DOC.tgz
       To: https://data.lhncbc.nlm.nih.gov/public/lsg/lexicon/${YEAR}/release/LEX_DOC.tgz

       and then unzip to https://data.lhncbc.nlm.nih.gov/public/lsg/lexicon/${YEAR}/release/LEX_DOC