Experiment on NER task using Huggingface state-of-the-art Natural Language Models
- Python ≥ 3.6
Create and activate a virtual environment (conda)
conda create --name py36_transformers-ner python=3.6
source activate py36_transformers-ner
If pip
is configured in your conda environment,
install dependencies from within the project root directory
pip install -r requirements.txt
The current pipeline is generating a Stanford NER compatible format.
We can start our experiment from this file. Small modifications should be applied
to the file so it can be processed by BERT NER. In particular the file do not use
B-LABEL
and I-LABEL
for teh first occurrence and following one of a label.
After putting the Stanford NER format file (e.g. StanfordNov19.txt
) in data
folder, execute the following command:
python ./preprocess/generate_from_stanford.py --input_data ./data/StanfordNov19.txt --output_dir ./data/
The script ouputs two files train.txt
and test.txt
that will be the input of the NER pipeline.
To execute the NER pipeline, run the following scripts:
python ./run_ner.py --data_dir ./data --model_type bert --model_name_or_path bert-base-cased --output_dir ./output --labels ./data/labels.txt --do_train --do_predict --save_steps 200000 --max_seq_length 512 --overwrite_output_dir --overwrite_cache
The script will output the results and predictions in the output directory.
Download and unzip the model, vocab and its config. Rename config file to config.json as expected from the script.
curl -Ol https://s3-us-west-2.amazonaws.com/ai2-s2-research/scibert/pytorch_models/scibert_scivocab_cased.tar
tar -xvf scibert_scivocab_cased.tar -C scibert_scivocab_cased
cd scibert_scivocab_cased/
tar -zxvf weights.tar.gz
mv bert_config.json config.json
rm weights.tar.gz
Download and unzip the model, vocab and its config. Rename config file to config.json as expected from the script.
Note that SpanBERT does not come with its own vocab.txt
file. Instead it reuses the same as BERT-large-cased model
curl -Ol https://dl.fbaipublicfiles.com/fairseq/models/spanbert_hf_base.tar.gz
mkdir spanbert_hf_base
tar -zxvf spanbert_hf_base.tar.gz -C spanbert_hf_base
cd spanbert_hf_base
curl -Ol https://raw.githubusercontent.com/pyvandenbussche/transformers-ner/master/data/bert_large_cased_vocab.txt
mv bert_large_cased_vocab.txt vocab.txt