DEGREES of MSci, MEng, BEng, BSc, MA and MA (Social Sciences)

TEXT AS DATA (M)

COMPSCI 5096

Wednesday, 20 May, 09:15 BST

1. This question is about tokenisation and similarity.

(a) This part concerns processing text. Consider the input string:

[He didn’t like the U.S. movie “Snakes on a train, revenge of Viper-man!”, now playing in the U.K.]

(i)  Provide a tokenised form of the above string. Identify and discuss two elements of the above string that present ambiguities. Justify your tokenisation decision for each.  [3] (ii)  Compare and contrast ‘standard’ word-based tokenisation with the tokenisation method used by BERT. Illustrate key differences using the example provided.  Analyse and  discuss why they differ and their relative advantages and disadvantages. (Hint: Recall we used BERT’s tokeniser in Lab 1 and in the in-class embedding exercise.)           [4]

(b) Consider the two tokenized documents:

S1: [a, woman, is, under, a, mayan, curse]

S2: [a, woman, sees, a, mayan, shaman, to, lift, the, curse]

Create a Dictionary from the two documents above (S1 and S2) with appropriate ordering. Give your answer in the form of a table with ID and token. Discuss the following properties  of the dictionary and provide reasons for the decision:  1) what is included in the dictionary and 2) the order of the dictionary.                  [3]

(c) Critically evaluate the Bag-of-Words (BoW) model as a term weighting feature model for documents. Discuss its strengths and give three weaknesses of the model and propose a modification that addresses each. You should relate each to Sci-kit Learn vectorizers and their important parameters.                                                                                                [4]

(d) You are measuring the similarity between two molecular compounds for drug discovery  research. They have been processed to create a series of unique structural ‘fingerprints’ and a one-hot encoding of the compounds is created. A compound has tens of thousands  of fingerprints on average and all the compounds are approximately the same size. Also,  most of the compounds in the dataset share more than 90% of fingerprints in common. A  lab partner suggests using Jaccard overlap to measure the similarity between compounds. First, critically discuss why Jaccard is or is not appropriate for this task and the challenges  it presents. Second, propose and justify a change to both the representation and similarity measure to address them.                                                                  [6]

2. This question is about language modelling and classification.

(a) This task involves developing an order error corrector for a popular burger chain, ‘out-and-in burger’. Below is a table of five separate order interactions transcribed from a mobile app.

Table 1: Five interactions for a burger restaurant ordering system.

Sample text collections statistics for a bigram model are below:

• V = 22 unique words (including reserved tokens)

• N = 45 tokens, including padding

(i)  Use the text provided in Table 1 above to compute word unigram probabilities. In a list  or table format complete the probability table with Laplace smoothing that has K = 0.5. Show your workings.  Discuss the impact on the probability values of increasing or decreasing the value of K. Describe the effect of K when these probabilities are used in  a spelling (error) correction task.

[5]

(ii)  A larger collection of restaurant ordering data is collected. It has the following statistics: N = 73194, V = 1996 from a total of 8565 documents (utterances).

Compute the bigram probability of the following sequence:

[i  might  like  a  cheeseburger]

with Stupid Backoff smoothing with default values. Collection statistics for the required  terms are provided below. Show your workings, including each bigram’s probability. Describe how and why a smoothing method is used here.    [6]

(b) Compare and contrast the APIs for SKLearn Transformers (e.g.  Count or TF-IDF) and Classifiers/Predictors (e.g. NaiveBayes, LogisticRegression). Include descriptions of their key interface functions with descriptions of their behaviour.  Discuss how they are used together to solve machine learning tasks on text.                            [3]

(c) Below is a snippet of code to vectorize and classify text with Scikit-learn.  Assume that tokenize_normalize and evaluation_summary have been defined, as we did in the labs.  The input data has been pre-processed into a vector of unnormalized text documents (each a single string).

from  sklearn .feature_extraction .text  import  CountVectorizer

from  sklearn .linear_model  import  LogisticRegression

#  Data  processing

data  =   . . .  #  Loads  a  vector  of  raw  text  documents

train_index  =  int(len(data)  *  0 . 1)

train_data  =  data[:train_index,:]

validation_data  =  data[int(train_index *0 .2):,:]

test_data  =  data[train_index:,:]

#  Assume  corresponding  labels  for  each  data  subset

train_labels,  test_labels,  validation_labels  =  . . .

#  Vectorization

one_hot_vectorizer  =  CountVectorizer(tokenizer=tokenize_normalize,

binary=True,  max_features=20)   one_hot_vectorizer .fit(train_data)

train_features  =  one_hot_vectorizer .transform(train_features)

validation_features  =  one_hot_vectorizer .fit_transform(validation_data) test_features  =  one_hot_vectorizer .transform(test_data)

#  Classification

lr  =  LogisticRegression(solver=’saga’,  max_iter=500)

lr_model  =  lr .fit(train_features,  train_labels)

evaluation_summary("LR  Train  summary",

lr_model .predict(train_features),  validation_labels)

lr_model  =  lr .fit(validation_features,  validation_features)

evaluation_summary("LR  Validation  summary",

lr_model .predict(validation_features),  validation_labels) lr_model  =  lr .fit(test_features,  test_labels)

evaluation_summary("LR  Test  summary",

lr_model .predict(validation_features),  test_labels)

Copy and paste the code above and fix its mistakes. Although there may be more, discuss three important mistakes with their consequence, one from each section (data processing, vectorization, classification).                     [6]

3. This question is about word embedding models and Natural Language Processing.

(a) Compare and contrast static word embeddings with contextual embedding models. Discuss the trade-offs between them for downstream tasks.                           [4]

(b) Using your knowledge of the self-attention mechanism, answer the following question considering the following sentence:

S1: [The president of the European Union spoke]

Use the following weight matrices and layer parameters to compute the unnormalised attention weights between the query “president” and the keys “spoke” and “the”. What can you infer from these values?

[3]

(c) Explain how and why attention-based encoders can be “stacked” to form layers in Trans- former models.                  [2]

(d) In this question we explore what can be done when faced with a completely “alien” scenario. Klingon is a language originating from TV series Star Trek. Many classic works such as  Hamlet, Much Ado About Nothing, Tao Te Ching, and Gilgamesh have been translated  by hand to Klingon. It is studied and formalised by the Klingon Language Institute (KLI) and was designed to be dissimilar from English. Below are some sample Klingon-English  translations.

Figure 1: Sample sentences in Klingon

We will use knowledge of text processing and NLP to understand what is being said by Klingons and the actors portraying them in the Star Trek series.

(i)  Describe the process for pre-training BERT on Klingon.  Briefly describe what is required and any changes needed for the model.           [3]

(ii)  We want to identify when an actor makes a mistake (uses an incorrect word) when reciting a Klingon sentence from the script. Describe how to apply BERT to identify and to suggest fixes for likely mistakes.               [4]

(iii)  We want to build a Klingon intent classifier to distinguish between ‘romance’, ‘anger’, and ‘other’ utterances. Describe how you would use an existing pre-trained Klingon BERT model for this task. Describe the data required and important challenges.     [4]