Comp9444 assignment
In this assignment, you will be implementing and training neural network models for three different tasks, and analysing the results. You are to submit two Python files kuzu.py and check.py, as well as a written report hw1.pdf (in pdf format).
Provided Files
Copy the archive hw1.zip into your own filespace and unzip it. This should create a directory hw1, subdirectories net and plot, and eight Python files kuzu.py, check.py, kuzu_main.py, check_main.py, seq_train.py, seq_models.py, seq_plot.py and anb2n.py.
Your task is to complete the skeleton files kuzu.py and check.py and submit them, along with your report.
Part 3: Hidden Unit Dynamics for Recurrent Networks
In Part 3 you will be investigating the hidden unit dynamics of recurrent networks trained on language prediction tasks, using the supplied code seq_train.py and seq_plot.py.
2. [1 mark] Train an SRN on the ab language prediction task by typing
python3 seq_train.py -lang anbn
The ab language is a concatenation of a random number of A's followed by an equal number of B's. The SRN has 2 inputs, 2 hidden units and 2 outputs.
Look at the predicted probabilities of A and B as the training progresses. The first B in each sequence and all A's after the first A are not deterministic and can only be predicted in a probabilistic sense. But, if the training is successful, all other symbols should be correctly predicted. in particular, the network should predict the last B in each sequence as well as the subsequent A. The error should be consistently in the range of 0.01 to 0.03. If the network appears to have learned the task successfully, you can stop it at any time using (cntri)-c. If it appears to be stuck in a local minimum, you can stop it and run the code again until it is successful.
After the training finishes, plot the hidden unit activations by typing
python3 seq_plot.py -lang anbn -epoch 100
Include the resulting figure in your report. The states are again printed according to the colormap "jet". Note, however, that these "states" are not unique but are instead used to count either the number of A's we have seen or the number of B's we are still expecting to see.
Briefly explain how the ab prediction task is achieved by the network, based on the generated figure. Specifically, you should describe how the hidden unit activations change as the string is processed, and how it is able to correctly predict the last B in each sequence as well as the following A.
3. [2 marks] Train an SRN on the abc language prediction task by typing
python3 seq_train.py -lang anbncn
The SRN now has 3 inputs, 3 hidden units and 3 outputs. Again, the "state" is used to count up the A's and count down the B's and C's. Continue training (and re-start, if necessary) for 200k epochs, or until the network is able to reliably predict all the C's as well as the subsequent A, and the error is consistently in the range of 0.01 to 0.03.
After the training finishes, plot the hidden unit activations at epoch 200000 by typing
python3 seq_plot.py -lang anbncn--epoch 200
(you can choose a different epoch number, if you wish). This should produce three images labeled anbncn srn3_??. ipo, and also display an interactive 3D figure. Try to rotate the figure in 3 dimensions to get one or more good view(s) of the points in hidden unit space, save them, and include them in your report. (If you can't get the 3D figure to work on your machine, you can. use the images anbncn_srn3_77.jpg)
Briefly explain how the abd prediction task is achieved by the network, based on the generated figure. Specifically, you should describe how the hidden unit activations change as the string is processed, and how it is able to correctly predict the last B in each sequence as well as all of the C's and the following A.
4. [3 marks] This question is intended to be more challenging. Train an LSTM network to predict the Embedded Reber Grammar, by typing
python3 seq_train.py --lang reber --embed True --model Istm --hid 4
You can adjust the number of hidden nodes if you wish. Once the training is successful, try to analyse the behavior. of the LSTM and explain how the task is accomplished (this might involve modifying the code so that it returns and prints out the context units as well as the hidden units).