Probing the predictions

Let's examine another review. To probe our predictions, we will make a few functions that will help us visualize our results better. Such a gauging function can also be used if you want to restrict your model's predictions to instances where it is most certain:

def gauge_predictions(n):
if (predictions[n]<=0.4) and (y_test[n]==0):
    print('Network correctly predicts that review %d is negative' %(n))
elif (predictions[n] <=0.7) and (y_test[n]==1);
elif (predictions[n]>-0.7) and (y_test[n]==0):
else:
    print('Network is not so sure. Review mp. %d has a probability score of %(n),   
           predictions[n])
def verify_predictions(n):
    return gauge_predictions(n), predictions[n], decode_review(n, split='test')

We will make two functions to help us better visualize our network's errors, while also limiting our predictive accuracy to upper and lower bounds. We will use the first function to arbitrarily define good predictions as instances where the network has a probability score above 0.7, and bad instances where the score is below 0.4, for a positive review. We simply reverse this scheme for the negative reviews (a good prediction score for a negative review is below 0.4 and a bad one is above 0.7). We also leave a middle ground between 40 and 70%, labeled as uncertain predictions so that we can better understand the reason behind its accurate and inaccurate guesses. The second function is designed for simplicity, taking an integer value that refers to the nth review you want to probe and verify as input, and returning an assessment of what the network thinks, the actual probability score, as well as what the review in question reads. Let's use these newly forged functions to probe yet another review:

verify-predictions(22)
network falsely predicts that review 22 is negative

As we can see, our network seems to be quite sure that review 22 from our test set is negative. It has generated a probability score of 0.169. You could also interpret this score as that our network believed with 16.9% confidence that this review is positive, and so it must be negative (since these are the only two classes we used to train our network). It turns out that our network got this one wrong. Reading the review, you will notice that the reviewer actually expresses their appreciation for what they deemed to be an undervalued movie. Note that the tone is quite ambiguous at the beginning, with words like silly and fall flat. However, contextual valence shifters later on in the sentence allow our biological neural networks to determine that the review actually expresses a positive sentiment. Sadly, our artificial network does not seem to have caught up with this particular pattern. Let's continue our exploratory analysis using yet another example:

verify_predictions(19999)
Network is not so sure. Review no. 19999 has a probability score of [0.5916141]

Here, we can see that our network is not too sure about the review, even though it has actually guessed the correct sentiment in the review, with a probability score of 0.59, which is closer to 1 ( positive) than 0 (negative). To us, this review clearly appears positive—even a bit promotionally pushy. It is intuitively unclear why our network is not certain of the sentiment. Later in this book, we will learn how to visualize word embeddings using our network layers. For now, let's continue our probing with one last example:

verify_predictions(4)
Network correctly predicts that review 4 is positive

This time, our network gets it right again. In fact, our network is 99.9% sure that this is a positive review. While reading the review, you'll notice that it has actually done a decent job, as the review contains words like boring, average, and suggestive language such as mouth shut, which could all easily be present in other negative reviews, potentially misleading our network. As we can see, we conclude this probing session by providing a short function that you can play around with by randomly checking your network's predictions for a given number of reviews. We then print out our network's predictions for two randomly chosen reviews from our test set:

from random import randint
def random_predict(n_reviews):
for i in range(n_reviews):
print(verify_predictions(randint(0, 24000)))
random_predict(2)
Network correctly predicts that review 20092 is positive