From the course: Natural Language Processing in Python

Unlock this course with a free trial

Join today to access over 26,400 courses taught by industry experts.

Solution: TF-IDF vectorizer

Solution: TF-IDF vectorizer - Python Tutorial

From the course: Natural Language Processing in Python

Start my 1-month free trial Buy for my team

Solution: TF-IDF vectorizer

“

For the last assignment for this section, we're going to be doing some steps that are pretty similar to the last assignment but we'll be using tf.idf instead of counts. So the first step is to vectorize the text using tf.idf vectorizer. To do this, I'm going to scroll back up here to the last assignment and copy all this code. All right. Now from here, I'm going to update all these counts with tf.idf, and then I'm also going to change all these C's to T's and document term matrices to tfidf. All right let's run that and you can see I get this document term matrix. It's 100 by almost 1300 just like before with our previous count vectorizer. The only difference is that these values here instead of being counts they're now tfidf scores. All right now with this let's modify the parameters to remove stop words and then set these two parameters as well. So I'm going to copy this down here and I'm going to add two to all the variable names. Let me run that and the first thing I'm going to do…

Contents