A cutoff refers to the approximate minimum amount of points needed to achieve a certain number of stars under the "review" section of the score page shown after completing a song. Therefore, a 5-star cutoff for a specific song, on a specific difficulty, refers to the minimum score you need to reach in order to receive a 5-star review.
For example: the 5-star cutoff for Welcome to the Jungle on Hard is 135,873. If you score at least that many points, you will receive a 5-star review.
Each cutoff value varies from song to song, as well as difficulty to difficulty, depending on things such as how long the song is, how many notes the song contains, etc.
For information on how cutoff values are calculated, see Base score.