首页 文章

在java中标记字符串后删除停用词

提问于
浏览
0

我想在标记字符串后删除停用词 . 我有外部文件.txt并读取它然后将它与标记化的字符串进行比较 . 如果标记化的单词与停用词相等,则将其删除 .

这是令牌化的代码

try{
            while ((msg =readBufferData.readLine()) != null) {
                int numberOfTokens;

                System.out.println("Before: "+msg);
                StringTokenizer tokens = new StringTokenizer(msg);

                numberOfTokens = tokens.countTokens();
                System.out.println("Tokens: "+numberOfTokens);

                System.out.print("After : ");
                while (tokens.hasMoreTokens()) {
                    msg = tokens.nextToken();
                    String msgLower = msg.toLowerCase();
                    String punctuationremove = punctuationRemover(msgLower);  
          //          buffWriter.write(punctuationremove+" "); --> write into file .txt
                    System.out.print(punctuationremove+" ");
                    removingStopWord(punctuationremove, readStopWordsFile());
                    numberOfTotalTokens++;   
                }
           //     buffWriter.newLine(); make a new line after tokening new message
                System.out.println("\n");
                numberOfMessages++;
            }
        // write close    buffWriter.close();
            System.out.println("Total Tokens: "+numberOfTotalTokens);
            System.out.println("Total Messages: "+numberOfMessages);
        }
        catch (Exception e){
            System.out.println("Error Exception: "+e.getMessage());
        }

然后我有一个用于读取停用词文件的代码

public static Set<String> readStopWordsFile() throws FileNotFoundException, IOException{
    String fileStopWords = "\\stopWords.txt";

    Set<String> stopWords = new LinkedHashSet<String>();
    FileReader readFileStopWord = new FileReader(fileStopWords);
    BufferedReader stopWordsFile = new BufferedReader(readFileStopWord);

    String line;

    while((line = stopWordsFile.readLine())!=null){
        line = line.trim();
        stopWords.add(line);
    }
    stopWordsFile.close();
    return stopWords;
}

如何将令牌与停用词集进行比较,并删除与停用词相同的令牌 . 你能帮助我吗,谢谢你

1 回答

  • 0

    您可以先读取停用词,然后检查您的令牌是否为禁用词 .

    Set<String> stopWords = readStopWordsFile();
    
      // some file reading logic
      while (tokens.hasMoreTokens()) {
           msg = tokens.nextToken();
           if(stopWords.contains(msg)){
             continue; // skip over a stopword token
           }
      }
    

相关问题