Text Classification and Distributional features techniques in Datamining and Warehousing
This work presents an incremental improvement for text categorization in data mining and warehousing by refining feature extraction techniques.
The paper addresses the problem of text classification errors caused by unimportant words in traditional TF-IDF methods by introducing distributional features that analyze word distribution across documents, and reports improved classification accuracy using k-nearest neighbor and k-means algorithms.
Text Categorization is traditionally done by using the term frequency and inverse document frequency.This type of method is not very good because, some words which are not so important may appear in the document .The term frequency of unimportant words may increase and document may be classified in the wrong category.For reducing the error of classifying of documents in wrong category. The Distributional features are introduced. In the Distribuional Features, the Distribution of the words in the whole document is analyzed. Whole Document is very closely analyzed for different measures like FirstAppearence, Last Appearance, Centriod, Count, etc.The measures are calculated and they are used in tf*idf equation and result is used in k- nearest neighbor and K-means algorithm for classifying the documents.