IRS for Computer Character Sequences Filtration: a new software tool and algorithm to support the IRS at tokenization process
This work addresses a specific preprocessing issue in information retrieval systems, but it is incremental as it applies existing pattern matching techniques to filter common character sequences.
The paper tackles the problem of tokenization in information retrieval systems by developing a software tool and algorithm that filters out IP addresses, web URLs, dates, and email addresses from text using pattern matching and filtration methods, resulting in cleaner text for subsequent tokenization.
Tokenization is the task of chopping it up into pieces, called tokens, perhaps at the same time throwing away certain characters, such as punctuation. A token is an instance of token a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing. New software tool and algorithm to support the IRS at tokenization process are presented. Our proposed tool will filter out the three computer character Sequences: IP-Addresses, Web URLs, Date, and Email Addresses. Our tool will use the pattern matching algorithms and filtration methods. After this process, the IRS can start a new tokenization process on the new retrieved text which will be free of these sequences.