Full-Text (Substring) Indexes in External Memory - Barsky, Marina

. .

Résumé :
Nowadays, textual databases are among the most rapidly growing collections of data. Some of these collections contain a new type of data that differs from classical numerical or textual data. These are long sequences of symbols, not divided into well-separated small tokens (words). The most prominent among such collections are databases of biological sequences, which are experiencing today an unprecedented growth rate. Starting in 2008, the 1000 Genomes Project has been launched with the ultimate goal of collecting sequences of additional 1,500 Human genomes, 500 each of European, African, and East Asian origin. This will produce an extensive catalog of Human genetic variations. The size of just the raw sequences in this catalog would be about 5 terabytes. Querying strings without well-separated tokens poses a different set of challenges, typically addressed by building full-text indexes, which provide effective structures to index all the substrings of the given strings. Since full-text indexes occupy more space than the raw data, it is often necessary to use disk space for their construction. However, until recently, the construction of full-text indexes in secondary storage was considered impractical due to excessive I/O costs. Despite this, algorithms developed in the last decade demonstrated that efficient external construction of full-text indexes is indeed possible. This book is about large-scale construction and usage of full-text indexes. We focus mainly on suffix trees, and show efficient algorithms that can convert suffix trees to other kinds of full-text indexes and vice versa. There are four parts in this book. They are a mix of string searching theory with the reality of external memory constraints. The first part introduces general concepts of full-text indexes and shows the relationships between them. The second part presents the first series of external-memory construction algorithms that can handle the construction of full-text indexes for moderately large strings in the order of few gigabytes. The third part presents algorithms that scale for very large strings. The final part examines queries that can be facilitated by disk-resident full-text indexes. Table of Contents: Structures for Indexing Substrings / External Construction of Suffix Trees / Scaling Up: When the Input Exceeds the Main Memory / Queries for Disk-based Indexes / Conclusions and Open Problems

Biographie:
Marina Barsky is a Post-Doctoral Fellow in the Department of Computer Science at the University of Illinois at Urbana[1]Champaign, US. She received her PhD in Computer Science from the University of Victoria, British Columbia, Canada in 2010. Her PhD research was dedicated to better construction of full-text indexes using disk. Currently she expands her expertise in database management to the field of data mining. Alex Thomo is an Associate Professor in the Department of Computer Science at the University of Victoria, British Columbia, Canada. He received his PhD in Computer Science from Concordia University of Montreal in 2003. Before joining UVic, he was a software engineer for Ericsson Inc, and Assistant Professor at Suffolk University in Boston. His main research is on theoretical and practical aspects of semistructured and graph databases, with a current focus on social and biological networks, automata-based techniques, and index structures for textual data. Ulrike Stege is an Associate Professor in the Department of Computer Science at the University of Victoria, British Columbia, Canada. She received her PhD in Computer Science from the ETH Z?rich - a Science and Technology University in Z?rich, Switzerland, in year 2000. Her main research interests are interdisciplinary, including the areas of Parameterized Complexity, Computational Biology, Cognitive Science and Human Problem Solving, and Computer Science Education....

Sommaire:
Structures for Indexing Substrings.- External Construction of Suffix Trees.- Scaling Up: When the Input Exceeds the Main Memory.- Queries for Disk-based Indexes.- Conclusions and Open Problems.