Matt's
Research Information
Research Group: University
of Maryland Biologically-Inspired Computing
Advisor:
James A. Reggia
Current Research:
The topic of my current research
is Rule Extraction. I am applying a competitive distribution of errors
learning method to standard backpropagation. This allows common
rule extraction methods such as "M of N" to be used with the trained network
to produce an understandable and valid rule set. More to follow.
Conference Publications:
Previous Research:
Word Segmentation and Recurrent Neural Nets
- Segmentation is the process of dividing a printed character string
into graphemes, each of which is associated with one (or rarely more) output
phonemes. The purpose of this study was to investigate what internal representation
of the segmentation process and character-to-phoneme correspondences would
be learned by a recurrent neural network as it was trained to produce the
correct temporal sequence of phonemes for printed words held fixed on its
input nodes. The resilient recurrent backpropagation network learned very
effectively to generate the correct pronunciation for 150 words. Some interesting
rules of pronunciation discovered by the network were extracted despite the
network's distributed representation.
- The acquisition of literacy depends on learning to associate strings
of printed characters with their pronunciations. In English this task is
complicated by the facts that (1) many individual characters have more than
one possible pronunciation, and (2) characters may be combined before being
associated with a single sound (e.g., PH /f/). The second of these factors
suggests that learning to read requires segmentation of strings into pronounceable
units. Since characters and phonemes are not in a one-to-one correspondence
in many words, it is not obvious that pronunciation information constrains
character-string segmentation. Few characters are always associated with
a single sound, and many characters are pronounced in several ways. Thus,
there is no unambiguous map from the set of words of the English language
to the set of phoneme sequences. The purpose of this study was to investigate
the internal representation of the segmentation process and graphemeto- phoneme
correspondences that would be learned/discovered by a recurrent neural network
as it was trained to produce the correct temporal sequence of phonemes to
pronounce printed words. This internal representation gives clues as to how
the network distinguishes particular word features and creates rules to segment
and pronounce words correctly. The internal representation can also give
clues as to whether or not the network simply memorizes whole input patterns
to associate with temporal phoneme sequences. These opposing methods form
the basis for the dual-route print-to-sound model of word pronunciation.
A second goal of this study was to critically evaluate the effectiveness
of some specific formulations of recurrent backpropagation (RBP) for this
task. To our knowledge, this is the first use of a recurrent neural network
of this kind to generate correct pronunciations of written words. Unlike
some past systems, such as NetTalk, there is no sliding window used to designate
which of the current characters to pronounce, i.e., the network must learn
to segment the characters as well as to pronounce them. In other words, the
network must learn the difficult task of generating the correct sequence
of output phonemes from a fixed representation of the input word, a very
challenging task. We evaluated how basic backpropagation, backpropagation
with momentum, backpropagation with an adaptive learning rate, and resilient
backpropagation all performed on this problem.
Competitive Error Distribution
Updated: 03/11/03 Matthew J. Radio