inpro.incremental.deltifier
Class ASRWordDeltifier

java.lang.Object
  inpro.incremental.deltifier.ASRWordDeltifier

All Implemented Interfaces:

edu.cmu.sphinx.frontend.SignalListener, edu.cmu.sphinx.instrumentation.Resetable, edu.cmu.sphinx.util.props.Configurable, ASRResultKeeper

Direct Known Subclasses:

FixedLagDeltifier, NonIncrementalDeltifier, SmoothingDeltifier

public class ASRWordDeltifier
extends java.lang.Object
implements edu.cmu.sphinx.util.props.Configurable, edu.cmu.sphinx.instrumentation.Resetable, ASRResultKeeper, edu.cmu.sphinx.frontend.SignalListener

an ASRWordDeltifier (and descendants) can be used to store the ASR's (or one of the ASR's) current hypothesis you would usually call deltify() (either on the result or a specific token from the n-best-list) and afterwards retrieve the WordIUs and edits since the previous call via getIUs() and getEdits() Descendents of this class implement methods described in NAACL-HLT 2009

Author:

Timo Baumann

Field Summary

(package private) int	currentFrame the number of the frame to which recognition has proceeded, restarts from zero with every recognition restart
(package private) int	currentOffset offset to account for token numbers restarting from zero on recognition restarts (we want IU times to increase between recognitions, not restart from zero)
private static org.apache.log4j.Logger	logger
protected boolean	recoFinal flag to avoid smoothing or fixed lags on final result
(package private) long	startTime
protected java.util.List<EditMessage<WordIU>>	wordEdits
protected IUList<WordIU>	wordIUs

Constructor Summary

ASRWordDeltifier()

Method Summary

void	deltify(edu.cmu.sphinx.result.Result result) update the stored ASR-representation with the new Result from ASR the current representation can afterwards be queried through getWordIUs() the difference from the previous state can be queried through getWordEdits() the actual deltification algorithm is described in deltify(Token)
protected void	deltify(edu.cmu.sphinx.decoder.search.Token token) update the stored ASR-representation with information from the ASR result's given Token the actual deltification algorithm works as follows: we first extract all the word- and phoneme tokens from the token sequence, each word-token is followed by the corresponding phoneme tokens let's assert that every word-token is followed by the corresponding phoneme tokens and that every SIL phoneme is preceded by a silence-word token we then compare the previous and the current hypothesis: the beginning of the previous and this word hypothesis should be equal, we only need to update phoneme timings eventually, the two hypotheses will differ: we construct remove-edits for the remaining old words we then construct the new words and add edits for them
int	getCurrentFrame() the frame count in the IU world; no restarts between recognitions and including non-VAD-times
double	getCurrentTime() the time that has passed in the IU world; no restarts between recognitions
private double	getTimeFromNewIt(java.util.ListIterator<edu.cmu.sphinx.decoder.search.Token> newIt) tries to get segmentEndTime from a list iterator; the iterator will be at the same position afterwards as it was before
protected java.util.List<edu.cmu.sphinx.decoder.search.Token>	getTokens(edu.cmu.sphinx.decoder.search.Token finalToken) adds a bunch of assertions to the output of ResultUtil.getTokenList(Token,boolean,boolean)
java.util.List<EditMessage<WordIU>>	getWordEdits() return the change of the hypothesis as a list of edits (on the word level) since the last call
java.util.List<WordIU>	getWordIUs() return the current hypothesis as a list of IUs (on the word level)
private static boolean	isSilenceSegment(edu.cmu.sphinx.decoder.search.Token t)
private static boolean	isSilenceWord(edu.cmu.sphinx.decoder.search.Token t)
void	newProperties(edu.cmu.sphinx.util.props.PropertySheet ps) this base implementation does not accept any parameters
void	reset()
void	setCollectTime(long collectTime)
void	setOffset(int currentOffset) there are two alternatives to set the offset: - put an OffsetAdapter into the FrontEnd (after VAD) - use signalOccurred() below; for this you have to call FrontEnd.addSignalListener(deltifier) somewhere (CurrentHypothesis-setup) here, the offset is given in centiseconds (frames)
void	signalOccurred(edu.cmu.sphinx.frontend.Signal signal)
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

wordIUs

protected final IUList<WordIU> wordIUs

wordEdits

protected final java.util.List<EditMessage<WordIU>> wordEdits

logger

private static final org.apache.log4j.Logger logger

currentFrame

int currentFrame

the number of the frame to which recognition has proceeded, restarts from zero with every recognition restart

currentOffset

int currentOffset

offset to account for token numbers restarting from zero on recognition restarts (we want IU times to increase between recognitions, not restart from zero)

startTime

long startTime

recoFinal

protected boolean recoFinal

flag to avoid smoothing or fixed lags on final result

Constructor Detail

ASRWordDeltifier

public ASRWordDeltifier()

Method Detail

newProperties

public void newProperties(edu.cmu.sphinx.util.props.PropertySheet ps)
                   throws edu.cmu.sphinx.util.props.PropertyException

this base implementation does not accept any parameters

Specified by:

newProperties in interface edu.cmu.sphinx.util.props.Configurable

Throws:

edu.cmu.sphinx.util.props.PropertyException

getTokens

protected java.util.List<edu.cmu.sphinx.decoder.search.Token> getTokens(edu.cmu.sphinx.decoder.search.Token finalToken)

adds a bunch of assertions to the output of ResultUtil.getTokenList(Token,boolean,boolean)

Parameters:

finalToken - the token to start the backwards trace from

Returns:

a list of tokens, word tokens precede their corresponding segment tokens TODO?!! currently, no provisions are taken for SILs to be preceded by a word! -> it would be really annoying to insert such words in the token list, we hence handle silence-insertion at a later step in the process -> should we do the opposite and strip silence words, so that we don't have to support them if they ever occur in the token list?

isSilenceWord

private static boolean isSilenceWord(edu.cmu.sphinx.decoder.search.Token t)

isSilenceSegment

private static boolean isSilenceSegment(edu.cmu.sphinx.decoder.search.Token t)

deltify

protected void deltify(edu.cmu.sphinx.decoder.search.Token token)

update the stored ASR-representation with information from the ASR result's given Token

the actual deltification algorithm works as follows:

• we first extract all the word- and phoneme tokens from the token sequence,
• each word-token is followed by the corresponding phoneme tokens
• let's assert that every word-token is followed by the corresponding phoneme tokens and that every SIL phoneme is preceded by a silence-word token
• we then compare the previous and the current hypothesis:
• the beginning of the previous and this word hypothesis should be equal, we only need to update phoneme timings
• eventually, the two hypotheses will differ:
• we construct remove-edits for the remaining old words
• we then construct the new words and add edits for them

Parameters:

token - the token from which the deltification starts

getTimeFromNewIt

private double getTimeFromNewIt(java.util.ListIterator<edu.cmu.sphinx.decoder.search.Token> newIt)

tries to get segmentEndTime from a list iterator; the iterator will be at the same position afterwards as it was before

deltify

public void deltify(edu.cmu.sphinx.result.Result result)

update the stored ASR-representation with the new Result from ASR

• the current representation can afterwards be queried through getWordIUs()
• the difference from the previous state can be queried through getWordEdits()

the actual deltification algorithm is described in deltify(Token)

Parameters:

result - current ASR hypothesis

See Also:

deltify(Token)

getWordEdits

public java.util.List<EditMessage<WordIU>> getWordEdits()

return the change of the hypothesis as a list of edits (on the word level) since the last call

Specified by:

getWordEdits in interface ASRResultKeeper

getWordIUs

public java.util.List<WordIU> getWordIUs()

return the current hypothesis as a list of IUs (on the word level)

Specified by:

getWordIUs in interface ASRResultKeeper

getCurrentFrame

public int getCurrentFrame()

the frame count in the IU world; no restarts between recognitions and including non-VAD-times

Specified by:

getCurrentFrame in interface ASRResultKeeper

getCurrentTime

public double getCurrentTime()

the time that has passed in the IU world; no restarts between recognitions

Specified by:

getCurrentTime in interface ASRResultKeeper

reset

public void reset()

Specified by:

reset in interface edu.cmu.sphinx.instrumentation.Resetable

setOffset

public void setOffset(int currentOffset)

there are two alternatives to set the offset: - put an OffsetAdapter into the FrontEnd (after VAD) - use signalOccurred() below; for this you have to call FrontEnd.addSignalListener(deltifier) somewhere (CurrentHypothesis-setup) here, the offset is given in centiseconds (frames)

signalOccurred

public void signalOccurred(edu.cmu.sphinx.frontend.Signal signal)

Specified by:

signalOccurred in interface edu.cmu.sphinx.frontend.SignalListener

setCollectTime

public void setCollectTime(long collectTime)

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object