NLP自然语言处理—N-gram language model.ppt
《NLP自然语言处理—N-gram language model.ppt》由会员分享,可在线阅读,更多相关《NLP自然语言处理—N-gram language model.ppt(23页珍藏版)》请在得力文库 - 分享文档赚钱的网站上搜索。
1、NLP自然语言处理N-gramlanguagemodelLanguage ModelsFormal grammars(e.g.regular,context free)give a hard“binary”model of the legal sentences in a language.For NLP,a probabilistic model of a language that gives a probability that a string is a member of a language is more useful.To specify a correct probabili
2、ty distribution,the probability of all sentences in a language must sum to 1.N-Gram Model FormulasWord sequencesChain rule of probabilityBigram approximationN-gram approximationEstimating ProbabilitiesN-gram conditional probabilities can be estimated from raw text based on the relative frequency of
3、word sequences.To have a consistent probabilistic model,append a unique start()and end()symbol to every sentence and treat these as additional words.Bigram:N-gram:Generative Model&MLEAn N-gram model can be seen as a probabilistic automata for generating sentences.Relative frequency estimates can be
4、proven to be maximum likelihood estimates(MLE)since they maximize the probability that the model M will generate the training corpus T.Initialize sentence with N1 symbolsUntil is generated do:Stochastically pick the next word based on the conditional probability of each word given the previous N 1 w
5、ords.Example from TextbookP(i want english food)=P(i|)P(want|i)P(english|want)P(food|english)P(|food)=.25 x.33 x.0011 x.5 x.68=.000031P(i want chinese food)=P(i|)P(want|i)P(chinese|want)P(food|chinese)P(|food)=.25 x.33 x.0065 x.52 x.68=.00019Train and Test CorporaA language model must be trained on
6、a large corpus of text to estimate good parameter values.Model can be evaluated based on its ability to predict a high probability for a disjoint(held-out)test corpus(testing on the training corpus would give an optimistically biased estimate).Ideally,the training(and test)corpus should be represent
7、ative of the actual application data.May need to adapt a general model to a small amount of new(in-domain)data by adding highly weighted small corpus to original training data.Unknown WordsHow to handle words in the test corpus that did not occur in the training data,i.e.out of vocabulary(OOV)words?
8、Train a model that includes an explicit symbol for an unknown word().Choose a vocabulary in advance and replace other words in the training corpus with.Replace the first occurrence of each word in the training data with.Evaluation of Language ModelsIdeally,evaluate use of model in end application(ex
9、trinsic,in vivo)RealisticExpensiveEvaluate on ability to model test corpus(intrinsic).Less realisticCheaperVerify at least once that intrinsic evaluation correlates with an extrinsic one.PerplexityMeasure of how well a model“fits”the test data.Uses the probability that the model assigns to the test
10、corpus.Normalizes for the number of words in the test corpus and takes the inverse.Measures the weighted average branching factor in predicting the next word(lower is better).Sample Perplexity EvaluationModels trained on 38 million words from the Wall Street Journal(WSJ)using a 19,979 word vocabular
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- NLP自然语言处理N-gram language model NLP 自然语言 处理 gram
限制150内