Category Archives: AI

使用LTP进行分词及词性标注

Posted on by — No Comments ↓

1、下载并解压
https://github.com/HIT-SCIR/ltp/releases
一定要注意，默认输出为utf-8，所以输出到屏幕上为乱码的。

2、语法
要看文档来这里：http://ltp.readthedocs.org/zh_CN/latest/install.html

ltp_test in LTP 3.3.1 - (C) 2012-2015 HIT-SCIR
The console application for Language Technology Platform.

usage: ./ltp_test <options>

options:
  --threads arg           The number of threads [default=1].
  --last-stage arg        The last stage of analysis. This option can be used
                          when the user onlywants to perform early stage
                          analysis, like only segment without postagging.value
                          includes:
                          - ws: Chinese word segmentation
                          - pos: Part of speech tagging
                          - ner: Named entity recognization
                          - dp: Dependency parsing
                          - srl: Semantic role labeling (equals to all)
                          - all: The whole pipeline [default]
  --input arg             The path to the input file.
  --segmentor-model arg   The path to the segment model
                          [default=ltp_data/cws.model].
  --segmentor-lexicon arg The path to the external lexicon in segmentor
                          [optional].
  --postagger-model arg   The path to the postag model
                          [default=ltp_data/pos.model].
  --postagger-lexicon arg The path to the external lexicon in postagger
                          [optional].
  --ner-model arg         The path to the NER model [default=ltp_data/ner.model
                          ].
  --parser-model arg      The path to the parser model
                          [default=ltp_data/parser.model].
  --srl-data arg          The path to the SRL model directory
                          [default=ltp_data/srl_data/].
  --debug-level arg       The debug level.
  -h [ --help ]           Show help information

3、测试例子
en.txt

Don't ever let somebody tell you you can't do something, not even me. 
You got a dream, you gotta protect it. 
People can’t do something themselves, they wanna tell you you can’t do it. 
If you want something, go get it. 
Period.

zh.txt

别让别人告诉你你成不了才，即使是我也不行。
如果你有梦想的话，就要去捍卫它。
那些一事无成的人想告诉你你也成不了大器。
如果你有理想的话，就要去努力实现。
就这样。

4、执行语句

ltp_test --input en.txt > enout.txt
ltp_test --input zh.txt > zhout.txt

5、测试结果
enpos.txt

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="Don&apos;t ever let somebody tell you you can&apos;t do something, not even me.">
                <word id="0" cont="Don&apos;t" pos="ws" ne="O" parent="1" relate="ATT" />
                <word id="1" cont="ever" pos="ws" ne="O" parent="7" relate="ATT" />
                <word id="2" cont="let" pos="ws" ne="O" parent="3" relate="ATT" />
                <word id="3" cont="somebody" pos="ws" ne="O" parent="4" relate="ATT" />
                <word id="4" cont="tell" pos="ws" ne="O" parent="5" relate="ATT" />
                <word id="5" cont="you" pos="ws" ne="O" parent="6" relate="ATT" />
                <word id="6" cont="you" pos="ws" ne="O" parent="7" relate="ATT" />
                <word id="7" cont="can&apos;t" pos="ws" ne="O" parent="8" relate="ATT" />
                <word id="8" cont="do" pos="ws" ne="O" parent="9" relate="ATT" />
                <word id="9" cont="something" pos="ws" ne="O" parent="-1" relate="HED" />
                <word id="10" cont="," pos="wp" ne="O" parent="9" relate="WP" />
                <word id="11" cont="not" pos="ws" ne="O" parent="13" relate="ATT" />
                <word id="12" cont="even" pos="ws" ne="O" parent="13" relate="ATT" />
                <word id="13" cont="me" pos="ws" ne="O" parent="9" relate="COO" />
                <word id="14" cont="." pos="wp" ne="O" parent="13" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="You got a dream, you gotta protect it.">
                <word id="0" cont="You" pos="ws" ne="O" parent="3" relate="ATT" />
                <word id="1" cont="got" pos="ws" ne="O" parent="2" relate="ATT" />
                <word id="2" cont="a" pos="ws" ne="O" parent="3" relate="ATT" />
                <word id="3" cont="dream" pos="ws" ne="O" parent="-1" relate="HED" />
                <word id="4" cont="," pos="wp" ne="O" parent="3" relate="WP" />
                <word id="5" cont="you" pos="ws" ne="O" parent="8" relate="ATT" />
                <word id="6" cont="gotta" pos="ws" ne="O" parent="8" relate="ATT" />
                <word id="7" cont="protect" pos="ws" ne="O" parent="8" relate="ATT" />
                <word id="8" cont="it" pos="ws" ne="O" parent="3" relate="COO" />
                <word id="9" cont="." pos="wp" ne="O" parent="3" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="People can’t do something themselves, they wanna tell you you can’t do it.">
                <word id="0" cont="People" pos="ws" ne="O" parent="1" relate="ATT" />
                <word id="1" cont="can" pos="ws" ne="O" parent="-1" relate="HED" />
                <word id="2" cont="’" pos="wp" ne="O" parent="6" relate="WP" />
                <word id="3" cont="t" pos="ws" ne="O" parent="4" relate="ATT" />
                <word id="4" cont="do" pos="ws" ne="O" parent="5" relate="ATT" />
                <word id="5" cont="something" pos="ws" ne="O" parent="6" relate="ATT" />
                <word id="6" cont="themselves" pos="ws" ne="O" parent="1" relate="COO" />
                <word id="7" cont="," pos="wp" ne="O" parent="6" relate="WP" />
                <word id="8" cont="they" pos="ws" ne="O" parent="9" relate="ATT" />
                <word id="9" cont="wanna" pos="ws" ne="O" parent="10" relate="ATT" />
                <word id="10" cont="tell" pos="ws" ne="O" parent="11" relate="ATT" />
                <word id="11" cont="you" pos="ws" ne="O" parent="12" relate="ATT" />
                <word id="12" cont="you" pos="ws" ne="O" parent="13" relate="ATT" />
                <word id="13" cont="can" pos="ws" ne="O" parent="6" relate="COO" />
                <word id="14" cont="’" pos="wp" ne="O" parent="17" relate="WP" />
                <word id="15" cont="t" pos="ws" ne="O" parent="17" relate="ATT" />
                <word id="16" cont="do" pos="ws" ne="O" parent="17" relate="ATT" />
                <word id="17" cont="it" pos="ws" ne="O" parent="13" relate="COO" />
                <word id="18" cont="." pos="wp" ne="O" parent="17" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="If you want something, go get it.">
                <word id="0" cont="If" pos="ws" ne="O" parent="3" relate="ATT" />
                <word id="1" cont="you" pos="ws" ne="O" parent="3" relate="ATT" />
                <word id="2" cont="want" pos="ws" ne="O" parent="3" relate="ATT" />
                <word id="3" cont="something" pos="ws" ne="O" parent="-1" relate="HED" />
                <word id="4" cont="," pos="wp" ne="O" parent="3" relate="WP" />
                <word id="5" cont="go" pos="ws" ne="O" parent="7" relate="ATT" />
                <word id="6" cont="get" pos="ws" ne="O" parent="7" relate="ATT" />
                <word id="7" cont="it" pos="ws" ne="O" parent="3" relate="COO" />
                <word id="8" cont="." pos="wp" ne="O" parent="3" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="Period.">
                <word id="0" cont="Period" pos="ws" ne="O" parent="-1" relate="HED" />
                <word id="1" cont="." pos="wp" ne="O" parent="0" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

zhpos.txt

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="别让别人告诉你你成不了才，即使是我也不行。">
                <word id="0" cont="别" pos="d" ne="O" parent="1" relate="ADV" />
                <word id="1" cont="让" pos="v" ne="O" parent="-1" relate="HED">
                    <arg id="0" type="ù" beg="0" end="0" />
                    <arg id="1" type="" beg="2" end="2" />
                    <arg id="2" type="" beg="3" end="9" />
                </word>
                <word id="2" cont="别人" pos="r" ne="O" parent="1" relate="DBL" />
                <word id="3" cont="告诉" pos="v" ne="O" parent="1" relate="VOB">
                    <arg id="0" type="R&#x07;" beg="4" end="4" />
                    <arg id="1" type="" beg="5" end="9" />
                </word>
                <word id="4" cont="你" pos="r" ne="O" parent="3" relate="IOB" />
                <word id="5" cont="你" pos="r" ne="O" parent="6" relate="SBV" />
                <word id="6" cont="成" pos="v" ne="O" parent="3" relate="VOB">
                    <arg id="0" type="ˆ&#x07;" beg="5" end="5" />
                    <arg id="1" type="" beg="9" end="9" />
                </word>
                <word id="7" cont="不" pos="d" ne="O" parent="8" relate="ADV" />
                <word id="8" cont="了" pos="v" ne="O" parent="6" relate="CMP" />
                <word id="9" cont="才" pos="n" ne="O" parent="6" relate="VOB" />
                <word id="10" cont="，" pos="wp" ne="O" parent="1" relate="WP" />
                <word id="11" cont="即使" pos="c" ne="O" parent="12" relate="ADV" />
                <word id="12" cont="是" pos="v" ne="O" parent="1" relate="COO">
                    <arg id="0" type="|&#x1C;S" beg="11" end="11" />
                </word>
                <word id="13" cont="我" pos="r" ne="O" parent="15" relate="SBV" />
                <word id="14" cont="也" pos="d" ne="O" parent="15" relate="ADV" />
                <word id="15" cont="不行" pos="a" ne="O" parent="12" relate="VOB">
                    <arg id="0" type="‘&#x07;" beg="13" end="13" />
                    <arg id="1" type="" beg="14" end="14" />
                </word>
                <word id="16" cont="。" pos="wp" ne="O" parent="1" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="如果你有梦想的话，就要去捍卫它。">
                <word id="0" cont="如果" pos="c" ne="O" parent="2" relate="ADV" />
                <word id="1" cont="你" pos="r" ne="O" parent="2" relate="SBV" />
                <word id="2" cont="有" pos="v" ne="O" parent="-1" relate="HED">
                    <arg id="0" type="&#x02;" beg="0" end="0" />
                    <arg id="1" type="" beg="1" end="1" />
                    <arg id="2" type="" beg="3" end="3" />
                    <arg id="3" type="" beg="5" end="5" />
                </word>
                <word id="3" cont="梦想" pos="n" ne="O" parent="2" relate="VOB" />
                <word id="4" cont="的" pos="u" ne="O" parent="2" relate="RAD" />
                <word id="5" cont="话" pos="n" ne="O" parent="2" relate="VOB" />
                <word id="6" cont="，" pos="wp" ne="O" parent="2" relate="WP" />
                <word id="7" cont="就要" pos="d" ne="O" parent="9" relate="ADV" />
                <word id="8" cont="去" pos="v" ne="O" parent="9" relate="ADV" />
                <word id="9" cont="捍卫" pos="v" ne="O" parent="2" relate="COO">
                    <arg id="0" type="£&#x07;V" beg="7" end="7" />
                    <arg id="1" type="" beg="10" end="10" />
                </word>
                <word id="10" cont="它" pos="r" ne="O" parent="9" relate="VOB" />
                <word id="11" cont="。" pos="wp" ne="O" parent="2" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="那些一事无成的人想告诉你你也成不了大器。">
                <word id="0" cont="那些" pos="r" ne="O" parent="3" relate="ATT" />
                <word id="1" cont="一事无成" pos="i" ne="O" parent="3" relate="ATT" />
                <word id="2" cont="的" pos="u" ne="O" parent="1" relate="RAD" />
                <word id="3" cont="人" pos="n" ne="O" parent="4" relate="SBV" />
                <word id="4" cont="想" pos="v" ne="O" parent="-1" relate="HED">
                    <arg id="0" type="‘&#x07;" beg="0" end="3" />
                    <arg id="1" type="" beg="5" end="12" />
                </word>
                <word id="5" cont="告诉" pos="v" ne="O" parent="4" relate="VOB">
                    <arg id="0" type="â&#x07;" beg="6" end="6" />
                    <arg id="1" type="" beg="7" end="12" />
                </word>
                <word id="6" cont="你" pos="r" ne="O" parent="5" relate="IOB" />
                <word id="7" cont="你" pos="r" ne="O" parent="9" relate="SBV" />
                <word id="8" cont="也" pos="d" ne="O" parent="9" relate="ADV" />
                <word id="9" cont="成" pos="v" ne="O" parent="5" relate="VOB">
                    <arg id="0" type="ù" beg="7" end="7" />
                    <arg id="1" type="" beg="8" end="8" />
                    <arg id="2" type="" beg="12" end="12" />
                </word>
                <word id="10" cont="不" pos="d" ne="O" parent="11" relate="ADV" />
                <word id="11" cont="了" pos="v" ne="O" parent="9" relate="CMP" />
                <word id="12" cont="大器" pos="n" ne="O" parent="9" relate="VOB" />
                <word id="13" cont="。" pos="wp" ne="O" parent="4" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="如果你有理想的话，就要去努力实现。">
                <word id="0" cont="如果" pos="c" ne="O" parent="2" relate="ADV" />
                <word id="1" cont="你" pos="r" ne="O" parent="2" relate="SBV" />
                <word id="2" cont="有" pos="v" ne="O" parent="-1" relate="HED">
                    <arg id="0" type="&#x13;&#x01;S" beg="0" end="0" />
                    <arg id="1" type="" beg="1" end="1" />
                    <arg id="2" type="" beg="3" end="3" />
                </word>
                <word id="3" cont="理想" pos="n" ne="O" parent="2" relate="VOB" />
                <word id="4" cont="的话" pos="u" ne="O" parent="2" relate="RAD" />
                <word id="5" cont="，" pos="wp" ne="O" parent="2" relate="WP" />
                <word id="6" cont="就要" pos="d" ne="O" parent="9" relate="ADV">
                    <arg id="0" type="" beg="4" end="4" />
                </word>
                <word id="7" cont="去" pos="v" ne="O" parent="9" relate="ADV">
                    <arg id="0" type="³&#x03;D" beg="4" end="4" />
                    <arg id="1" type="" beg="6" end="6" />
                </word>
                <word id="8" cont="努力" pos="a" ne="O" parent="9" relate="ADV" />
                <word id="9" cont="实现" pos="v" ne="O" parent="2" relate="COO">
                    <arg id="0" type=" &#x01;D" beg="4" end="4" />
                    <arg id="1" type="" beg="6" end="6" />
                    <arg id="2" type="" beg="8" end="8" />
                </word>
                <word id="10" cont="。" pos="wp" ne="O" parent="2" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

<?xml version="1.0" encoding="utf-8" ?>
<xml4nlp>
    <note sent="y" word="y" pos="y" ne="y" parser="y" wsd="n" srl="y" />
    <doc>
        <para id="0">
            <sent id="0" cont="就这样。">
                <word id="0" cont="就" pos="d" ne="O" parent="1" relate="ADV" />
                <word id="1" cont="这样" pos="r" ne="O" parent="-1" relate="HED" />
                <word id="2" cont="。" pos="wp" ne="O" parent="1" relate="WP" />
            </sent>
        </para>
    </doc>
</xml4nlp>

使用NiuParser进行分词及词性标注

Posted on by — 4 Comments ↓

1、下载并解压
http://www.niuparser.com/

2、语法

[USAGE]
         NiuParser-v1.3.0-mt-win.exe    <Action>        <OPTIONS>
[ACTION]
        --WS    :  Word Segmentation.
        --POS   :  Part-Of-Speech Tagging.
        --NER   :  Named Entity Recognition.
        --CHK   :  Chunking (shallow syntactic parsing).
        --CP    :  Constituent Parser.
        --DP    :  Dependency Parser.
        --SRL   :  Semantic Role Label.
[OPITION]
>>   Get Options of Word Segmentation
                 NiuParser-v1.3.0-mt-win.exe    --WS
>>   Get Options of POS Tagging
                 NiuParser-v1.3.0-mt-win.exe    --POS
>>   Get Options of Named Entity Recognition
                 NiuParser-v1.3.0-mt-win.exe    --NER
>>   Get Options of Base Phrase Chunking
                 NiuParser-v1.3.0-mt-win.exe    --CHK
>>   Get Options of Constituent Parser
                 NiuParser-v1.3.0-mt-win.exe    --CP
>>   Get Options of Dependency Parser
                 NiuParser-v1.3.0-mt-win.exe    --DP

3、测试例子
en.txt

Don't ever let somebody tell you you can't do something, not even me. 
You got a dream, you gotta protect it. 
People can’t do something themselves, they wanna tell you you can’t do it. 
If you want something, go get it. 
Period.

zh.txt

别让别人告诉你你成不了才，即使是我也不行。
如果你有梦想的话，就要去捍卫它。
那些一事无成的人想告诉你你也成不了大器。
如果你有理想的话，就要去努力实现。
就这样。

4、执行语句

NiuParser-v1.3.0-mt-win.exe --WS -c niuparser.config -in en.txt -out enws.txt
NiuParser-v1.3.0-mt-win.exe --POS -c niuparser.config -in enws.txt -out enpos.txt

NiuParser-v1.3.0-mt-win.exe --WS -c niuparser.config -in zh.txt -out zhws.txt
NiuParser-v1.3.0-mt-win.exe --POS -c niuparser.config -in zhws.txt -out zhpos.txt

5、测试结果
enpos.txt

Don't/NR ever/NN let/VV somebody/NR tell/NR you/NR you/NR can't/NN d/NN o/VV something/JJ ,/NN not/VV even/NR me./NN 
You/NR go/NN t/NN a/AD dream,/VV you/NR gotta/NR protect/NN it./NN 
People/NR can/NR ’/PU t/NN d/NN o/VV something/JJ t/NN hemselves/NN ,/PU the/DT y/NN wanna/NR tell/NR you/NR you/NR can/NR ’/PU t/NN do/VV it./NN 
If/NR you/NR want/VV something/JJ ,/NN go/NN get/VV it./NN

zhpos.txt

别/AD 让/VV 别人/NN 告诉/VV 你/PN 你/PN 成/VV 不/AD 了/VV 才/AD ，/PU 即使/CS 是/VC 我/PN 也/AD 不/AD 行/VV 。/PU 
如果/CS 你/PN 有/VE 梦想/NN 的话/SP ，/PU 就/AD 要/VV 去/VV 捍卫/VV 它/PN 。/PU 
那些/DT 一事无成/CD 的/DEG 人/NN 想/VV 告诉/VV 你/PN 你/PN 也/AD 成/VV 不/AD 了/VV 大器/NN 。/PU 
如果/CS 你/PN 有/VE 理想/NN 的话/SP ，/PU 就/AD 要/VV 去/VV 努力/AD 实现/VV 。/PU

使用THULANC进行分词及词性标注

Posted on by — No Comments ↓

1、首先到这里下载程序及模型
http://thulac.thunlp.org/

2、解压，我用的是java版本的程序

3、基本语法

java -jar THULAC_lite_java_run.jar [-t2s] [-seg_only] [-deli delimeter] [-user userword.txt] -input input_file -output output_file

其中：
-t2s                将句子从繁体转化为简体
-seg_only           只进行分词，不进行词性标注
-deli delimeter     设置词与词性间的分隔符，默认为下划线_
-filter             使用过滤器去除一些没有意义的词语，例如“可以”。
-user userword.txt  设置用户词典，用户词典中的词会被打上uw标签。词典中每一个词一行，UTF8编码(python版暂无)
-model_dir dir      设置模型文件所在文件夹，默认为models/
-input input_file   设置从文件读入，默认为命令行输入
-output output_file 设置输出到文件中，默认为命令行输出

4、测试例子
en.txt

Don't ever let somebody tell you you can't do something, not even me. 
You got a dream, you gotta protect it. 
People can’t do something themselves, they wanna tell you you can’t do it. 
If you want something, go get it. 
Period.

zh.txt

别让别人告诉你你成不了才，即使是我也不行。
如果你有梦想的话，就要去捍卫它。
那些一事无成的人想告诉你你也成不了大器。
如果你有理想的话，就要去努力实现。
就这样。

5、执行语句

java -jar THULAC_lite_java_run.jar -input en.txt -output enout.txt
java -jar THULAC_lite_java_run.jar -input zh.txt -output zhout.txt

6、测试结果
enout.txt

Don_n '_w t_g ever_nz let_x somebody_x tell_np you_np you_np can_np '_w t_g d_g o_v something_x ,_w not_np even_np me._np 
You_np got_np a_v dream_np ,_w you_np gotta_x protect_x it._x 
People_x can??_n t_g d_g o_v something_x themselves_x ,_w they_x wanna_n tell_np you_np you_np can??_n t_g do_v it._m 
If_v you_np want_x something_x ,_w go_v get_np it._m 
Period._x

zhout.txt

别_d 让_v 别人_r 告诉_v 你你_r 成_v 不_d 了_v 才_n ，_w 即使_c 是_v 我_r 也_d 不行_a 。_w 
如果_c 你_r 有_v 梦想_n 的_u 话_n ，_w 就要_d 去_v 捍卫_v 它_r 。_w 
那些_r 一事无成_id 的_u 人_n 想_v 告诉_v 你你_r 也_d 成_v 不_d 了_v 大器_n 。_w 
如果_c 你_r 有_v 理想_n 的_u 话_n ，_w 就要_d 去_v 努力_a 实现_v 。_w 
就_d 这样_r 。_w

使用NLTK进行分词及词性标注

Posted on by — No Comments ↓

1、首先是安装
1.1、安装Python 3.4
注意要用32位版本
http://www.python.org/downloads/

1.2、安装Numpy
注意两点，一是不一定所有版本都有windows安装包，二是要找支持python3.4的安装包
http://sourceforge.net/projects/numpy/files/NumPy/

1.3、安装NLTK
注意3.2版本有bug，不要用。
http://pypi.python.org/pypi/nltk

2、下载NLT Data
方法1：
在python中运行：

import nltk
nltk.download()

方法2：
到下面的地址，直接去找链接，然后自己下载解压
https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml

3、进行分词
3.1、设置环境变量

set PYTHON_HOME=C:\NeoLanguages\Python34_x86
set PATH=%PYTHON_HOME%;%PATH%
set NLTK_DATA=D:\NLP\NLTK\nltk_data
@python

3.2、py文件

#!usr/bin/python

import nltk

#测试句子
sentence = "Don’t ever let somebody tell you you can’t do something, not even me. \
You got a dream, you gotta protect it. People can’t do something themselves, \
they wanna tell you you can’t do it. If you want something, go get it. Period."

#分词
tokens = nltk.word_tokenize(sentence)

#词性标注
tagged = nltk.pos_tag(tokens)

#句法分析
entities = nltk.chunk.ne_chunk(tagged)

3.3、逐句运行

D:\MyProjects\NLP\NLTK>python
Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> sentence = "Don’t ever let somebody tell you you can’t do something, not e
ven me. \
... You got a dream, you gotta protect it. People can’t do something themselves
, \
... they wanna tell you you can’t do it. If you want something, go get it. Peri
od."
>>> tokens = nltk.word_tokenize(sentence)
>>> tagged = nltk.pos_tag(tokens)
>>> entities = nltk.chunk.ne_chunk(tagged)

>>> tokens
['Don’t', 'ever', 'let', 'somebody', 'tell', 'you', 'you', 'can’t', 'do', 'som
ething', ',', 'not', 'even', 'me', '.', 'You', 'got', 'a', 'dream', ',', 'you',
'got', 'ta', 'protect', 'it', '.', 'People', 'can’t', 'do', 'something', 'thems
elves', ',', 'they', 'wan', 'na', 'tell', 'you', 'you', 'can’t', 'do', 'it', '.
', 'If', 'you', 'want', 'something', ',', 'go', 'get', 'it', '.', 'Period', '.']

>>> tagged
[('Don’t', 'NNP'), ('ever', 'RB'), ('let', 'VB'), ('somebody', 'NN'), ('tell',
'VB'), ('you', 'PRP'), ('you', 'PRP'), ('can’t', 'VBP'), ('do', 'VB'), ('someth
ing', 'NN'), (',', ','), ('not', 'RB'), ('even', 'RB'), ('me', 'PRP'), ('.', '.'
), ('You', 'PRP'), ('got', 'VBD'), ('a', 'DT'), ('dream', 'NN'), (',', ','), ('y
ou', 'PRP'), ('got', 'VBD'), ('ta', 'JJ'), ('protect', 'NN'), ('it', 'PRP'), ('.
', '.'), ('People', 'NNS'), ('can’t', 'VBP'), ('do', 'VBP'), ('something', 'NN'
), ('themselves', 'PRP'), (',', ','), ('they', 'PRP'), ('wan', 'VBP'), ('na', 'T
O'), ('tell', 'VB'), ('you', 'PRP'), ('you', 'PRP'), ('can’t', 'VBP'), ('do', '
VB'), ('it', 'PRP'), ('.', '.'), ('If', 'IN'), ('you', 'PRP'), ('want', 'VBP'),
('something', 'NN'), (',', ','), ('go', 'VBP'), ('get', 'VB'), ('it', 'PRP'), ('
.', '.'), ('Period', 'NNP'), ('.', '.')]

>>> entities
Tree('S', [('Don’t', 'NNP'), ('ever', 'RB'), ('let', 'VB'), ('somebody', 'NN'),
 ('tell', 'VB'), ('you', 'PRP'), ('you', 'PRP'), ('can’t', 'VBP'), ('do', 'VB')
, ('something', 'NN'), (',', ','), ('not', 'RB'), ('even', 'RB'), ('me', 'PRP'),
 ('.', '.'), ('You', 'PRP'), ('got', 'VBD'), ('a', 'DT'), ('dream', 'NN'), (',',
 ','), ('you', 'PRP'), ('got', 'VBD'), ('ta', 'JJ'), ('protect', 'NN'), ('it', '
PRP'), ('.', '.'), ('People', 'NNS'), ('can’t', 'VBP'), ('do', 'VBP'), ('someth
ing', 'NN'), ('themselves', 'PRP'), (',', ','), ('they', 'PRP'), ('wan', 'VBP'),
 ('na', 'TO'), ('tell', 'VB'), ('you', 'PRP'), ('you', 'PRP'), ('can’t', 'VBP')
, ('do', 'VB'), ('it', 'PRP'), ('.', '.'), ('If', 'IN'), ('you', 'PRP'), ('want'
, 'VBP'), ('something', 'NN'), (',', ','), ('go', 'VBP'), ('get', 'VB'), ('it',
'PRP'), ('.', '.'), Tree('PERSON', [('Period', 'NNP')]), ('.', '.')])
>>>