nutch-default.xml配置参数解释(部分)

<property>

<name>http.max.delays</name>

<value>100</value>

<description>Thenumber of times a thread will delay when trying to

fetch a page. Each time it finds that a host is busy,it will wait

fetcher.server.delay. After http.max.delays attepts,it will give

up on the page fornow.</description>

</property>

<property>

<name>http.content.limit</name>

<value>65536</value>

<description>Thelength limit for downloaded content,in bytes.

If this value isnonnegative (>=0),content longer than it will be truncated;

otherwise,notruncation at all.

</description>

</property>

<property>

<name>db.default.fetch.interval</name>

<value>30</value>

<description>Thedefault number of days between re-fetches of a page.

</description>

</property>

<property>

<name>fetcher.threads.fetch</name>

<value>10</value>

<description>Thenumber of FetcherThreads the fetcher should use.

This is alsodetermines the maximum number of requests that are

made at once(each FetcherThread handles one connection).</description>

</property>

<property>

<name>fetcher.threads.per.host</name>

<value>1</value>

<description>Thisnumber is the maximum number of threads that

should beallowed to access a host at one time.</description>

</property>

<property>

<name>fetcher.verbose</name>

<value>false</value>

<description>Iftrue,fetcher will log more verbosely.</description>

</property>

<property>
<name>parser.threads.parse</name>
<value>10</value>
<description>Numberof ParserThreads ParseSegment shoulduse.</description>
</property>

<property>
<name>fs.default.name</name>
<value>local</value>
<description>Thename of the default file system. Either the
literal string "local" or a host:port forNDFS.</description>
</property>

<property>
<name>ndfs.name.dir</name>
<value>/tmp/nutch/ndfs/name</value>
<description>Determines where on the local filesystem theNDFS name node
should store thename table.</description>
</property>

<property>
<name>ndfs.data.dir</name>
<value>/tmp/nutch/ndfs/data</value>
<description>Determines where on the local filesystem anNDFS data node
should store itsblocks.</description>
</property>

<property>

<name>indexer.max.tokens</name>

<value>10000</value>

<description>

The maximum numberof tokens that will be indexed for a single field

in a document.This limits the amount of memory required for

indexing,so thatcollections with very large files will not crash

the indexingprocess by running out of memory.

Note that thiseffectively truncates large documents,excluding

from the indextokens that occur further in the document. If you

know your sourcedocuments are large,be sure to set this value

high enough toaccomodate the expected size. If you set it to

Integer.MAX_VALUE,then the only limit is your memory,but you

should anticipatean OutOfMemoryError.

</description>

</property>

<property>

<name>indexer.mergeFactor</name>

<value>50</value>

<description>Thefactor that determines the frequency of Lucene segment

merges. This mustnot be less than 2,higher values increase indexing

speed but lead toincreased RAM usage,and increase the number of

open file handles(which may lead to "Too many open files" errors).

NOTE: the"segments" here have nothing to do with Nutch segments,they

are a low-leveldata unit used by Lucene.

</description>

</property>

<property>

<name>indexer.minMergeDocs</name>

<value>50</value>

<description>Thisnumber determines the minimum number of Lucene

Documents bufferedin memory between Lucene segment merges. Larger

values increaseindexing speed and increase RAM usage.

</description>

</property>

<property>

<name>indexer.maxMergeDocs</name>

<value>2147483647</value>

<description>Thisnumber determines the maximum number of Lucene

Documents to bemerged into a new Lucene segment. Larger values

increase batchindexing speed and reduce the number of Lucene segments,

which reduces thenumber of open file handles; however,this also

decreasesincremental indexing performance.

</description>

</property>

<property>

<name>searcher.summary.context</name>

<value>5</value>

<description>

The number ofcontext terms to display preceding and following

matching terms ina hit summary.

</description>

</property>

<property>

<name>searcher.summary.length</name>

<value>20</value>

<description>

The total numberof terms to display in a hit summary.

</description>

</property>

<property>

<name>plugin.folders</name>

<value>plugins</value>

<description>Directories where nutch plugins are located. Each

element may be arelative or absolute path. If absolute,it is used

as is. Ifrelative,it is searched for on the classpath.</description>

</property>

<property>

<name>plugin.includes</name>

<value>protocol-http|urlfilter-regex|parse-(text|html|js)|index-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>

<description>Regular expression naming plugin directorynames to

include. Anyplugin not matching this expression is excluded.

In any case youneed at least include the nutch-extensionpoints plugin. By

default Nutchincludes crawling just HTML and plain text via HTTP,

and basic indexingand search plugins. In order to use HTTPS please enable

protocol-httpclient,but be aware of possible intermittentproblems with the

underlyingcommons-httpclient library.

</description>

</property>

<property>

<name>parser.character.encoding.default</name>

<value>windows-1252</value>

<description>Thecharacter encoding to fall back to when no other information

isavailable</description>

</property>

<property>

<name>parser.html.impl</name>

<value>neko</value>

<description>HTMLParser implementation. Currently the following keywords

are recognized:"neko" uses NekoHTML,"tagsoup" uses TagSoup.

</description>

</property>

<property>

<name>extension.clustering.hits-to-cluster</name>

<value>100</value>

<description>Number of snippets retrieved for the clusteringextension

if clusteringextension is available and user requested results

to beclustered.</description>

</property>

<property>

<name>extension.ontology.extension-name</name>

<value></value>

<description>Usethe specified online ontology extension. If empty,

the firstavailable extension will be used. The "name" here refersto an 'id'

attribute of the'implementation' element in the plugin descriptor XML

file.</description>

</property>

<property>

<name>query.url.boost</name>

<value>4.0</value>

<description>Used as a boost for url field in Lucene query.

</description>

</property>

<property>

<name>query.anchor.boost</name>

<value>2.0</value>

<description>Used as a boost for anchor field in Lucene query.

</description>

</property>

<property>

<name>query.title.boost</name>

<value>1.5</value>

<description>Used as a boost for title field in Lucene query.

</description>

</property>

<property>

<name>query.host.boost</name>

<value>2.0</value>

<description>Used as a boost for host field in Lucene query.

</description>

</property>

<property>

<name>query.phrase.boost</name>

<value>1.0</value>

<description>Used as a boost for phrase in Lucene query.

Multiplied byboost for field phrase is matched in.

</description>

</property>

<property>

<name>lang.analyze.max.length</name>

<value>2048</value>

<description>The maximum bytes of data to uses to indentify

the language (0means full content analysis).

The larger is thisvalue,the better is the analysis,but the

slowest it is.

</description>

</property>

<property>

<name>searcher.dir</name>

<value>crawl</value>

<description>

Path to root ofcrawl. This directory is searched (in

order) for eitherthe file search-servers.txt,containing a list of

distributed searchservers,or the directory "index" containing

merged indexes,orthe directory "segments" containing segment

indexes.

</description>

</property>