首页 文章

Apache Solr DataImportHandler不会从数据库索引数据

提问于
浏览
0

要将数据从我的数据库导入Apache Solr,我使用的是DataImportHandler . DataImportHandler的集成是成功的,索引也是如此 . 但“处理的文档总数”为0,我在日志中找不到错误 .

我使用Apache Solr 3.5和Drupal,很多项目已经编入索引 . 因此,schema.xml已经有很多字段 . 使用DataImportHandler,我想添加更多内容,因此还需要添加更多内容 . 为了测试,我有一个测试数据库,其中包含一列“id”(主键)和另一列“test_name”(文本) .

在其他帖子中,我经常看到类似的问题,但没有什么可以帮助我,所以我希望,你可以 . 我不知道我的字段匹配是否有问题,或者是否是另一个问题 .

这是我的dataconfig.xml:

<dataConfig>
    <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" encoding="UTF-8"
                     url="jdbc:mysql://localhost/test_db" 
                        user="user" password="password" batchSize="-1"/>
   <document>
     <entity pk="id" query="SELECT `id` AS ID,`test_name` AS TEST_NAME FROM `test`">
        <field column="ID" name="id" />
        <field column="TEST_NAME" name="test_name" />
     </entity>
  </document>
</dataConfig>

我的schema.xml:

<?xml version="1.0" encoding="UTF-8" ?>

<schema name="drupal-3.0-beta9-solr3" version="1.3">

  <types>

    <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>

    <!-- boolean type: "true" or "false" -->
    <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true" omitNorms="true"/>
    <!--Binary data type. The data should be sent/retrieved in as Base64 encoded Strings -->
    <fieldtype name="binary" class="solr.BinaryField"/>


    <!-- numeric field types that can be sorted, but are not optimized for range queries -->
    <fieldType name="integer" class="solr.TrieIntField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>
    <fieldType name="float" class="solr.TrieFloatField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>
    <fieldType name="long" class="solr.TrieLongField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>
    <fieldType name="double" class="solr.TrieDoubleField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>

    <fieldType name="sint" class="solr.SortableIntField" sortMissingLast="true" omitNorms="true"/>
    <fieldType name="slong" class="solr.SortableLongField" sortMissingLast="true" omitNorms="true"/>
    <fieldType name="sfloat" class="solr.SortableFloatField" sortMissingLast="true" omitNorms="true"/>
    <fieldType name="sdouble" class="solr.SortableDoubleField" sortMissingLast="true" omitNorms="true"/>

    <fieldType name="tint" class="solr.TrieIntField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/>
    <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/>
    <fieldType name="tlong" class="solr.TrieLongField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/>
    <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/>


    <fieldType name="date" class="solr.DateField" sortMissingLast="true" omitNorms="true"/>

    <!-- A Trie based date field for faster date range queries and date faceting. -->
    <fieldType name="tdate" class="solr.TrieDateField" omitNorms="true" precisionStep="6" positionIncrementGap="0"/>


    <!-- A text field that only splits on whitespace for exact matching of words -->
    <fieldType name="text_ws" class="solr.TextField" omitNorms="true" positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>

    <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>

        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="stopwords.txt"
                enablePositionIncrements="true"
                />
        <filter class="solr.WordDelimiterFilterFactory"
                protected="protwords.txt"
                generateWordParts="1"
                generateNumberParts="1"
                catenateWords="1"
                catenateNumbers="1"
                catenateAll="0"
                splitOnCaseChange="1"
                preserveOriginal="1"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.DictionaryCompoundWordTokenFilterFactory" dictionary="german-common-nouns.txt" minWordSize="5" minSubwordSize="2" maxSubwordSize="15" onlyLongestMatch="true"/>

        <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="35" side="front"/>
        <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="stopwords.txt"
                enablePositionIncrements="true"
                />
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
    </fieldType>

    <!-- An unstemmed text field - good if one does not know the language of the field -->
    <fieldType name="text_und" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
        <filter class="solr.WordDelimiterFilterFactory"
                protected="protwords.txt"
                generateWordParts="1"
                generateNumberParts="1"
                catenateWords="1"
                catenateNumbers="1"
                catenateAll="0"
                splitOnCaseChange="0"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="stopwords.txt"
                enablePositionIncrements="true"
                />
        <filter class="solr.WordDelimiterFilterFactory"
                protected="protwords.txt"
                generateWordParts="1"
                generateNumberParts="1"
                catenateWords="0"
                catenateNumbers="0"
                catenateAll="0"
                splitOnCaseChange="0"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>

    <fieldType name="edge_n2_kw_text" class="solr.TextField" omitNorms="true" positionIncrementGap="100">
     <analyzer type="index">
       <tokenizer class="solr.KeywordTokenizerFactory"/>
       <filter class="solr.LowerCaseFilterFactory"/>
       <filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="25" />
     </analyzer>
     <analyzer type="query">
       <tokenizer class="solr.KeywordTokenizerFactory"/>
       <filter class="solr.LowerCaseFilterFactory"/>
     </analyzer>
    </fieldType>
   <!--  Setup simple analysis for spell checking -->

   <fieldType name="textSpell" class="solr.TextField" positionIncrementGap="100">
     <analyzer>
       <tokenizer class="solr.StandardTokenizerFactory" />
       <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
       <filter class="solr.LengthFilterFactory" min="4" max="20" />
       <filter class="solr.LowerCaseFilterFactory" />
       <filter class="solr.RemoveDuplicatesTokenFilterFactory" />
     </analyzer>
   </fieldType>

    <fieldType name="sortString" class="solr.TextField" sortMissingLast="true" omitNorms="true">
      <analyzer>
        <tokenizer class="solr.KeywordTokenizerFactory"/>

        <filter class="solr.LowerCaseFilterFactory" />
        <!-- The TrimFilter removes any leading or trailing whitespace -->
        <filter class="solr.TrimFilterFactory" />

      </analyzer>
    </fieldType>

    <!-- A random sort type -->
    <fieldType name="rand" class="solr.RandomSortField" indexed="true" />

    <fieldtype name="ignored" stored="false" indexed="false" class="solr.StrField" />

    <fieldType name="point" class="solr.PointType" dimension="2" subFieldType="tdouble"/>

    <fieldType name="location" class="solr.LatLonType" subFieldType="tdouble"/>

    <fieldtype name="geohash" class="solr.GeoHashField"/>

 </types>


 <fields>

   <field name="id" type="string" indexed="true" stored="true" required="true" />
   <!-- entity_id is the numeric object ID, e.g. Node ID, File ID -->
   <field name="entity_id"  type="long" indexed="true" stored="true" />
   <!-- entity_type is 'node', 'file', 'user', or some other Drupal object type -->
   <field name="entity_type" type="string" indexed="true" stored="true" required="true" />
   <!-- bundle is a node type, or as appropriate for other entity types -->
   <field name="bundle" type="string" indexed="true" stored="true"/>
   <field name="bundle_name" type="string" indexed="true" stored="true"/>

   <field name="site" type="string" indexed="true" stored="true"/>
   <field name="hash" type="string" indexed="true" stored="true"/>
   <field name="url" type="string" indexed="true" stored="true"/>
   <!-- label is the default field for a human-readable string for this entity (e.g. the title of a node) -->
   <field name="label" type="text" indexed="true" stored="true" termVectors="true" omitNorms="true"/>
   <!-- The string version of the title is used for sorting -->
   <copyField source="label" dest="sort_label"/>
   <!-- content is the default field for full text search - dump crap here -->
   <field name="content" type="text" indexed="true" stored="true" termVectors="true"/>
   <field name="teaser" type="text" indexed="false" stored="true"/>

   <field name="path" type="string" indexed="true" stored="true"/>
   <field name="path_alias" type="text" indexed="true" stored="true" termVectors="true" omitNorms="true"/>

   <!-- new fields for grouping pdfs -->
   <field name="article_group" type="string" indexed="true" stored="true" termVectors="true" multiValued="false"/>
   <field name="pdf_path" type="string" indexed="true" stored="true"/>
   <copyField source="path" dest="article_group"/>

   <field name="tid"  type="long" indexed="true" stored="true" multiValued="true"/>

   <field name="test_name" type="text" indexed="true" stored="true" termVectors="true"/>

   <field name="taxonomy_names" type="text" indexed="true" stored="false" termVectors="true" multiValued="true" omitNorms="true"/>
   <!-- Copy terms to a single field that contains all taxonomy term names -->
   <copyField source="tm_vid_*" dest="taxonomy_names"/>

   <!-- Here, default is used to create a "timestamp" field indicating
        when each document was indexed.-->
   <field name="timestamp" type="tdate" indexed="true" stored="true" default="NOW" multiValued="false"/>

    <!-- This field is used to build the spellchecker index -->
   <field name="spell" type="textSpell" indexed="true" stored="true" multiValued="true"/>

  <!-- copyField commands copy one field to another at the time a document
        is added to the index.  It's used either to index the same field differently,
        or to add multiple fields to the same field for easier/faster searching.  -->
   <copyField source="label" dest="spell"/>
   <copyField source="content" dest="spell"/>


   <!-- A set of fields to contain text extracted from HTML tag contents which we
        can boost at query time. -->
   <dynamicField name="tags_*" type="text"   indexed="true" stored="false" omitNorms="true"/>

   <!-- For 2 and 3 letter prefix dynamic fields, the 1st letter indicates the data type and
        the last letter is 's' for single valued, 'm' for multi-valued -->

   <!-- We use long for integer since 64 bit ints are now common in PHP. -->
   <dynamicField name="is_*"  type="long"    indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="im_*"  type="long"    indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="iss_*" type="slong"   indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="ism_*" type="slong"   indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="ss_*"  type="string"  indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="sm_*"  type="string"  indexed="true"  stored="true" multiValued="true"/>
   <!-- Normal text fields are for full text - the relevance of a match depends on the length of the text -->
   <dynamicField name="ts_*"  type="text"    indexed="true"  stored="true" multiValued="false" termVectors="true"/>
   <dynamicField name="tm_*"  type="text"    indexed="true"  stored="true" multiValued="true" termVectors="true"/>
   <!-- Unstemmed text fields for full text - the relevance of a match depends on the length of the text -->
   <dynamicField name="tus_*" type="text_und" indexed="true"  stored="true" multiValued="false" termVectors="true"/>
   <dynamicField name="tum_*" type="text_und" indexed="true"  stored="true" multiValued="true" termVectors="true"/>
   <!-- These text fields omit norms - useful for extracted text like taxonomy_names -->
   <dynamicField name="tos_*" type="text"    indexed="true"  stored="true" multiValued="false" termVectors="true" omitNorms="true"/>
   <dynamicField name="tom_*" type="text"    indexed="true"  stored="true" multiValued="true" termVectors="true" omitNorms="true"/>
   <!-- Special-purpose text fields -->
   <dynamicField name="tes_*" type="edge_n2_kw_text" indexed="true" stored="true" multiValued="false" omitTermFreqAndPositions="true" />
   <dynamicField name="tem_*" type="edge_n2_kw_text" indexed="true" stored="true" multiValued="true" omitTermFreqAndPositions="true" />
   <dynamicField name="tws_*" type="text_ws" indexed="true" stored="true" multiValued="false"/>
   <dynamicField name="twm_*" type="text_ws" indexed="true" stored="true" multiValued="true"/>
   <!-- trie dates are preferred, so give them the 2 letter prefix -->
   <dynamicField name="ds_*"  type="tdate"   indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="dm_*"  type="tdate"   indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="bm_*"  type="boolean" indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="bs_*"  type="boolean" indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="its_*" type="tlong"   indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="itm_*" type="tlong"   indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="fts_*" type="tfloat"  indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="ftm_*" type="tfloat"  indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="pts_*" type="tdouble" indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="ptm_*" type="tdouble" indexed="true"  stored="true" multiValued="true"/>
   <!-- Binary fields can be populated using base64 encoded data. Useful e.g. for embedding
        a small image in a search result using the data URI scheme -->
   <dynamicField name="xs_*"  type="binary"  indexed="false" stored="true" multiValued="false"/>
   <dynamicField name="xm_*"  type="binary"  indexed="false" stored="true" multiValued="true"/>
   <!-- In rare cases a sfloat rather than tfloat is needed for sortMissingLast -->
   <dynamicField name="fss_*" type="sfloat"  indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="fsm_*" type="sfloat"  indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="pss_*" type="sdouble" indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="psm_*" type="sdouble" indexed="true"  stored="true" multiValued="true"/>
   <!-- In rare cases a date rather than tdate is needed for sortMissingLast -->
   <dynamicField name="dds_*" type="date"    indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="ddm_*" type="date"    indexed="true"  stored="true" multiValued="true"/>
   <!-- In case a 32 bit int is really needed, we provide these fields. 'h' is mnemonic for 'half word', i.e. 32 bit on 64 arch -->
   <dynamicField name="hs_*" type="integer" indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="hm_*" type="integer" indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="hss_*" type="sint"   indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="hsm_*" type="sint"   indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="hts_*" type="tint"   indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="htm_*" type="tint"   indexed="true"  stored="true" multiValued="true"/>

   <!-- Begin added fields to use features in Solr 3.4+ -->
   <dynamicField name="points_*" type="point" indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="pointm_*" type="point" indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="locs_*" type="location" indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="locm_*" type="location" indexed="true"  stored="true" multiValued="true"/>
   <dynamicField name="geos_*" type="geohash" indexed="true"  stored="true" multiValued="false"/>
   <dynamicField name="geom_*" type="geohash" indexed="true"  stored="true" multiValued="true"/>
   <!-- End added fields for Solr 3.4+ -->
   <!-- Sortable version of the dynamic string field -->
   <dynamicField name="sort_*" type="sortString" indexed="true" stored="false"/>
   <copyField source="ss_*" dest="sort_*"/>
  <!-- A random sort field -->
   <dynamicField name="random_*" type="rand" indexed="true" stored="true"/>
   <!-- This field is used to store access information (e.g. node access grants), as opposed to field data -->
   <dynamicField name="access_*" type="integer" indexed="true" stored="false" multiValued="true"/>


   <!-- The following causes solr to ignore any fields that don't already match an existing
        field name or dynamic field, rather than reporting them as an error.
        Alternately, change the type="ignored" to some other type e.g. "text" if you want
        unknown fields indexed and/or stored by default -->
   <dynamicField name="*" type="ignored" multiValued="true" />

 </fields>

 <!-- Field to use to determine and enforce document uniqueness.
      Unless this field is marked with required="false", it will be a required field
   -->
 <uniqueKey>id</uniqueKey>

 <!-- field for the QueryParser to use when an explicit fieldname is absent -->
 <defaultSearchField>content</defaultSearchField>

 <!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
 <solrQueryParser defaultOperator="AND"/>

</schema>

我有一个uniquekey(id)和一个defaultSearchField .

我的输出:

<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">8</int>
</lst>
<lst name="initArgs">
<lst name="defaults">
<str name="config">dataconfig.xml</str>
</lst>
</lst>
<str name="command">full-import</str>
<str name="status">idle</str>
<str name="importResponse"/>
<lst name="statusMessages">
<str name="Total Requests made to DataSource">1</str>
<str name="Total Rows Fetched">3</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2013-11-14 14:24:04</str>
<str name="Total Documents Processed">0</str>
<str name="Total Documents Failed">3</str>
<str name="Time taken ">0:0:0.45</str>
</lst>
<str name="WARNING">
This response format is experimental. It is likely to change in the future.
</str>
</response>

我已经尝试了很多不同的东西,没有任何效果..请帮助我!

3 回答

  • 0

    请检查schema.xml中的必填字段,因为Solr不处理缺少必需fieald的文档 . (这就是为什么你获取行=#但处理的文档总数= 0)

    所以你的错误就是你要定义两个必填字段(id,entity_type),如下所示:

    <field name="id" type="string" indexed="true" stored="true" **required="true"** />
    <field name="entity_type" type="string" indexed="true" stored="true" **required="true"** />
    

    在您的实体中,您只是使用'id'字段,但未使用'entity_type':

    <entity pk="id" query="SELECT `id`,`test_name` FROM `test`">
        <field column="id" name="id" />
        <field column="test_name" name="test_name" />
        <!-- the other required field (entity_type) is missing here -->
     </entity>
    

    所以你要么在你的实体中索引这个字段'entity_type',

    或者在您的情况下,您应该从schema.xml中的定义中删除所需的选项

  • 2

    您缺少必填字段 . 应始终使用/分配必填字段 .

  • 1

    在field标记下的schema.xml中,您只需要保留data-config中存在的那些字段 .

    <fields>
       <field name="id" type="string" indexed="true" stored="true" required="true" />
       <field name="test_name" type="string" indexed="true" stored="true" required="true" />
    </fields>
    

    为了简单起见,请尝试像这样修改部分数据配置

    <document>
         <entity pk="id" query="SELECT `id`,`test_name` FROM `test`">
            <field column="id" name="id" />
            <field column="test_name" name="test_name" />
         </entity>
      </document>
    

相关问题