我在Solr3.4中的中文/日文文本索引有困难。我正在使用DIH导入数据,连接块是
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost/db_development?useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8" user="user" useUnicode="true" characterEncoding="UTF-8" encoding="UTF-8" password="password" zeroDateTimeBehavior="convertToNull" name="app" />
此字段的fieldtype defn为
<fieldType name="text_commongrams" class="solr.TextField"> <analyzer> <charFilter class="solr.HTMLStripCharFilterFactory" /> <tokenizer class="solr.ICUTokenizerFactory" /> <filter class="solr.ICUTransformFilterFactory" id="Traditional-Simplified"/> <filter class="solr.ICUFoldingFilterFactory"/> <filter class="solr.ASCIIFoldingFilterFactory"/> <filter class="solr.ICUNormalizer2FilterFactory" name="nfkc_cf" mode="compose"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory" /> <filter class="solr.TrimFilterFactory" /> <filter class="solr.LowerCaseFilterFactory" /> <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true" /> <filter class="solr.CommonGramsFilterFactory" words="stopwords_en.txt" ignoreCase="true" /> <filter class="solr.StopFilterFactory" words="stopwords_en.txt" ignoreCase="true" /> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" splitOnNumerics="0" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" preserveOriginal="1" /> </analyzer> </fieldType>
MySQL字符编码详细信息如
+--------------------------+-----------------------------------------+ | Variable_name | Value | +--------------------------+-----------------------------------------+ | character_set_client | latin1 | | character_set_connection | latin1 | | character_set_database | latin1 | | character_set_filesystem | binary | | character_set_results | latin1 | | character_set_server | utf8 | | character_set_system | utf8 | | character_sets_dir | /opt/local/share/mysql5/mysql/charsets/ | +--------------------------+-----------------------------------------+
我使用Java参数-dfile.encoding=utf-8
启动Solr。
输入文本为JavaOne Tokyo 2012
当我将其导入到Solr中,并使用ID查询该文档时,我看到的文本为JavaOne Tokyo 2012??®?™:??,©?????????
谁能告诉我我哪里错了?
所以我最后不得不修改我的MySQL表,将字符串存储在UTF8中。有关如何将现有表从latin1转换为utf8的详细信息可以在这里找到。