Re: [platform-help-dev] Lucene analyzers for double-byte languages?

["Dorian Birsan" <birsan@xxxxxxxxxx>]

Erik, other groups have successfully
tested 2.0.2 infocenter on many languages, including those your mentioned,
so something must be different in the setup.
Since things work fine in the stand-alone,
it appears that the locale passed to the infocenter is not correct. Unlike
the stand-alone, the locale is picked up from the request, not from the
host machine. So you must ensure your browser sends the appropriate locale.
An alternative to detecting the browser
locale is to proxy the infocenter by another webapp, and have a dispatcher
servlets that wraps the incoming request, changes the locale to a desired
locale and then delegates to the real infocenter.
There is a fix in 3.0 to fix some locale
related issues, but in your case the problem is likely caused by your browser
not sending the expected locale.

-Dorian

Erik Hennum/Oakland/IBM@IBMUS Sent by: platform-help-dev-admin@xxxxxxxxxxx 07/16/2003 10:19 PM Please respond to platform-help-dev

To:        platform-help-dev@xxxxxxxxxxx         cc:                 Subject:        Re: [platform-help-dev] Lucene analyzers for double-byte languages?

Hi, Dorian:

That's reassuring!

And, in fact, when I try the search using the standalone version of Eclipse
2.0.2, I can select a Japanese string from a topic, copy it into the search
field, run the search, and return a list of search results.  When
I
selected a search results, the matched Japanese string is correctly
highlighted in the displayed topic.

However, if I do the same thing in the InfoCenter version of Eclipse 2.0.2,
the query string doesn't seem to get to the backend Eclipse web
application. The status bar shows a correct-looking URL for a time, but
nothing comes back to change the prompt in the search results frame.  The
Tomcat console reports the error "java.io.CharConversionException:
isHexDigit" during conversion of parameters (I've appended the full
exception).

The Java topics do contain some untranslated English strings.  If
I query
on an English string in the InfoCenter version, the search succeeds,
listing the Japanese topics in the search results frame and highlighting
the matched string in the displayed topic.

Could there be an issue with encoding, transmitting, and decoding the query
string via the web server / servlet container?

I noticed that escape is deprecated in ECMAScript v3 (equivalent to
Netscape 6 _javascript_ 1.5 or IE 5.5 JScript 5.5) and so tried the
encodeURIComponent() _javascript_ function in search.jsp but got the same
result.

I'm trying to confirm that InfoCenter Eclipse 2.0.2 on WebSphere
Application Server is also failing to pass the query string through to
the
Eclipse web application.  Our tester reports the following message:

    There was an error in your action:
    Java.lang.IllegalArgumentException

Any suggestions on how we might fix this problem?  We're quite late
in a
release cycle.


Thanks,


Erik Hennum
ehennum AT us.ibm.com


java.io.CharConversionException: isHexDigit
        at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:124)
        at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:87)
        at
org.apache.tomcat.util.http.Parameters.processParameters(Parameters.j
ava:408)
        at
org.apache.tomcat.util.http.Parameters.processParameters(Parameters.j
ava:495)
        at
org.apache.tomcat.util.http.Parameters.handleQueryParameters(Paramete
rs.java:278)
        at
org.apache.coyote.tomcat4.CoyoteRequest.parseRequestParameters(Coyote
Request.java:1920)
        at
org.apache.coyote.tomcat4.CoyoteRequest.getParameterNames(CoyoteReque
st.java:942)
        at
org.apache.coyote.tomcat4.CoyoteRequest.getParameterMap(CoyoteRequest
.java:922)
        at
org.apache.coyote.tomcat4.CoyoteRequestFacade.getParameterMap(CoyoteR
equestFacade.java:193)
        at
org.apache.catalina.core.ApplicationHttpRequest.setRequest(Applicatio
nHttpRequest.java:525)
        at
org.apache.catalina.core.ApplicationHttpRequest.<init>(ApplicationHtt
pRequest.java:125)
        at
org.apache.catalina.core.ApplicationDispatcher.wrapRequest(Applicatio
nDispatcher.java:921)
        at
org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD
ispatcher.java:547)
        at
org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis
patcher.java:498)
        at
org.apache.jsp.search_results_jsp._jspService(search_results_jsp.java
:48)
        at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:136)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:202)
        at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:2
89)
        at
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:240)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:247)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:193)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:260)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContex
t.invokeNext(StandardPipeline.java:643)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.jav
a:480)
        at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:191)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContex
t.invokeNext(StandardPipeline.java:643)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.jav
a:480)
        at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

        at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:
2397)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:180)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContex
t.invokeNext(StandardPipeline.java:643)
        at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatche
rValve.java:170)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContex
t.invokeNext(StandardPipeline.java:641)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:171)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContex
t.invokeNext(StandardPipeline.java:641)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.jav
a:480)
        at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:174)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContex
t.invokeNext(StandardPipeline.java:643)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.jav
a:480)
        at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

        at
org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:22
3)
        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:405)
        at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ssConnection(Http11Protocol.java:380)
        at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java
:508)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadP
ool.java:533)
        at java.lang.Thread.run(Thread.java:513)



                    
                     
                     
                     
                     
                     
  
                    
 "Dorian Birsan"            
                     
                     
                     
                 
                    
 <birsan@xxxxxxxxxx>            
To:       platform-help-dev@xxxxxxxxxxx      
                     
           
                    
 Sent by:                
       cc:            
                     
                     
                   
                    
 platform-help-dev-admin@        Subject:  Re:
[platform-help-dev] Lucene analyzers for double-byte languages?  
                    
 eclipse.org                
                     
                     
                     
                 
                    
                     
                     
                     
                     
                     
  
                    
                     
                     
                     
                     
                     
  
                    
 07/16/2003 10:25 AM              
                     
                     
                     
           
                    
 Please respond to              
                     
                     
                     
             
                    
 platform-help-dev              
                     
                     
                     
             
                    
                     
                     
                     
                     
                     
  
                    
                     
                     
                     
                     
                     
  





Erik,

Search should work in the languages you listed, as eclipse provides a
default analyzer.
The English and German analyzers are a bit smarter, as they deal with
stemming, stop words, etc.
You could certainly pick up 3rd party plugins and contribute them as
plugins in your product (that's the intention of the analyzer extension
point).

-Dorian


                    
                     
                     
          
   Erik                  
                     
                     
     
   Hennum/Oakland/IBM@IBMUS          To:
                     
            
   Sent by:                
 platform-help-dev@xxxxxxxxxxx          
      
   platform-help-dev-admin@e         cc:  
                     
          
   clipse.org                
       Subject:        [platform-help-dev]
  
                    
        Lucene analyzers for double-byte languages?
  
                    
                     
                     
          
   07/16/2003 12:51 PM              
                     
                
   Please respond to              
                     
                  
   platform-help-dev              
                     
                  
                    
                     
                     
          








Help Developers:

With regard to

   http://dev.eclipse.org/mhonarc/lists/platform-help-dev/msg00082.html

has there been any success in creating Lucene analyzers for Japanese,
Traditional Chinese, Simplified Chinese, and Korean?

Our need is to enable search in these languages on Eclipse 2.0.2

If there aren't any analyzers, would we need to hack the JSPs to disable
or
hide the search UI?  (Merely to confirm.)

I did notice in looking at the Lucene site that an analyzer is available
for Simplified Chinese:

   http://marc.theaimsgroup.com/?l=lucene-dev&m=100705753831746&q=p3

I presume that would need a wrapper to plug into the
org.eclipse.help.luceneAnalyzer extension point.


Thanks,


Erik Hennum
ehennum AT us.ibm.com


_______________________________________________
platform-help-dev mailing list
platform-help-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-help-dev



_______________________________________________
platform-help-dev mailing list
platform-help-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-help-dev