Excessive noise in log files due to attempts to reach non-existent host
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	• bzimport
	Mar 6 2013, 6:30 PM

Description

Author: ram

Description:
In the Lucene global config file lsearch-global-2.1.conf we map disabled indexes to a non-existent host named "search1000x". This causes a large number of exception stack traces to be dumped to the log file, for example on search1016:

ram@search1016:/a/search/log$ grep -c 'java.rmi.UnknownHostException: Unknown host: search1000x;' *
log:851206
log.2013-03-02:1130806
log.2013-03-03:1126806
log.2013-03-04:rOPUP112560683d3b
log.2013-03-05:1124406

This can be remedied by adding a property (such as 'nullHost') to explicitly identify a non-existent host and have the Java code ignore that host.

Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=47293

Details

Reference: bz45795

Event Timeline

• bzimport raised the priority of this task from to High.Nov 22 2014, 1:23 AM

• bzimport added a project: Wikimedia-lucene-search-2.

• bzimport set Reference to bz45795.

• bzimport added a subscriber: Unknown Object (MLST).

• bzimport created this task.Mar 6 2013, 6:30 PM

ram wrote:

Here is a sample of the stack trace from NetworkStatusThread:

2013-03-06 19:41:23,390 [Thread-14] WARN org.wikimedia.lsearch.interoperability.RMIMessengerClient - Cannot contact RMI registry for host search1000x : Unknown host: search1000x; nested exception is:

java.net.UnknownHostException: search1000x

2013-03-06 19:41:23,390 [Thread-14] WARN org.wikimedia.lsearch.search.NetworkStatusThread - Host search1000x for fiu_vro-titles.tspart2 still down.
java.rmi.UnknownHostException: Unknown host: search1000x; nested exception is:

java.net.UnknownHostException: search1000x
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:598)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at org.wikimedia.lsearch.interoperability.RMIMessengerClient.messengerFromCache(RMIMessengerClient.java:135)
at org.wikimedia.lsearch.interoperability.RMIMessengerClient.getSearcherPoolStatus(RMIMessengerClient.java:367)
at org.wikimedia.lsearch.search.NetworkStatusThread.pingHosts(NetworkStatusThread.java:82)
at org.wikimedia.lsearch.search.NetworkStatusThread.run(NetworkStatusThread.java:52)

Caused by: java.net.UnknownHostException: search1000x

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:195)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.wikimedia.lsearch.interoperability.CustomSocketFactory.createSocket(CustomSocketFactory.java:46)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 8 more

And here is one from SearcherCache:

2013-03-06 20:38:38,482 [pool-1-thread-48] WARN org.wikimedia.lsearch.search.SearcherCache - Cannot get searcher status for en-titles.tspart2 on search1000x : Unknown host: search1000x; nested exception is:

java.net.UnknownHostException: search1000x

java.rmi.UnknownHostException: Unknown host: search1000x; nested exception is:

java.net.UnknownHostException: search1000x
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:598)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at org.wikimedia.lsearch.interoperability.RMIMessengerClient.messengerFromCache(RMIMessengerClient.java:135)
at org.wikimedia.lsearch.interoperability.RMIMessengerClient.getSearcherPoolStatus(RMIMessengerClient.java:367)
at org.wikimedia.lsearch.search.SearcherCache.initializeRemote(SearcherCache.java:445)
at org.wikimedia.lsearch.search.SearcherCache.initializeRemote(SearcherCache.java:432)
at org.wikimedia.lsearch.search.SearcherCache.getRandomHost(SearcherCache.java:338)
at org.wikimedia.lsearch.search.WikiSearcher.makeMultiSearcher(WikiSearcher.java:65)
at org.wikimedia.lsearch.search.WikiSearcher.<init>(WikiSearcher.java:100)
at org.wikimedia.lsearch.search.SearchEngine.fetchTitles(SearchEngine.java:898)
at org.wikimedia.lsearch.search.SearchEngine.search(SearchEngine.java:731)
at org.wikimedia.lsearch.search.SearchEngine.search(SearchEngine.java:136)
at org.wikimedia.lsearch.frontend.SearchDaemon.processRequest(SearchDaemon.java:106)
at org.wikimedia.lsearch.frontend.HttpHandler.handle(HttpHandler.java:197)
at org.wikimedia.lsearch.frontend.HttpHandler.run(HttpHandler.java:114)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Caused by: java.net.UnknownHostException: search1000x

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:195)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.wikimedia.lsearch.interoperability.CustomSocketFactory.createSocket(CustomSocketFactory.java:46)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 20 more

ram wrote:

Change-Id: Ia68eef0307fe69843ccfcd678b62c42e91e1d3ec
Change-Id: I2764c45a9ce4391b88576b9122142b47e900956e

Patch pushed for review.

ram wrote:

This patch is merged and now in production. Log files now are much less noisy.

[Merging "MediaWiki extensions/Lucene Search" into "Wikimedia/lucene-search2", see bug 46542. You can filter bugmail for: search-component-merge-20130326 ]

Related URL: https://gerrit.wikimedia.org/r/59533 (Gerrit Change Ie6be47113e57b8419d041909bff458b171b263a0)

Excessive noise in log files due to attempts to reach non-existent hostClosed, ResolvedPublicActions

Description

Details

Event Timeline

Excessive noise in log files due to attempts to reach non-existent host
Closed, ResolvedPublic
Actions