Today’s article is based on the numbers of problems that I faced while configuring fast search. This is the first time I have done this configuration but there were string of problems I encountered while resolving the issue. I would like to share the complete troubleshooting steps so that your time will be saved, issue will be resolved, you can meet your deadline in case of any, reduce the frustration J
SP attributes:
- SharePoint Server 2010 with recent CU
- Fast Search Server 2010
Problem description:
I have configured the Fast Search and added the content sources. I was able to crawl but found out Search results are not coming / working.
Error Message:
"Internal Server error exception" while searching from SharePoint sites.
Troubleshooting done:
1. Verified certificate installation and configuration on both FAST and SharePoint servers
2. Examined Service application proxy, TCP Port status and IPV6 status of FAST and SharePoint servers
3. Verified that FAST Service Account is part of FASTSearchAdministrators and FASTSearchKeywordAdministrators group
4. While troubleshooting FAST service and SharePoint Search Service status identified that SAMWorker process at FAST ADMIN Node was in DEAD state.
About SAMWorker:
Generates a security filter for each user to trim down the query results returned to a user.
Inputs End user claims tokens from the QRProxyService.
User security filter requests from the QRServer.
Accepts configuration changes from the SAM Admin.
Outputs Generates a user’s security filter which is joined with the user’s query in the QRServer to do security trimming.
Dependencies: LogService, SAM Admin Service, QRServer, QRServerProxy.
5. We Provide Full Control for Machine Keys in the following location on FAST Server.
(location" “Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\ Machine Keys”)
6. After this we used a new browser and browsed to the site.
Got the error "The search request was unable to connect to the Search Service"
7. When examined SharePoint ULS, was seeing error below,
02/24/2014 09:30:05.06 w3wp.exe (0x1CD8) 0x1364 SharePoint Server Search Query g4hr Monitorable FastSearchGateway::GetQueryResult--Communication Exception searching on QRServer: http://SharePoint.amolgalaxy:13287. Putting qrserver on the exclude list for a while. Exception: System.ServiceModel.FaultException: The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs. Server stack trace: at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter) at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at IProxyRemote.ProcessRequest(String request) at Microsoft.SharePoint.Search.Extended.Query.Http.WcfSearchEngine.GetSearchResponse(IQuery query, String searchUrlPath, String urlQueryStr, Int32 timeout) at Microsoft.SharePoint.Search.Extended.Query.Http.AbstractUrlBasedSearchEngine.SearchInternal(IQuery query, IResultParser parser, String urlStr) at Microsoft.SharePoint.Search.Extended.Query.Http.AbstractUrlBasedSearchEngine.Search(IQuery query) at Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.GetQueryResult(IQuery query). f72be699-6dae-42d0-8bd6-501f61958e3d
02/24/2014 09:30:05.06 w3wp.exe (0x1CD8) 0x1364 SharePoint Server Search Query g4hk Monitorable AbstractSearchGateway::Search--Exception when searching for sharepoint site:"http://amolgalaxy/sites/admin-apps". Ex: Microsoft.Office.Server.Search.Query.SearchServiceNotFoundException: Service fault communicating with query service http://SharePoint.amolgalaxy:13287; The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs. at Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.GetQueryResult(IQuery query) at Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.ExecuteSearch(SearchRequest request, String queryString) at Microsoft.Office.Server.Search.Query.Gateway.AbstractSearchGateway.Search(SearchRequest request) f72be699-6dae-42d0-8bd6-501f61958e3d
8. We confirmed that the Proxy for Search is up and running by executing Get-SPServiceApplicationProxy
- Checked the proxy setting on the IE and was not configured.
- Tried to start the SamWorker (which is dead) but it failed to start
- In the event viewer found the following error
Log Name: Application
Source: FASTSearchSAMWorker
Date:
Event ID: 0
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer:
Description:
Service cannot be started. System.InvalidOperationException: Error Starting up. Unable to initialize Startup Object Microsoft.SharePoint.Search.Extended.Security.Base.WCF.WCFServiceManager,Microsoft.SharePoint.Search.Extended.Security.Base"
at Microsoft.SharePoint.Search.Extended.Security.WorkerService.WorkerService.OnStart(String[] args)
at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)
9. Followed steps below in an attempt to resolve this error,
- Stopped the Fast search service.
- Modified the Fast folder (%fastsearch%) permissions to add 'FASTSearchAdministrators' group members one by one, with full control.
- Set child objects to inherit permissions
- Removed the tilda files begining with ‘~’ from %fastsearch% folder
- Started the Fast services for SharePoint
It did not help resolve the issue. SamWorker remained in stopped state.
NCTRL
Status showing SAM worker 'dead'
state.
Services.msc> FAST Search SAM Worker
Service is not started. Manually startign it is not helping to get sam worker running.
When
ran commands below in given order,
-nctrl stop samworker
-nctrl stop samadmin
-nctrl start samadmin
-nctrl start samworker
The SAMWorker process gets in to running state and goes back to dead
state in 2-3 minutes.
You
are not able to get any search results in the SharePoint farm. Crawling and
indexing works as expected.
This is a new deployment and the issue explained above was
identified during the deployment phase.
-Checked %FASTSEARCH%/var/logs/syslog/ found only 'authorization-admin_amolgalaxy.txt' . So,
this system never has SAMworker running.
If so, we should see at least authorization-worker_ amolgalaxy.txt in
same folder.
-Tried to set FAST logging from warning to debug
Ran commands below
at customer fast server,
-Set-FASTSearchSecurityLogLevel -DefaultLogLevel
debug --> Got error "cannot
set log level."
-Set-FASTSearchSecurityLogLevel
-WarningNameSpaceLogLevel debug -->
Got error "cannot set log level.”
Checked the config file in location Config File(s) Location and verified that they are good by comparing
with a good FAST server's configuration files.
For
SAM ADMIN:
%FASTSEARCH%/bin/Microsoft.Sharepoint.Search.Extended.Security.AdminService.exe.config
%FASTSEARCH%/components/sam/admin and
For SAM WORKER
%FASTSEARCH%/bin/Microsoft.Sharepoint.Search.Extended.Security.workerService.exe.config
%FASTSEARCH%/components/sam/worker
-Followed http://support.microsoft.com/kb/2656160
and ran "psconfig.ps1 -action p" command via Powershell as FAST For SharePoint service account. The command completed successfully.
-Registered
ServicePrincipalNames for CN=s-fastdevsvc,OU=App
Services,OU=CUSFA,DC=carrier,DC=UTC,DC=com
using SETSPN command.
That's it - Issue Resolved! Verified that the search results are returned as expected.
If you have any queries/questions regarding the above mentioned information then please let me know, Thank you.
