Fast Search Internal Server error exception - Resolved - SharePoint 2010

Today’s article is based on the numbers of problems that I faced while configuring fast search. This is the first time I have done this configuration but there were string of problems I encountered while resolving the issue. I would like to share the complete troubleshooting steps so that your time will be saved, issue will be resolved, you can meet your deadline in case of any, reduce the frustration J

SP attributes:

- SharePoint Server 2010 with recent CU

- Fast Search Server 2010

Problem description:

I have configured the Fast Search and added the content sources. I was able to crawl but found out Search results are not coming / working.

Error Message:

"Internal Server error exception" while searching from SharePoint sites.

Troubleshooting done:

1.   Verified certificate installation and configuration on both FAST and SharePoint servers

Ref: http://technet.microsoft.com/en-us/library/ff381244%28v=office.14%29.aspx

2.   Examined Service application proxy, TCP Port status and IPV6 status of FAST and SharePoint servers

Ref: http://blogs.technet.com/b/rubeno/archive/2012/02/23/troubleshooting-the-connection-between-the-fast-search-server-contentdistributor-and-sharepoint-2010.aspx

3.   Verified that FAST Service Account is part of FASTSearchAdministrators and FASTSearchKeywordAdministrators group

Ref: http://technet.microsoft.com/en-us/library/ff599535%28v=office.14%29.aspx#BKMK_Useraccounts

4.   While troubleshooting FAST service and SharePoint Search Service status identified that SAMWorker process at FAST ADMIN Node was in DEAD state.

About SAMWorker:

Generates a security filter for each user to trim down the query results returned to a user.

Inputs  End user claims tokens from the QRProxyService. 

User security filter requests from the QRServer. 

Accepts configuration changes from the SAM Admin.        

Outputs Generates a user’s security filter which is joined with the user’s query in the QRServer to do security trimming.

Dependencies: LogService, SAM Admin Service, QRServer, QRServerProxy.

5.   We Provide Full Control for Machine Keys in the following location on FAST Server.

(location" “Documents and Settings\All Users\Application  Data\Microsoft\Crypto\RSA\ Machine Keys”)

Ref: http://msdn.microsoft.com/en-us/library/windows/desktop/bb204778%28v=vs.85%29.aspx

6.   After this we used a new browser and browsed to the site.

Got the error "The search request was unable to connect to the Search Service"

7.   When examined SharePoint ULS, was seeing error below,

02/24/2014 09:30:05.06 w3wp.exe (0x1CD8) 0x1364 SharePoint Server Search Query g4hr Monitorable FastSearchGateway::GetQueryResult--Communication Exception searching on QRServer: http://SharePoint.amolgalaxy:13287. Putting qrserver on the exclude list for a while. Exception: System.ServiceModel.FaultException: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.    Server stack trace:      at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)     at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at IProxyRemote.ProcessRequest(String request)     at Microsoft.SharePoint.Search.Extended.Query.Http.WcfSearchEngine.GetSearchResponse(IQuery query, String searchUrlPath, String urlQueryStr, Int32 timeout)     at Microsoft.SharePoint.Search.Extended.Query.Http.AbstractUrlBasedSearchEngine.SearchInternal(IQuery query, IResultParser parser, String urlStr)     at Microsoft.SharePoint.Search.Extended.Query.Http.AbstractUrlBasedSearchEngine.Search(IQuery query)     at Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.GetQueryResult(IQuery query). f72be699-6dae-42d0-8bd6-501f61958e3d

02/24/2014 09:30:05.06 w3wp.exe (0x1CD8) 0x1364 SharePoint Server Search Query g4hk Monitorable AbstractSearchGateway::Search--Exception when searching for sharepoint site:"http://amolgalaxy/sites/admin-apps". Ex: Microsoft.Office.Server.Search.Query.SearchServiceNotFoundException: Service fault communicating with query service http://SharePoint.amolgalaxy:13287; The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.     at Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.GetQueryResult(IQuery query)     at Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.ExecuteSearch(SearchRequest request, String queryString)     at Microsoft.Office.Server.Search.Query.Gateway.AbstractSearchGateway.Search(SearchRequest request) f72be699-6dae-42d0-8bd6-501f61958e3d

8.   We confirmed that the Proxy for Search is up and running by executing Get-SPServiceApplicationProxy

- Checked the proxy setting on the IE and was not configured.

- Tried to start the SamWorker (which is dead) but it failed to start

- In the event viewer found the following error

Log Name:      Application

Source:        FASTSearchSAMWorker

Date:         

Event ID:      0

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:     

Description:

Service cannot be started. System.InvalidOperationException: Error Starting up.  Unable to initialize Startup Object Microsoft.SharePoint.Search.Extended.Security.Base.WCF.WCFServiceManager,Microsoft.SharePoint.Search.Extended.Security.Base"

at Microsoft.SharePoint.Search.Extended.Security.WorkerService.WorkerService.OnStart(String[] args)

at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)

9. Followed steps below in an attempt to resolve this error,

- Stopped the Fast search service.

- Modified the Fast folder (%fastsearch%) permissions to add 'FASTSearchAdministrators' group members one by one, with full control.

- Set child objects to inherit permissions

- Removed the tilda files begining with ‘~’ from %fastsearch% folder

- Started the Fast services for SharePoint

It did not help resolve the issue. SamWorker remained in stopped state.

NCTRL Status   showing SAM worker  'dead'  state.

        Services.msc> FAST Search SAM Worker Service is not started. Manually startign it is not helping to  get sam worker running.

When ran commands below in given order,

 

-nctrl stop samworker

-nctrl stop samadmin

-nctrl start samadmin

-nctrl start samworker

   

The SAMWorker process gets in to running state and goes back to dead state in 2-3 minutes.

You are not able to get any search results in the SharePoint farm. Crawling and indexing works as expected.

This is a new deployment and the issue explained above was identified during the deployment phase.

-Checked %FASTSEARCH%/var/logs/syslog/ found only   'authorization-admin_amolgalaxy.txt' . So, this system never has SAMworker running.  If so, we should see at least authorization-worker_ amolgalaxy.txt in same folder.

-Tried to set FAST logging from warning to debug

 

Ran commands below at customer fast server,

 

-Set-FASTSearchSecurityLogLevel -DefaultLogLevel debug    --> Got error "cannot set log level."

 -Set-FASTSearchSecurityLogLevel -WarningNameSpaceLogLevel  debug --> Got error "cannot set log level.”

Checked the config file in location Config File(s) Location   and verified that they are good by comparing with a good FAST server's configuration files.

  For SAM ADMIN:

  %FASTSEARCH%/bin/Microsoft.Sharepoint.Search.Extended.Security.AdminService.exe.config 

%FASTSEARCH%/components/sam/admin  and

  For SAM WORKER

  %FASTSEARCH%/bin/Microsoft.Sharepoint.Search.Extended.Security.workerService.exe.config

%FASTSEARCH%/components/sam/worker

-Followed  http://support.microsoft.com/kb/2656160     and ran   "psconfig.ps1 -action p"   command via Powershell  as FAST For SharePoint  service account.  The command completed successfully.

-Registered ServicePrincipalNames for CN=s-fastdevsvc,OU=App Services,OU=CUSFA,DC=carrier,DC=UTC,DC=com  using SETSPN command.

That's it - Issue Resolved!  Verified that the search results are returned as expected.

If you have any queries/questions regarding the above mentioned information then please let me know, Thank you.