Fast Search Internal Server error exception - Resolved - SharePoint 2010

Today’s article is based on the numbers of problems that I faced while configuring fast search. This is the first time I have done this configuration but there were string of problems I encountered while resolving the issue. I would like to share the complete troubleshooting steps so that your time will be saved, issue will be resolved, you can meet your deadline in case of any, reduce the frustration J

SP attributes:

- SharePoint Server 2010 with recent CU

- Fast Search Server 2010

Problem description:

I have configured the Fast Search and added the content sources. I was able to crawl but found out Search results are not coming / working.

Error Message:

"Internal Server error exception" while searching from SharePoint sites.

Troubleshooting done:

1.   Verified certificate installation and configuration on both FAST and SharePoint servers

Ref: http://technet.microsoft.com/en-us/library/ff381244%28v=office.14%29.aspx

2.   Examined Service application proxy, TCP Port status and IPV6 status of FAST and SharePoint servers

Ref: http://blogs.technet.com/b/rubeno/archive/2012/02/23/troubleshooting-the-connection-between-the-fast-search-server-contentdistributor-and-sharepoint-2010.aspx

3.   Verified that FAST Service Account is part of FASTSearchAdministrators and FASTSearchKeywordAdministrators group

Ref: http://technet.microsoft.com/en-us/library/ff599535%28v=office.14%29.aspx#BKMK_Useraccounts

4.   While troubleshooting FAST service and SharePoint Search Service status identified that SAMWorker process at FAST ADMIN Node was in DEAD state.

About SAMWorker:

Generates a security filter for each user to trim down the query results returned to a user.

Inputs  End user claims tokens from the QRProxyService. 

User security filter requests from the QRServer. 

Accepts configuration changes from the SAM Admin.        

Outputs Generates a user’s security filter which is joined with the user’s query in the QRServer to do security trimming.

Dependencies: LogService, SAM Admin Service, QRServer, QRServerProxy.

5.   We Provide Full Control for Machine Keys in the following location on FAST Server.

(location" “Documents and Settings\All Users\Application  Data\Microsoft\Crypto\RSA\ Machine Keys”)

Ref: http://msdn.microsoft.com/en-us/library/windows/desktop/bb204778%28v=vs.85%29.aspx

6.   After this we used a new browser and browsed to the site.

Got the error "The search request was unable to connect to the Search Service"

7.   When examined SharePoint ULS, was seeing error below,

02/24/2014 09:30:05.06 w3wp.exe (0x1CD8) 0x1364 SharePoint Server Search Query g4hr Monitorable FastSearchGateway::GetQueryResult--Communication Exception searching on QRServer: http://SharePoint.amolgalaxy:13287. Putting qrserver on the exclude list for a while. Exception: System.ServiceModel.FaultException: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.    Server stack trace:      at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)     at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at IProxyRemote.ProcessRequest(String request)     at Microsoft.SharePoint.Search.Extended.Query.Http.WcfSearchEngine.GetSearchResponse(IQuery query, String searchUrlPath, String urlQueryStr, Int32 timeout)     at Microsoft.SharePoint.Search.Extended.Query.Http.AbstractUrlBasedSearchEngine.SearchInternal(IQuery query, IResultParser parser, String urlStr)     at Microsoft.SharePoint.Search.Extended.Query.Http.AbstractUrlBasedSearchEngine.Search(IQuery query)     at Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.GetQueryResult(IQuery query). f72be699-6dae-42d0-8bd6-501f61958e3d

02/24/2014 09:30:05.06 w3wp.exe (0x1CD8) 0x1364 SharePoint Server Search Query g4hk Monitorable AbstractSearchGateway::Search--Exception when searching for sharepoint site:"http://amolgalaxy/sites/admin-apps". Ex: Microsoft.Office.Server.Search.Query.SearchServiceNotFoundException: Service fault communicating with query service http://SharePoint.amolgalaxy:13287; The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.     at Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.GetQueryResult(IQuery query)     at Microsoft.Office.Server.Search.Query.Gateway.FastSearchGateway.ExecuteSearch(SearchRequest request, String queryString)     at Microsoft.Office.Server.Search.Query.Gateway.AbstractSearchGateway.Search(SearchRequest request) f72be699-6dae-42d0-8bd6-501f61958e3d

8.   We confirmed that the Proxy for Search is up and running by executing Get-SPServiceApplicationProxy

- Checked the proxy setting on the IE and was not configured.

- Tried to start the SamWorker (which is dead) but it failed to start

- In the event viewer found the following error

Log Name:      Application

Source:        FASTSearchSAMWorker

Date:         

Event ID:      0

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:     

Description:

Service cannot be started. System.InvalidOperationException: Error Starting up.  Unable to initialize Startup Object Microsoft.SharePoint.Search.Extended.Security.Base.WCF.WCFServiceManager,Microsoft.SharePoint.Search.Extended.Security.Base"

at Microsoft.SharePoint.Search.Extended.Security.WorkerService.WorkerService.OnStart(String[] args)

at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)

9. Followed steps below in an attempt to resolve this error,

- Stopped the Fast search service.

- Modified the Fast folder (%fastsearch%) permissions to add 'FASTSearchAdministrators' group members one by one, with full control.

- Set child objects to inherit permissions

- Removed the tilda files begining with ‘~’ from %fastsearch% folder

- Started the Fast services for SharePoint

It did not help resolve the issue. SamWorker remained in stopped state.

NCTRL Status   showing SAM worker  'dead'  state.

        Services.msc> FAST Search SAM Worker Service is not started. Manually startign it is not helping to  get sam worker running.

When ran commands below in given order,

 

-nctrl stop samworker

-nctrl stop samadmin

-nctrl start samadmin

-nctrl start samworker

   

The SAMWorker process gets in to running state and goes back to dead state in 2-3 minutes.

You are not able to get any search results in the SharePoint farm. Crawling and indexing works as expected.

This is a new deployment and the issue explained above was identified during the deployment phase.

-Checked %FASTSEARCH%/var/logs/syslog/ found only   'authorization-admin_amolgalaxy.txt' . So, this system never has SAMworker running.  If so, we should see at least authorization-worker_ amolgalaxy.txt in same folder.

-Tried to set FAST logging from warning to debug

 

Ran commands below at customer fast server,

 

-Set-FASTSearchSecurityLogLevel -DefaultLogLevel debug    --> Got error "cannot set log level."

 -Set-FASTSearchSecurityLogLevel -WarningNameSpaceLogLevel  debug --> Got error "cannot set log level.”

Checked the config file in location Config File(s) Location   and verified that they are good by comparing with a good FAST server's configuration files.

  For SAM ADMIN:

  %FASTSEARCH%/bin/Microsoft.Sharepoint.Search.Extended.Security.AdminService.exe.config 

%FASTSEARCH%/components/sam/admin  and

  For SAM WORKER

  %FASTSEARCH%/bin/Microsoft.Sharepoint.Search.Extended.Security.workerService.exe.config

%FASTSEARCH%/components/sam/worker

-Followed  http://support.microsoft.com/kb/2656160     and ran   "psconfig.ps1 -action p"   command via Powershell  as FAST For SharePoint  service account.  The command completed successfully.

-Registered ServicePrincipalNames for CN=s-fastdevsvc,OU=App Services,OU=CUSFA,DC=carrier,DC=UTC,DC=com  using SETSPN command.

That's it - Issue Resolved!  Verified that the search results are returned as expected.

If you have any queries/questions regarding the above mentioned information then please let me know, Thank you.

Event ID 5586

Issue: Event ID: 5586 Generated in the SharePoint server.

Error: Event ID 5586 generated in SharePoint server application event logs.

Event Type:      Error

Event Source:  Windows SharePoint Services 3

Event Category:            Database

Event ID:          5586

Date:                09/10/2013

Time:               10:24:22

User:                N/A

Computer:        ++++++++++++

Description:

Unknown SQL Exception 1035 occurred. Additional error information from SQL Server is included below.

Background:

As a part of daily health check on the server, I have noticed allot many occurrences of the event ID 5586 on multiple servers.

This is something occurs on regular intervals but not specific to one server.

By looking some of the reference on the internet I happened to find the TechNet Article which talk about the possible resolution of this error.

Resolution:

This error occurs in both the versions of the SharePoint i.e. 2007 and 2010; however in my case it was reported for the SharePoint 2007.

Remedies for SharePoint 2007: http://technet.microsoft.com/en-us/library/cc561042(v=office.12).aspx    

Remedies for SharePoint Server 2010: http://technet.microsoft.com/en-us/library/ee513056%28v=office.14%29.aspx   

Great thanks to MS to keep the record of the most relevant event IDs and sharing the possible remedies

If you feel any difficulty in following any of resolution listed in the articles, do let me know. I would be more happy to help you put with.

Products Applies to:

1)      SharePoint 2007

2)      SharePoint Server 2010

3)      Windows Sever 2003

4)      Windows Server 2008

5)      SQL Server 2005 and SQL Server 2008.

Hide Title column in list in SharePoint 2010

Unlike SharePoint’s predefined lists, your custom list has only one column when you first create it — the Title column. Unfortunately, you can’t delete the Title column or change its data type, but you can rename or hide it.

Here are the complete details as how to hide the title column from the list. 

Please check and let me know in case of any queries/questions so that we can discuss and proceed further, Thank you.

-It’s actually a two-step resolution-

First step: Removing from the view

1.    Open the list

2.    List settings

3.    Views Section

4.    Click on ALLITEMS view

5.    Uncheck the title column

6.    Click OK

Second Step: Actual Hiding

1.    Open the list

2.    List settings

3.    Advanced settings

4.    Check the ‘allow management of content types’ option to Yes.

5.    Click OK

6.    Now refer to ‘item’ which is present under the content type section

7.    Click on the item content type

8.    Click on ‘Item’ under the columns

9.    Here you will get the option to hide it completely 

Product Applies To:

·         SharePoint Server 2010

·         SharePoint Foundation 2010

·         SharePoint Server 2013

Content databases contain orphaned items

Problem Description:

The SharePoint Health Analyzer detected an error.  Content databases contain orphaned items.

In some situations, a content database that is used by Microsoft SharePoint Foundation may become corrupted. The corrupted database may contain orphaned items. For example, a document may not have a parent document library or a list may not have a parent Microsoft SharePoint Foundation Web site.  Consequently, you may be unable to create new items with the same URL as the orphaned items, and space in the content database will be consumed unnecessarily.

Error Message:

Content databases contain orphaned items

Resolution:

Perfect article provided by Microsoft on this which helps me to resolve this issue:

http://technet.microsoft.com/en-in/library/ff805076(v=office.14).aspx

Product Applies To:

SharePoint Server 2010.

SharePoint Foundation 2010.

A positional parameter cannot be found that accepts argument SharePoint 2010

Problem Description:

Let me tell you that you will face this error while executing PowerShell commands. I have experienced the same while executing ‘add solution’ command.

Error Message:

A positional parameter cannot be found that accepts argument

You might laugh when I will tell you the resolutions in reference to the above as it’s so simple.

Resolution:

-Check the literal path that you have provided, most probably that’s the only reason when this command throws an exception.

Make sure that you providing the path in double quotation marks. That’s it.

Please amend the changes and you will successfully execute the command that you are working on.

In case of any queries/questions then please let me know, Thank you.

Kerberos configuration known issues (SharePoint Server 2010)

Kerberos authentication and non-default ports

There is a known issue where some Kerberos clients (.NET Framework, Internet Explorer 7 and 8 included) do not correctly form service principal names when attempting to authenticate with Kerberos enabled web applications that are configured on non-default ports (ports other than 80 and 443). The root of the problem is the client does not properly form the SPN in the TGS request by specifying it without the port number (as seen in the Sname of the TGS request).
Example:
If the web application is running at http://intranet.contoso.com:1234, the client will request a ticket for a service with a SPN equal to http/intranet.contoso.com instead of http/intranet.contoso.com:1234.
Details regarding the issue can be found in the following articles:

• Internet Explorer 6 cannot use the Kerberos authentication protocol to connect to a Web site that uses a non-standard port in Windows XP and in Windows Server 2003 (http://support.microsoft.com/kb/908209/en-us)
• Configure Kerberos authentication (Office SharePoint Server 2007) (http://go.microsoft.com/fwlink/p/?LinkId=196987)

To work around this issue, register SPNs with and without port number. Example:

• http://intranet.contoso.com:12345

• http/intranet
• http/intranet.contoso.com
• http/intranet:12345
• http/intranet.contoso.com:12345

We recommend that you register the non-default port to ensure that if the issue is resolved in some future service pack or hot fix, the applications using the workaround will still continue to function.
Note that this workaround will not work if the following conditions are true:

• There is more than one web application running on a non-default port
• The web applications either bind to the host name of the server or bind to the same host header (on different ports)
• The web application IIS application pools use different service accounts

• http://server.contoso.com:5000 AppPool Id: contoso\svcA
• http://server.contoso.com:5001 AppPool Id: contoso\svcB

If these conditions are true, following the recommendation in this workaround will yield duplicate SPNs registered to different service accounts which will break Kerberos authentication.
If you have multiple web sites sharing a common host name running on multiple ports, and you use different IIS application pool identities for the web applications, then you cannot use Kerberos authentication on all web sites. (One application can use Kerberos, the rest will require another authentication protocol.) To use Kerberos on all applications in this scenario, you would need to either:

1. Run all web applications under 1 shared service account
2. Run each site with its own host header
   
   
   Reference : http://technet.microsoft.com/en-us/library/gg502606%28v=office.14%29.aspx

SharePoint 2010 - Browser Compatibility

This article explains known limitations of some browsers with SharePoint 2010.

For most interactions with SharePoint an internet browser is required and the choice of browser will affect the experience. Most functionality is available in all browsers, however there are exceptions where certain features require Active X technology which is only available in Internet Explorer and means that some of SharePoint's functionality is lost when using non-IE browsers.

Browser support for SharePoint Server 2010 can be divided into three different levels, as follows:

• Supported - A supported Web browser is a Web browser that is supported to work with SharePoint Server 2010, and all features and functionality work.
• Supported with known limitations - These are non-Internet Explorer browsers such as FireFox, Safari and Chrome. They are supported for use with SharePoint but some functionality is missing in these browsers such as opening documents directly in Office applications (although viewing and editing office documents within the browser is fully supported, as is downloading and editing). Documentation on how to resolve these issues is readily available from Microsoft.
• Not tested - A Web browser that is not tested means that its compatibility with SharePoint Server 2010 is untested, and there may be issues with using the particular Web browser. SharePoint Server 2010 works best with up-to-date, standards-based Web browsers.

 

Courtesy : https://answers.uchicago.edu/page.php?id=24860

Thanks Brent for sharing this

How to overwrite or disable the List View Threshold value for particular list in SharePoint 2010 Using PowerShell

If you want to overwrite or disable the threshold limit of List View items for particular list then execute the following cmdlets in Powershell.

$WebApplication = Get-SPWeb http://<SiteCollectionName>

$List = $WebApplication.Lists["List Name"]

$List.EnableThrottling = $false

$List.Update()

There is one property of SPList called "Enable Throttling" which has to be set to false to overwrite the threshold limit set on Web Application for all the lists.

I hope this is helpful and If you have any questions then please let me know, Thank you.

Applies to SharePoint Server 2010.

The SharePoint Health Analyzer detected an error. One or more services have started or stopped unexpectedly

Problem Description:

The SharePoint Health Analyzer detected an error. One or more services have started or stopped unexpectedly.

Error Message:

The following services are managed by SharePoint, but their running state does not match what SharePoint expects: SPAdminV4. This can happen if a service crashes or if an administrator starts or stops a service using a non-SharePoint interface. If SharePoint-managed services do not match their expected running state, SharePoint will be unable to correctly distribute work to the service.

SharePoint was unable to automatically repair this error.

To stop or start a service managed by SharePoint, use the SharePoint service management interface in the SharePoint Central Administration Site. If a service has crashed, restart the service manually on the affected servers by running "net start [service name]" from a command prompt. For more information about this rule, see http://go.microsoft.com/fwlink/?LinkID=142683.

Checkpoints:

- All services are running fine.

- All SharePoint sites as well as web applications are running without any issues.

Reason:

Basically it’s a clear cut indication that the one of the required service is un-reachable.

Resolutions you can try as there are multiple things by which we can resolve this issue:

- Check the administration service is running or not in services console. if its stopped/disable then please restart it.

- try to reboot the server and check the results again.

- try restarting the services by using command prompt

- after the above things and if you are still facing the problem then please check the SP logs and surely you will find the  exact error which will tell you the root cause behind it.

If you have any issues/queries regarding the above mentioned information then please let me know, Thank you

Product applies to:

- SP2010

- SP2013

- SharePoint Foundation 2010

Site Collection Codes

Parameter value	Description
GLOBAL#0	Global template
STS#0	Team Site
STS#1	Blank Site
STS#2	Document Workspace
MPS#0	Basic Meeting Workspace
MPS#1	Blank Meeting Workspace
MPS#2	Decision Meeting Workspace
MPS#3	Social Meeting Workspace
MPS#4	Multipage Meeting Workspace
CENTRALADMIN#0	Central Admin Site
WIKI#0	Wiki Site
BLOG#0	Blog
SGS#0	Group Work Site
TENANTADMIN#0	Tenant Admin Site
ACCSRV#0	Access Services Site
ACCSRV#1	Assets Web Database
ACCSRV#3	Charitable Contributions Web Database
ACCSRV#4	Contacts Web Database
ACCSRV#6	Issues Web Database
ACCSRV#5	Projects Web Database
BDR#0	Document Center
OFFILE#0	(obsolete) Records Center
OFFILE#1	Records Center
OSRV#0	Shared Services Administration Site
PPSMASite#0	PerformancePoint
BICenterSite#0	Business Intelligence Center
PWA#0	Project Web App Site
PWS#0	Microsoft Project Site
SPS#0	SharePoint Portal Server Site
SPSPERS#0	SharePoint Portal Server Personal Space
SPSMSITE#0	Personalization Site
SPSTOC#0	Contents area Template
SPSTOPIC#0	Topic area template
SPSNEWS#0	News Site
CMSPUBLISHING#0	Publishing Site
BLANKINTERNET#0	Publishing Site
BLANKINTERNET#1	Press Releases Site
BLANKINTERNET#2	Publishing Site with Workflow
SPSNHOME#0	News Site
SPSSITES#0	Site Directory
SPSCOMMU#0	Community area template
SPSREPORTCENTER#0	Report Center
SPSPORTAL#0	Collaboration Portal
SRCHCEN#0	Enterprise Search Center
PROFILES#0	Profiles
BLANKINTERNETCONT	Publishing Portal
SPSMSITEHOST#0	My Site Host
ENTERWIKI#0	Enterprise Wiki
SRCHCENTERLITE#0	Basic Search Center
SRCHCENTERLITE#1	Basic Search Center
SRCHCENTERFAST#0	FAST Search Center
visprus#0	Visio Process Repository