SharePoint 2013: Programmatically set crawl cookies for forms authenticated web sites

Last week I was having difficulties in crawling forms authenticated web sites. When configuring a crawl rule to store the authentication cookies the login page was returning multiple cookies with same name but different domain.

Add Crawl Rule

This gave issues in a later stage (during crawl) because all cookies would be sent along with the request, the target system had issues correctly identifying us due to these “duplicate” cookies.

You can easily check the request information that is sent during crawl by starting fiddler on the crawl server and configuring the proxy settings to http://localhost:8888 (default Fiddler settings).

 Search Proxy Setting

In the end we chose for an alternate method of configuring the cookies, namely through PowerShell. This gave us the ultimate flexibility to configure exactly the cookies we wanted to pass along with the crawl requests.

asnp microsoft.sharepoint.powershell -ea 0 | Out-Null $ssa = get-spenterprisesearchserviceapplication $crPath = 'http://authenticatedwebsite*' # Get or create crawl rule $cr = Get-SPEnterpriseSearchCrawlRule -SearchApplication $ssa | ? { $_.Path -eq $crPath } if ($cr -eq $null) { $cr = New-SPEnterpriseSearchCrawlRule -Path $crPath -SearchApplication $ssa -Type InclusionRule -AuthenticationType CookieRuleAccess -FollowComplexUrls $true } # Set cookie credentials $cr.SetCredentials('CookieRuleAccess', 'myUser=crawlUser; myPwd=crawlPassword', 'http://cookie-set-via-powershell')



Links to this post


CAPTCHA Image Validation