Sitecore Adaptive Images – Huge Crawling Logs Problem

Some time ago I have noticed that on our production environment the Crawling Log files were abnormally big – between 500 MB and 1 GB. After some digging was found out that they were just filling up with exceptions while crawling the media library. The exceptions look like this:

WARN Could not compute value for ComputedIndexField: urllink for indexable: sitecore://master/{8EA15044-EE2C-41DB-81D6-0A9C42062814}?lang=en&ver=1
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
Source: Sitecore.Kernel
at Sitecore.Context.PageMode.get_IsNormal()
at adaptiveImages.AdaptiveImagesMediaProvider.GetMediaUrl(MediaItem item)
at Sitecore.ContentSearch.ComputedFields.UrlLink.ComputeFieldValue(IIndexable indexable)
at Sitecore.ContentSearch.LuceneProvider.LuceneDocumentBuilder.AddComputedIndexFields()

Considering the case that there are a tons of media (over 2000 items) and languages (there is a separate exception for each language) we had some pretty serious index time performance issues and hard disk space problems.

After some debugging I have found out that the Sitecore.Context.PageMode.IsNormal was throwing null reference. The huge problem was that there is no way to protect against null pointers here because you cannot check the Sitecore.Context and the Sitecore.Context.PageMode for nulls. The good part is that the PageMode.IsNormal method code is using the Sitecore.Context.Site (which is the actual null in this case) to check the PageMode.

public static bool IsNormal
{
get
{
return Context.Site.DisplayMode == DisplayMode.Normal;
}
}

With this information the fix became pretty trivial. As try-catch is pretty expensive operation – we just needed to check if the Context.Site is not null and the Context.Site.DisplayMode is normal. The fixes took place in the two GetMediaUrl overrides. Here is the code for the modified methods which were causing the exception.

/// <summary>
/// Gets a media URL.
/// </summary>
/// <param name="item">The media item.</param>
/// <returns>
/// The media URL.
/// </returns>
public override string GetMediaUrl(MediaItem item)
{
Assert.ArgumentNotNull(item, "item");

// Fix for checking the page mode
bool isNormal = Context.Site != null && Context.Site.DisplayMode == DisplayMode.Normal;

//If media item is not an image or the page context is not normal, then return
if (!IsImage(item) || !isNormal)
return base.GetMediaUrl(item);

MediaUrlOptions mediaUrlOptions = new MediaUrlOptions();

return GetMediaUrl(item, mediaUrlOptions);
}

/// <summary>
/// Gets the media URL.
/// </summary>
/// <param name="item">The item.</param>
/// <param name="mediaUrlOptions">The media URL options.</param>
/// <returns></returns>
public override string GetMediaUrl(MediaItem item, MediaUrlOptions mediaUrlOptions)
{
Assert.ArgumentNotNull(item, "item");
Assert.ArgumentNotNull(mediaUrlOptions, "mediaUrlOptions");

// Fix for checking the page mode
bool isNormal = Context.Site != null && Context.Site.DisplayMode == DisplayMode.Normal;

//If media item is not an image or the page context is not normal, then return
if (!IsImage(item) || !isNormal || Context.Database == null || Context.Database.Name != _database)
return base.GetMediaUrl(item, mediaUrlOptions);

//If resolution cookie is not set
if (!IsResolutionCookieSet())
{
//If mobileFirst is set to FALSE or user agent is identifying as a desktop, return with largest break-point resolution
if (!_mobileFirst || IsDesktopBrowser())
{
mediaUrlOptions.MaxWidth = GetLargestBreakpoint();
return base.GetMediaUrl(item, mediaUrlOptions);
}
//Return with mobile-first breakpoint (Smallest)
mediaUrlOptions.MaxWidth = GetMobileFirstBreakpoint();
return base.GetMediaUrl(item, mediaUrlOptions);
}

// If Max-width is not set or Max-width is greater than the selected break-point, then set the Max-width to the break-point
if (mediaUrlOptions.MaxWidth == 0 || mediaUrlOptions.MaxWidth > GetScreenResolution())
mediaUrlOptions.MaxWidth = GetScreenResolution();

// If Max-width is not set and the 'maxWidth' setting is not empty, then set the Max-width property to the maxWidth
if (mediaUrlOptions.MaxWidth == 0 && !string.IsNullOrEmpty(_maxWidth))
{
int maxWidth = 0;
if (int.TryParse(_maxWidth, out maxWidth))
{
// If pixel ratio is normal
if (GetCookiePixelDensity() == 1)
mediaUrlOptions.MaxWidth = maxWidth;
else
mediaUrlOptions.MaxWidth = maxWidth * GetCookiePixelDensity();
}

}

return base.GetMediaUrl(item, mediaUrlOptions);
}

Basically we are using an isNormal variable to check the Context.Site display mode instead of using the Context.PageMode.IsNormal property:

bool isNormal = Context.Site != null && Context.Site.DisplayMode == DisplayMode.Normal;

By doing this I kept the module’s consistency and logic as we just added a null check for the Context.Site. After the fix our logs became around 2 Kb again and our indexing speed came back to normal!

Enjoy!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s