Dissecting Hard Rock Memorabilia and Silverlight Deep Zoom - Part 5

Trying to find out which image is clicked? Maybe this code snippet will help...

msi.MouseLeftButtonUp += delegate(object sender, MouseButtonEventArgs e)
{
    Point p = this.msi.ElementToLogicalPoint(e.GetPosition(this.msi));
    int subImageIndex = SubImageHitTest(p);
    if (subImageIndex >= 0)
        // look up a custom corresponding data structure or xml and return the metadata associated with this image
}

int SubImageHitTest(Point p)
{
    for (int i=0; i<msi.SubImages.Count; i++) {
        MultiScaleSubImage subImage = msi.SubImages[i];
        
        double scaleBy = 1 / subImage.ViewportWidth;
        Rect subImageRect = new Rect(-subImage.ViewportOrigin.X * scaleBy, -subImage.ViewportOrigin.Y * scaleBy, 
                                        1 * scaleBy, (1 / subImage.AspectRatio) * scaleBy);
        if (subImageRect.Contains(p))
            return i;
    }

    return -1;
}

Dissecting Hard Rock Memorabilia and Silverlight Deep Zoom - Part 4

Update Oct 14, 2008: Removed the example from this post to reduce load time of this page. Please use the example from this post, DeepZoom sample ported to Silverlight 2 Beta 2, to try the functionality.

Click on the Randomize button to randomly arrange the images.

Click on the Full Screen button to get into full screen mode. I am not sure if there is bug in Silverlight or I am doing something wrong but it seems like the MultiScaleImage height and width is not reset after it gets out of the full screen mode! Also, the keyboard seems to be disabled in full screen mode (except for the ESC key)!

Click on the Set Source button to set the source for the MultiScaleImage control to the one specified in the text box to the left of the button. This should be a fully qualified URI and should point to a valid items.bin file (http://.../items.bin). I could not get this to work across domains even after placing a crossdomain.xml file in the root folder of my alternate web host but maybe I was doing something wrong. It seems to work if the file is in the same host but for some reason it is not arranging the images. You can try this url and see what I mean - http://thepintospatronus.com/deepzoom/test2/items.bin

What I learned from this exercise so far...

• How to use some of the Siverlight 2 controls, position them on the screen so that they scale, and apply borders to them!
• How to enable full screen mode. The msdn help section refers to some Javascript code to achieve this but I wanted to do it in managed code. Finally figured it out...

  buttonFullScreen.Click += (sender, e) => Application.Current.Host.Content.IsFullScreen = !Application.Current.Host.Content.IsFullScreen;
     
  Application.Current.Host.Content.FullScreenChanged (sender, e) => // Will be called when the control renders in full screen;
  Application.Current.Host.Content.Resized (sender, e) => // Will be called when the control gets out of full screen;

• How to host this puppy! Initially I tried silverlight.live.com but then realized that I couldn't upload the deep zoom images up there. So I used my host to upload the images but then realized that the control located at silverlight.live.com couldn't access the images from my host - I even played around with the crossdomain.xml with no success. I then decided to host everything on my host (bluehost.com) since they did support the .xap mime type without any additional tweaking

Dissecting Hard Rock Memorabilia and Silverlight Deep Zoom - Part 3

In Part 2 we saw the code for arranging the images in a nice clean layout. This post will discuss when to call the ArrangeImages() method.

When the page that contains the Silverlight control is first loaded, the MultiScaleImageControl will load the images and render it on the screen. The issue here is that the ArrangeImages method hasn't had a chance to do its magic yet! Hence we have to intercept the MultiScaleImage control somehow before it renders the images. This can be achieved by the following 2 steps:

1. Set UseSprings to false in the Xaml

<MultiScaleImage x:Name="msi" UseSprings="false"/>

2. Intercept the MultiScaleImage.Motionfinished Event. Call ArrangeImages in this handler

msi.MotionFinished += delegate(object sender, RoutedEventArgs e)
{
  // This is required to ensure that ArrangeImages is called only once
  if(msi.UseSprings == false) {
      ArrangeImages();
      msi.UseSprings = true;
  }
};

In addition, the ArrangeImages method can be called on demand - as in a button click or after the dataset is filtered.

Here is a sample image of what the output looks like after ArrangeImages is called on 17 arbitrarily cropped images

Dissecting Hard Rock Memorabilia and Silverlight Deep Zoom - Part 2

In Part 1 of this series, I discussed the logic to arrange the images in a grid - where the height of all images in a row is the same but can change across rows. The objective is to maintain the aspect ratio of the image and to not stretch it. This post provides some sample code snippets that implement the algorithm.

Some considerations

1. The code caters to spacing between images even thought this was not discussed in the logic
2. The _imagesToShow and _imagesToHide collections should be populated with MutliScaleSubImages prior to calling ArrangeImages
3. There is 1 condition that is not addressed: if the height of all rows exceed the height of the container - it can be ignored for now if clipping is acceptable
4. This code is based off the sample code published by Scott Hanselman and hence the use of the variable msi (which is an instance of MultiScaleImage)
5. I am sure this code and/or logic can be improved upon so please feel free to leave your comments

List<MultiScaleSubImage> _imagesToShow = new List<MultiScaleSubImage>();
List<MultiScaleSubImage> _imagesToHide = new List<MultiScaleSubImage>();

public void ArrangeImages()
{
   double containerAspectRatio = msi.Width / msi.Height;
   double spaceBetweenImages = 0.005;

   List<SubImage> subImages = new List<SubImage>();
   _imagesToShow.ForEach(subImage => subImages.Add(new SubImage(subImage)));
  
   // Capture the total width of all images
   double totalImagesWidth = 0.0;
   subImages.ForEach(subImage => totalImagesWidth += subImage.Width);

   // Calculate the total number of rows required to display all the images
   int numRows = (int)Math.Sqrt((totalImagesWidth / containerAspectRatio)+1);
  
   // Create the rows
   List<Row> rows = new List<Row>(numRows);
   for (int i=0; i<numRows; i++)
       rows.Add(new Row(spaceBetweenImages));

   double widthPerRow = totalImagesWidth / numRows;
   double imagesWidth = 0;
   // Separate the images into rows. The total width of all images in a row should not exceed widthPerRow
   for (int i=0, j=0; i<numRows; i++, imagesWidth=0) {
       while (imagesWidth < widthPerRow && j < subImages.Count) {
           rows[i].AddImage(subImages[j]);
           subImages[j].RowNum = i;
           imagesWidth += subImages[j++].Width;
       }
   }

   // At this point in time the subimage height is 1 
   // If we assume that the total height is also 1 we need to scale the subimages to fit within a total height of 1
   // If the total height is 1, the total width is aspectRatio. Hence (aspectRatio)/(total width of all images in a row) is the scaling factor.
   // Added later: take into account spacing between images
   rows.ForEach(Row => Row.Scale(containerAspectRatio));

   // Calculate the total height, with space between images, of the images across all rows
   // Also adjust the colNum for each image
   double totalImagesHeight = -spaceBetweenImages(numRows - 1) * spaceBetweenImages;
   rows.ForEach(Row => totalImagesHeight += Row.Height + spaceBetweenImages);
   Debug.Assert(totalImagesHeight <= 1.0); // TODO: Handle this condition

   // The totalImagesHeight should not exceed 1. 
   // if it does, we need to scale all images by a factor of (1 / totalImagesHeight)
   // This takes care of consideration #3 in the text above
   if (totalImagesHeight > 1) {
        subImages.ForEach(subImage => subImage.Scale(1 / (totalImagesHeight+spaceBetweenImages)));
        totalImagesHeight = (numRows - 1) * spaceBetweenImages;
        rows.ForEach(Row => totalImagesHeight += Row.Height);
        Debug.Assert(totalImagesHeight <= 1);
   }

   // Calculate the top and bottom margin
   double margin = (1 - totalImagesHeight) / 2;

   // First hide all the images that should not be displayed
   _imagesToHide.ForEach(subImage => subImage.Opacity = 0.0);

   // Then display the displayable images to scale
   for (int i=0; i<_imagesToShow.Count; i++) {
       _imagesToShow[i].Opacity = 1.0;    // in case this was hidden earlier

       double X = rows[subImages[i].RowNum].CalcX(subImages[i].ColNum);
       double Y = margin + ((subImages[i].RowNum == 0) ? 0 : rows[subImages[i].RowNum-1].Height + spaceBetweenImages);
       for (int j=0; j<subImages[i].RowNum; j++)
            Y += spaceBetweenImages + rows[j].Height;
      
       _imagesToShow[i].ViewportWidth = containerAspectRatio/ subImages[i].Width;
//      AnimateImage(_imagesToShow[i], new Point(-(X / subImages[i].Width), -(Y / subImages[i].Width)));    // for animation, use this statement instead of the next one
       _imagesToShow[i].ViewportOrigin = new Point(-(X / subImages[i].Width), -(Y / subImages[i].Width));
   }
}

The helper classes:

public class SubImage
{
   private Size _imageSize = new Size(0.0, 1.0);    //normalized height is 1 for all images
   private int _colNum = 0;
   private int _rowNum = 0;
  
   public SubImage(MultiScaleSubImage image)
   {
       // Capture the normalized size of each image (fixed height of 1)
       // This normalization is required since we want the height of all images to be the same but the widths can differ
       _imageSize.Width = image.AspectRatio;
   }

   public int ColNum
   {
       get { return _colNum; }
       set { _colNum = value; }
   }

   public int RowNum
   {
       get { return _rowNum; }
       set { _rowNum = value; }
   }
  
   public double Width  { get { return _imageSize.Width; } }
   public double Height { get { return _imageSize.Height; } }

   public void Scale(double scaleBy)
   {
       _imageSize.Width *= scaleBy;
       _imageSize.Height *= scaleBy;
   }
}

public class Row
{
   List<SubImage> _subImages = new List<SubImage>();
   double _spaceBetweenImages = 0;
  
   public Row(double spaceBetweenImage)
   {
       _spaceBetweenImages = spaceBetweenImage;
   }
  
   public double TotalWidth
   {
       get
       {
           double totalWidth = 0;
           _subImages.ForEach(image => totalWidth += image.Width);
           return totalWidth;
       }
   }

   public int ImageCount {get{return _subImages.Count;}}
   public double Height {get{return (ImageCount <= 0) ? 0 : _subImages[0].Height;}}
   public double TotalSpaceBetweenImages {get{return (ImageCount <= 0) ? 0 : (ImageCount-1) * _spaceBetweenImages;}}

   public void AddImage(SubImage subImage)
   {
       _subImages.Add(subImage);
       subImage.ColNum = _subImages.Count - 1;
   }
  
   public void Scale(double canvasWidth)
   {
       double scaleBy = (canvasWidth - TotalSpaceBetweenImages) / TotalWidth;
       foreach (SubImage subImage in _subImages)
           subImage.Scale(scaleBy);
   }
  
   public double CalcX(int colNum)
   {
       Debug.Assert(colNum < _subImages.Count);

       double X = 0;
       for (int i=0; i<colNum; i++) {
           X += _subImages[i].Width + _spaceBetweenImages;
       }
       return X;
   }
}

Deep Zoom layout.bin file decoded

This is a follow-up to my previous post (Deep Zoom items.bin file decoded)...
The reason for decoding the layout.bin is that the files items.bin and layout.bin are dependent on each other. For the adventurous few who may now decide to play around with the items.bin file, it may be necessary to modify the layout.bin file as well!

Bytes (Hex)

00-04  ItemCount

Item information (Repeat for each item)
05-08  Item Id
09-0C  ViewportOrigin.X
0D-10  ViewportOrigin.Y
11-14  ViewportWidth

Pretty straightforward! I played around with these files and was able to add and delete items easily.

Deep Zoom items.bin file decoded

Thanks to George Bell (check out the comment section on this Expression Team blog), I became aware of a undocumented command line tool, sparseimagetool.exe, that is bundled with the Deep Zoom composer. This is a nifty tool which can generate a collection given a simple xml file. This helps when one needs to process a large number of images. The xml file should be created along the lines of the SparseImageSceneGraph.xml that is output by the Deep Zoom composer.

Usage: SparseImageTool CreateCollection

While I was playing with this tool, I noticed a 'Use XML' format option. I tried it and it generated a items.xml file instead of a items.bin file. While this xml file cannot be used as input to the MultiScaleImage I thought it would be interesting to see if I could decode the items.bin file from this xml file. I don't know what I am going to achieve by decoding this file but it does bring up interesting possibilities; one possibility could be - to implement filtering by modifying the items.bin file itself instead of playing around with the Opacity of each MultiScaleSubImage. This might seem overkill but is worth investigating for a large collection!

This is what my sample items.xml file looks like:

<?xml version="1.0" encoding="UTF-8"?>
<Collection Version="2" UseStringsFile="0" MinLevel="0" MaxLevel="8" PageSizeLog2="8" PageFormat="jpg" PageQuality="1" NextItemId="6" ItemCount="6">
<Thumbnails>
   <Thumbnail Id="0" MinLevel="0" MaxLevel="8" SizeX="0.6953125" SizeY="0.521484375"/>
   <Thumbnail Id="1" MinLevel="0" MaxLevel="8" SizeX="0.6953125" SizeY="0.521484375"/>
   <Thumbnail Id="2" MinLevel="0" MaxLevel="8" SizeX="0.6953125" SizeY="0.521484375"/>
   <Thumbnail Id="3" MinLevel="0" MaxLevel="8" SizeX="0.6953125" SizeY="0.521484375"/>
   <Thumbnail Id="4" MinLevel="0" MaxLevel="8" SizeX="0.6953125" SizeY="0.521484375"/>
   <Thumbnail Id="5" MinLevel="0" MaxLevel="8" SizeX="0.6953125" SizeY="0.521484375"/>
</Thumbnails>
<Items>
   <Item Id="0" Init="test2_images\DSCF0067.sdi" Relative="1" Type="ImagePixelSource" Thumb="0"/>
   <Item Id="1" Init="test2_images\DSCF0078.sdi" Relative="1" Type="ImagePixelSource" Thumb="1"/>
   <Item Id="2" Init="test2_images\DSCF0082.sdi" Relative="1" Type="ImagePixelSource" Thumb="2"/>
   <Item Id="3" Init="test2_images\DSCF0087.sdi" Relative="1" Type="ImagePixelSource" Thumb="3"/>
   <Item Id="4" Init="test2_images\DSCF0090.sdi" Relative="1" Type="ImagePixelSource" Thumb="4"/>
   <Item Id="5" Init="test2_images\DSCF0061.sdi" Relative="1" Type="ImagePixelSource" Thumb="5"/>
</Items>
</Collection>

I used the above XML file to decode the corresponding items.bin file

Bytes (Hex)

00     Version            (2)
01-04  UseStringsFile     (0)
05-08  MinLevel           (0)
09-0C  MaxLevel           (8)
0D-10  PageSizeLog2       (8)
11-14  ?? - PageQuality?  (1)
15-1C  ?? - PageFormat?   (jpg)
1D-20  NextItemId         (6)
21-24  ItemCount          (6)

Thumbnail information for 1 thumbnail (Repeat for each thumbnail)
25-28  Thumbnail Id       (0)
29-2C  MinLevel           (0)
2D-30  MaxLevel           (8)
31-34  SizeX              (0.6953125)
35-38  SizeY              (0.521484375)

Item information (Repeat for each item)
9D-A0  Item Id            (0)
A0-A4  Init Length        (24)
A4-BC  Init               (test2_images\DSCF0067.sdi)
BD     Relative           (1)
BE-C1  Type Length        (16)
C2-D1  Type               (ImagePixelSource)
D2-D5  Thumb              (0)

Dissecting Hard Rock Memorabilia and Silverlight Deep Zoom - Part 1

I have been following some of the Silverlight presentations on Mix 2008 and was totally blown away with the Hard Rock Memorabilia demo. Vertigo had created a truly compelling site! I was hoping that they would put some of their source code online, but since this did not happen, I decided to play around with Silverlight and Deep Zoom myself and attempt to create a similar experience.

There were some parts of the puzzle that needed to be addressed before a similar experience could be created:

1. Mouse and keyboard handlers, including Mouse Wheel support, for panning and zooming the Deep Zoom image
2. Figuring out how to manipulate the Deep Zoom images as a collection instead of a single large image
3. Arranging the images in a grid, where the height of all images in a row is the same but can change across rows. This gives a clean look to the images
4. Animating the images so that they appear to slide into position
5. Adding extra information to each image so that they can be filtered on
   

Part 1 is easily solved since a lot of people have already provided solutions for the mouse and keyboard handlers. Some of the contributors who caught my attention: Scott Hanselman, Soul Solutions Blog, and the Expression Team at Microsoft.

I figured out Part 2 myself but noticed that the Expression Team already posted a solution for this. The DeepZoom composer provides a way to export the images as a collection. The images can then be accessed using the MultiScaleImage.SubImages collection.

The Expression Team has provided a partial solution to Part 3. They arranged the images in a grid but did not arrange it such that the height of all images in a row remain the same. I will attempt to provide an algortihm for this in this post and maybe provide some code snippet in the next post. I do not have access to Visual Studio 2008 at this time and hence am using notepad++ to write code. I am actually working off Hanselman's source code so I will wait until I create my own project before posting the entire source code. In the meantime, I will try to provide some code snippets...

Once again, the Expression Team has provided a very good sample for animating images, which addresses Part 4. This is one area I was struggling with since I lack animation skills. I am learning more everyday but for now I am happy to borrow this part of the code from the Expression Team!

I haven't reached Part 5 yet but I suspect that the Name/Tag property on the MultiScaleSubImage can be used to store additional information regarding the image. At the very least, a pointer of some sort can be stored in the Tag property which can then be used to retrieve more information on the image.

Now for the algorithm for Part 3 (arrange images in a grid, where the height of all images in a row is the same but can change across rows)

1. Infer the MultiScaleImage aspect ratio - MultiScaleImage.Width / MultiScaleImage.Height
   
2. Capture the normalized width of each image - What I mean by this is that the height of each image should remain the same. Assuming that we want the height to be a fixed '1', the width would equal the aspect ratio of the image (not the MultiScaleImage aspect ratio but the aspect ratio of the SubImage itself)
   
3. Capture the total normalized width of all images - This is the sum of the normalized widths of each image
   
4. Calculate the number of rows required to display the images - For this we need to normalize the heights of all images within the MultiScaleImage container. This can be achieved by dividing the total normalized width of images by the MultiScaleImage aspect ratio: Number of rows = Sqrt of [total image width (3.) / MultiScaleImage aspect ratio (1.)]
5. Capture the width per row - total images width (3.) / Number of rows (4.)
   
6. For each row, capture all the images that will fit in that row. This is based on the width of each image (2.) and the max width per row (5.)
7. At this point in time, each row will contain a set of images which may or may not take up the width allocated to the entire row. Scale the rows by a factor of MultiScaleImage aspect ratio (1.) / total normalized width of images in a row. This will ensure that images in a row will fill up the width of the entire row. This will also normalize the height of all rows such that the sum of height of all rows will not exceed '1' (the normalized height of the MultiScaleImage)
   
8. After step 8, all the rows should contain images that fill up the width of the entire row but the row height between rows may differ. By this we ensure that the images will fill the width of the container (MultiScaleImage) but may not fill up the height. Hence we need to calculate the top and bottom margin for display purposes
   
9. Total Margin = 1 - total normalized height of rows. Top and bottom margin = Total margin / 2
10. Un-normalize the images by scaling them back to the actual container size and display them - SubImage.ViewportWidth = MultiScaleImage aspect ratio / SubImage.width, SubImage.ViewportWidth = Point(X / SubImage.width, Y / SubImage.width) where X and Y are calculated based on the row/col value and the prior images width

That should do it! I will try to post some code snippet in subsequent posts.

A few tips (I discovered along the way...)

• SubImage can be hidden/shown by controlling the Opacity property. This is useful for implementing filter functionality
• Instead of setting the MultiScaleImage Source property in the Xaml, a better place to set it is in the MultiScaleImage.Loaded event. This way the entire html page with the Silverlight control will be loaded and then the MultiScaleImage control will try to load the images. This enhances the overall user experience