Taking Server Side Screenshots of Websites

Note: This is an old post from my previous blog. I haven’t looked into this for a while and it might no longer work. This was tested with Ubuntu 11.10.

I was recently doing some testing to see if it was possible to take server side screenshots of websites. After a bit of fudging around, managed to do it…. Now, I can’t take credit for figuring this out. Other people have done this before me and some have posted blog posts about this, but technology changes so fast that some of the blog posts that outlined how to do this no longer work. I got most of the way from a blog post by Gregory Mazurek.

Tools:

  • Ubuntu Linux (I used Ubuntu 11.10) – 32 bit (so we can run Flash without any problems)
  • Xvfb
  • Firefox
  • Flash
  • ImageMagick

Theoretically, it sounds very simple. Xvfb lets you run X windows but rather than rendering to a display (servers might not have a display), it renders to a virtual framebuffer. So…you run Firefox in Xvfb, then take a screenshot of it as it’s running. Simple right? lol…

Step 1 – Install all the stuff you need

$ sudo apt-get install xvfb
$ sudo apt-get install firefox
$ sudo apt-get install adobe-flashplugin    #You may need to enable the canonical partner repositories. Uncomment appropriate lines in /etc/apt/sources.list
$ sudo apt-get install imagemagick

Step 2 – Get the xvfb process running

Note: Credit to Gregory Mazurek for this shell script…

Create a file in /etc/init.d called xvfb. Put this in:

#!/bin/bash
#
# /etc/rc.d/init.d/xvfb
#
# chkconfig: 345 98 90
# description: starts virtual framebuffer process to
# enable server
#
#
#
# Source function library.
#.  /etc/init.d/functions
XVFB_OUTPUT=/tmp/Xvfb.out
XVFB=/usr/bin/X11/Xvfb
XVFB_OPTIONS=":5 -screen 0 1080x1440x24 -fbdir /var/run"

start()  {
echo -n "Starting : X Virtual Frame Buffer "
$XVFB $XVFB_OPTIONS >>$XVFB_OUTPUT 2>&1&;
RETVAL=$?
echo
return $RETVAL
}

stop()   {
echo -n "Shutting down : X Virtual Frame Buffer"
echo
killall Xvfb
echo
return 0
}

case "$1" in
start)
start
;;
stop)
stop
;;
status)
status xvfb
;;
restart)
    stop
    start
    ;;

*)
echo "Usage: xvfb {start|stop|status|restart}"
exit 1
;;
esac
exit $?

Once you have this shell script in place, you can simply start and stop the xvfb server easily with:

sudo /etc/init.d/xvfb start    # (or sudo service xvfb start)
sudo /etc/init.d/xvfb stop

Great!

Now to run Firefox…

Step 3 – Firefox and Flash

I’ll assume that you did the apt-get install to get both Firefox and flash-plugin. Once you have those, test the configuration like this…

  1. Make sure that the xvfb server is running
  2. Run this: DISPLAY=:5 nohup firefox http://www.youtube.com &

Explanation:

DISPLAY=:5 This tells xvfb to render to display 5 (virtual) nohup silence the output firefox loads firefox http://youtube.com loads youtube which tells you whether you have Flash or not & Load this in the background

Firefox should be running in the background now. If it’s not, you may have to debug and find out why. Remove the nohup and the & to see the output to shell if you need to debug.

Now comes the moment of truth. Taking a screenshot:

DISPLAY=:5 import -window root screenshot.png

This will dump the desktop to screenshot.png. Check it.

Kill firefox by typing fg then CTRL+C.

Step 4 – Firefox configuration

We have to override the default Firefox configuration settings. E.g. it has one that will try to resume a crash.

Firefox stores its preferences in the user root ~/.mozilla. On each system and each user it’s different. Look in: ~/.mozilla/firefox/

In there, you should see a directory with funny numbers and letters followed by a .default. Mine is 6mqvagz4.default. Chdir into there. Create a file called user.js.

The problem here is that we either have to edit files manually or have to get Firefox to generate the appropriate settings for us. If you are already using Ubuntu at home, this is much simpler because you can set up Firefox however you like, then take the entire preferences directory and copy its contents into the server’s. This is what I ended up doing. That route is by far the easiest and recommended.

Otherwise:

In prefs.js, edit these:

pref("browser.startup.page",0);
pref("browser.sessionstore.resume_from_crash", false);

Now…another annoyance is that the Firefox window may not be at the size you want it. So you need to edit the file localstore.rdf. What I ended up doing was running Firefox on my Mac at home, deleting the localstore.rdf file, having Firefox generate a new one. On quitting, it saves it out. This is my localstore.rdf file that I ended up using:

<?xml version="1.0"?>
<RDF:RDF xmlns:NC="http://home.netscape.com/NC-rdf#" xmlns:RDF="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <RDF:Description RDF:about="about:config#lockCol" ordinal="3" />
  <RDF:Description RDF:about="about:config#prefCol" ordinal="1" sortDirection="ascending" />
  <RDF:Description RDF:about="about:config#valueCol" ordinal="7" />
  <RDF:Description RDF:about="chrome://browser/content/browser.xul#sidebar-title" value="" />
  <RDF:Description RDF:about="about:config#typeCol" ordinal="5" />
  <RDF:Description RDF:about="chrome://browser/content/browser.xul">
    <NC:persist RDF:resource="chrome://browser/content/browser.xul#main-window"/>
    <NC:persist RDF:resource="chrome://browser/content/browser.xul#sidebar-box"/>
    <NC:persist RDF:resource="chrome://browser/content/browser.xul#sidebar-title"/>
  </RDF:Description>
  <RDF:Description RDF:about="chrome://browser/content/browser.xul#main-window"
                   screenY="0"
                   width="1080"
                   screenX="0"
                   height="1440"
                   sizemode="maximized" />
  <RDF:Description RDF:about="about:config">
    <NC:persist RDF:resource="about:config#prefCol"/>
    <NC:persist RDF:resource="about:config#lockCol"/>
    <NC:persist RDF:resource="about:config#typeCol"/>
    <NC:persist RDF:resource="about:config#valueCol"/>
  </RDF:Description>
</RDF:RDF>

Notice the width and height of the window. I set it to this portrait width/height so that I could grab square images.

Step 5 – Further Automation

From here, you can do further automation to make the process of taking a screenshot really simple. I created this script:

#!/bin/bash
#screengrab

FIREFOX_OUTPUT=/tmp/firefox.out
TIMESTAMP=$(date +%s)
DELAY=5

#Check for args
if [ $# -ne 1 ]
        then
                echo "You need to pass a URL in. E.g. urlscreengrab www.google.com"
                exit
fi

#Calculate how many instances of grab we have running
COUNT=$(ps -e | grep -c 'grab.sh')
COUNT=$((COUNT-2))
#I don't know why but the above returns 2 when there are 0 processes running.
#if grab.sh is already running, wait for it to finish
WAIT=$((DELAY*COUNT))
if [ "$WAIT" -gt "0" ]
        then
                WAIT=$((WAIT+2)) #Add 2 seconds to wait for other grab processes to finish
fi
echo "Waiting $WAIT seconds for other grab processes to finish"
sleep $WAIT

#Load up firefox
echo "Loading Firefox"
DISPLAY=:5 firefox $1 >>$FIREFOX_OUTPUT 2>&1&;
FFPID=$!
echo "Standby...waiting $DELAY seconds for browser to load..."
sleep $DELAY
DISPLAY=:5 import -window root -crop 1000x1000+4+86 $TIMESTAMP.png
#DISPLAY=:5 import -window root $TIMESTAMP.png
#killall firefox-bin
kill $FFPID
echo "$TIMESTAMP.png"

To use this script, type: ./urlscreengrab http://www.youtube.com

With the above configuration, it will automatically take the screenshot and crop it into a 1000×1000 square png file. It also has process checking, so it will check for existing Firefox processes and wait until the other grab processes finish. It will also wait 5 seconds for a web page to finish loading before taking the shot. This is needed on many web pages because it takes time to load the page.

Step 6 – Hook up to your web scripts

From here, I just call the web script from PHP using the exec() command. Make sure that Apache is running using the same user as you set up your above scripts so that it can use the correct Mozilla profile. To check, create a php script with the following:

<?php
echo exec('whoami');

This should give you the user of the owner of the apache/php process.

My web script for grabbing the image:

<?php
/*
Script for grabbing a website image
*/

if (!isset($_GET['url']))
{
        echo "No URL provided";
        exit();
}

//chdir into the correct dir
chdir('images');

//Grab the site
$exec = "/home/ubuntu/grab.sh ".$_GET['url'];
echo "Executing: ".$exec."";
$lastline = exec($exec, $out, $status);

foreach ($out as $line)
{
        echo $line."";
}
echo "Finished with status: $status ";

if ($status == 0)
{
        echo "<a href=\"images/$lastline\">Your image is here</a>";
}

There you go…have fun! 🙂

  • mrdrozdov

    Does this still work?

  • Pascal Roget

    You might want to set toolkit.startup.max_resumed_crashes to -1 in prefs.js to prevent Firefox from displaying the “Safe mode” pop-up after a few screenshots are taken.

    Also, that $_GET[‘url’] parameter should be checked to prevent random people from running code on your server.