Yes, I’ve been ranting about RSS feeds for a while and how essential they are for keeping up with Web sites. But sometimes you want to follow a site or page that doesn’t have an RSS feed. In that case you’ll want to use a Web page monitor. A Web page monitor is just what it sounds like — a service or software package that watches a Web page for any changes (usually there are options for how it detects changes) and then informs you when there are.
There are a number of different page monitoring services available — there’s even Windows software (WebSite Watcher is still going strong!) The one I like to use to keep up with page changes is a service called VisualPing, which is available at https://visualping.io/ .
In this article I’m going to walk you through how VisualPing works and give you an example of how I use it.
(USUAL DISCLAIMER: I pay VisualPing for its services. It does not pay me. This is not a paid advertisement or endorsement; I just like VisualPing. Okay?)
What I’m Watching
You might think that all major Web sites have RSS feeds. No. You might also think that all Web sites have some kind of mechanism for alerting you to changes. Also no. There are a number of Web sites — colleges and universities, organizations, etc — that I want to watch that don’t have useful RSS feeds or mailing lists.
I don’t have time to periodically visit these Web sites. That’s why I use VisualPing to do it for me. VisualPing visits the site, checks to see if a change has been made, and if a change has been made VisualPing takes a screen shot and e-mails it to me. I review the screen shot and if it looks useful I visit the site.
For this article we’re going to use Kickstarter as an example. Kickstarter has no native RSS feeds that I can find. There’s a project for creating Kickstarter RSS feeds ( http://kicktracker.inaimathi.com/ ) and there’s a site devoted to tracking Kickstarter projects ( https://www.kicktraq.com/ ) and that has RSS feeds, but I wasn’t aware of those projects when I decided I really needed to monitor Kickstarter for news on archive and database projects. So I turned to VisualPing.
VisualPing’s Homepage
I’ve had a VisualPing account for – uh – a couple of years now. I’ve created a lot of jobs with it. And that’s why if you logged into my account and looked at my home page you’d find things a bit overwhelming.
I’ll be getting into how VisualPing works in a moment, but I wanted to show this screen shot first for you longtime readers who might have tried page monitoring software and services 10 to 15 years ago. At the time they were less sophisticated, and even if they were sophisticated, you had to do a lot of work to make sure they found just the kind of page changes you wanted. Back then you might have had to weigh whether a page monitor was going to be worth the trouble as it might provide far too many “false positives” that you would waste your time reviewing. VisualPing is much more advanced than that and it’s much easier to precisely monitor a page (and get a lot of data – VisualPing specifies page changes down to the pixel count.)
Back to the example. Kickstarter, wasn’t it?
Looking for Kicky New Archives
I can go to Kickstarter and do a search for the word archive. I can even sort the results by the newest projects. And when I do I’m left with a page that looks like this:
This is great for the moment, but what is the page going to look like a week from now? VisualPing will help me with that. I make a note of the URL ( https://www.kickstarter.com/discover/advanced?term=archive&sort=newest&page=1 ) and take that to my VisualPing home page.
Setting Up a Page to Monitor in VisualPing
From the VisualPing home page I’m going to click on the Create New Job link toward the top of the home page.
The new job page asks for some basic information: the URL you want to monitor, the address you want updates sent to, and how often you want to monitor the page.
Once you’ve put in an URL and clicked GO, VisualPing will load the page and give you an overlay that looks like this:
The bright part of the page is the part that VisualPing will monitor. (This is the “visual” option; we’ll take a look at another option in a moment.) The dimmer part is the part that VisualPing will ignore when it’s looking for page updates. You can click and drag the corners to make that space larger or smaller. In this case I’ll just monitor the part of the page that lists the three most recent projects, as I know new projects will show up at the beginning of the list.
In this case, I see that Kickstarters involving archives aren’t exactly super-hot. And this search results page doesn’t look like it has enough activity to be checked daily, so I’ll check it every week. (VisualPing charges not by time periods but how often it checks pages for you, so if you check pages too often you could end up wasting money. More about VisualPing pricing later in the article.)
But hold on; you’re not quite done yet. There’s an advanced option for your Web monitoring job. Let’s take a quick look.
VisualPing gives you the option to compare the Visual version of the site (what it looks like) to the Web version of its site (what the page code itself shows.) VisualPing recommends that if you’re getting too many false positives to switch to the Web version of the page comparison. (I find that if I’m careful about what parts of the pages I monitor, I don’t have too many false positives.) You can read more about the two kinds of comparisons at https://visualping.io/faq#webcompare .
The advanced search also lets you specify a lag time after page load before a screen shot is taken (it can be up to five seconds), whether or not to use a proxy (if the page won’t accept connections from VisualPing) and the kind of change VisualPing should be monitoring for (from Tiny to Major. I leave it on tiny unless I’m getting false alarms for things like number changes.)
When VisualPing finds a change to a monitored page, it will email you with screenshots of the page pre- and post- detected change. But it also keeps a record of changes. Let’s look at one of the pages I monitor: the Delaware Public Archives page, at http://archives.delaware.gov/news/index.shtml .
VisualPing shows me the history of page changes at a glance, along with a thumbnail so I can really get a glance. I can make sure the page is being checked regularly, and review my settings for this page. (In this case the change history is telling me clearly I do not need to be checking this page once a day. I’m going to roll it back to once a week.) From this view I can also do some more advanced adjustments to the monitoring (like specifying where alerts should go, and blocking certain page elements from the VisualPing checker) and change the section of the page being checked.
Now, why did I roll the page check back to once a week? Because as I noted before, VisualPing pricing is based on the number of checks you do, not by a specified time period. Did you see this bit at the top of my dashboard?
VisualPing’s pricing ( https://visualping.io/pricing ) is based on how many checks a day / week / month you think you’ll need. I’m in the “Intensive” tier, which looks like a lot until you realize that I’m monitoring 124 pages and will probably add more. I try to preserve my checks by reviewing change histories and changing daily checks to weekly checks if some pages don’t look like they change much.
It Just Works
Setting up a job with VisualPing is deceptively easy; provide a page URL, select the area of the page you want monitored, choose a frequency, and away you go. But once you get under the hood a bit or look at the information on the dashboard, you’ll discover that VisualPing is aggregating a lot of data and making it easy to keep tabs on Web page changes. It’s currently my favorite way to monitor sites which don’t have an RSS feed.
