Getting 100,000s Pages Indexed Using the Instagram API

I created a new website, powered by the Instagram API.

It is a Instagram Online Web Viewer, an alternative web-based interface for a popular usually-mobile app. The website has around 5 to 10 million pages so far. I just wrote some PHP spider scripts that can spider more and more users, tags and photos, then I have a browser plugin to auto-refresh the web-page running the script every minute or so. Just low enough to stay within the API request limits. A CRON JOB would probably work too.

I wrote the spider scripts, then wrote the front end PHP code to output the pages and interlink all pages by displaying both related and random pages.

Once everything was working to a decent standard, I linked to the Tags and Users sections from the homepage.

Within a week of adding these internal links, so Google could find the sub-sections with millions of content pages, they have indexed 200,000 of the pages, and organic traffic has jumped from 1 unique per day to 10 to 50 to 100+.

This is all with 0 backlinks. With some decent backlinks, the rankings should become more decent and traffic should rise exponentially further. Right now, there are lots of long tail rankings, rarely ranked high, but enough of them to drive some traffic.

I see some competing sites getting huge amounts of traffic. Though they have also accumulated 10s and 100s of thousands of links over time. They even manage to rank for things like ‘Kim Kardashian’.

So How Did I Get The Pages Indexed?

I didn’t do anything, except make several million pages findable. I didn’t even add any backlinks. The content is not quite unique, but is slightly so. It is API powered, so I did what I could.


0 responses so far ↓

  • There are no comments yet...Kick things off by filling out the form below.

Leave a Comment