Feedsby Internet Solutions

RSS Feed Creator Beyond Page One: Whole-Section Feeds

November 14, 20267 min readRSS feeds
RSS Feed Creator Beyond Page One: Whole-Section Feeds

Short answer: A list page usually shows only the latest ten or twelve items, but RSS Feed Creator does not stop there when your plan has room for more. It follows the “next page” links of the list, and where the website has a sitemap it adds further addresses from the same section, newest first, reading each item’s title, description, image and date from the page itself. The result is a feed that represents a section of the website, not just its front-page teasers. The number of items you actually get is capped by your plan: 20 on the free plan and 100 on paid plans.

Why the first page is not enough

Think about what you want from a feed. For a news page you may want only the newest stories, and page one is enough. But there are plenty of cases where you need more: a researcher who wants the last three months of announcements from an agency, a shop owner who wants every item of a supplier’s news section, or a team that wants a feed that is not empty just because the site published two things this week.

A list page also hides items. Blogs paginate, shops paginate, job boards paginate. If a tool only reads the first page, your feed can look thin or change erratically when a site reorders its front page. RSS Feed Creator was built to treat “the website” as the unit, not “the visible list”. That is the idea behind how RSS Feed Creator finds items on pages without RSS, extended to more than one page.

Step one: follow the pagination

After reading the first page, the tool looks for the next page of the list. It recognises the usual signs: a “next” link declared in the page, page addresses of the form /page/2/ and addresses with a page number in the query string. For each further page it repeats the detection, adds the items that are new and carries on until one of these happens:

Items are matched by their address, so a story that appears on two pages is counted once. This protects the feed from duplicates and from pages that repeat a “featured” block on every page.

Step two: use the section’s sitemap

When pagination does not provide enough items and the website has a sitemap, RSS Feed Creator uses it. If the address you entered is a section, such as /news/ or /blog/, it takes the addresses from the sitemap that belong to the same section and that are not already in the list. It orders them by the date the sitemap gives for each, newest first, and then reads each page’s own meta information: the title, the description, the picture and the publication date.

That is why well-built sites work so well. A page with a proper title, a description and a share image gives a feed item that looks like a real article. If a page lacks a date in its meta tags, the date from the sitemap is used instead. Details of pages are remembered for about a week, so a refresh does not read the same pages again and again.

For a comparison of the two techniques in general, see sitemap vs RSS: which is better for detecting new pages?. RSS Feed Creator uses both and gives you the result as one feed.

What your plan changes

The depth of a feed is limited by the plan, and that is easy to remember:

Plan Items per feed Refresh
Free Up to 20 Every 3 hours
PostRSS add-on (€5 per month) Up to 100 Every 30 minutes

On the free plan, you get the newest 20 items, which is plenty for monitoring. On a paid plan, up to 100 items are kept in each feed, and a section with a long history can fill it. The Store and Business plans are made for product feeds and have their own product limits; the full list is on the pricing page.

Choosing the right address

To get the most from a whole-section feed, paste the address of the section, not of the site. A section address tells the tool which part of the sitemap belongs to your list. For a blog, that is the blog index. For a job board, it is the vacancies list. For a supplier, it is the news or announcements section.

The home page works as a source, but it mixes everything and offers little structure for the section logic to follow. Look at the preview, which shows how many items were found and lists the first of them. If the total is the limit of your plan, the section has at least that many items and the feed will be full. If the total is small, the site may simply be small. The step is covered in the RSS Feed Creator preview.

Whole-section feeds in practice

A few typical uses show the value:

A worked example

Imagine a regional agency that publishes announcements at agency.example.org/news/ and has no feed. The first page shows twelve items and a “next” link. You paste the section address, choose “A web page” and open the preview. The heading reports items found on the page, and the total is higher than twelve, because the tool has followed the next-page links. You scan the cards, see sensible headlines with dates, and create the feed.

On the free plan the feed holds the newest 20 of those items. After upgrading to the PostRSS add-on it can hold up to 100, and the next refresh fills the gap. You did not change the source address or build anything new, which is the point: the depth follows the plan, and the feed address stays the same. You can then add the address to a reader, merge it with other agencies’ feeds or send it to PostRSS.

Checking what you got

After creating the feed, the management page shows the number of items, the time of the last update and the latest entries with their dates. Compare the oldest visible date with what you expect. If the feed reaches back a few weeks, the whole-section logic has worked. If it holds only the first page, either your plan’s limit was reached early or the site offers no further pages or sitemap addresses.

It also helps to open two or three older items and confirm that the title and date on the card match the article. Items taken from the sitemap are only as good as the page’s own meta tags, so a spot check is cheap insurance. The rest of the work, such as keeping only the items with certain words, is done with the filters described in keyword filters in RSS Feed Creator.

Limits and good manners

No automatic method is perfect. Sitemaps are not always complete, and some pages carry little meta information, so an item may have a title but no picture. Sections built entirely with scripts offer nothing to read. Treat the feed as a very good draft of the section and use the preview and the management page to see what you really have.

RSS Feed Creator reads public pages at a measured pace and refreshes feeds on schedule: every 3 hours on the free plan and every 30 minutes on paid plans. The bot identifies itself as FeedsBot. If you operate the website that is being read, the best help you can give is a clean sitemap, proper meta tags and, if possible, an official feed.

Related reading

The bottom line

RSS Feed Creator reads beyond page one: it follows pagination, adds section pages from the sitemap and fills each item from the page’s own meta information. Paste a section address, check the preview and let your plan’s item limit decide how deep the feed goes.

FAQ

Does the tool always read more than the first page?

It does when your plan has room for more items and the site offers next-page links or a sitemap. The plan’s item limit stops it.

How many items can one feed hold?

Up to 20 on the free plan and up to 100 on paid plans.

Which address should I paste?

Paste the address of the section that lists your items, such as a blog or news index, rather than the home page.

Why might an item have no picture or date?

Items added from the sitemap take their details from the page’s meta tags. If the page has none, the item is saved without them, although the sitemap date is used when the page has no date.

Are duplicates removed?

Yes. Items are matched by their address, so the same page is counted once.

#Media Monitoring#Page-to-Feed#RSS Feed Generator#Website Monitoring
Create your first feed — free.A feed from any page. Every product in every catalog.
Start free

More from the blog

All articles →
Internet Solutions

More from our team

Built by Internet Solutions. Try the rest of our products — each one saves you time in a different way.

internet-solutions.net ↗