# Shale Experts - https://www.shaleexperts.com/
#
# Editorial content (articles, news, plays) is meant to be crawled, indexed and
# shared. Account, admin and unfinished surfaces are not.
#
# Note on filtered/duplicate URLs: they are deliberately NOT disallowed here.
# Blocking a URL stops crawlers reading its rel=canonical, which is what
# actually consolidates the duplicates. Let them be crawled and let the
# canonical tag do the consolidating.
#
# WHEN A Disallow HERE IS THE RIGHT INSTRUMENT, because that rule has already
# been misapplied once (see the /store/ note below):
#
# yes the URL has no unique content of its own AND/OR requesting it mutates
# state (carts, checkout, logout, form posts), or it is a private or
# admin surface;
# yes the URL belongs to an UNBOUNDED space -- a catch-all route that answers
# 200 for every path anyone can invent -- and the pages in it carry no
# rel=canonical. Nothing consolidates what has no canonical, and an
# infinite soft-404 space cannot be crawled to exhaustion. This is the
# /store/ case below and it was missed the first time round;
# no the URL is a duplicate or near-duplicate of something indexable AND it
# carries a rel=canonical. Use the canonical, stay crawlable;
# no the URL is real content that simply should not rank. Use
# on a page that stays crawlable
# -- a blocked URL can still be indexed URL-only from an external link,
# and blocking guarantees the crawler never sees the noindex.
#
# Every status quoted in this file was measured anonymously against the local
# replica on 2026-07-27. Re-measure before trusting a claim; the last set of
# justifications here was wrong.
User-agent: *
#
# There is deliberately no `Allow: /` in this group, though there used to be.
# Absent any matching rule a crawler allows by default, so it bought nothing
# under RFC 9309 / Google / Bing (all of which resolve conflicts by LONGEST
# match) -- while under the older first-match semantics that Python's stdlib
# robotparser and various small crawlers still implement, a leading `Allow: /`
# matches every URL before any Disallow below is ever considered, silently
# turning this entire group into a no-op. Verified with both parsers: with
# `Allow: /` present, a first-match parser reported /administration/index,
# /store/checkout and /dev3/latestnews as all crawlable. Do not put it back.
#
# RELATEDLY, AND THIS IS WHY THE SEPARATORS BELOW ARE `#` AND NOT BLANK LINES:
# RFC 9309's grammar permits empty lines inside a group, but the older
# record-based parsers treat the first blank line as the end of the record and
# silently drop every rule after it -- a single blank line directly under
# `User-agent: *` discards the whole group. Comment lines do not have that
# effect (they are stripped after the blank-line test, then skipped). Measured
# on five variants; the only spelling that yields the intended answer under
# BOTH parser families is: no `Allow: /`, and no blank line between
# `User-agent: *` and the last rule of the group. Keep it that way.
#
# The paywall's destination page, and the one /user/ URL that must stay
# crawlable. Measured: /user/plans is 200, 38 KB, carries a self-referencing
# rel=canonical, and is a real "Plans & Pricing" page with plan names and
# prices. It is also where the paywall sends anonymous visitors: every one of
# the 11,915 gated articles 302s to /user/plans?utm_source=premium_article.
# Blocking the destination of a sitewide redirect is the worst available
# outcome -- Googlebot follows 11,915 redirects into a URL it is forbidden to
# fetch, and the highest commercial-intent page on the site can never rank.
# The Allow is longer than the Disallow below, so it wins in Google's and
# Bing's most-specific-rule matching, and the prefix covers the query string.
Allow: /user/plans
#
# Account, admin and transactional surfaces. The statuses are recorded because
# the previous version of this block justified it with "crawling them just
# burns budget on redirects to the login/plans pages" and that was false for
# most of them:
#
# 302 /user/index, /user/account, /user/accountedit, /user/changepassword,
# /user/renew -- the only ones the "redirects" claim was ever true of
# 200 /user/login (login form), /user/subscribe, /user/company_list. The
# last two are the same "Shale Experts is a Members ONLY resource"
# interstitial with the controller method interpolated into the copy
# ("Interested in the Subscribe?" / "Interested in the Company_list?"),
# i.e. duplicates of each other with no content of their own
# 200 /administration/index -- the admin login form
# 404 /administration/, /db_tools, /webServices, /user/profile
# 200 /tellaFriend (empty body), /maprequest (3.5 KB request form)
# 500 /bookmark
#
# None of these is a duplicate of an indexable page, so nothing here depends on
# a crawler reading a canonical tag. They are blocked under the first test at
# the top of the file, not in spite of the second.
#
# TRAILING SLASHES: mostly gone, on purpose. `Disallow: /maprequest/` does not
# match `/maprequest`, and `/maprequest/` 301s to `/maprequest`, so the slashed
# form blocked only the redirect and left the page itself crawlable. Same for
# /tellaFriend (200), /bookmark (500 -- a crawlable 500 is a server-health
# signal against the whole property), /db_tools, /webServices, /administration,
# /dev3 and /articles_preview. The bare prefix covers both spellings.
#
# /user is the exception and keeps its slash: `Disallow: /user` would also
# match /user_nav, which is routes.php:699 -> frontend/userNav, a 200 "Featured
# Plays" content page. The bare /user is handled with an end-anchor instead.
# `$` is RFC 9309 and is honoured by Google and Bing; the pre-RFC parsers read
# it as a literal and so match nothing, which leaves them able to fetch exactly
# one URL, /user, which 302s to /user/account -- blocked for them too. That is
# the whole cost of the one rule in this file that is not portable.
# (/bookmark-list, routes.php:706, is caught by `Disallow: /bookmark` -- that is
# intended: it 302s to /user/login.)
Disallow: /user/
Disallow: /user$
Disallow: /administration
Disallow: /db_tools
Disallow: /tellaFriend
Disallow: /bookmark
Disallow: /maprequest
Disallow: /webServices
#
# /searchservice is the Elasticsearch proxy behind the site's own search UI, and
# it is not a page. Measured anonymously 2026-08-20: GET /searchservice answers
# 200 with 11,238 bytes of raw ES response --
# {"took":6,"timed_out":false,...,"hits":{"total":935310,...,"_index":"slides_1786135191"...
# -- with no , no canonical and no meta robots. routes.php:46-71 maps
# thirteen routes under it and every one is machine surface: five `_search`
# endpoints, get_companies, get_categories, fetch_pres, authlevel, sign, login.
# `/searchservice/presentation?f=<filename>.pdf`, the gated PDF fetch, answers
# 307 to /user/login.
#
# This is the "no unique content of its own, or a private surface" case at the
# top of the file, and the crawl cost is not hypothetical. In one 30-minute
# production window Googlebot managed 33 requests in total -- roughly 1.1 a
# minute, against a 49,091-URL sitemap -- and spent SIXTEEN of them on
# /searchservice/presentation URLs that every single time redirected to the
# login page. Meanwhile 32,215 presentation pages sat in Search Console as
# "Discovered - currently not indexed" for want of exactly that budget. Blocking
# this reclaims about half of what Google currently spends here, and there is no
# downside: nothing under it can rank, and nothing links to it for readers.
#
# Bare prefix, per the trailing-slash note above: `/searchservice/` would not
# match `/searchservice`, and the unslashed form is the one that returns the
# JSON. There is no `/searchservice_*` route for the bare prefix to catch by
# accident (checked against routes.php), so the /user vs /user_nav problem does
# not arise here.
#
# NOT a security control. robots.txt is a crawling convention, and this endpoint
# remains publicly readable to anyone who requests it -- including the index
# names and document counts above. That it is exposed at all is a separate
# question from whether crawlers should spend budget on it.
Disallow: /searchservice
#
# /store/ is a SEPARATE CodeIgniter application (sites/store, served by
# `location /store` in docker/web/site.conf), which is why none of it carries
# the seo_helper.php markup the main site emits.
#
# THE DECIDING FACT, which an earlier link crawl of this app could not see:
# sites/store/application/config/routes.php ends in
# $route['(:any)'] = "products/index/show_cart_new/$1";
# and CI 2's (:any) is `.+`, which matches across slashes. So EVERY path under
# /store/ answers HTTP 200. Measured anonymously on the replica, 2026-07-27:
# /store/zzz-nonexistent-xyz 200, 13,763 bytes
# /store/a/b/c/d 200, 13,715 bytes
# That is an unbounded soft-404 space, and no /store/ page emits rel=canonical
# or meta robots at all (grep for either on /store/ and /store/pd/89: zero
# matches). The previous version of this block argued from "let the canonical
# tag consolidate the duplicates" -- an instrument that does not exist here --
# and from a crawl of 33 LINKED URLs, which says nothing about the infinite set
# of unlinked ones a crawler will find in logs, old links and referrer spam.
#
# So: a blanket Disallow, plus a narrow Allow list. The earlier reasoning was
# right about WHAT to keep crawlable and wrong about how to bound the rest, and
# both halves are preserved below.
#
# ORDER MATTERS. Google and Bing resolve Allow/Disallow conflicts by LONGEST
# match, so the order is irrelevant to them; the older first-match parsers take
# the first rule that matches, so the Allows must come FIRST or they never fire.
# Written this way both families reach the same answer.
#
# WHAT STAYS CRAWLABLE, and why:
# /store 200, 21,234 bytes. The catalogue landing page, linked from
# the main nav on every page of the site
# (views/includes/nav.php). Note `Disallow: /store/` does
# NOT match `/store` -- and that is the good outcome, since
# /store, /store/, /store/index.php and /store/products all
# return the identical 21,234-byte document with no
# canonical to consolidate them. Exactly one spelling stays
# crawlable; the other three are blocked.
# /store/pd/<id> the real product pages. Measured: /store/pd/89 200 15,749 B,
# /store/pd/149 200 15,824 B, /store/pd/168 200 16,089 B,
# /store/pd/188 200 15,618 B, /store/pd/190 200 15,682 B,
# /store/pd/205 200 15,990 B, /store/pd/208 200 15,762 B.
# Names, descriptions and prices -- the only pages on the
# site that describe what the business sells.
# /store/css/ render-blocking assets. Google renders before it ranks, so
# /store/js/ blocking a product page's stylesheet is not a neutral act.
# /store/images/ Referenced by sites/store/application/views (css, js,
# /store/fonts/ images) and from within screen.css (../images/, ../fonts/).
# These are static files served by `try_files $uri` from the
# `./sites/store:/var/store` mount, ahead of the catch-all;
# on disk: css/screen.css 68,852 B, js/jquery-1.6.min.js
# 91,671 B, fonts/fontawesome-webfont.woff 44,432 B. The
# replica cannot demonstrate that -- its web container has
# no /var/store mount at all, so these paths fall through to
# the catch-all and measure 200 text/html ~13.7 KB there.
# Stated as measured, and stated as a replica artefact.
# Crawlers do not enumerate asset directories; they fetch
# what a page references, which is a finite set.
#
# WHAT THE BLANKET NOW COVERS that the 13 targeted rules did not. Measured
# anonymously on the replica, 2026-07-27:
# /store/logout 302 -> /store/ unsets session userdata
# (the main site's /user/logout is blocked; this
# was not)
# /store/test_email 500 a crawlable error page
# /store/order_summary 500, 8,046 B
# /store/download 500
# /store/item_download 500
# /store/findState_shipping/TX 500
# /store/user/forgotpassword 200, 11,119 B password-reset surface.
# `Disallow: /user/` never matched /store/user/
# /store/user/resetPassword 200, 11,157 B
# /store/user/account 302 -> /store/index.php/
# /store/user/my-orders 302 -> /store/index.php/
# /store/user/order-detail 302 -> /store/index.php/
# /store/changepassword 302 -> /store/index.php/
# /store/account/edit 302 -> /store/index.php/
# /store/order_list 302 -> /store/
# /store/edit_billing 302 -> /store/index.php/register
# /store/edit_shipping 302 -> /store/index.php/register
# /store/update_billing 302 -> /store/index.php/register
# /store/update_shipping 302 -> /store/index.php/register
# /store/download/1 302 -> /store/
# /store/item_download/abc 200, 0 bytes an unauthenticated
# force_download wrapper
# /store/update_cart 302 -> /store/show_cart_view
# /store/remove_single_product/1 302 -> /store/
# /store/checkout 200, 21,070 B
# /store/checkout_guest 200, 59,972 B
# /store/checkout_process_guest 200, 59,803 B
# /store/processform 200, 63,089 B
# /store/processform_test 404, 1,235 B
# /store/register 200, 63,136 B
# /store/login 200, 13,721 B
# /store/login/1 200, 3,077 B
# /store/show_cart_view 200, 12,326 B
# /store/show_cart/1 200, 13,739 B
# /store/add_cart_item 200, 12,326 B
# /store/empty_cart 200, 10,441 B
# /store/continue_shopping 200, 97 B
# /store/product_description 200, 13,763 B (catch-all soft 404)
# /store/findState 200, 13,733 B (catch-all soft 404)
# /store/findState/TX 200, 187 B an AJAX fragment
# /store/search 200, 132,508 B search results
# /store/ 200, 21,234 B duplicate of /store
# /store/index.php 200, 21,234 B duplicate of /store
# /store/products 200, 21,234 B duplicate of /store
# /store/pd 200, 14,622 B product page with no id
# /store/play/2 200, 13,680 B empty category
# /store/Maps 200, 13,718 B empty category
# /store/application/... 403 (nginx)
# /store/system/... 403 (nginx)
# Not one of those is a page that should rank, and eight of them are 5xx, 302 or
# state-mutating. The 11 empty-category URLs (/store/Maps, /store/Services,
# /store/GIS-Data, /store/Digital-Data, /store/Drilling-Activity,
# /store/Drilling-Permits, /store/PDF/Digital-Maps, /store/Wall-Map,
# /store/play/{2,5,9,17,171}) are 13.7 KB each and indistinguishable from the
# catch-all soft 404; the standing fix is still a noindex on the empty-result
# branch in sites/store/application/views/includes/header.php, and when that
# ships they can be moved back out from behind the blanket. Blocking them costs
# nothing now, because with no canonical there is nothing for a crawl to
# consolidate.
#
# COST NOTE, unchanged and still true: every /store/ response sets 2-4 cookies
# and is served `Cache-Control: no-store`, so it can never enter the shared page
# cache and every crawl is a full CI + MySQL render. Worth paying for product
# pages, not for an infinite cart.
Allow: /store/pd/
Allow: /store/css/
Allow: /store/js/
Allow: /store/images/
Allow: /store/fonts/
Disallow: /store/
#
# Unfinished/duplicate article surfaces. /dev3 renders the same articles as
# /articles through an older template that does not apply the paywall, so it
# must never be crawled or indexed. (It is being removed/gated separately -
# this is defence in depth, not the fix.)
#
# This is the one place a Disallow is used on duplicates, and it survives the
# test at the top of the file on its own terms: /dev3/latestnews (200, 151 KB)
# and /dev3/epnews (200, 97 KB) emit a SELF-referencing rel=canonical, so
# crawling them would not consolidate anything -- the instrument the rule
# prefers is actively wrong here. The article leaf is now gated in code
# (dev3/controllers/articles.php:25 302s anonymous visitors to /user/plans),
# the listings are not.
#
# /articles_preview is the one rule in this file that the rule at the top does
# NOT justify, and it is left in place with eyes open. Measured:
# /articles_preview/<slug>_<id> returns 200 with the full 197 KB article body
# and a CROSS-canonical to the real /articles/ URL -- the tag is correct, so
# crawling really would consolidate it, and blocking really does stop Google
# reading it. It stays blocked only because it is a whole second copy of all
# 7,612 free articles on a path that does not match $se_cacheable_uri in
# docker/web/site.conf (`~^/articles/[^/?]+$`), so every one of those crawls
# would be an uncached full CI + MySQL render of a page that must never win.
# The actual fix is to delete the alias route (application/config/routes.php:488
# maps articles_preview/(:any)_(:any) onto the same controller), after which
# this line should go too. Until then this is a crawl-budget decision, not a
# consolidation one, and it should not be cited as precedent for blocking any
# other duplicate.
Disallow: /dev3
Disallow: /articles_preview
#
# ---------------------------------------------------------------------------
# CARRIED OVER FROM origin/dev's robots.txt IN THE MERGE. These were dev's
# rules, not this release's, and they are NOT re-measured to the standard the
# rest of this file holds itself to. Each is annotated with why it survived and
# what would have to be checked to keep or drop it.
# ---------------------------------------------------------------------------
#
# /mobile/ - the legacy mobile application is retired. docker/web/site.conf now
# answers `location ^~ /mobile` with a 301 onto the equivalent path on the
# responsive site, so every /mobile/* URL is a pure redirect with no content of
# its own. Passes the first test at the top of the file. Blocking it stops
# crawlers spending budget bouncing through the redirect; the destination URLs
# are all reachable and crawlable directly.
Disallow: /mobile/
#
# /resizeimage/ - the image transform endpoint. `?imgurl=<anything>` is an
# unbounded query space that runs a GD resize per request and can never enter
# the shared page cache, which is exactly the shape the second test at the top
# of the file is about.
#
# THE ONE THING TO CHECK BEFORE TRUSTING THIS: it blocks image RESOURCES, not
# pages, and application/helpers/my_helper.php:5216-5217 still emits
# `<img src="<base>resizeimage/?imgurl=...">` for ad creatives, with more
# callsites across plays/ and dev3/ views. Googlebot renders before it ranks, so
# a blocked <img> is a blocked render resource, and these URLs also stop being
# eligible for Google Images. That is judged acceptable here because the
# canonical pipeline is /img/ (image_helper.php calls the resizer a shim kept
# alive only for compatibility) and /img/ stays fully crawlable, and because
# AdSense's own crawlers match their more specific groups below and are
# unaffected by this line. If a measurement ever shows article hero images still
# resolving through /resizeimage/, drop this rule rather than the images.
Disallow: /resizeimage/
#
# Static/internal directories that exist on disk under /var/www but are not
# pages: sites/website/includes and sites/website/cabcache. Nothing links to
# them, nothing there carries a canonical, and a crawler that finds one gets
# either a raw asset or the CodeIgniter catch-all. /cache does not currently
# exist; the rule is harmless and forward-looking.
Disallow: /includes/
Disallow: /cabcache/
Disallow: /cache/
#
# The design-system page (sites/website/js/design-system.js and friends). A
# developer surface, not content. Bare prefix, no trailing slash, per the
# TRAILING SLASHES note above.
Disallow: /styleguide
#
# NOT CARRIED OVER: dev's `Disallow: /user/plans`. That rule predates the
# paywall work in this release and is now actively harmful - /user/plans is the
# 302 destination of all 11,915 gated articles. See the `Allow: /user/plans`
# block near the top of this group for the measurements. If you are reconciling
# this file against dev's, that omission is deliberate.
#
# Also not carried over: dev's `Disallow: /administration/` and
# `Disallow: /db_tools/`. The rules are here, but without the trailing slash,
# because the slashed form does not match the unslashed URL and both of those
# answer on the unslashed spelling. Same coverage, strictly wider.
# Google AdSense's crawler is a separate agent from Googlebot. It must be able
# to fetch any page carrying ad slots in order to serve contextually relevant
# ads; if it inherits the User-agent:* rules above it will report crawler
# errors and fall back to untargeted ads.
User-agent: Mediapartners-Google
Allow: /
# AdSense/Ad Manager fetcher for ad-serving decisions.
User-agent: AdsBot-Google
Allow: /
# YANDEX / SEMRUSH / AHREFS ARE NOW ALLOWED, and the blanket
# `User-agent: Yandex / Disallow: /` and `SemrushBot / Disallow: /` groups that
# stood here since the original file are gone.
#
# That is origin/dev's decision, taken with a stated reason, and this release
# had no counter-reason - it simply inherited the old blocks without examining
# them. Two things make it safe to adopt rather than merely accept:
#
# 1. It is paid for in infrastructure, not hope. docker/web/site.conf now runs
# a per-IP limit_req zone at 10 r/s specifically for scraper user-agents
# (bot|crawler|spider|slurp|yandex|semrush|ahrefs|mj12|dotbot), separate
# from the 30 r/s browser zone, answering 429 above that. A blanket
# Disallow was the only lever available before; it is no longer the only
# one, and a rate limit binds a crawler that ignores robots.txt too, which
# a Disallow never did.
# 2. The cost side changed underneath the old rule. Articles, /news, /epnews
# and /company/<slug> are served from the shared fastcgi cache now, so the
# marginal cost of a crawl of the pages these tools actually want is a
# cache HIT, not a full CodeIgniter + MySQL render.
#
# Note that the courted crawlers (Googlebot, Bingbot, AdsBot, Applebot ...) are
# deliberately NOT in that 10 r/s bucket - site.conf classifies them onto the
# browser key ahead of the generic pattern. Yandex and Semrush are.
#
# If this is ever reverted, revert the site.conf rate classes with it; the two
# were decided together.
# The site's own sitemap index. Production's robots.txt advertised a
# third-party Attracta sitemap that now resolves to an empty <sitemapindex>, and
# never referenced a first-party one at all.
#
# WHY /sitemap.php AND NOT /sitemap.xml, WHICH IS THE NICER URL
#
# Because this file and the config that serves /sitemap.xml do not deploy at the
# same time, and this file wins the race.
#
# Production CD is `git fetch && git reset --hard origin/master`, once a minute,
# with no container rebuild (CRON_JOBS.md:63-70). sites/website is bind-mounted
# (`./sites/website:/var/www`), so THIS file goes live within 60 seconds of a
# merge. The location block that serves /sitemap.xml lives in
# docker/web/site.conf, which is mounted as /etc/nginx/conf.d/site.template and
# turned into the running site.conf by an envsubst that only executes in the web
# container's start command (docker-compose.yml:44-95). The generated tree is
# `./ephemeral/sitemap:/var/sitemap`, also a mount, and ephemeral/ is gitignored
# so a git reset cannot create it. Both need the containers recreated. Nothing
# in the documented CD does that, so the window is not short -- it is open until
# somebody runs a deploy by hand.
#
# Measured in that exact state (sitemap location blocks removed from the running
# nginx, /var/sitemap absent from the php container, everything else current):
#
# /sitemap.xml HTTP 404, Content-Type: text/html, 98,201 bytes of the
# CodeIgniter 404 page. The advertised sitemap is a 98 KB HTML
# document.
# /sitemap.php HTTP 200, Content-Type: application/xml, X-SE-Sitemap:
# fallback -- sitemap.php rendering the tracked
# sitemap-fallback.tpl.xml for whichever host asked, which is a
# self-contained <urlset> of 12 section roots, not an index, so
# it cannot name a child that does not exist yet.
#
# /sitemap.php reaches PHP through the generic `~ \.php$` regex location, and
# sitemap.php itself needs nothing but files that git reset already delivered.
# That is what makes it the rung that answers 200 with valid XML in every
# intermediate state. (Verified by building a site.conf with the whole sitemap
# section deleted, reloading nginx and re-measuring -- not by assuming the regex
# location has always been there. If a future site.conf drops it, re-check.)
#
# One more reason the .xml URL is not merely equivalent-but-early: in the
# pre-recreate state the CHILD sitemaps 404 as HTML too (measured:
# /sitemaps/articles-free-001.xml -> HTTP 404, text/html, 98,421 bytes). An
# index whose children all 404 is worse than a small valid urlset, and only the
# /sitemap.php rung degrades to the latter.
#
# TO SWITCH TO /sitemap.xml, in a follow-up, in this order:
# 1. recreate the web and php containers so site.conf and the /var/sitemap
# mounts are live;
# 2. confirm on production: `curl -sI https://www.shaleexperts.com/sitemap.xml`
# returns 200 with `X-SE-Sitemap: generated`, and one child URL from the
# index returns 200 as well;
# 3. only then change the line below, and let it deploy on its own.
# Do not do 3 before 1. There is no ordering guarantee available to this file.
#
# Only one URL is advertised, whichever it is: Google caches robots.txt for up
# to 24 hours and pointing at both would submit the same URLs twice. Both serve
# the identical document with 200 and no redirect, so nothing is lost by
# naming the older one.
Sitemap: https://www.shaleexperts.com/sitemap.php
# Google News sitemap (key=news-sitemap). A SECOND Sitemap line, not a second
# spelling of the first: it is a different document with different content
# (only articles published in the last 48 hours, per the Google News schema),
# so the "only one URL" rule above does not apply to it. It is DYNAMIC -- a
# CodeIgniter route (routes.php -> modules/seo/controllers/News_sitemap.php),
# not part of the weekly static build, because a 48-hour sliding window cannot
# be served from a weekly file. Unlike /sitemap.xml it has no deploy-ordering
# problem: the route rides the same git checkout as this file and reaches PHP
# through `location /` -> index.php, which every intermediate nginx config
# already has. An empty news day answers 200 with a valid empty <urlset>,
# never a 404.
Sitemap: https://www.shaleexperts.com/news-sitemap.xml