> ## Documentation Index
> Fetch the complete documentation index at: https://www.ayrshare.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Meta Media Crawler Blocked (Instagram / Threads)

> How to fix error code 479 (media could not be fetched — the dedicated non-retryable crawler-block code), the transient code 440, and related Instagram 138 / Threads 379, caused by robots.txt or bot rules blocking Meta's media crawler.

Error code **479** is the dedicated, **non-retryable** Ayrshare code returned when Meta cannot fetch your media URL — most commonly because `robots.txt` or a bot-blocking rule on your server is denying the crawler (`facebookexternalhit`). It is returned only after Ayrshare's automatic re-host fallback has also failed, which strongly indicates a host-side block or an unreachable source.

When Ayrshare publishes to Instagram or Threads, it does not blindly hand your URL to Meta and give up. Ayrshare automatically retries the fetch and, if Meta still cannot reach the media, **re-hosts a copy on its own CDN** and asks Meta to fetch that instead. Code **479** means even that fallback could not get Meta to fetch the media — so retrying the same request will not help until the underlying hosting issue is fixed (`retryAvailable: false`, HTTP 400).

<Note>
  This page covers failures whose error message or details mention that the social network could not download the media, typically referencing `facebookexternalhit`, `robots.txt`, `"Restricted by robots.txt"`, `"HTTP error code 403"`, or Meta subcode `2207052`. For aspect-ratio or format errors on Instagram code 138, see [Instagram Media Guidelines](/docs/media-guidelines/instagram) or [Threads Media Guidelines](/docs/media-guidelines/threads) instead.
</Note>

## Symptom

When the crawler is blocked, you'll see errors like these:

```json Error 479 (primary — dedicated, non-retryable) theme={"system"}
{
  "status": "error",
  "errors": [{
    "action": "post",
    "code": 479,
    "retryAvailable": false,
    "message": "The social network could not fetch the media from this URL, even after Ayrshare re-hosted it on its own CDN (Instagram/Meta subcode 2207052). Ensure the file is publicly reachable by Meta's crawlers (facebookexternalhit / Facebot) — check your robots.txt and any WAF/bot rules — not only in a browser. Retrying the same URL will not help until hosting is fixed.",
    "details": "Media download has failed.: The media could not be fetched from the provided URI. Restricted by robots.txt (HTTP error code 403). Meta subcode 2207052.",
    "platform": "instagram",
    "status": "error"
  }],
  "postIds": [],
  "id": "..."
}
```

```json Error 440 (transient ingestion — retryable) theme={"system"}
{
  "status": "error",
  "errors": [{
    "action": "post",
    "code": 440,
    "retryAvailable": true,
    "message": "The social network could not ingest the media in time. This is usually transient (for example Meta subcode 2207032 'download too slow' or 2207003 'create media fail'). Retry the post.",
    "details": "Media download has failed.: Media download took too long / could not create media...",
    "platform": "instagram",
    "status": "error"
  }],
  "postIds": [],
  "id": "..."
}
```

```json Instagram Error 138 (fallback — less specific upstream response) theme={"system"}
{
  "status": "error",
  "errors": [{
    "retryAvailable": true,
    "status": "error",
    "code": 138,
    "details": "Media download has failed.: The media could not be fetched from the provided URI. Video download failed with: HTTP error code 403. Restricted by robots.txt",
    "action": "post",
    "platform": "instagram",
    "message": "Instagram Error: Instagram cannot process your post at this time. Please try your post again."
  }],
  "postIds": [],
  "id": "..."
}
```

```json Threads Error 379 theme={"system"}
{
  "status": "error",
  "errors": [{
    "status": "error",
    "code": 379,
    "message": "Error posting to Threads.",
    "action": "post",
    "platform": "threads"
  }],
  "postIds": [],
  "id": "..."
}
```

<Note>
  Understanding the split between these codes:

  <ul class="custom-bullets">
    <li>**Code 479 — dedicated media-fetch / crawler-block (non-retryable).** This is the code for the failure covered by this page. It is returned with `retryAvailable: false` when Meta could not fetch the media (subcode `2207052` or a media-fetch text pattern) **even after Ayrshare re-hosted it on its own CDN**. Its message explicitly names `facebookexternalhit` / `Facebot` and `robots.txt` — if you see 479, you're on the right page. Fix hosting before retrying.</li>
    <li>**Code 440 — transient ingestion (retryable).** Now used for genuinely transient ingestion exhaustion — Meta subcodes `2207032` ("download too slow") / `2207003` ("create media fail"), or unclassified exhaustion. It carries `retryAvailable: true`; retry via [`/post/retry`](/docs/apis/post/retry-post).</li>
    <li>**Code 138 — less-specific Instagram fallback.** Emitted for the same root cause when the upstream response is less specific. 138 is also used for aspect-ratio / format issues, so the media-fetch variant is identifiable by `"Restricted by robots.txt"` or `"HTTP error code 403"` in `details`.</li>
    <li>**Code 379 — Threads.** Does not include a `details` field. If Threads fails alongside an Instagram 479 or 138, the root cause is typically the same.</li>
  </ul>
</Note>

## Why This Happens

When you publish to Instagram or Threads via Ayrshare, Meta's servers fetch your media from the URL you provide. This server-side fetch uses the `facebookexternalhit` User-Agent. If your server's `robots.txt` disallows this crawler — or a WAF/bot-protection rule blocks it — Meta cannot download the file and the publish fails.

Facebook Page publishing uses a different ingestion path, which is why the same `mediaUrl` may work for Facebook but fail for Instagram and Threads.

## Fix: Update Your robots.txt

### Recommended: Allow Meta explicitly, keep others open

Add these rules to your `robots.txt` file:

```txt robots.txt theme={"system"}
User-agent: facebookexternalhit
Allow: /

User-agent: *
Allow: /
```

This explicitly allows Meta's crawler while keeping your site open to other crawlers (Google, Bing, etc.).

### Advanced: Lock down to social publishers only

If you want to block most crawlers but allow social media platforms:

```txt robots.txt theme={"system"}
User-agent: facebookexternalhit
Allow: /

User-agent: Twitterbot
Allow: /

User-agent: LinkedInBot
Allow: /

User-agent: Pinterest
Allow: /

User-agent: *
Disallow: /
```

<Warning>
  Use a single `User-agent: *` block, placed at the end of the file. RFC 9309-compliant crawlers merge multiple wildcard groups into one, but not every parser in the wild is RFC-compliant — duplicate wildcard groups are a common source of rules being dropped or applied inconsistently.
</Warning>

## Verify Meta Can Fetch Your URL

Before retrying your post, verify that Meta's crawler can now access your media. Run this command, replacing `$URL` with your full media URL:

```bash theme={"system"}
curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" \
  -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" \
  "$URL"
```

<ul class="custom-bullets">
  <li>**Healthy response:** HTTP 200 or 206 with binary data in the body.</li>
  <li>**Blocked response:** HTTP 403 or an empty/HTML error page.</li>
</ul>

<Note>
  Per Meta's documentation, `robots.txt` changes may take up to 24 hours to propagate through Meta's crawler cache. If verification succeeds but your post still fails, wait and retry later.
</Note>

## If This Doesn't Fix It

If you've updated `robots.txt` and verified with the `curl` command but still see failures:

<ul class="custom-bullets">
  <li>**24-hour propagation delay** — Meta caches `robots.txt`. Wait up to 24 hours after making changes before retrying.</li>
  <li>**WAF or bot-fight rules** — Cloudflare Bot Fight Mode, AWS WAF managed bot rule groups, and similar services may block Meta's crawler IP ranges even if `robots.txt` allows it. Check your WAF logs and add an exception for `facebookexternalhit`.</li>
  <li>**Hotlink protection / Referer checks** — Some CDNs block requests from data-center IPs or without a valid `Referer` header. Whitelist Meta's crawler or disable hotlink protection for media paths.</li>
  <li>**Signed-URL / presigned-URL expiry** — If your media URL has an expiration timestamp (common with S3 presigned URLs), ensure it doesn't expire before Meta's crawler can fetch it. For scheduled posts, generate URLs that remain valid until well after the scheduled time.</li>
  <li>**Managed media hosting** — If you use a service like Cloudinary, Imgix, or similar where you cannot edit `robots.txt`, check their documentation for a Meta/Facebook crawler allow-list setting.</li>
</ul>

If none of these resolve the issue, [contact Ayrshare support](https://www.ayrshare.com/contact) and include:

* The failing `postId` from the error response
* The output of the `curl` verification command above
* Your `robots.txt` contents

## Retry a Failed Post

Retry behavior depends on which code you received:

<ul class="custom-bullets">
  <li>**Code 440 (transient, `retryAvailable: true`)** — the ingestion simply ran out of time or hit a temporary Meta error. Retry directly using the [Retry Post endpoint](/docs/apis/post/retry-post); no changes to your hosting are needed.</li>
  <li>**Code 479 (`retryAvailable: false`)** — Meta could not fetch the media even after Ayrshare re-hosted it on its own CDN, so retrying the unchanged blocked URL will fail again. **Fix hosting first**: allow Meta's crawlers (`facebookexternalhit` / `Facebot`) in `robots.txt` and your WAF, or re-host the media on a Meta-reachable CDN. Verify with the `curl` command above, then retry via the [Retry Post endpoint](/docs/apis/post/retry-post).</li>
</ul>

## See Also

<ul class="custom-bullets">
  <li>[Instagram API](/docs/apis/post/social-networks/instagram)</li>
  <li>[Threads API](/docs/apis/post/social-networks/threads)</li>
  <li>[Instagram Media Guidelines](/docs/media-guidelines/instagram)</li>
  <li>[Threads Media Guidelines](/docs/media-guidelines/threads)</li>
  <li>[Ayrshare Error Codes](/docs/errors/errors-ayrshare#media-fetch--crawler-access-errors)</li>
  <li>[Video Publishing Fails](/docs/help-center/technical-support/video_publishing_fails)</li>
</ul>