boyang-hu/website-rebuild-skill

1. ⭐ **只取原始字节**:一切抓取走 `id_`(identity)回放旗——    `https://web.archive.org/web/<ts>id_/<原URL>` 返回捕获的原始字节,无改写、无工具条。

`https://web.archive.org/web/<ts>id_/<原URL>` 返回捕获的原始字节,无改写、无工具条。

删掉那条 `<link rel=preload href="…googletagmanager…">` 之后,`verify-offline` 的 class 1 变绿了。 **而真正的外联还在**:载荷里有个 `next/script` 描述符,水合时会注入 `<script src="https://www.googletagmanager.com/gtag/js?id=…">`,再由它拉起

**而真正的外联还在**:载荷里有个 `next/script` 描述符,水合时会注入 `<script src="https://www.googletagmanager.com/gtag/js?id=…">`,再由它拉起 `analytics.google.com` 与 `stats.g.doubleclick.net`。探针的 `external requests` 一直在报这三条。

`<script src="https://www.googletagmanager.com/gtag/js?id=…">`,再由它拉起

- **种子**：全部已知页面路由 + 已知关键资产路径（rogier 60 个初始路径；lando 从 `/` 爬 7 页并用 `/404-page-not-found` 探测出 404 模板）。 - **提取正则集**：对每个文本响应（HTML/JS/CSS/SVG/JSON）提取 `href/src/poster/content` 属性、CSS `url()`、动态 `import()`、`new Worker("...")`、`fetch("...")`、资产目录前缀

- **提取正则集**：对每个文本响应（HTML/JS/CSS/SVG/JSON）提取 `href/src/poster/content` 属性、CSS `url()`、动态 `import()`、`new Worker("...")`、`fetch("...")`、资产目录前缀字面量（`/assets|_astro|audio|content|fonts|images|models|workers/` 类）、按扩展名白名单匹配的绝对 URL【rogier】【noomo】。

- deck 深处资源（`about-us/process/step1..4.png` 等 6 个）靠解析 bundle 模板字面量补齐【kimi】。 - 基址变量拼接：lando 的 GL 资产基址 `vQ="https://lando.itsoffbrand.io/gl"`（4 GLB + 3 HDRI + 解码器 + MSDF 字体）与 Rive 基址 `mj=".../rive/"`（8 个 .riv）都是变量拼接，静态正则不可见——从 bundle 读出基址后枚举补

- 基址变量拼接：lando 的 GL 资产基址 `vQ="https://lando.itsoffbrand.io/gl"`（4 GLB + 3 HDRI + 解码器 + MSDF 字体）与 Rive 基址 `mj=".../rive/"`（8 个 .riv）都是变量拼接，静态正则不可见——从 bundle 读出基址后枚举补抓【lando】。

- [ ] **查询参数化的资产变换接口**：图片 CDN 把尺寸/裁剪/格式写在 query 里（`x.jpg?width=320|600|1200`、`?crop=center`、`&format=webp`），**同 pathname 不同字节**。按 pathname 落盘会让整组变体坍缩成一个文件，而下游零 404 门照样绿（§5.1）。做法：映射与落盘**查询感知**，并把"同 pathname 多变体"单独清点（objectandarchive：142 条 CD

- [ ] **不带尾斜杠的裸主机基址常量**：代码常写 `const B="https://cdn.example.com"`、`window.shopUrl='https://site.com'` 再拼路径；只匹配"带尾斜杠"形式的提取/改写规则对它天然失明——objectandarchive 实测因此漏了 4 个遥测外联 + **2 个到线上源站的主题资产请求（那份资产一直在盘上）**。同类还有 JSON 转义的协议相对写法 `\/\/host\/`。做法：提取与改写规

<a href="https://host/">https://host/</a>

- 图片：`https://cdn.sanity.io/images/<projectId>/<dataset>/<sha1-40>-<W>x<H>.<ext>[?w=&h=&fit=&auto=format…]`

- **必须 GET，禁止用 HEAD（`curl -sIL`）作唯一判据**：API Gateway/CloudFront 前端对 HEAD 返回假 404（kprverse：HEAD 404 / GET 200）【probe】。

- **最终 URL 同一性校验**：`final` 落点域 ≠ 目标域即 X 信号（darknetflix 301→netflix.com、umami-land 整域 301→google.com）【probe】。`curl -sIL` 表面 200 会掩盖 301 退役信号。

**【lamalama】D 信号第 1 条按字面命中、理由不成立（2026-09-06）**：目标 `https://lamalama.com/` 本身就是 WordPress 页——generator meta `WPML ver:4.9.6` + `WP Rocket 3.23.1.1`，`wp-content` 出现 1,306 次，robots 是 Yoast 模板。但步骤 2 双抓 **byte-identical**（431,958 B，WP Rocket 页面缓

实测：它抓到两侧本地化实现不一致（一侧留 `href="http://host"`、另一侧写出 `href=""`），而外壳字节门全绿

1. **D1a 同源绝对/协议相对 → 根相对**：`https://<host>/`、`http://<host>/`、`//<host>/` → `/`。**必须同时处理 JSON 转义形式** `https:\/\/<host>\/` → `\/`（内联 JSON-LD / 配置块里全是这种写法，漏了就留下真实外域引用）。**四种形态一个都不能少**：绝对 / 协议相对 / **转义绝对** `https:\/\/host\/` / **转义协议相对** `\/\/h

2. **D1b 外部 Shopify CDN / 其它外部主机 → 本地目录**：`https://cdn.shopify.com/` 与 `//cdn.shopify.com/` → `/cdn-shopify/`（含转义形式），对应镜像的 `assets/cdn.shopify.com/` 树。**转义形式对外部主机同样成立，别只给源站主机开**：objectandarchive 的 D1a 一开始只处理了源站主机的转义写法，外部主机漏掉，5 页共 25 处 `"inp

3. **D1c 裸主机基址常量 → 本地基址**【objectarchive】：遥测与主题代码常把基址写成**不带尾斜杠**的常量再拼路径（`"https://otlp-http-production.shopifysvc.com"`、`window.shopUrl='https://<host>'`）。只改写"带尾斜杠"形式时，objectandarchive 实测漏了 4 个遥测外联 + 2 个到线上源站的主题资产请求（那份资产一直在盘上）。**改写规则按主机匹配，不要

.../media/1f9dadf367424346-s.p.04       尾巴被切掉 https://host/static/media/9010da…       "/_next" 在上一个 push 里

https://host/static/media/9010da…       "/_next" 在上一个 push 里

https://host/9dc1a6fb114b646f-s.p…      整个路径前缀都在上一个 push 里

`https://cdn.jsdelivr.net/npm/${pkg}@${ver}/dist/dotlottie-player.wasm`

⛔ 静态扫描读到的是 `https://cdn.jsdelivr.net/npm/$` —— 到第一个 `${` 为止的前缀。

- **种子**：全部已知页面路由 + 已知关键资产路径。 - **提取正则集**：对每个文本响应（HTML/JS/CSS/SVG/JSON）提取 `href/src/poster/content` 属性、CSS `url()`、动态 `import()`、`new Worker("...")`、`fetch("...")`、资产目录前缀字面量（`/assets|_astro|audio|content|fonts|images|models|workers/` 类）、按扩展

- **提取正则集**：对每个文本响应（HTML/JS/CSS/SVG/JSON）提取 `href/src/poster/content` 属性、CSS `url()`、动态 `import()`、`new Worker("...")`、`fetch("...")`、资产目录前缀字面量（`/assets|_astro|audio|content|fonts|images|models|workers/` 类）、按扩展名白名单匹配的绝对 URL【rogier】【noomo】。

- [ ] **查询参数化的资产变换接口**：图片 CDN 把尺寸/裁剪/格式写在 query 里（`x.jpg?width=320|600|1200`、`?crop=center`、`&format=webp`），**同 pathname 不同字节**。按 pathname 落盘会让整组变体坍缩成一个文件，而下游零 404 门照样绿（§5.1）。做法：映射与落盘**查询感知**，并把"同 pathname 多变体"单独清点【objectarchive】 - [ ] **`s

- [ ] **不带尾斜杠的裸主机基址常量**：代码常写 `const B="https://cdn.example.com"`、`window.shopUrl='https://site.com'` 再拼路径；只匹配"带尾斜杠"形式的提取/改写规则对它天然失明。同类还有 JSON 转义的协议相对写法 `\/\/host\/`。做法：提取与改写规则覆盖**裸主机 / 带尾斜杠 / 协议相对 / JSON 转义**四种形态，且**探针要报完整 URL 而不只是 host 直方图

- ⭐ **同一个 403 有两种相反的药**：一族 CDN 缺 same-origin Referer 就 403（landonorris——于是爬虫带上了 Referer），另一族**带浏览器式请求头才 403、裸 curl 反而 200**（video.twimg.com,rauchg 实测）。单一请求头配置对其中一族永远是错的——`mirror-site.mjs` 的 get() 现在带**请求头梯子**：标准 profile 撞 401/403 时用最小 profi

⭐ **长度前缀行没有终止符。** 下一行的行首紧贴在声明的末尾——**长度本身就是分隔符**, 这正是它被写下来的全部理由。于是把 `https://media.host/x` 缩短成 `/ext/media.host/x`,

这正是它被写下来的全部理由。于是把 `https://media.host/x` 缩短成 `/ext/media.host/x`,

<link rel="preload" href="https://www.googletagmanager.com/gtag/js?id=…">

6. ⭐ **Worker 对象上空字段的 error 事件,第一嫌疑是脚本 URL 本身**:worker    脚本加载失败产生的 error 事件没有 message/filename/lineno,长得和"跨域    脱敏"一模一样——先 curl 一下 worker 脚本 URL 再谈别的(basement 实测:

脚本加载失败产生的 error 事件没有 message/filename/lineno,长得和"跨域    脱敏"一模一样——先 curl 一下 worker 脚本 URL 再谈别的(basement 实测:    构建重建吞掉软链 → 间歇 404 → 被误判为 draco 跨域错误登记了一轮)。

脱敏"一模一样——先 curl 一下 worker 脚本 URL 再谈别的(basement 实测:

- 图片：`https://cdn.sanity.io/images/<projectId>/<dataset>/<sha1-40>-<W>x<H>.<ext>[?w=&h=&fit=&auto=format…]`

（dataset 常为 `production`） - 文件/视频：`https://cdn.sanity.io/files/<projectId>/<dataset>/<sha1-40>.<ext>` - 内容 API：`https://<projectId>.api.sanity.io/v<日期>/data/query/<dataset>?query=<GROQ>`；

- 文件/视频：`https://cdn.sanity.io/files/<projectId>/<dataset>/<sha1-40>.<ext>`

UA='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36' TARGET='https://example.com/awarded-path'   # 必须是获奖/目标路径本身，不是只探根域

TARGET='https://example.com/awarded-path'   # 必须是获奖/目标路径本身，不是只探根域

curl -sL -A "$UA" -o probe/a.html \

- **必须 GET，禁止用 HEAD（`curl -sIL`）作唯一判据**：API Gateway/CloudFront 前端对 HEAD 返回假 404【probe】。

- **最终 URL 同一性校验**：`final` 落点域 ≠ 目标域即 X 信号【probe】。`curl -sIL` 表面 200 会掩盖 301 退役信号。

sleep 5 curl -sL -A "$UA" -o probe/b.html "$TARGET"

curl -sL -A "$UA" -o probe/b.html "$TARGET"

BUNDLE='https://example.com/assets/main.xxxx.js' curl -s -A "$UA" -o probe/bundle.js -w 'size=%{size_download}\n' "$BUNDLE"

curl -s -A "$UA" -o probe/bundle.js -w 'size=%{size_download}\n' "$BUNDLE"

# 响应 <1KB → 极可能是缺 Referer 的拒绝页（landonorris 返回 32 字节拒绝页造成假阴性），补齐 Referer 重试 [ "$(wc -c < probe/bundle.js)" -lt 1024 ] && curl -s -A "$UA" -e "${TARGET%/*}/" -o probe/bundle.js "$BUNDLE" # minification 形态预检（未混淆产物可跳过 beautify，省一道工）

[ "$(wc -c < probe/bundle.js)" -lt 1024 ] && curl -s -A "$UA" -e "${TARGET%/*}/" -o probe/bundle.js "$BUNDLE"

1. 存活性判定到**路径粒度**，且用 GET 不用 HEAD（API 网关对 HEAD 假 404）。 2. `curl -sIL` 表面 200 会掩盖 301 退役信号——必须校验**最终 URL 与目标主体同一性**。

2. `curl -sIL` 表面 200 会掩盖 301 退役信号——必须校验**最终 URL 与目标主体同一性**。

1. **D1a 同源绝对/协议相对 → 根相对**：`https://<host>/`、`http://<host>/`、`//<host>/` → `/`。**必须同时处理 JSON 转义形式** `https:\/\/<host>\/` → `\/`（内联 JSON-LD / 配置块里全是这种写法，漏了就留下真实外域引用）。**四种形态一个都不能少**：绝对 / 协议相对 / **转义绝对** `https:\/\/host\/` / **转义协议相对** `\/\/h

2. **D1b 外部 Shopify CDN / 其它外部主机 → 本地目录**：`https://cdn.shopify.com/` 与 `//cdn.shopify.com/` → `/cdn-shopify/`（含转义形式），对应镜像的 `assets/cdn.shopify.com/` 树。**转义形式对外部主机同样成立，别只给源站主机开**【objectarchive】（实证：`case-studies/shopify-platform.md` §2；为什么每一道

3. **D1c 裸主机基址常量 → 本地基址**【objectarchive】：遥测与主题代码常把基址写成**不带尾斜杠**的常量再拼路径（`"https://otlp-http-production.shopifysvc.com"`、`window.shopUrl='https://<host>'`）。**改写规则按主机匹配，不要求尾斜杠**（只匹配尾斜杠形式的漏网实证：`case-studies/shopify-platform.md` §2）；验收侧的配套要求见 `m

> **D1c 与"转义写法"是同族不同格，两格都要单独想过**：D1c 是**没有尾斜杠**（`"https://host"` + 代码自己拼路径），D1a/D1b 的转义形式是**斜杠被转义**（`https:\/\/host\/`）。两者都会让"只匹配 `https://host/`"的提取 / 改写 / 断言规则天然失明，且失明时的表现都是绿灯。断言面见 `verification-gates.md` §1.6（第 4 类给了可执行查法）。

<link rel="preconnect" href="https://shop.app" crossorigin="anonymous">

<link href="https://monorail-edge.shopifysvc.com" rel="dns-prefetch">

- **弃单 beacon**：`pagehide` 监听器 → `navigator.sendBeacon("https://monorail-edge.shopifysvc.com/v1/produce", {schema_id:"online_store_buyer_site_abandonment/1.1", …})`。它自带 guard「performance 条目里没有 monorail 记录才发」——**离线复刻恰好满足该条件，所以是必发不是可能发**。load

**③ 出站 `<a href>` 锚点** —— 如页脚的 `https://www.shopify.com/legal/privacy`。这是**源站内容**，按宪法第 3 条（源站有的都要有）**保留，不算外联**。断言脚本必须按元素类型判定，不能"字符串里含外部域即红"，否则会逼出删内容的错误修法。

2. **列主题资产**：`/cdn/shop/t/<N>/assets/` 下所有 js/css 全量下载；同时看是"单 bundle"还是"section 粒度多脚本"——这决定 M1 的坐标系粒度。**第三种可能：主题资产里根本没有承载签名行为的东西**（只有 vendor 与上游存量），行为全在页面内联块里 → 走 `reverse-engineering.md` §0.1 的**无 bundle 平行分支**建坐标系【objectarchive】。 3. **栈判定

4. **抄近路检查**：有 sourcemap 就先 curl `.map` 验证可取（simply-chocolate 的 432KB map 带完整 `sourcesContent`）；是 Dawn 或其 fork 就直接读 `github.com/Shopify/dawn`——这两种情况下 M1 的 beautify 环节近乎免费。**对 fork 站，读上游源码不只是抄近路，它同时是 §0.2 判据 1 的对照基准**（拿哪个版本对照由第 1 步的血统年代决定）【o

const ports = [5173, 5174, 5175];            // 另一块是 [5176, 5177]   (async () => {     for (const port of ports) { try { await import(`http://localhost:${port}/src/main.ts`); return; } catch {} }

(async () => {     for (const port of ports) { try { await import(`http://localhost:${port}/src/main.ts`); return; } catch {} }     console.warn('[Carousel3D] Vite dev server not found, falling back to asset');

for (const port of ports) { try { await import(`http://localhost:${port}/src/main.ts`); return; } catch {} }

| **2 内联自包含遥测** | 遥测实现整段内联在 HTML 里，不依赖被 stub 的外部脚本；`sendBeacon`/`fetch` 直打绝对外部域 | grep 构建产物里的外部绝对 URL 字符串，逐条判定是否可触发 | | **3 兜底路径外联** | 只在某脚本加载失败时才触发，正常跑不出现 | 读代码判定触发条件，不能只靠跑一遍 | | **4 转义的绝对写法**：`https:\/\/host\/…`、`\/\/host\/…`（JSON 字符串里的斜杠

| **4 转义的绝对写法**：`https:\/\/host\/…`、`\/\/host\/…`（JSON 字符串里的斜杠转义）【objectarchive】 | **双重隐身**：① 探针看不见——这类字节多半躺在 JSON 载荷 / 结构化数据里，宿主 DOM 不渲染就永远不发请求（甚至根本不是请求）；② **静态面也看不见**——`grep 'https://host'` 一条都不命中，于是"外部绝对 URL 清单"看起来是空的 | 按下面的三步查：**先按四种拼写 

**它与 `shopify-platform.md` §2 D1c 是同族不同格**：D1c 讲的是基址常量**没有尾斜杠**（`"https://host"` 再由代码拼路径），本格讲的是**斜杠被转义**（`https:\/\/host\/`）。两格的共同点是让"只匹配 `https://host/`"的提取 / 改写 / 断言规则天然失明，**且失明时的表现都是绿灯**。

- **dump-scene-graph.mjs（运行时场景图 dump 成数值账本）**——评估后不纳入：shopifydesign 那份是源站 bundle 里某个内部函数的逐字转写，换个站点连挂载点都不存在。范式（先 dump 源站数值再移植再数值验收）已由 `dump-timelines.mjs` 代表；需要时按目标站的引擎重写一份。 - **rogier 的 capture.mjs / analyze-home-bands.mjs**（行亮度剖面分析）——依赖 sh

- **rogier 的 capture.mjs / analyze-home-bands.mjs**（行亮度剖面分析）——依赖 sharp，违反零依赖哲学，未纳入；等价能力可用 `lib/png.mjs` + 自写剖面重做。 - **racingshop 的 gapfill.mjs（协议相对 URL 归一重解）**——评估后不纳入：它修的是爬虫把 `//host/path` 拼成 `https://origin//host/path` 的 bug，而 `mirror-sit

- **racingshop 的 gapfill.mjs（协议相对 URL 归一重解）**——评估后不纳入：它修的是爬虫把 `//host/path` 拼成 `https://origin//host/path` 的 bug，而 `mirror-site.mjs` 已在提取阶段就把协议相对 URL 归一成 `https://host/path`，根因不再产生，留着只会诱导别人跑一个针对不存在故障的补丁。若历史镜像里已有这类损坏条目，一次性重解那份 manifest 即可，不需

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

Executable file

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

// fingerprint.mjs — Step 0 指纹侦察（references/scope-and-fingerprint.md §2 的 // 六步 curl 协议）的跨平台等价实现。为无 POSIX 工具链的环境而写：Windows

// 六步 curl 协议）的跨平台等价实现。为无 POSIX 工具链的环境而写：Windows

//    的行都只是把 §3 的硬判据对应的证据摆到你面前，不替你下结论。 // //   node fingerprint.mjs --target https://example.com/awarded-path \

// //   node fingerprint.mjs --target https://example.com/awarded-path \ //        [--bundle https://example.com/assets/main.xxx.js[,more.js]] \

//   node fingerprint.mjs --target https://example.com/awarded-path \

//        [--bundle https://example.com/assets/main.xxx.js[,more.js]] \

// 中文规格（自 scripts/README.md 迁入，v0.3.21；本表另一拼写：`fingerprint.mjs`） // Step 0 六步探测协议的跨平台等价实现（GET 存活 + 重定向链与终点域同一性、双抓 diff、物种/年代、HTML 技术指纹、bundle 初检；出现次数计数与 <1KB Referer 重试内置；Sanity CMS 证据采集——projectId/dataset/API 主机/auto=format/_key，三种拼写归一，命中即

// Step 0 六步探测协议的跨平台等价实现（GET 存活 + 重定向链与终点域同一性、双抓 diff、物种/年代、HTML 技术指纹、bundle 初检；出现次数计数与 <1KB Referer 重试内置；Sanity CMS 证据采集——projectId/dataset/API 主机/auto=format/_key，三种拼写归一，命中即指路 sanity-platform.md）。**只采证据不出判级**——判级仍走 scope-and-fingerprint.m

// Step 0 指纹侦察（`references/scope-and-fingerprint.md` §2 六步 curl 协议）的跨平台等价——无 POSIX 工具链（Windows PowerShell 无 curl/cmp/fold/tr/perl）也能跑：GET 存活 + 手动重定向链与终点域同一性、双抓确定性 diff、物种/年代 grep、HTML 技术指纹（剥注释枚举 `<script src>`/内联 `import()`、框架模式×引擎范式标记，计数一律

// `node fingerprint.mjs --target https://example.com/awarded-path --bundle https://example.com/assets/main.xxx.js`

const t0 = performance.now();   for (let i = 0; i < 10; i++) {     const res = await fetch(cur, {

for (let i = 0; i < 10; i++) {     const res = await fetch(cur, {       redirect: "manual",

const res = await fetch(cur, {

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

*  * Usage:  *   node gapfill-video.mjs --master https://cdn.example.com/vp/<id>/<id>.m3u8

* Usage:  *   node gapfill-video.mjs --master https://cdn.example.com/vp/<id>/<id>.m3u8  *     [--out mirror]        mirror root (must match mirror-site.mjs)

*   node gapfill-video.mjs --master https://cdn.example.com/vp/<id>/<id>.m3u8

*     [--out mirror]        mirror root (must match mirror-site.mjs)  *     [--origin https://example.com]  same-origin path rule + Referer header;  *                                     defaults to the first master's origin

*     [--origin https://example.com]  same-origin path rule + Referer header;

* HLS/DASH 流媒体阶梯补录（master → rendition → 分片），静态爬虫的结构性盲区  * HLS 流媒体阶梯补录：master m3u8 → 递归取 variant/备用音轨/I-frame 播放列表 → 逐段下载 `.ts`/`.m4s`（含 EXT-X-MAP 初始化段、EXT-X-KEY）→ 追加进 manifest 账本。补的是静态爬虫的结构性盲区：HTML 里只有 master，其余全由播放器运行时 fetch，只有探针 404 才暴露（`

* HLS 流媒体阶梯补录：master m3u8 → 递归取 variant/备用音轨/I-frame 播放列表 → 逐段下载 `.ts`/`.m4s`（含 EXT-X-MAP 初始化段、EXT-X-KEY）→ 追加进 manifest 账本。补的是静态爬虫的结构性盲区：HTML 里只有 master，其余全由播放器运行时 fetch，只有探针 404 才暴露（`serve.mjs` 的 MIME 表已含 `.m3u8`/`.ts`/`.m4s`/`.mpd`，补录后即可本

* `node gapfill-video.mjs --master https://cdn.x.com/vp/<id>/<id>.m3u8 --origin https://example.com`（`--dry-run` 先看阶梯全貌）

if (!MASTERS.length) {   console.error(     'usage: gapfill-video.mjs --master https://cdn.example.com/vp/id/id.m3u8 [--out mirror]\n' +

console.error(     'usage: gapfill-video.mjs --master https://cdn.example.com/vp/id/id.m3u8 [--out mirror]\n' +       '       [--origin https://example.com] [--workers 4] [--delay 60] [--force] [--dry-run] [--manifest path]'

'usage: gapfill-video.mjs --master https://cdn.example.com/vp/id/id.m3u8 [--out mirror]\n' +

'       [--origin https://example.com] [--workers 4] [--delay 60] [--force] [--dry-run] [--manifest path]'

async function get(url) {   const res = await fetch(url, {

const res = await fetch(url, {

#!/usr/bin/env node

eval(s.js);

Executable file

Executable file

for (let i = 0; i < attempts; i++) {     try {       const res = await fetch(`http://127.0.0.1:${port}/json/version`, { signal: AbortSignal.timeout(2000) });

try {       const res = await fetch(`http://127.0.0.1:${port}/json/version`, { signal: AbortSignal.timeout(2000) });       const { webSocketDebuggerUrl } = await res.json();

const res = await fetch(`http://127.0.0.1:${port}/json/version`, { signal: AbortSignal.timeout(2000) });

#!/usr/bin/env node

// writing into it, so rm throws ENOTEMPTY and a passing run exits non-zero

Executable file

Executable file

// // Shapes covered, and why each one is here: //   1. absolute            https://host/path

// Shapes covered, and why each one is here: //   1. absolute            https://host/path //   2. protocol-relative   //host/path        (quoted or parenthesised)

//   1. absolute            https://host/path

* Build an extractor bound to one site.  *  *   origin      "https://example.com" (no trailing slash) — base for root-relative refs

*  *   origin      "https://example.com" (no trailing slash) — base for root-relative refs  *   originHost  hostname of the origin

*   origin      "https://example.com" (no trailing slash) — base for root-relative refs

// ⛔ A TEMPLATE PREFIX IS NOT AN ADDRESS. A URL assembled at runtime —     //     `https://cdn.jsdelivr.net/npm/${pkg}@${ver}/dist/x.wasm`

//     `https://cdn.jsdelivr.net/npm/${pkg}@${ver}/dist/x.wasm`

// strips manufactured 1,593 phantom `…quality(70` URLs (v0.1.68's paren       // lesson, third location). And the match can OVERRUN a closing paren the       // URL never opened: inline `style="…url(https://…x.webp);--aspect:…"`

// lesson, third location). And the match can OVERRUN a closing paren the       // URL never opened: inline `style="…url(https://…x.webp);--aspect:…"`       // yielded 98 phantom `x.webp);--aspect` URLs (fourth location) — the

// URL never opened: inline `style="…url(https://…x.webp);--aspect:…"`

// unconditional: sentence punctuation around a URL in prose, never path.       // ⚠ decodeEntities can INTRODUCE the very boundary chars the raw match       // excluded: `"image":"https://…x.webp",…` matches

// ⚠ decodeEntities can INTRODUCE the very boundary chars the raw match       // excluded: `"image":"https://…x.webp",…` matches       // straight through, and only the decode turns " back into `"`.

// excluded: `"image":"https://…x.webp",…` matches

// The (?!\/) guard is load-bearing: protocol-relative refs (//host/path)     // also start with "/", and without it they get joined onto ORIGIN as     // https://host//host/path — 77 phantom 404s on the first Shopify target

// also start with "/", and without it they get joined onto ORIGIN as     // https://host//host/path — 77 phantom 404s on the first Shopify target     // (racingshop-rebuild). Shape 2 already handled those.

// https://host//host/path — 77 phantom 404s on the first Shopify target

// 3b. root-relative paths as PLAIN STRING LITERALS, not in an attribute.     // Shape 3 requires src=/href=/poster=…, which is right for markup and blind     // to code: `new Workbox("/sw.js")`, `fetch("/data/index.json")`,

// Shape 3 requires src=/href=/poster=…, which is right for markup and blind     // to code: `new Workbox("/sw.js")`, `fetch("/data/index.json")`,     // `img.src = "/img/sprite.svg"` match nothing there.

// to code: `new Workbox("/sw.js")`, `fetch("/data/index.json")`,

//     //   .../media/1f9dadf367424346-s.p.04        (tail cut off)     //   https://host/static/media/9010da…        ("/_next" was in the push before)

//   .../media/1f9dadf367424346-s.p.04        (tail cut off)     //   https://host/static/media/9010da…        ("/_next" was in the push before)     //   https://host/9dc1a6fb114b646f-s.p…       (the whole path prefix was)

//   https://host/static/media/9010da…        ("/_next" was in the push before)

//   https://host/9dc1a6fb114b646f-s.p…       (the whole path prefix was)

Executable file

* ⭐ A length-prefixed row has NO terminator. The next row's header begins at  * the declared end — the length IS the separator, which is the whole reason it  * is written down. So shortening `https://media.host/x` to `/ext/media.host/x`

* the declared end — the length IS the separator, which is the whole reason it  * is written down. So shortening `https://media.host/x` to `/ext/media.host/x`  * inside a row leaves the reader consuming the next row's header as text, and

* is written down. So shortening `https://media.host/x` to `/ext/media.host/x`

Executable file

Executable file

Executable file

// 裸主机引用（含路径引用在内的全部出现）：flight 的 :HC preconnect 提示只写     // `"https://cdn.sanity.io"`，不带 /images/<projectId>/ 路径——首页可以零资产

// `"https://cdn.sanity.io"`，不带 /images/<projectId>/ 路径——首页可以零资产

for (const p of fetchProfiles(url, { origin, typeHint })) {     try {       const res = await fetch(url, { ...init, headers: { ...p.headers, ...(init.headers || {}) }, redirect });

try {       const res = await fetch(url, { ...init, headers: { ...p.headers, ...(init.headers || {}) }, redirect });       if (res.ok || (res.status >= 300 && res.status < 400)) return { res, profile: p.name, error: "" };

const res = await fetch(url, { ...init, headers: { ...p.headers, ...(init.headers || {}) }, redirect });

Executable file

Executable file

#!/usr/bin/env node

const get = async (p) => {     const c = AbortSignal.timeout(1500);     const res = await fetch(`http://127.0.0.1:${port}${p}`, { signal: c });

const c = AbortSignal.timeout(1500);     const res = await fetch(`http://127.0.0.1:${port}${p}`, { signal: c });     return res;

const res = await fetch(`http://127.0.0.1:${port}${p}`, { signal: c });

for (;;) {     try {       const list = await (await fetch(`http://127.0.0.1:${port}/json`, { signal: AbortSignal.timeout(2000) })).json();

try {       const list = await (await fetch(`http://127.0.0.1:${port}/json`, { signal: AbortSignal.timeout(2000) })).json();       seen = list.map((t) => `${t.type} ${t.url}`);

const list = await (await fetch(`http://127.0.0.1:${port}/json`, { signal: AbortSignal.timeout(2000) })).json();

async function warnOnForeignPid(port, pid, tool) {   try {     const v = await (await fetch(`http://127.0.0.1:${port}/json/version`, { signal: AbortSignal.timeout(2000) })).json();

try {     const v = await (await fetch(`http://127.0.0.1:${port}/json/version`, { signal: AbortSignal.timeout(2000) })).json();     const cdp = await connectCdp(v.webSocketDebuggerUrl, { defaultTimeoutMs: 3000 });

const v = await (await fetch(`http://127.0.0.1:${port}/json/version`, { signal: AbortSignal.timeout(2000) })).json();

try {     const u = new URL(IDENTITY_PATH, baseUrl);     const res = await fetch(u, { signal: AbortSignal.timeout(3000) });

const u = new URL(IDENTITY_PATH, baseUrl);     const res = await fetch(u, { signal: AbortSignal.timeout(3000) });     if (res.ok) return await res.json();

const res = await fetch(u, { signal: AbortSignal.timeout(3000) });

} catch {}   try {     const res = await fetch(baseUrl, { signal: AbortSignal.timeout(3000) });

[Credential-related source omitted]

const res = await fetch(baseUrl, { signal: AbortSignal.timeout(3000) });

Executable file

* targets" no matter which shapes fired — a difference-based count cannot work  * here, because localising an EXTERNAL host leaves the host string in place  * (`https://cdn.x` -> `/ext/cdn.x`).

* here, because localising an EXTERNAL host leaves the host string in place  * (`https://cdn.x` -> `/ext/cdn.x`).  */

* (`https://cdn.x` -> `/ext/cdn.x`).

// ⚠ `to || "/"` when NO path follows. Replacing `https://host` with "" for an

// origin host leaves `href=""`, which is not "the home page" — it is "this   // page", a silently broken link. Caught cross-side by the payload gate: the   // mirror kept `href="http://www.chungiyoo.com"` (serve.mjs only rewrites the

// page", a silently broken link. Caught cross-side by the payload gate: the   // mirror kept `href="http://www.chungiyoo.com"` (serve.mjs only rewrites the   // trailing-slash form) while the build produced `href=""`, so the two

// mirror kept `href="http://www.chungiyoo.com"` (serve.mjs only rewrites the

// site's own relaunch, which prints the address it is talking about:   //   //     <a href="https://host/">https://host/</a>

//   //     <a href="https://host/">https://host/</a>   //

//     <a href="https://host/">https://host/</a>

// ⛔ THESE GO THROUGH THE LENGTH-AWARE PATH TOO. It is not only localisation   // that edits a length-prefixed payload: the thing that had to be deleted here   // was a `<link rel="preload" href="https://www.googletagmanager.com/…">`

// that edits a length-prefixed payload: the thing that had to be deleted here   // was a `<link rel="preload" href="https://www.googletagmanager.com/…">`   // sitting INSIDE a flight row, and deleting it blanket-style shortens the row

// was a `<link rel="preload" href="https://www.googletagmanager.com/…">`

Executable file

Executable file

//   /cdn/shop/x.jpg                 ->  cdn/shop/x.jpg //   /collections/foo?page=2         ->  collections/foo@@page=2/index.html //   https://cdn.other.com/a.js?b=1  ->  assets/cdn.other.com/a@@b=1.js

//   /collections/foo?page=2         ->  collections/foo@@page=2/index.html //   https://cdn.other.com/a.js?b=1  ->  assets/cdn.other.com/a@@b=1.js //

//   https://cdn.other.com/a.js?b=1  ->  assets/cdn.other.com/a@@b=1.js

Executable file

#!/usr/bin/env node

*  * Usage:  *   node mirror-site.mjs --origin https://example.com [--out mirror]

* Usage:  *   node mirror-site.mjs --origin https://example.com [--out mirror]  *     [--hosts cdn.example.com,media.example.net]  extra asset hosts to follow

*   node mirror-site.mjs --origin https://example.com [--out mirror]

* BFS 爬虫镜像（资产白名单 + 迭代到不动点；`redirect:manual` + 三本账，含逐文件 sha256；⭐ `redirects.tsv` 与 manifest 一样**跨运行累积**——`--scope` 补页曾把它截成只剩表头，载荷门在 `/work` 撞 404 才发现）。`--scope <前缀>` 把**页面**队列限制在目标路径下（微站挂在企业 CMS 域下时必用；⛔ 只限页面不限资产）。**账本累积**（`--seeds` 补漏不再截短上一轮

* BFS 爬虫镜像源站（页面/跨域资产，文本资产迭代到不动点；对要求同源 Referer 的资产域补齐 Referer 头、404 模板探测）。**`redirect: "manual"` 硬纪律**：重定向只记进 `redirects.tsv` 并把目标重新入队，绝不把 301 的 body 写在来源路径下。产出三本账：`mirror-manifest.json`（含逐文件 sha256）、`inventory.tsv`、`redirects.tsv`，外加 `urlpa

* js|mjs|json|svg|html?` 扩展名白名单，`.atom`/`.xml`/`.rss`/`.txt` 与无扩展名路由整类不被打开，而闭包门用的是同一张表所以查不出来；`application/octet-stream` 按**没有声明**处理，它是"服务器不知道"不是"这是二进制"）。v0.3.16：绝对 `--out` 按给定路径用；`--rounds/--workers` 须为 ≥1 的整数（否则 exit 2）；三本账每 100 个文件与 Ctrl-

const ORIGIN_RAW = flag('origin', null); if (!ORIGIN_RAW) {   console.error('usage: mirror-site.mjs --origin https://example.com [--out mirror] [--hosts a,b] [--pages /x,/y] [--probe-404 /slug] [--seeds urls.txt] [--rounds 4] [--workers 8] 

if (!ORIGIN_RAW) {   console.error('usage: mirror-site.mjs --origin https://example.com [--out mirror] [--hosts a,b] [--pages /x,/y] [--probe-404 /slug] [--seeds urls.txt] [--rounds 4] [--workers 8] [--scope /path/] [--query-ignore v,cb | -

console.error('usage: mirror-site.mjs --origin https://example.com [--out mirror] [--hosts a,b] [--pages /x,/y] [--probe-404 /slug] [--seeds urls.txt] [--rounds 4] [--workers 8] [--scope /path/] [--query-ignore v,cb | --query-only width,hei

// refuses requests WITHOUT a same-origin Referer (landonorris), another // refuses requests WITH browser-shaped headers (video.twimg.com served a // bare curl and 403'd the polite profile — measured on rauchg). So a 4xx on

// refuses requests WITH browser-shaped headers (video.twimg.com served a // bare curl and 403'd the polite profile — measured on rauchg). So a 4xx on // the standard profile gets ONE retry on a minimal profile before the URL is

// bare curl and 403'd the polite profile — measured on rauchg). So a 4xx on

// Root-relative `href="/x"` AND absolute/protocol-relative same-origin // `href="https://host/x"` — WordPress and most CMS themes emit the latter for

// `href="https://host/x"` — WordPress and most CMS themes emit the latter for

const url = ORIGIN + (path === '/' ? '/' : path);     try {       const res = await fetch(url, { headers: { 'user-agent': BROWSER_UA }, redirect: 'manual' });

try {       const res = await fetch(url, { headers: { 'user-agent': BROWSER_UA }, redirect: 'manual' });       if (res.status >= 300 && res.status < 400) {

const res = await fetch(url, { headers: { 'user-agent': BROWSER_UA }, redirect: 'manual' });

#!/usr/bin/env node

#!/usr/bin/env node

// // Usage: //   node netcapture.mjs --origin https://example.com [--mirror mirror]

// Usage: //   node netcapture.mjs --origin https://example.com [--mirror mirror] //     [--routes /,/about,/contact]      routes to visit (default "/")

//   node netcapture.mjs --origin https://example.com [--mirror mirror]

// 真实浏览器 CDP 抓包对账补录运行时资源（**CDN 站必须传 `--hosts`**，否则只观测同源流量、会报假 GAP=0） // 真实浏览器 CDP 抓包，与磁盘镜像 diff 对账（HAVE/GAP），补运行时拼接 URL。**`--hosts` 在 CDN 站上不是可选项**：记录范围是 host 白名单（语义同 mirror-site 的 `ASSET_HOSTS`，把同一份 host 清单传给它）；不在白名单的 host 只计数并在末尾列出，未传 `--

// `node netcapture.mjs --origin https://example.com --hosts cdn.x.com --routes /,/about`

const ORIGIN_RAW = flag("origin", null); if (!ORIGIN_RAW) {   console.error("usage: netcapture.mjs --origin https://example.com [--mirror mirror] [--routes /,/a] [--viewports desktop,mobile] [--steps 12] [--dwell 1500] [--settle 9000] [--ho

if (!ORIGIN_RAW) {   console.error("usage: netcapture.mjs --origin https://example.com [--mirror mirror] [--routes /,/a] [--viewports desktop,mobile] [--steps 12] [--dwell 1500] [--settle 9000] [--hosts cdn.x.com,media.y.net] [--out file.ts

console.error("usage: netcapture.mjs --origin https://example.com [--mirror mirror] [--routes /,/a] [--viewports desktop,mobile] [--steps 12] [--dwell 1500] [--settle 9000] [--hosts cdn.x.com,media.y.net] [--out file.tsv] [--fetch] [--swift

// whole capture down with an uncaught TypeError — losing every route, not     // just the bad request. A capture tool must survive its subject.     // Field case: a tag manager built https://senses%20trackingscript.<host>/…

// just the bad request. A capture tool must survive its subject.     // Field case: a tag manager built https://senses%20trackingscript.<host>/…     // — a percent-encoded SPACE inside the hostname, 8x per session.

// Field case: a tag manager built https://senses%20trackingscript.<host>/…

const sitePath = inflight.get(p.requestId);     if (!sitePath) return;     requests.set(sitePath, {

if (!sitePath) return;     requests.set(sitePath, {       path: sitePath,

requests.set(sitePath, {

} else if (msg.method === "Network.loadingFinished") {     const sitePath = inflight.get(p.requestId);     const rec = sitePath && requests.get(sitePath);

const sitePath = inflight.get(p.requestId);     const rec = sitePath && requests.get(sitePath);     if (rec) rec.bytes = p.encodedDataLength || 0;

const rec = sitePath && requests.get(sitePath);

const rows = [...requests.values()].sort((a, b) => a.path.localeCompare(b.path));

// m.path is an absolute URL (records are keyed by host + path).     // ⛔ PER-URL TOLERANCE + PERIODIC LEDGERING. The first version had neither:     // one thrown fetch (DNS, TLS, reset) aborted the WHOLE loop before

// ⛔ PER-URL TOLERANCE + PERIODIC LEDGERING. The first version had neither:     // one thrown fetch (DNS, TLS, reset) aborted the WHOLE loop before     // appendLedger ever ran, stranding every file already written as

// one thrown fetch (DNS, TLS, reset) aborted the WHOLE loop before

#!/usr/bin/env node

#!/usr/bin/env node

// <out>/metric.json. // //   node pixelcompare.mjs --a http://localhost:5173/ --b http://localhost:5175/

// //   node pixelcompare.mjs --a http://localhost:5173/ --b http://localhost:5175/ //     [--name home]                    view name (keys metric.json + filenames)

//   node pixelcompare.mjs --a http://localhost:5173/ --b http://localhost:5175/

// ⛔ **冻 JS 时钟冻不住 CSS 动画**——`animation` 跑在浏览器动画时间线上，不经过 JS。症状是**同侧对照比跨侧还大**且最差格相同。该旗标把所有动画 `paused` + 固定负延迟钉在同一相位（⚠ 它改变被渲染内容，这正是目的：两侧定格在同一位置）。 // 双服务器 A/B 截图 + 64×40 网格量化（适合活体场景）+ 并排合成图 + metric.json；`--max-mean` 可作门。**开拍前先证明 A/B 是两个进程**：同 

// `node pixelcompare.mjs --a http://127.0.0.1:25002/ --b http://127.0.0.1:25001/ --name home`；1728×1080 加 `--format jpeg --quality 92`

await waitFor(async () => (await fetch(URL_A)).ok, 10000, 'server A ' + URL_A);

await waitFor(async () => (await fetch(URL_B)).ok, 10000, 'server B ' + URL_B);

Executable file

#!/usr/bin/env node

* CDP 无头探针：console/异常/网络采集 + Log 域监听（SRI 拦截盲区修复）、`--eval/--evalAfter/--shot/--mobile`、CLEAN 判定退出码进 CI。`--no-external` 把"任何离开本服务 origin 的请求"记为失败（断网门要求的零外联，此前无人断言）；`--walk N` 全页滚动走查（`--scroll` 只跳单点，跳过的场景根本不挂载）。调试端口按 side 分配（side 从目标 URL 的端口自动

* 逐参数走 argv，旗标可以在 URL 前面  * `node probe.mjs http://127.0.0.1:25001/ --shot out.png --walk 24 --no-external`；大视口截图 `--shot out.jpg --format jpeg --quality 92`  */

* `node probe.mjs http://127.0.0.1:25001/ --shot out.png --walk 24 --no-external`；大视口截图 `--shot out.jpg --format jpeg --quality 92`

// the next argument. // ⛔ WALK THE ARGV; do not `find` the first token without dashes. With a flag // ahead of the URL (`--wait 9000 http://…`) that token is the flag's VALUE, and

// ⛔ WALK THE ARGV; do not `find` the first token without dashes. With a flag // ahead of the URL (`--wait 9000 http://…`) that token is the flag's VALUE, and // the probe died on `new URL('9000')` with a bare TypeError — after the flag

// ahead of the URL (`--wait 9000 http://…`) that token is the flag's VALUE, and

case 'Network.requestWillBeSent': {       const u = m.params.request.url;       requests.set(m.params.requestId, u);

requests.set(m.params.requestId, u);

}     case 'Network.loadingFailed': {       const u = requests.get(m.params.requestId) || '?';

case 'Network.loadingFailed': {       const u = requests.get(m.params.requestId) || '?';       if (!m.params.canceled) failures.push(`FAILED ${m.params.errorText} ${u}`);

const u = requests.get(m.params.requestId) || '?';

Executable file

for (const p of profilesFor(url)) {     try {       const r = await fetch(url, { headers: p.headers, redirect: "manual" });

try {       const r = await fetch(url, { headers: p.headers, redirect: "manual" });       if (r.ok) {

const r = await fetch(url, { headers: p.headers, redirect: "manual" });

#!/usr/bin/env node

// 中文规格（自 scripts/README.md 迁入，v0.3.21；本表另一拼写：`serve.mjs`） // 零依赖静态服务器（MIME/Range/服务层改写/重定向回放），兼任源站参照服。`--rewrite FROM::TO` 是**登记式字面量替换**，为的是一类本地化触及不到的东西——**源程序按自己的域名分支**（`location.hostname=="x.com" && (CDN=...)`，镜像不在那个域名上于是整个子系统走空路径）；**首次命

// 零依赖静态服务器（MIME/Range/服务层改写/重定向回放），兼任源站参照服。`--rewrite FROM::TO` 是**登记式字面量替换**，为的是一类本地化触及不到的东西——**源程序按自己的域名分支**（`location.hostname=="x.com" && (CDN=...)`，镜像不在那个域名上于是整个子系统走空路径）；**首次命中打印**，因为沉默与生效此前无法区分。`--fallback-root` 让复刻侧只放产出、资产全部从只读镜像读（`

// 零依赖静态服务器：MIME 补全（含 HLS 阶梯与 `.mov`）、Range、redirects.tsv 重定向回放（FROM 写绝对 URL 或裸路径都能命中）、`/ext/<host>/` 服务层改写（镜像磁盘神圣不改）、`--stub-ext-hosts` 把"改写进 `/ext/` 但故意不镜像"的遥测 host 回 JS stub（否则要么真外联、要么 404）、`?__probe` 注入 probe-shim、404.html 回放。**`--side 

// project, or pass --stub-ext-hosts; register each one as a deviation. // The rewrite is what makes this work even for runtime-built URLs: a loader // that concatenates a "https://telemetry.example/tag/" literal with an id has

// The rewrite is what makes this work even for runtime-built URLs: a loader // that concatenates a "https://telemetry.example/tag/" literal with an id has // the literal rewritten in the JS response, so the built URL is redirected too,

// that concatenates a "https://telemetry.example/tag/" literal with an id has

// (`//example.com/cdn/...`, common on platforms that serve the store CDN under // the custom domain). Served from 127.0.0.1 those resolve to // http://example.com/..., i.e. THE OFFLINE MIRROR SILENTLY PHONES THE LIVE SITE

// the custom domain). Served from 127.0.0.1 those resolve to // http://example.com/..., i.e. THE OFFLINE MIRROR SILENTLY PHONES THE LIVE SITE // for bytes that are already on disk. Rewriting them to root-relative is a

// http://example.com/..., i.e. THE OFFLINE MIRROR SILENTLY PHONES THE LIVE SITE

// unhandled is an outbound request the render-level gates cannot show you: // //   plain          https://host/path        http://host/path

// //   plain          https://host/path        http://host/path //   protocol-rel   //host/path

//   plain          https://host/path        http://host/path

//   protocol-rel   //host/path //   JSON-escaped   https:\/\/host\/path  and  \/\/host\/path //   BARE CONSTANT  "https://host"           then code does host + "/v1/metrics"

//   JSON-escaped   https:\/\/host\/path  and  \/\/host\/path //   BARE CONSTANT  "https://host"           then code does host + "/v1/metrics" //

//   BARE CONSTANT  "https://host"           then code does host + "/v1/metrics"

// a theme asset URL inside an inline script; the bare constant is how telemetry // SDKs and platform boilerplate store their base URL (`window.shopUrl = // 'https://example.com'`). Measured on objectandarchive with only the

// SDKs and platform boilerplate store their base URL (`window.shopUrl = // 'https://example.com'`). Measured on objectandarchive with only the // trailing-slash forms handled: the last 6 outbound requests of an otherwise

// 'https://example.com'`). Measured on objectandarchive with only the

// //     let CDN_PATH = ""; //     window.location.hostname == "example.com" && (CDN_PATH = "https://cdn.example");

//     let CDN_PATH = ""; //     window.location.hostname == "example.com" && (CDN_PATH = "https://cdn.example"); //

//     window.location.hostname == "example.com" && (CDN_PATH = "https://cdn.example");

const eh = h.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");     text = text       .replace(new RegExp(`https?://([\\w.+-]+(?::[^@/\\s"']*)?)@${eh}/`, "gi"), (m, ui) => `http://${ui}@${HOST}:${PORT}/ext/${h}/`)

text = text       .replace(new RegExp(`https?://([\\w.+-]+(?::[^@/\\s"']*)?)@${eh}/`, "gi"), (m, ui) => `http://${ui}@${HOST}:${PORT}/ext/${h}/`)       .replace(new RegExp(`https?:\\\\/\\\\/([\\w.+-]+(?::[^@\\s"']*)?)@${eh}\\\\/`, "gi"), (m

.replace(new RegExp(`https?://([\\w.+-]+(?::[^@/\\s"']*)?)@${eh}/`, "gi"), (m, ui) => `http://${ui}@${HOST}:${PORT}/ext/${h}/`)

// DSN is the canonical case: `https://<key>@o3794….ingest.us.sentry.io/…`   // sits in a chunk, the stub host is listed, and the request still went out —   // because `https://host/` never occurs in the text; `https://key@host/` does.

// sits in a chunk, the stub host is listed, and the request still went out —   // because `https://host/` never occurs in the text; `https://key@host/` does.   // Normalize the userinfo away for LISTED hosts only (plain and \/-escaped

// because `https://host/` never occurs in the text; `https://key@host/` does.

}   for (const h of ORIGIN_HOSTS) {     text = text.replaceAll(`https://${h}/`, "/").replaceAll(`http://${h}/`, "/");

text = text.replaceAll(`https://${h}/`, "/").replaceAll(`http://${h}/`, "/");

// The NO-PATH form: `https://host` with nothing after it means the home

}   for (const h of EXT_HOSTS) {     text = text.replaceAll(`https://${h}/`, `/ext/${h}/`).replaceAll(`http://${h}/`, `/ext/${h}/`);

for (const h of EXT_HOSTS) {     text = text.replaceAll(`https://${h}/`, `/ext/${h}/`).replaceAll(`http://${h}/`, `/ext/${h}/`);     text = text

text = text.replaceAll(`https://${h}/`, `/ext/${h}/`).replaceAll(`http://${h}/`, `/ext/${h}/`);

// (`x@@dpl=…<U+200B>.png`) because that is the URL it was fetched by, so an exact

const server = http.createServer(async (req, res) => {   try {     const url = new URL(req.url, `http://${req.headers.host || "localhost"}`);

const url = new URL(req.url, `http://${req.headers.host || "localhost"}`);

to = EXT_HOSTS.includes(u.hostname)           ? `/ext/${u.hostname}${u.pathname}${u.search}`           : `http://${req.headers.host || "localhost"}${u.pathname}${u.search}${u.hash}`;

? `/ext/${u.hostname}${u.pathname}${u.search}           : `http://${req.headers.host || "localhost"}${u.pathname}${u.search}${u.hash}`;

: `http://${req.headers.host || "localhost"}${u.pathname}${u.search}${u.hash}`;

server.listen(PORT, HOST, () => {   console.log(`serving ${ROOT}  [side ${IDENTITY.side.toUpperCase()}]`);   console.log(`  http://${HOST}:${PORT}/`);

console.log(`serving ${ROOT}  [side ${IDENTITY.side.toUpperCase()}]`);   console.log(`  http://${HOST}:${PORT}/`);   console.log(`  port ${labelPort(PORT)}`);

console.log(`  http://${HOST}:${PORT}/`);

Executable file

#!/usr/bin/env node

#!/usr/bin/env node

import { readFile, writeFile, mkdir, rm } from "node:fs/promises";

#!/usr/bin/env node

Executable file

#!/usr/bin/env node

* probe --no-external.  *  *   node scripts/sweep-routes.mjs --base http://127.0.0.1:6571 --pages docs/pages.json

*  *   node scripts/sweep-routes.mjs --base http://127.0.0.1:6571 --pages docs/pages.json  *        [--wait 6000] [--interact "<js, runs after wait>"] [--interact-wait 4000]

*   node scripts/sweep-routes.mjs --base http://127.0.0.1:6571 --pages docs/pages.json

case "Network.requestWillBeSent": {       const u = m.params.request.url;       requests.set(m.params.requestId, u);

const u = m.params.request.url;       requests.set(m.params.requestId, u);       if (/^https?:/.test(u) && new URL(u).origin !== SELF_ORIGIN) {

requests.set(m.params.requestId, u);

}     case "Network.loadingFailed": {       const u = requests.get(m.params.requestId) || "?";

case "Network.loadingFailed": {       const u = requests.get(m.params.requestId) || "?";       if (!m.params.canceled) {

const u = requests.get(m.params.requestId) || "?";

for (const route of routes) {   pageErrors = []; failures = []; allowedFailures = []; lifecycle = []; navigations = 0;   requests.clear(); external.clear(); allowedExternal.clear();

requests.clear(); external.clear(); allowedExternal.clear();

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

import { readFile, mkdtemp, rm } from "node:fs/promises";

#!/usr/bin/env node

#!/usr/bin/env node

*     <id>:T<hex>,<exactly that many UTF-8 BYTES of text>  *  * Rewriting `https://media.host/x` to `/ext/media.host/x` inside such a row

*  * Rewriting `https://media.host/x` to `/ext/media.host/x` inside such a row  * shortens the text while `T<hex>` still claims the old count. The reader takes

* Rewriting `https://media.host/x` to `/ext/media.host/x` inside such a row

*  *   node scripts/verify-lenprefix.mjs --dir site  *   node scripts/verify-lenprefix.mjs --base http://127.0.0.1:8081 --routes /,/careers

*   node scripts/verify-lenprefix.mjs --dir site  *   node scripts/verify-lenprefix.mjs --base http://127.0.0.1:8081 --routes /,/careers  *

*   node scripts/verify-lenprefix.mjs --base http://127.0.0.1:8081 --routes /,/careers

const targets = []; if (DIR) for await (const f of htmlFiles(path.resolve(DIR))) targets.push({ label: path.relative(path.resolve(DIR), f), get: () => readFile(f, "utf8") }); if (BASE) for (const r of ROUTES) targets.push({ label: r, get: (

if (BASE) for (const r of ROUTES) targets.push({ label: r, get: () => fetch(new URL(r, BASE)).then((x) => x.text()) });

#!/usr/bin/env node

*   node verify-mirror.mjs --mirror mirror --resample 8 --resample-delay 2000  *  *   [--origin https://example.com]  default: the manifest's own `origin`

*  *   [--origin https://example.com]  default: the manifest's own `origin`  *   [--hosts a,b]                   extra hosts for the closure pass (default:

*   [--origin https://example.com]  default: the manifest's own `origin`

const ORIGIN = (flag("origin", manifest.origin || "") || "").replace(/\/+$/, ""); if (!ORIGIN) {   console.error("FATAL: no origin — pass --origin https://example.com (the manifest has none).");

if (!ORIGIN) {   console.error("FATAL: no origin — pass --origin https://example.com (the manifest has none).");   process.exit(2);

console.error("FATAL: no origin — pass --origin https://example.com (the manifest has none).");

console.log(       `         An HTML body under an image / font / script URL is a refusal page, a login wall\n` +         `         or an SPA catch-all — not the asset. Re-fetch (Referer? cookies? rate limit?)\n` +

`         An HTML body under an image / font / script URL is a refusal page, a login wall\n` +         `         or an SPA catch-all — not the asset. Re-fetch (Referer? cookies? rate limit?)\n` +         `         or register the URL as unf

`         or an SPA catch-all — not the asset. Re-fetch (Referer? cookies? rate limit?)\n` +

//   // So an excuse now means exactly what it is written as:   //   host only  (`telemetry.example.net`, `https://telemetry.example.net/`)

// So an excuse now means exactly what it is written as:   //   host only  (`telemetry.example.net`, `https://telemetry.example.net/`)   //              -> the whole host is excused. That is what "accepted

//   host only  (`telemetry.example.net`, `https://telemetry.example.net/`)

//              -> the whole host is excused. That is what "accepted   //                 degradation / stubbed vendor" decides, at host scope.   //   full URL   (`https://cdn.example.com/a/8914/files`)

//                 degradation / stubbed vendor" decides, at host scope.   //   full URL   (`https://cdn.example.com/a/8914/files`)   //              -> EXACT. It excuses itself and nothing below it.

//   full URL   (`https://cdn.example.com/a/8914/files`)

//              -> EXACT. It excuses itself and nothing below it.   //   trailing * (`https://cdn.example.com/legacy/pack/*`)   //              -> prefix, DECLARED. Listed loudly on every run, because each

//   trailing * (`https://cdn.example.com/legacy/pack/*`)

// --- gate 5: sampled re-fetch (opt-in) --------------------------------------

const want = f.profile || "std";       const prof = fetchProfiles(url, { origin: ORIGIN, typeHint: f.type || "" }).find((x) => x.name === want);       const res = await fetch(url, {

const prof = fetchProfiles(url, { origin: ORIGIN, typeHint: f.type || "" }).find((x) => x.name === want);       const res = await fetch(url, {         headers: prof ? prof.headers : { "user-agent": BROWSER_UA, accept: "*/*", referer: ORIGIN

const res = await fetch(url, {

#!/usr/bin/env node

import { readFile, readdir, writeFile, mkdtemp, rm } from "node:fs/promises";

#!/usr/bin/env node

async function get(base, p) {   if (isHttp(base)) {     const r = await fetch(base + p, { headers: { "user-agent": UA } });

if (isHttp(base)) {     const r = await fetch(base + p, { headers: { "user-agent": UA } });     return { status: r.status, text: await r.text() };

const r = await fetch(base + p, { headers: { "user-agent": UA } });

#!/usr/bin/env node

*  * Usage:  *   node scripts/verify-offline.mjs --base http://127.0.0.1:29001 --routes /,/x,/y

* Usage:  *   node scripts/verify-offline.mjs --base http://127.0.0.1:29001 --routes /,/x,/y  *

*   node scripts/verify-offline.mjs --base http://127.0.0.1:29001 --routes /,/x,/y

* **零外联门的静态一半**：枚举产出字节里每个外部绝对 URL 并逐条裁决（§1.6 的四类断言面里，资源级探针看不见的那三类）  * **零外联门的静态半边**（资源级半边是 `probe.mjs --no-external`）：产出字节里的外部绝对 URL 普查——class 1 连接热身（preconnect/dns-prefetch）、class 2/3 调用点（sendBeacon/fetch/Image 携带外部字面量），census 里每个 host 必须在

* **零外联门的静态半边**（资源级半边是 `probe.mjs --no-external`）：产出字节里的外部绝对 URL 普查——class 1 连接热身（preconnect/dns-prefetch）、class 2/3 调用点（sendBeacon/fetch/Image 携带外部字面量），census 里每个 host 必须在 `mirror/external.txt` 有登记行  * `node verify-offline.mjs --base http:

* `node verify-offline.mjs --base http://127.0.0.1:<port> --routes /,/about`

return i >= 0 && args[i + 1] !== undefined ? args[i + 1] : d; }; const BASE = (flag("base", "http://127.0.0.1:29001") || "").replace(/\/+$/, "");

}; const BASE = (flag("base", "http://127.0.0.1:29001") || "").replace(/\/+$/, ""); const ROUTES = flag("routes", "/").split(",").filter(Boolean);

const BASE = (flag("base", "http://127.0.0.1:29001") || "").replace(/\/+$/, "");

try {   await fetch(BASE, { redirect: "manual", signal: AbortSignal.timeout(5000) });

await fetch(BASE, { redirect: "manual", signal: AbortSignal.timeout(5000) });

for (const route of ROUTES) {   const res = await fetch(BASE + route, { redirect: "manual" });

const res = await fetch(BASE + route, { redirect: "manual" });

#!/usr/bin/env node

* So: expand both sides, compare the STRUCTURE, and report where they diverge.  *  *   node scripts/verify-payload.mjs --a http://127.0.0.1:24001 --b http://127.0.0.1:24002 \

*  *   node scripts/verify-payload.mjs --a http://127.0.0.1:24001 --b http://127.0.0.1:24002 \  *        --routes /,/works,/about,/contact

*   node scripts/verify-payload.mjs --a http://127.0.0.1:24001 --b http://127.0.0.1:24002 \

const get = async (base, route) => {   const res = await fetch(base + route);

const res = await fetch(base + route);

#!/usr/bin/env node

#!/usr/bin/env node

* (verification-gates.md §1.6 class 4).  *  *   node scripts/verify-refs-served.mjs --base http://127.0.0.1:6376 --dir site [--allow mirror/external.txt]

*  *   node scripts/verify-refs-served.mjs --base http://127.0.0.1:6376 --dir site [--allow mirror/external.txt]  *

*   node scripts/verify-refs-served.mjs --base http://127.0.0.1:6376 --dir site [--allow mirror/external.txt]

* **引用可达门**:把产出字节里的每一条资源引用**逐条问服务器**(一次 GET,不开浏览器)。⭐ 关键在于**问服务器,而不是再实现一遍它的解析**——一个自己走镜像的离线检查就是第二份 url→path 实现,而第二份实现就是一次等着被报成窟窿的分歧(实测它把 28 张在场的图报成缺失,只因不知道服务器的查询变体回退)。⚠ 它看不见运行时拼出来的 URL,那是资源级探针的活(§1.6 class 4)  * **引用可达门**：产出字节里每条资产引用逐条**向真服务

* **引用可达门**：产出字节里每条资产引用逐条**向真服务器请求**（GET + Range 0-0）。⭐ 问服务器，不重实现它——离线走盘的检查是 url→path 的第二份实现，第二份实现就是一个待报的假洞。`--allow` 消费与镜像门同一份 `external.txt`（源站自身 404 的引用是登记的偏差，不是要发明文件补的洞）  * `node verify-refs-served.mjs --base http://127.0.0.1:<port> --a

* `node verify-refs-served.mjs --base http://127.0.0.1:<port> --allow mirror/external.txt`

const args = process.argv.slice(2); const flag = (n, d) => { const i = args.indexOf("--" + n); return i >= 0 ? args[i + 1] : d; }; const BASE = flag("base", "http://127.0.0.1:6376").replace(/\/$/, "");

const flag = (n, d) => { const i = args.indexOf("--" + n); return i >= 0 ? args[i + 1] : d; }; const BASE = flag("base", "http://127.0.0.1:6376").replace(/\/$/, ""); const DIR = path.resolve(flag("dir", "site"));

const BASE = flag("base", "http://127.0.0.1:6376").replace(/\/$/, "");

const ref = list[idx++];     let code = 0;     try { code = (await fetch(BASE + ref, { method: "GET", headers: { range: "bytes=0-0" } })).status; } catch { code = -1; }

let code = 0;     try { code = (await fetch(BASE + ref, { method: "GET", headers: { range: "bytes=0-0" } })).status; } catch { code = -1; }     n++;

try { code = (await fetch(BASE + ref, { method: "GET", headers: { range: "bytes=0-0" } })).status; } catch { code = -1; }

#!/usr/bin/env node

//   set CONFIG.server to have the gate boot it first. // // Measure redirects on the ORIGIN with `curl -sI` / fetch(redirect:"manual")

// // Measure redirects on the ORIGIN with `curl -sI` / fetch(redirect:"manual") // before filling in CONFIG.redirects — server-side routing behavior leaves

// Measure redirects on the ORIGIN with `curl -sI` / fetch(redirect:"manual")

base: `http://127.0.0.1:${PORT}`,

for (let i = 0; i < 80; i += 1) {     try {       await fetch(CONFIG.base + "/", { redirect: "manual" });

try {       await fetch(CONFIG.base + "/", { redirect: "manual" });       up = true;

await fetch(CONFIG.base + "/", { redirect: "manual" });

} else {   try {     await fetch(CONFIG.base + "/", { redirect: "manual" });

try {     await fetch(CONFIG.base + "/", { redirect: "manual" });   } catch {

await fetch(CONFIG.base + "/", { redirect: "manual" });

for (const [route, mirrorFile] of CONFIG.pages) {     const res = await fetch(CONFIG.base + route, { redirect: "manual" });

const res = await fetch(CONFIG.base + route, { redirect: "manual" });

for (const [from, wantCode, wantTo] of CONFIG.redirects) {     const res = await fetch(CONFIG.base + from, { redirect: "manual" });

const res = await fetch(CONFIG.base + from, { redirect: "manual" });

for (const p of CONFIG.notFound) {     const res = await fetch(CONFIG.base + p, { redirect: "manual" });

const res = await fetch(CONFIG.base + p, { redirect: "manual" });

for (const p of CONFIG.publicAssets) {     const res = await fetch(CONFIG.base + p);

const res = await fetch(CONFIG.base + p);

#!/usr/bin/env node

#!/usr/bin/env node

env: process.env.PORT || null, }); const BASE = `http://127.0.0.1:${PORT}`;

}); const BASE = `http://127.0.0.1:${PORT}`; console.log(`[verify-ssr] server under test ${BASE}  (${PORT_LABEL})`);

const BASE = `http://127.0.0.1:${PORT}`;

try {   await fetch(BASE, { redirect: "manual" });

await fetch(BASE, { redirect: "manual" });

for (const [route, mirrorFile] of pages) {   const a = await fetch(BASE + route).then((r) => r.text());

const a = await fetch(BASE + route).then((r) => r.text());

const notFound = await fetch(BASE + NOT_FOUND_ROUTE);

#!/usr/bin/env node

import { readFile, readdir, stat, cp, mkdtemp, rm } from "node:fs/promises";

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

*  * 1. ⭐ RAW BYTES ONLY: every fetch uses the `id_` (identity) replay flag —  *    `https://web.archive.org/web/<ts>id_/<original>` returns the capture's

* 1. ⭐ RAW BYTES ONLY: every fetch uses the `id_` (identity) replay flag —  *    `https://web.archive.org/web/<ts>id_/<original>` returns the capture's  *    original bytes, no rewriting, no toolbar. Never mirror the replay HTML.

*    `https://web.archive.org/web/<ts>id_/<original>` returns the capture's

* exponential backoff on 429/503. A rescue is not a race.  *  *   node scripts/wayback-mirror.mjs --origin https://darknetflix.io \

*  *   node scripts/wayback-mirror.mjs --origin https://darknetflix.io \  *        [--hosts cdn.example.com] [--anchor 20200626202014 | auto]

*   node scripts/wayback-mirror.mjs --origin https://darknetflix.io \

* 中文规格（自 scripts/README.md 迁入，v0.3.21；本表另一拼写：`wayback-mirror.mjs`）  * **X 类抢救:死站 → 标准镜像**(`references/archival-rescue.md`)。CDX 枚举 → 锚点+时间窗逐 URL 选连贯捕获(auto 锚点 = 根页 200 最密年代取中位,抢注者时代的 301 垃圾靠状态码+窗口出局)→ `id_` 旗抓**原始字节**(绝不镜像被注入改写的回放 HTML)→ 产出与

* **X 类抢救:死站 → 标准镜像**(`references/archival-rescue.md`)。CDX 枚举 → 锚点+时间窗逐 URL 选连贯捕获(auto 锚点 = 根页 200 最密年代取中位,抢注者时代的 301 垃圾靠状态码+窗口出局)→ `id_` 旗抓**原始字节**(绝不镜像被注入改写的回放 HTML)→ 产出与 mirror-site 同构的 mirror/ + 账本 + `wayback-provenance.json`(逐文件捕获时间戳/d

* `node wayback-mirror.mjs --origin https://dead.example --anchor auto --window-days 365`

const ORIGIN = (flag("origin", "") || "").replace(/\/$/, ""); if (!ORIGIN) { console.error("usage: wayback-mirror.mjs --origin https://dead.example [--hosts cdn.x] [--anchor auto] ..."); process.exit(2); }

if (!ORIGIN) { console.error("usage: wayback-mirror.mjs --origin https://dead.example [--hosts cdn.x] [--anchor auto] ..."); process.exit(2); }

for (let t = 0; t < tries; t++) {     try {       const r = await fetch(url, { redirect: "manual", headers: { "user-agent": "website-rebuild-skill/wayback-rescue (research; low-rate)" } });

try {       const r = await fetch(url, { redirect: "manual", headers: { "user-agent": "website-rebuild-skill/wayback-rescue (research; low-rate)" } });       if (r.status === 429 || r.status === 503) { await sleep(2000 * 2 ** t); continue; 

const r = await fetch(url, { redirect: "manual", headers: { "user-agent": "website-rebuild-skill/wayback-rescue (research; low-rate)" } });

for (const u of seeds) {     if (manifest2[u]) continue;     const r = await politeFetch(`https://web.archive.org/web/${anchor}id_/${u}`, { tries: 4 });

if (manifest2[u]) continue;     const r = await politeFetch(`https://web.archive.org/web/${anchor}id_/${u}`, { tries: 4 });     await sleep(350);

const r = await politeFetch(`https://web.archive.org/web/${anchor}id_/${u}`, { tries: 4 });

const loc = r.headers.get("location");       if (!loc) continue;       final = await politeFetch(new URL(loc, "https://web.archive.org").href, { tries: 4 });

if (!loc) continue;       final = await politeFetch(new URL(loc, "https://web.archive.org").href, { tries: 4 });       await sleep(350);

final = await politeFetch(new URL(loc, "https://web.archive.org").href, { tries: 4 });

const captures = []; // {original, timestamp, status, mimetype, digest, host} for (const host of HOSTS) {   const cdxUrl = `https://web.archive.org/cdx/search/cdx?url=${encodeURIComponent(host + "/*")}&output=json&fl=original,timestamp,stat

for (const host of HOSTS) {   const cdxUrl = `https://web.archive.org/cdx/search/cdx?url=${encodeURIComponent(host + "/*")}&output=json&fl=original,timestamp,statuscode,mimetype,digest`;   const r = await politeFetch(cdxUrl, { tries: 6 });

const cdxUrl = `https://web.archive.org/cdx/search/cdx?url=${encodeURIComponent(host + "/*")}&output=json&fl=original,timestamp,statuscode,mimetype,digest`;

// real digest exists whenever the URL was truly there; revisits are skipped // and the earlier identical capture wins through normal selection. // ⛔ SPELLING TWINS COLLIDE ON DISK. The archive stores `http://x/` and

// and the earlier identical capture wins through normal selection. // ⛔ SPELLING TWINS COLLIDE ON DISK. The archive stores `http://x/` and // `http://x:80/` as two originals; `f.eot` and `f.eot?` (the IE eot hack)

// ⛔ SPELLING TWINS COLLIDE ON DISK. The archive stores `http://x/` and

// `http://x:80/` as two originals; `f.eot` and `f.eot?` (the IE eot hack)

let picked = null, buf = null, r = null;     for (const cc of cands.slice(0, 4)) {       const rawUrl = `https://web.archive.org/web/${cc.timestamp}id_/${cc.original}`;

for (const cc of cands.slice(0, 4)) {       const rawUrl = `https://web.archive.org/web/${cc.timestamp}id_/${cc.original}`;       r = await politeFetch(rawUrl);

const rawUrl = `https://web.archive.org/web/${cc.timestamp}id_/${cc.original}`;

const base = holeUrl.split("/").pop().split("?")[0];   if (!base || base.length < 5) continue;   const q = `https://web.archive.org/cdx/search/cdx?url=${encodeURIComponent(new URL(holeUrl).host + "/*")}&output=json&filter=original:.*/${enco

if (!base || base.length < 5) continue;   const q = `https://web.archive.org/cdx/search/cdx?url=${encodeURIComponent(new URL(holeUrl).host + "/*")}&output=json&filter=original:.*/${encodeURIComponent(base).replace(/\./g, "\\.")}$&fl=origina

const q = `https://web.archive.org/cdx/search/cdx?url=${encodeURIComponent(new URL(holeUrl).host + "/*")}&output=json&filter=original:.*/${encodeURIComponent(base).replace(/\./g, "\\.")}$&fl=original,timestamp,statuscode&limit=50`;

if (!cands.length) continue;   const [archivedAs, ts] = cands[0];   const fr = await politeFetch(`https://web.archive.org/web/${ts}id_/${archivedAs}`); await sleep(350);

const [archivedAs, ts] = cands[0];   const fr = await politeFetch(`https://web.archive.org/web/${ts}id_/${archivedAs}`); await sleep(350);   if (!fr || fr.status !== 200) continue;

const fr = await politeFetch(`https://web.archive.org/web/${ts}id_/${archivedAs}`); await sleep(350);

#!/usr/bin/env node

#!/usr/bin/env node

import { mkdir, readdir, symlink, copyFile, rm } from "node:fs/promises";

#!/usr/bin/env node

#!/usr/bin/env node

#!/usr/bin/env node

// （同 url+w+q，忽略 Vercel 的 dpl 部署 id——源站发了什么才是参照），镜像没有的档位才向本机 // `next start` 的优化器取（登记：这些档位是重建侧生成物），按 lib/urlpath 的查询变体命名落进静态树。 //   node tools/harvest-optimized-images.mjs [--base http://127.0.0.1:3311] [--site rebuild/static-site] [--mirro

//   node tools/harvest-optimized-images.mjs [--base http://127.0.0.1:3311] [--site rebuild/static-site] [--mirror mirror]

const args = process.argv.slice(2); const flag = (k, d) => { const i = args.indexOf("--" + k); return i >= 0 ? args[i + 1] : d; }; const BASE = flag("base", "http://127.0.0.1:3311"), SITE = flag("site", "rebuild/static-site"), MIRROR = flag

const flag = (k, d) => { const i = args.indexOf("--" + k); return i >= 0 ? args[i + 1] : d; }; const BASE = flag("base", "http://127.0.0.1:3311"), SITE = flag("site", "rebuild/static-site"), MIRROR = flag("mirror", "mirror"); const POLICY =

const BASE = flag("base", "http://127.0.0.1:3311"), SITE = flag("site", "rebuild/static-site"), MIRROR = flag("mirror", "mirror");

let ok = 0, fromMirror = 0, fail = 0; for (const u of [...urls].sort()) {   const q = new URL("http://x" + u).searchParams;

for (const u of [...urls].sort()) {   const q = new URL("http://x" + u).searchParams;   const key = `${q.get("url")}|${q.get("w")}|${q.get("q")}`;

const q = new URL("http://x" + u).searchParams;

if (mirrorByKey.has(key)) { buf = await readFile(join(MIRROR, mirrorByKey.get(key))); fromMirror++; }   else {     const r = await fetch(BASE + u).catch(() => null);

else {     const r = await fetch(BASE + u).catch(() => null);     if (!r || !r.ok) { fail++; console.log(`  FAIL ${r ? r.status : "net"} ${u.slice(0, 90)}`); continue; }

const r = await fetch(BASE + u).catch(() => null);

buf = Buffer.from(await r.arrayBuffer());   }   const rel = localRelPath("http://x" + u, "x", POLICY);

}   const rel = localRelPath("http://x" + u, "x", POLICY);   await mkdir(dirname(join(SITE, rel)), { recursive: true });

const rel = localRelPath("http://x" + u, "x", POLICY);

#!/usr/bin/env node

for (const { path: rel } of await readInventory(root)) if (rel && !OWNER.has(rel)) OWNER.set(rel, root); } const ORIGIN_URL = (MANIFEST.origin || "https://example.invalid").replace(/\/$/, "") + "/";

} const ORIGIN_URL = (MANIFEST.origin || "https://example.invalid").replace(/\/$/, "") + "/"; const ORIGIN_HOST = new URL(ORIGIN_URL).hostname;

const ORIGIN_URL = (MANIFEST.origin || "https://example.invalid").replace(/\/$/, "") + "/";

#!/usr/bin/env node

import { readFile, writeFile, mkdir, rm } from "node:fs/promises";

#!/usr/bin/env node

Executable file

#!/usr/bin/env node

#!/usr/bin/env node