Sermon pipeline

What it is and who uses it

One Node.js app that moves each campus’s Sunday recording from the recorder to the public. A person trims the audio in a browser, and the app does the rest, with two steps still partly by hand.

The four campuses, with the code the app and file names use:

• DT, Downtown Minneapolis. A HyperDeck Studio records video (MP4). This is the only campus with video.
• NL, North Lakes (Shoreview). A GLD80 console makes a WAV that staff upload to Google Drive.
• LT, St. Paul. The campus was called Lowertown until 2026-10-25. The code LT did not change. The console makes a WAV that staff upload to Google Drive.
• CH, Columbia Heights. Staff upload an MP3 to Google Drive. Since 2026-11-10 a date-named WAV is also accepted and converted to MP3 at trim.

Where each recording ends up:

• A sermon post on resources.hopecc.com (WordPress, Advanced Sermons Manager plugin).
• The campus’s podcast feed on Transistor, which also hosts the audio the sermon page plays.
• Vimeo, for Downtown video only.
• The “HopeCC - Twin Cities” Subsplash app.

Hope has no livestream (recorded 2026-10-07), so these are all the channels.

Who uses it: Austin Sego runs the Sunday steps. Preachers and campus staff send details and files by email or Google Drive; they do not sign in. No one else is recorded as an operator.

How to get access

The dashboard is https://publish.hopecc.com, behind Cloudflare Access (team hopecc, Google Workspace sign-in). The Access policy allows austin@hopecc.com only.

To add a person:

1. Open the HopeCC Zero Trust dashboard and find the self-hosted Access app on publish.hopecc.com.
2. Add the person’s email to its Allow policy.

Anyone on that policy can do everything Austin can, including publishing to the live site. Per-campus roles were scoped in October 2026 and never built.

Other doors, each held by Austin:

• SSH to the NAS as hccadmin, by key. Passwordless sudo covers the Docker binary only.
• The GitHub repo hopecc-twincities/sermon-pipeline (private).
• The vendor accounts: Auphonic, Transistor, Vimeo, Subsplash, Cloudflare R2. The Subsplash sign-in and the Synology C2 backup login are in Austin’s Bitwarden. For the others, the README gap note says several credentials sit only in Austin’s Bitwarden and laptop .env files.

How to get help

No help desk or on-call contact is recorded. Austin Sego is the only named owner, and no backup admin is recorded.

To diagnose, in this order:

1. Open the Logs tab in the dashboard (server log and HyperDeck log).
2. Open the Process tab for Auphonic and DT video state, and the This week board for each step’s state.
3. Read the repo README.md and CLAUDE.md at github.com/hopecc-twincities/sermon-pipeline.

Vendor support contacts are not recorded, except that the Subsplash account has a named customer success contact on its Subscription page (Dashboard, Settings, Subscription).

Where it runs

On utility-nas, the Synology RS1221+ in the Downtown IT closet (172.20.40.122, AVL VLAN 40), in Docker. The Mac mini utility-server ran it until the cutover on 2026-08-19 and has since been wiped.

• Container: sermon-pipeline-sermon-pipeline-1, restart: unless-stopped, port 3000, runs as the NAS account svc-pipeline (uid 1029), never as root.
• Code: a git checkout at /volume1/sermon-pipeline/app, mounted into the container at /app. A git pull takes effect without an image rebuild.
• Image: Node 22, ffmpeg, git, Ghostscript, and rclone v1.75.0 (pinned; Debian’s older rclone fails HyperDeck FTP transfers).
• Working folders on the NAS: /volume1/sermon-pipeline/staging (HyperDeck pulls, mounted at /app/staging) and /volume1/sermon-pipeline/rclone-config.
• Public address: https://publish.hopecc.com, a published application on the existing workspace-mcp Cloudflare Tunnel, pointing at http://172.20.40.122:3000.
• LAN fallback: http://utility-nas:3000, only when the tunnel is down (see the runbook “Switch to the LAN fallback”).
• Storage: Cloudflare R2 bucket sermons (862 GB on 2026-09-24) holds working files. The long-term archive is /volume1/media-archive/sermons/ on the NAS (4,003 files, 1,049 GB on 2026-09-25).
• Development: clone at /usr/local/sermon-pipeline on Austin’s MacBook, or ~/work/sermon-pipeline on utility-agent. Claude Code is not installed on the NAS itself.

R2 layout, where {loc} is the lowercase campus code:

• {loc}/raw_recordings/ raw files.
• {loc}/peaks/ waveform data for the trimmer.
• {loc}/trimmed/ output of the trim, named YYYYMMDDLL.ext, for example 20260426DT.
• {loc}/ready_to_publish/ Auphonic output, plus the remuxed DT MP4.
• dt/auphonic_out/ Auphonic’s own drop for DT audio.

Accounts and credentials

Values are never written here. Each item names where it lives.

• Pipeline .env, /volume1/sermon-pipeline/app/.env on utility-nas (owner 1029:100, mode 640, mounted read-only into the container). It holds:
  • R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET.
  • WP_SITE_URL, WP_USERNAME, WP_APP_PASSWORD.
  • AUPHONIC_API_KEY and AUPHONIC_PRESET_DT/NL/LT/CH.
  • TRANSISTOR_API_KEY and TRANSISTOR_SHOW_DT/NL/LT/CH.
  • VIMEO_ACCESS_TOKEN.
  • CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUD, and AUTH_MODE for the sign-in check.
• WordPress application password, for the WordPress user claude-readonly. The name is misleading, because the user can write. The password is shared with the Website Infrastructure work, so rotating it breaks every user at once. WordPress never shows it again after creation, so read it from the NAS .env. The rotation steps are in the runbook “Edit the NAS .env”.
• rclone config, /volume1/sermon-pipeline/rclone-config/rclone.conf. It holds the Google Drive OAuth token (gdrive:), the R2 keys (r2:), and the HyperDeck FTP login. The container mounts the folder, not the file, so token refreshes persist.
• hccadmin’s own rclone config, ~/.config/rclone/rclone.conf on the NAS. It holds gdrive-austin for the Austin-OS vault mirror. Append to it and never overwrite it.
• GitHub token for the NAS, a classic token with repo scope from the hopeccmpls GitHub account, at /volume1/sermon-pipeline/app/.git/.git-credentials. Its expiry is not recorded and not visible from the NAS. A different token, GITHUB_PAT in the laptop .env at /usr/local/sermon-pipeline/.env, was revoked (GitHub returned 401 on 2026-10-31) and nothing in the repo reads it.
• Cloudflare Access app for publish.hopecc.com in the HopeCC Zero Trust account (team hopecc). The Allow policy and the app’s AUD tag are set in the dashboard.
• Cloudflare tunnel, the workspace-mcp tunnel, container workspace-mcp-tunnel, compose folder /volume1/docker/workspace-mcp.
• Vimeo token backup, /volume1/sermon-pipeline/app/state/.env.bak-20260928, the .env from before the token swap. Delete it, and revoke the old token at developer.vimeo.com under My Apps, once the new token has held for a few weeks.
• Cloudways break-glass SSH, user master_khfnzmmbcs on 157.245.5.206, using the shared key kept in the Website Infrastructure folder on Austin’s Mac. The WordPress app for resources is cjftjjdjrk.
• Scratch .env at ~/Documents/Claude/Projects/Sermon Publishing Pipeline/.env on Austin’s Mac, holding a Cloudflare API token and account ID for the HopeCC account. The app does not read it.
• Subsplash admin sign-in. Used by hand and by the Sunday automation. It is kept in Austin’s Bitwarden. A Subsplash run asks Austin for the sign-in with a ping, and the sign-in took about a minute on 2026-11-08.

Depends on

Services this one needs:

• utility-nas, the host and the home of the long-term archive
• Cloudflare R2, Cloudflare Access, and the Cloudflare Tunnel on the NAS
• Cloudflare zone hopecc.com, for cache purges after sermon pages change
• resources.hopecc.com on the Cloudways server hopecc-production (app cjftjjdjrk), with the sermon-rest-api and sermon-audio-redirects mu-plugins
• Google Drive, the shared drive “Recordings” (NL, LT, CH raw files) and the Google account behind the rclone token
• The HyperDeck Studio at 172.20.40.239 on VLAN 40
• Auphonic, Transistor, Vimeo, and Subsplash
• GitHub, for deploys
• hopecc.com/dtslides, the page the app reads to find the Downtown slide deck

What depends on it:

• Subsplash Sunday automation, which waits for the website post this pipeline creates
• Podcast apps, which read the Transistor feeds
• The legacy Pinecast feed redirects and old audio URLs, which now point at Transistor files this pipeline publishes to

Schedule

All schedules run inside the app, not in DSM Task Scheduler. Times are America/Chicago (the container sets TZ).

• Drive ingest for NL, LT, CH: every 15 minutes, all week. It looks back 21 days at most.
• HyperDeck pull for DT: every 5 minutes, Sundays 10:00 to 14:00. A pull skips files younger than 10 minutes.
• Auto-distribute: every 5 minutes. It runs the podcast, Vimeo (DT only), and publish steps for every post marked Ready, and pulls the DT slide deck.
• Auphonic status check: every 60 seconds while a production is running.
• Subsplash Sunday automation: Claude scheduled tasks subsplash-sunday, subsplash-sunday-5pm and subsplash-sunday-probe on Austin’s Mac, Sundays 1:00 to 4:30 PM every 30 minutes. The 2026-11-08 test run built and read back DT, LT and NL, and the three items were published that evening with the board marked done. The first unattended run is due 2026-11-15.

The week’s human work, in order, is trim each campus, fill in Details, mark the post Ready, then check the result. Preacher details arrive by email on Sunday morning, mostly before about 1 PM. Downtown slides are pulled automatically once a deck exists. A deck changed on the Friday or Saturday before also counts.

Runbooks

Normal Sunday run

1. Check the container is up: ssh hccadmin@utility-nas 'sudo -n /usr/local/bin/docker ps'. Open https://publish.hopecc.com and sign in with Google.
2. Open the This week board. It lists every campus’s nine steps (recording, trim, auphonic, checks, details, podcast, vimeo, publish, subsplash) and says whose turn it is.
3. Wait for the recordings. Downtown arrives from the HyperDeck after 10:00. NL, LT and CH arrive from Drive within 15 minutes of an upload. The Ingest tile on the Trim tab shows the last pull, and “Pull now” forces one.
4. Trim each campus on the Trim tab. Set the in and out points and press trim. NL, LT and CH go to Auphonic by themselves afterward. Downtown also submits by itself after an in-app trim (deployed 2026-10-04; a check on a live Downtown Sunday is still open in the task list). If a card sits on “Not submitted”, press Submit to Auphonic on it.
5. Watch the Process tab until each campus reads Done. Downtown video goes Processing, Remuxing video, Done. For Downtown, press Check on the video. It must pass all four checks (H.264, AAC 48 kHz, size close to the trimmed file, not silent) before it goes to Vimeo.
6. On the Details tab, pick the campus and the service date and fill in the form from the preacher’s email. Save the slides and the discussion guide from the email first and pick them in the form, which renames them YYYYMMDDLL.ext. Pick topics from the list only. Press Create draft.
7. Tick “Ready: these details are final” once slides and guide are attached. Ready is what lets the app go on. Marking it early ships an episode without those links to feeds that cannot be fully taken back.
8. Let the app finish. Every 5 minutes it creates and publishes the Transistor episode, sends Downtown video to Vimeo, and publishes the post. Each step shows on the Distribute tab and on the board. If the podcast step says Transistor is still processing, wait a minute and press it again; it reuses the same episode.
9. Subsplash. The Sunday automation builds and publishes each campus’s item after the website post is live. If it did not, use the Distribute tab’s Subsplash card, which lists every value with a copy button, and enter the item by hand. For Downtown, import the video first and remove the auto-generated audio track before importing the podcast audio.
10. Prove it. Open each sermon page logged out and confirm the audio plays (and the Vimeo embed for Downtown). Confirm the new episode is the only one for that campus in Transistor. Confirm processed files are named for the service date, such as 20260927NL.

If a step fails, its error shows on the step. Do not call the publish or podcast endpoints to test whether they refuse. The vault records an incident from that on 2026-09-23.

Fallback: publish by hand if the app is down

Use this when the app cannot be restored before the sermon is due. It was last run in full on 2026-09-13 from a Claude session without access to the NAS. Podcast hosting has changed since, and the steps below use the current host.

1. Find what each stage produced. Auphonic lists recent productions at https://auphonic.com/api/productions.json?limit=12 with an Authorization: Bearer header (the key is AUPHONIC_API_KEY). Each production has mp3, vtt, html and json outputs; download links redirect, so use curl -L. Raw and trimmed files are in the R2 bucket sermons.
2. If the audio is not yet trimmed or processed, trim it with ffmpeg stream copy (-c copy, no re-encode) from a local copy and submit the trimmed file to Auphonic with the campus’s preset. Auphonic names the output from the preset’s filename pattern, so find the file by production, not by guessing a name.
3. Collect the details. Preacher emails come from Gmail. Columbia Heights details arrive as a Google Drive share notice from the CH tech account, with title, speaker and passage in the shared file’s name.
4. Upload the slides and guide to WordPress media as YYYYMMDDLL.ext. Send a normal browser User-Agent, or Cloudflare answers 403 or error 1010. Upload as multipart form data. Slide PDFs over 1 MB should be compressed first.
5. Create the sermon post as a draft with POST /wp-json/wp/v2/sermons. Campus and topics must match existing terms exactly. Meta keys all start with asp_: asp_sermon_bible_passage, asp_sermon_mp4 (audio URL), asp_sermon_pdf, asp_sermon_bulletin, asp_sermon_vimeo, and asp_sermon_video_type_select set to vimeo. Series artwork is the featured image copied from the newest earlier post in that series.
6. Podcast. In Transistor, create an episode in the campus’s show from the processed MP3, publish it, and put its media.transistor.fm URL in asp_sermon_mp4. Show notes follow the template: optional blurb, series, speaker, campus, a slides link, an optional guide link, and the campus page on hopecc.com.
7. Downtown video. Check the file in dt/ready_to_publish/ has H.264 video and AAC audio, is close to the trimmed file in size, and is not silent. Then create the Vimeo upload with a presigned R2 link as a pull upload, and write the Vimeo URL and asp_sermon_video_type_select to the post.
8. Set the post to published only after the audio URL is set. Read it back and open the live page logged out.
9. Subsplash. Open the campus series page, create a media item, and fill in title, subtitle (Downtown, St. Paul, North Lakes or Heights), date, speaker, scripture, existing topics only, audio URL, slides as a Document, and the guide as a Web Link titled “Discussion Guide”. Then press Publish.

Expect about 75 minutes for all four campuses by hand.

Add or rename a campus

A campus lives in four places that must agree. The app fails closed when they do not, because the campus list in WordPress is a fixed set and the app will not create terms.

To add a campus:

1. Pick a two-letter code. The code is lowercased for R2 folders and file names.
2. Create the campus term in WordPress under the sermon_campus taxonomy on resources.hopecc.com. The app’s name for it must match the term exactly.
3. In Transistor, create the campus’s show. Copy its numeric ID from the show’s URL, or call GET /api/distribute/transistor/shows on the app.
4. In Auphonic, create or copy a preset for the campus, with its output mapped to {loc}/ready_to_publish/. Copy its UUID from the preset’s URL.
5. In config/config.js in the repo, add the campus: filename pattern for raw files, minimum raw file size (so test recordings stay hidden), Auphonic preset, Transistor show, R2 prefixes, and the wpCampus name. Set vimeo: true only if it records video.
6. Add AUPHONIC_PRESET_XX and TRANSISTOR_SHOW_XX to the NAS .env (see the runbook “Edit the NAS .env”).
7. For a Drive campus, create Shared Drives/Recordings/Sermons/<Campus>/Raw Recordings/ and add its path to the ingest config. For a video campus, the HyperDeck pull and the Downtown-only checks need adapting. Sources do not record how.
8. Deploy (runbook “Deploy a code change”). The mu-plugin sermon-rest-api does not change, because it registers meta fields, not campuses.
9. Prove it. Drop a test file in the Drive folder, press “Pull now”, and confirm it appears on the Trim tab with a waveform.

To rename a campus, the app code stays the same:

1. Keep the two-letter code. St. Paul kept LT after the 2026-10-25 rename, so file names, R2 folders and env keys did not change.
2. Change the display name in the places people see it: the WordPress sermon_campus term, the wpCampus string in config/config.js (the two must match exactly, or publishing fails with a 400 listing valid names), the Subsplash item subtitle, and the Transistor show title.
3. Done for St. Paul and deployed 2026-11-01 (33a02dd): config display name and short name “St. Paul”, the show-notes link hopecc.com/stpaul (the old /lowertown link redirects there), the Subsplash subtitle for new items, and the Trim tab label.
4. Still open for St. Paul in the task list: the Lowertown footer in 220 of 434 Transistor episodes (the 10/25 and 11/1 episodes were fixed by hand); the WordPress sermon_campus term, which still reads Lowertown - St. Paul with 433 posts (keep its slug lowertown-st-paul, and change wpCampus and scripts/archive-week.py in the same deploy); the Auphonic preset album “Hope Lowertown”; about 433 back-catalog Subsplash items whose subtitle still says Lowertown; and the Drive folder Sermons/Lowertown, to be renamed only together with the gdrivePath config change in one deploy, after telling the St. Paul tech.

Deploy a code change

1. Push the change to main in hopecc-twincities/sermon-pipeline. Avoid deploying on Sunday before the last campus is published.
2. Confirm nothing is running. GET /api/auphonic/active must be empty, and no ingest or HyperDeck pull may be in progress.
3. Pull inside the container: ssh hccadmin@utility-nas 'sudo -n /usr/local/bin/docker exec sermon-pipeline-sermon-pipeline-1 git -C /app pull --ff-only'.
4. Restart: ssh hccadmin@utility-nas 'sudo -n /usr/local/bin/docker restart sermon-pipeline-sermon-pipeline-1'. The app reads .env and config only at startup.
5. A DSM alert “Container stopped unexpectedly” about 12 seconds later is expected. It fires for any restart not started from the Container Manager screen. A real stop has no restart command in /var/log/auth.log about 15 seconds before the alert. Alerts on 2026-11-01 (twice) and 2026-11-08 (five times) came while publishing was running, and no cause is recorded. Check the container is up before marking anything Ready.
6. If package.json or package-lock.json changed, run docker compose down -v then docker compose up -d --build in /volume1/sermon-pipeline/app. A Dockerfile change needs docker compose build then up -d.
7. Prove it. Open the dashboard, check the Logs tab for a clean start line, and exercise the changed path.

Edit the NAS .env

The file belongs to 1029:100 with mode 640, so hccadmin cannot write it, and the container mounts it read-only on purpose.

1. Run a throwaway container as the file’s owner with the folder mounted read-write: sudo docker run --rm -i -u 1029:100 -v /volume1/sermon-pipeline/app:/w alpine sh -c '...' operating on /w/.env.
2. Write the file in place with >, not mv, so owner and mode stay the same.
3. Restart the container. The app reads .env only at startup.
4. To check a key without the running process, use docker exec -w /app ... node -e 'require("dotenv").config(); ...'. A bare docker exec node -e does not load .env and reports a false 401.
5. If you rotate the WordPress application password, update all of these in one sitting: the NAS .env, any Claude Code worktree .env links, and any laptop copy. Then run the GET /wp-json/wp/v2/users/me check (expect 200) and publish one test sermon before Sunday.

Switch to the LAN fallback when the tunnel is down

Sign-in is enforced since 2026-10-18. A browser on the LAN never has the Access cookie, so the LAN path is closed unless you open it on purpose.

1. Run scripts/set-auth-mode.sh report in the repo on the NAS. It takes about 15 seconds including the restart, and refuses while Auphonic, HyperDeck or ingest is running.
2. Use http://utility-nas:3000.
3. When the tunnel is back, run scripts/set-auth-mode.sh enforce.
4. Prove it. GET /api/me from inside the container reports who the app thinks you are. Loopback inside the container passes in every mode.

While report is on, anyone who can reach port 3000 can publish.

When it breaks

The Trim tab shows no file for North Lakes, St. Paul or Heights

1. Look at the Ingest tile on the Trim tab. If the last pull is old or red, press “Pull now” and read the Logs tab.
2. Check the file is in the Drive folder, and that its name matches the campus pattern. NL and LT take 12_Apr_08.25.54.wav or the same with a campus suffix. CH takes M-D(-YY).mp3, or the same as .wav. An .m4a is skipped. On 2026-11-08 Heights shared its recording as .m4a, and getting it into the trimmer was still an open task on 2026-11-10.
3. A file under the campus size floor is hidden on purpose (DT 500 MB, NL and LT 100 MB, CH 20 MB; DT’s archive floor is 100 MiB).
4. If every Drive call fails, the shared rclone client ID may have been retired. rclone has warned since 2026 that its shared Google client will stop working. The plan to move to the church’s own OAuth client (Google project hopecc-sermon-pipeline, audience Internal) was written 2026-11-02. Its steps are still open in the task list, so the shared client is still in use.

Downtown recording is missing or will not trim ("moov atom not found")

1. Open the HyperDeck log in the Logs tab. A file pulled while the deck was still recording is truncated. Since 2026-10-25 the pull re-downloads files whose size changed, skips files under 10 minutes old, and re-pulls a staged file that fails with “moov atom not found” up to twice.
2. If a partial file is still staged, delete it from /volume1/sermon-pipeline/staging and force a pull with POST /api/hyperdeck/run (it bypasses the Sunday gate).
3. A failed ingest does not set the scheduler’s lastError, so the board can look fine. Read the log.
4. Check the HyperDeck itself at 172.20.40.239. If the deck was not recording, no pull will help. This happened on 2026-08-16.

The Downtown video is silent or 43 MB

A good Downtown MP4 in dt/ready_to_publish/ passes three tests.

1. Size is close to the trimmed file (about 5 to 8 GB). A file near 43 MB is Auphonic’s cover-art video, not a remux.
2. ffprobe shows h264 video and aac audio. MP3 audio inside an MP4 plays silent in QuickTime, Safari and Subsplash even though ffprobe calls it valid.
3. A 20-second slice is not silent. Auphonic output sits around -19 dB mean with a peak near -1 dB.

To recover, run scripts/dt-remux-recover.sh <BASENAME> [MP3_R2_KEY] from the repo. It finishes a remux from the trimmed MP4 left in tmp/, checks free space first, and refuses to upload an output under half the source size. Check both server.log and server.error.log. A remux failure logs to the error file only.

Auphonic does not start or never finishes

1. Open the Process tab. A card on “Not submitted” after a trim needs Submit to Auphonic pressed.
2. Check GET /api/auphonic/active. In-flight jobs are saved to state/auphonic-state.json and resume after a restart.
3. Authentication is a Bearer header with AUPHONIC_API_KEY. HTTP Basic returns 403. An older note that said Basic was wrong.
4. Auphonic reports Done when processing ends, but its copy to R2 arrives minutes later. The app downloads the processed file from Auphonic’s API, so a missing R2 key right after Done is not a failure.

A slide upload is refused with HTTP 413

1. The WordPress host runs an Imunify360 firewall that rejected raw uploads over about 1 MB after an automatic update on 2026-11-06. The cap is a best guess. Exact setting not established.
2. The app now sends uploads as multipart and compresses PDFs over 1 MB with Ghostscript (96 ppi). Either alone gets most files through.
3. A .docx bulletin, or a PDF still over about 1 MB after compression, can still fail if the firewall starts capping multipart bodies. The error then says “refused as too large”.
4. Fix by shrinking the file by hand, or by raising the issue with Cloudways support. Do not rely on an exemption the next unattended Imunify update could reset.

The sermon page shows no audio though the post saved

1. Every sermon meta key starts with asp_. Unprefixed keys save without error and render nothing. This was the bug from the first build until 2026-08-09.
2. WordPress drops any meta key not registered in sermon-rest-api.php, the mu-plugin in wp-content/mu-plugins/ on resources.hopecc.com. Renaming a key in the app does nothing until that file is redeployed.
3. Redeploy it in place with cat new.php > sermon-rest-api.php, so file ownership stays. An scp or mv makes a master-owned file that needs Cloudways, Application Settings, Reset Permission.
4. After a change to a published page or file, purge Cloudflare. Sermon pages sat in Cloudflare’s cache for about two days on 2026-09-22, and WordPress uploads are cached for a year.

The Transistor step errors or does nothing

1. “Still processing” means Transistor has not finished pulling the audio. Press the step again after a minute. The app reuses a same-title episode from the last 7 days.
2. An unset TRANSISTOR_SHOW_<LOC> makes the app skip the campus with a warning, not fail. Check the .env.
3. Transistor allows 10 requests per 10 seconds and answers 429 with a 10-second lockout. This does not matter at weekly volume.
4. A new feed item should have no season or episode number. Numbers were cleared on every show on 2026-09-22.

The dashboard at publish.hopecc.com will not load

1. Check the Cloudflare Tunnel container: docker ps on the NAS should list workspace-mcp-tunnel. Stopping or recreating the workspace-mcp compose stack takes publish.hopecc.com down with it.
2. If the tunnel is down, use the LAN fallback runbook.
3. If it loads but says “bad token: wrong audience”, CF_ACCESS_AUD in the .env does not match the Access app. The value is the kid parameter in the Access sign-in redirect.
4. A Mac that looked up the name before its DNS record existed can cache the empty answer. Flush with sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder, or test with dig @1.1.1.1.

A script or curl gets 403 from resources.hopecc.com or Pinecast

1. Cloudflare answers 403 or error 1010 to non-browser user agents. Send a normal browser User-Agent. For Pinecast, a podcast-app agent such as AppleCoreMedia/1.0 works.
2. A 403 from curl’s default agent is the bot filter, not a broken feed.

git pull fails on the NAS, or Subsplash will not take a topic

1. If git pull is refused, the GitHub token may have expired. It is a classic token with no expiry visible from the NAS. While it is expired the container keeps serving old code with no alarm. Generate a new token for the hopeccmpls account and write it to /volume1/sermon-pipeline/app/.git/.git-credentials.
2. Subsplash has a hard limit of 100 topics and Hope is at it. Never create a topic. If the dropdown offers only “Add X”, skip that topic. Remove an unused topic first if one must be added.
3. Subsplash form quirks: typed values are lost until Save draft is pressed, the date box compares as a calendar date, and the scripture picker closes with Tab.

Backup and recovery

What is where, and the second copy of each:

• Current week’s files: R2 sermons. The NAS archive receives them through scripts/archive-week.py. A filing run on 2026-09-25 put 575 files (851.64 GB) from R2 into masters/ and raw/ on the NAS with zero mismatches.
• Published audio: Transistor (media.transistor.fm) and the NAS. 3,268 of the 3,289 catalog rows with a Pinecast URL were matched to Transistor files on 2026-09-22 (the rest are listed in transistor-repoint-worklist.csv), and 1,857 sermon posts played from Transistor on 2026-10-27.
• Legacy audio, 2003 to 2024, and podcast episodes: /volume1/media-archive/sermons/legacy-audio/ and podcast-episodes/ on the NAS, each file with an md5 manifest. _archive-meta/scripts/03-verify-archive.sh re-hashes everything. It read 4,003 verified, 0 mismatched, 0 missing on 2026-09-25. Also at /volume1/media-archive/sermons/: README.txt and CATALOG.html, a searchable index that opens offline. Regenerate it with python3 _archive-meta/scripts/08-build-catalog.py.
• Pinecast: now on the free plan, redirect only. Do not delete the account. Its 11 feeds send a 301 to Transistor, and old Subsplash items still play audio from storage.pinecast.net (all 1,828 URLs checked on 2026-10-27). The backup of every episode is in podcast-episodes/.
• Downtown video: Vimeo holds 863 real videos (3.68 TB, 2010 to 2026) and is the only copy of everything before 2025. The NAS holds 25 Downtown masters from 2026. A plan to copy the Vimeo library to /volume1/media-archive/vimeo-archive/ was written 2026-09-29. The task is open and no copy is recorded as run.
• The .env and rclone config: copies exist only on the NAS and, for the .env, on Austin’s MacBook at /usr/local/sermon-pipeline/.env. state/.env.bak-20260928 is a dated backup on the NAS.
• The code: GitHub hopecc-twincities/sermon-pipeline. The GitHub organization is owned by Austin’s personal account.

Gaps, stated plainly:

• The NAS has no offsite backup. Hyper Backup had zero jobs on 2026-09-24. The 2026-09-29 recommendation was Synology C2 at the measured 207.54 GB scope (the sermons tier minus raw/ and the Downtown MP4s, plus configuration), with client-side encryption. On 2026-11-05 Austin created the C2 account, built the Hyper Backup task Synology C2 (daily 03:30, client-side encryption on, excludes *.mp4 and raw/) and started the first run. The restore test on the MacBook is not recorded as done, and the 2026-11-09 README still lists the missing offsite backup as a gap. The *.mp4 filter also drops ProPresenter videos in media-archive/propresenter, which Austin chose to leave out on 2026-11-05.
• Raw recordings have no offsite copy by decision. They sit on the NAS with RAID and daily snapshots for 12 months.
• Pruning of R2 (raw and trimmed after 21 days, ready_to_publish after 90 days) is decided. It needs the NAS offsite backup to pass a restore test first. No prune is recorded as run.
• Two Downtown masters are weak. 20260524DT survives only as a 2.2 Mbps encode once raw is pruned, and 20260823DT on Vimeo is the trimmed cut, not the master.

To restore the app on a new host: clone the repo, restore the .env and rclone.conf, create the NAS account and folders as in docker-compose (COMPOSE_PROJECT_NAME=sermon-pipeline, STAGING_DIR, RCLONE_CONFIG_DIR, and INGEST_AUTOSTART=false until you are ready, because starting the app starts ingesting), run docker compose up -d --build, then prove each stage with a forced run. The 2026-08-11 migration runbook in the vault is the worked example.

Costs and contracts

• Vimeo: $300 per year for the plan that holds the Downtown library. 0.59 TB of 4.40 TB was free on 2026-09-29, roughly a year to eighteen months at current uploads (estimate). Renewal date not recorded.
• Cloudflare R2: about $11 per month at 747 GB on 2026-08-31, projected near $32 per month within a year without pruning. The bucket was 862 GB on 2026-09-24.
• Transistor: flat-rate plan with unlimited shows, chosen because per-show pricing elsewhere would multiply cost across 11 shows. Price and renewal date not recorded. API key limit is 10 requests per 10 seconds.
• Auphonic: plan and cost not recorded.
• Subsplash: Giving plus Subsplash One 1,000, $6,926.30 per year, renewing 2027-07-26. This is the church app contract and covers more than sermons. API access costs a one-time onboarding fee of $2,000. The API was parked on 2026-10-09 for that reason, and on 2026-11-02 the account contact offered $1,000 with no trial. Undecided.
• Pinecast: free plan since 2026-10-20. Keep it.
• Cloudflare Access and Tunnel: assumed on the free tier, not re-verified.
• Offsite backup: Synology C2, 300 GB tier at about $29.99 per year (assumed, not re-verified). The account was created 2026-11-05. The backup was about 225 GB then. A 1 TB tier near $79.99 per year would be needed only if the ProPresenter videos were added.

Decisions and history

Newest first.

• 2026-11-10. Heights accepts a date-named WAV as a fallback. The trim re-encodes it to 256 kbps MP3, so later stages are unchanged. A Finder Quick Action, “Make MP3 for Upload”, converts a WAV on a Mac (needs brew install lame). A real Heights WAV through ingest and trim had not been seen yet.
• 2026-11-08. Subsplash Sunday tasks created and the first three Subsplash items published through them. Downtown slides are pulled automatically from the deck at hopecc.com/dtslides (widened the same day to count a deck changed on the Friday or Saturday before). Media uploads switched to multipart and PDFs over 1 MB are compressed, after an Imunify360 firewall update started refusing larger raw uploads. The Subsplash Sunday dry run passed for Downtown, St. Paul and North Lakes.
• 2026-11-05. Synology C2 offsite backup task built and first run started. Restore test pending.
• 2026-11-02. Plan written to move the Google Drive rclone remote to the church’s own OAuth client before the shared one is retired. Nothing changed.
• 2026-11-01. St. Paul display names deployed (33a02dd). The LT recording was missing, so Austin chose a 10-second silent episode titled with “No recording. See show notes” so the slides and guide reached the feed. NL, DT and CH went out through auto-distribute and Subsplash was published for all four campuses. The container stopped unexpectedly twice that day, and five more times on 2026-11-08. No cause is recorded and the task is open.
• 2026-10-25. St. Paul replaced the name Lowertown. The code LT was kept. The same day, a HyperDeck pull froze a partial copy of the Downtown recording, and the pull was fixed to re-download changed files and skip files still being written.
• 2026-10-20 and 2026-10-27. Pinecast downgraded to the free plan. Checks the next week found nothing broken, and the last redirect-map entries that pointed at Pinecast were moved to Transistor.
• 2026-10-18. Remote access went live at publish.hopecc.com behind Cloudflare Access, and sign-in was enforced. The LAN path is a deliberate switch. The app also began running the podcast, Vimeo and publish steps by itself once a post is marked Ready.
• 2026-10-09. Subsplash API parked because of the $2,000 onboarding fee. On 2026-11-02 the account contact offered to cut it to $1,000, with no trial. Undecided.
• 2026-10-04. Downtown submits to Auphonic automatically after an in-app trim (deployed the same day with the This week fixes).
• 2026-09-24 and 2026-09-29. Storage roles set: the NAS is the record and R2 is a short-lived outbox. Vimeo library to be copied to the NAS and kept out of the offsite backup. Synology C2 recommended over Backblaze B2 for the offsite backup.
• 2026-09-22. Podcast hosting moved from Pinecast to Transistor in one evening, Columbia Heights first. All 11 Pinecast feeds were set to redirect by 2026-09-24. Campus term lookup became closed, so a wrong campus name fails instead of creating a duplicate term.
• 2026-09-18 and 2026-09-25. The legacy sermon archive (3,428 files) and the live R2 tiers (575 files) were copied to the NAS and verified.
• 2026-10-25. First full Sunday with all four campuses marked Ready in one go. Auto-distribute finished the podcast and publish steps for all four between 15:19 and 15:23 and Downtown’s Vimeo upload at 15:24 with no failures, and Subsplash was published for all four.
• 2026-09-13. Last full Sunday run done by hand from a Claude session. The 2026-09-27 run was the first through the app’s Distribute tab.
• 2026-08-19. Cutover from the Mac mini to utility-nas. The HyperDeck schedule was built into the app because DSM 7.4 cannot create scheduled tasks from the command line.
• 2026-08-09. Found that every WordPress meta key written since the first build was wrong and no sermon page rendered it. Fixed with asp_-prefixed keys.
• Earlier choices still in force. Trim by ffmpeg stream copy with no re-encode. Auphonic by API, never watch folders. DT audio goes to Auphonic alone and is remuxed onto the video as AAC. Raw files are never overwritten. Vimeo only for Downtown. YouTube to replace Vimeo for new Downtown video was decided and not built.

Where sources disagree:

• The repo README still says the podcast step is manual on Pinecast with unset show IDs. Newer vault entries say Transistor has been live since 2026-09-22 with show IDs set. This page follows the vault.
• The 2026-10-09 decision kept Subsplash as a copy-paste step. The 2026-11-01 and 2026-11-08 entries and the repo README of 2026-11-09 describe a browser-driven Claude skill that does the entry. This page follows the newer entries and keeps the copy-paste card as the fallback.