# Firmware collection: download → extract → organize → cloud

For building/organizing large firmware libraries (e.g. avto1000.com mega packs, GIFROM
torrents). All tools bundled; all designed to survive a session (run under `nohup`).

## Target structure

```
<root>/<Brand>/<Model>/<CalibrationID>/<Category>/[Calibrator] name__<sha10>.bin
```
- **Brand/Model**: classified from path (Kia-Hyundai, Lada, Ford…; Cerato, Vesta…).
- **CalibrationID**: read from the BIN itself (ASCII at ~`0x3FE2F`, e.g. `GTD-B46CQS3-C000`),
  more reliable than the path. (`organize_firmware.py` does this.)
- **Category**: Stock / Tune / Euro2 / ImmoOff / Other (keyword + cal-id-only heuristics).
- **Calibrator**: Paulus, Ledokol, MMK, MotorMaster, RSW, Chelyaba… from path tokens.
- **sha10**: 10-char SHA for dedup across overlapping packs.

## Tools

- `tools/organize_firmware.py` — walks an `_extracted/` dir, classifies, copies into the
  structure with dedup. Reuse its functions (`classify_brand`, `detect_calibration_id`,
  `file_hash`, …) by exec-ing its header (lines 1–219) — its main body runs at import, so
  don't `import` it raw.
- `tools/collection_pipeline.py` — streaming daemon: for each downloaded archive →
  extract → organize → `rclone move` to cloud → delete local. Dedup via a persistent
  `_seen_hashes.txt`. Survives the session. **It is guarded with `if __name__=="__main__"`** —
  keep it that way (importing an unguarded daemon spawns a second copy that races).
- `tools/download_mega.sh` — mega.nz mass downloader with IP rotation (`megadl --proxy`) and
  a 900s stall watchdog; resumes via `.done` markers.
- `tools/ip_rotation_tunnel.sh` — auto-reconnecting SSH tunnel to a rotating SOCKS proxy.

## IP rotation (beats mega's ~1GB/IP/day quota)

mega free tier throttles per IP (HTTP 509). Route `megadl` through a proxy whose outbound
**rotates** exit IPs. If you have an xray/v2ray box with a balancer set to `random` over
multiple outbounds, tunnel that SOCKS port locally and point megadl at it:
```bash
# tunnel remote 127.0.0.1:<rotate-port> → local 12090, auto-reconnect:
bash tools/ip_rotation_tunnel.sh    # edit host/ports inside first
# verify rotation: repeated calls should show DIFFERENT IPs
for i in 1 2 3; do curl -s --socks5-hostname 127.0.0.1:12090 https://api.ipify.org; echo; done
# megadl picks it up via MEGA_PROXY (default socks5://127.0.0.1:12090 in download_mega.sh)
```
A random-balancer SOCKS inbound rotates exit per connection → effectively N× the quota.
megadl tolerates per-connection IP changes mid-download (mega storage GET is range-based).

## Extraction gotchas

- **Use `unar` FIRST, `7z` as fallback.** 7z chokes on many RAR compression methods
  ("Unsupported Method"); unar (The Unarchiver) handles them. Password for avto1000 packs: `veer90`.
- **Recurse into nested archives.** Big packs are archives-within-archives; extract, then
  scan the output for `.rar/.zip/.7z` and extract those too (bounded depth). Without this a
  3.6GB pack yields 0 firmwares.
- **Don't delete on 0 files.** If organize finds 0 NEW firmwares it's usually all-duplicate
  (content already uploaded via an overlapping pack) — keep/skip, don't treat as success-delete.
- **Commit dedup hashes only AFTER a confirmed upload.** Otherwise a failed upload poisons
  `_seen_hashes.txt` and the retry sees "0 new". (collection_pipeline.py already does this.)
- **rclone to Google Drive throttles** on thousands of tiny files (per-file API ~2–3/s even
  at `--transfers=16`). Expect the upload tail to be the slow part; it's not stuck.
- `zsh` aborts `rm -f *.a *.b` entirely if one glob has no match — use `find … -delete`.
- macOS TCC blocks `~/Downloads`; work elsewhere.
