Backup descentralizatDecentralized backup
O a doua cutie ThorStorage pe care internetul nu o poate atinge. Ea se conectează singură la storage-ul principal prin FTPS criptat, pe program, și trage doar ce s-a schimbat. Fără porturi deschise spre exterior, fără expunere — o oglindă unidirecțională, incrementală. A second ThorStorage box the internet can't reach. It connects out to your main storage over encrypted FTPS, on a schedule, and pulls only what changed. No inbound ports, no exposure — a one-way, incremental mirror.
Ideea în două cuvinte: backup-ul trage, nu primește. Mașina de backup pornește ea conexiunea către storage-ul principal — principalul nu se conectează niciodată la ea. Așa, chiar dacă principalul e compromis, nu are cum să ajungă la copia de siguranță. The idea in one line: the backup pulls, it's never pushed to. The backup box opens the connection to the main storage — the main box never connects back to it. So even if the main box is compromised, it has no path into your backup.
Cum aratăHow it looks
Conexiunea pleacă mereu de la B spre A. B nu are niciun port deschis spre internet, deci din exterior pur și simplu nu există. The connection always starts at B and goes to A. B has no inbound port open to the internet, so from the outside it simply isn't there.
De ce e sigurWhy it's safe
Incremental, din v2.44: jobul are un STATE_DIR mic și persistent unde ține un index al fișierelor. La prima rulare aduce tot; la fiecare rulare ulterioară transferă doar ce e nou sau modificat — nu tot de la zero.
Incremental, since v2.44: the job keeps a small persistent STATE_DIR holding a file index. The first run fetches everything; every run after that transfers only what's new or changed — never the whole set again.
Ai nevoie deYou'll need
ftp_sync.py — sincronizarea incrementală prin FTP/FTPS. Îl lipești ca job în Scheduler.ftp_sync.py — the incremental FTP/FTPS sync. Paste it as a Scheduler job.Pas cu pasStep by step
În ThorStorage → Utilizatori, creează un cont FTP dedicat, cu „acasă" pe folderul pe care vrei să-l oglindești. Notează host-ul, utilizatorul și parola. Ai grijă ca FTPS să fie activ (implicit e). Folosește un cont separat, doar pentru backup — nu contul tău principal. In ThorStorage → Users, create a dedicated FTP account whose home is the folder you want mirrored. Note the host, username and password. Make sure FTPS is on (it is by default). Use a separate account just for the backup — not your main one.
Alege (sau creează) folderul de output al jobului — acolo se vor aduna fișierele copiate. Poate fi un cont FTP local pe B, ca să le poți lua și de acolo. Confirmă că mașina B nu are niciun port forwardat din router spre ea. Pick (or create) the job's output folder — that's where the copied files accumulate. It can be a local FTP account on B so you can fetch them from there too. Confirm B has no port forwarded to it from the router.
ftp_sync.pyOn B: a Scheduler job running ftp_sync.pyScheduler → Job nou → lipește ftp_sync.py și editează doar blocul CONFIG:
Scheduler → New job → paste ftp_sync.py and edit only the CONFIG block:
HOST = "29.42.17.2" # IP-ul principalului (A)the main box (A) USER = "backup" # contul de la pasul 1the account from step 1 REMOTE_DIR = "/" # ce folder de pe Awhich folder on A USE_TLS = True # FTPS — lasă-l pornit— keep it on
Parola o pui în variabila de mediu FTP_PASSWORD a jobului (nu în text). Apoi: Folder de output = folderul de la pasul 2 · Rețea = pornită · Program = ex. la fiecare oră · bifează „Trimite-mi email dacă un job pică".
Put the password in the job's FTP_PASSWORD environment variable (not in the text). Then: Output folder = the folder from step 2 · Network = on · Schedule = e.g. hourly · tick "Email me if a run fails".
Rulează jobul o dată manual („Run now"). Prima oară descarcă tot; jobul își ține un index în STATE_DIR, așa că rulările următoare aduc doar fișierele noi sau schimbate. Logul arată exact ce s-a transferat:
Run the job once by hand ("Run now"). The first time it downloads everything; the job keeps an index in STATE_DIR, so later runs fetch only new or changed files. The log shows exactly what moved:
De acum, la fiecare rulare programată, B trage singur ce e nou de pe A și îl adaugă în oglindă. Dacă vreo rulare pică, primești email. Fișierele șterse pe A rămân în oglinda de pe B (marcate în index) — un backup nu-ți pierde datele când dispar de pe sursă. From now on, on every scheduled run, B pulls whatever is new on A and adds it to the mirror. If a run fails, you get an email. Files deleted on A stay in B's mirror (flagged in the index) — a backup doesn't lose your data when the source does.
Ține B în afara internetului. Toată siguranța vine din faptul că B nu are niciun port deschis spre exterior. Nu-i face port-forward „ca să-l vezi de acasă" — dacă ai nevoie de acces la B, folosește un VPN spre rețeaua lui, nu o expunere directă. Keep B off the internet. All of the safety comes from B having no inbound port. Don't port-forward it "so I can see it from home" — if you need to reach B, use a VPN into its network, not a direct exposure.
Scriptul completThe full script
ftp_sync.py
Ăsta e scriptul pe care îl lipești la pasul 3. Editează doar blocul CONFIG de sus (host, utilizator, folder). Parola o pui în variabila de mediu FTP_PASSWORD a jobului. În Scheduler, OUTPUT_DIR și STATE_DIR vin automat din mediu — nu le atinge.
This is the script you paste in step 3. Edit only the CONFIG block at the top (host, user, folder). Put the password in the job's FTP_PASSWORD environment variable. In the Scheduler, OUTPUT_DIR and STATE_DIR arrive automatically from the environment — leave them alone.
Descarcă ftp_sync.pyDownload ftp_sync.py
#!/usr/bin/env python3
"""
ftp_sync.py — Incremental FTP/FTPS mirror for the ThorStorage Scheduler.
Downloads ONLY what is missing or changed on the remote server.
ftp_sync.py — Oglindă incrementală FTP/FTPS pentru Scheduler-ul ThorStorage.
Descarcă DOAR ce lipsește sau s-a schimbat pe serverul remote.
HOW IT WORKS IN THE SCHEDULER (important):
- Downloaded files are written to OUTPUT_DIR. After each run, ThorStorage PUBLISHES
the contents of OUTPUT_DIR into the job's output folder (adds/replaces, never deletes),
so the full mirror builds up over time.
- OUTPUT_DIR is EMPTY on every run (you don't see last run's files), so the
"already downloaded?" decision CANNOT rely on local files — it relies on an INDEX
(remote size + date) kept in STATE_DIR.
- STATE_DIR is a small folder that PERSISTS between runs. It holds .ftp_index.json,
so the second run downloads only the delta, not everything.
CUM FUNCȚIONEAZĂ ÎN SCHEDULER (important):
- Fișierele descărcate se scriu în OUTPUT_DIR. După fiecare rulare, ThorStorage
PUBLICĂ conținutul lui OUTPUT_DIR în folderul de output al jobului (adaugă/înlocuiește,
nu șterge), așa se adună oglinda completă în timp.
- OUTPUT_DIR e GOL la fiecare rulare (nu vezi fișierele descărcate data trecută), deci
decizia „am descărcat deja?" NU se poate baza pe fișierele locale — se bazează pe un
INDEX (dimensiune + dată remote) ținut în STATE_DIR.
- STATE_DIR e un folder mic care PERSISTĂ între rulări. Aici se ține .ftp_index.json,
așa că la a doua rulare descărcăm doar delta, nu tot.
Edit the CONFIG block. Password: put it in CONFIG or in the FTP_PASSWORD env var.
Editează blocul CONFIG. Parola: pune-o în CONFIG sau în variabila de mediu FTP_PASSWORD.
"""
import ftplib
import hashlib
import json
import logging
import os
import sys
from datetime import datetime, timezone
# ================== CONFIG ==================
HOST = "ftp.example.com" # EN: server IP or hostname | RO: IP sau hostname server
PORT = 21
USER = "username" # EN: FTP account name | RO: numele contului FTP
PASSWORD = "" # EN: leave "" and use FTP_PASSWORD at runtime
# RO: lasă "" și folosește FTP_PASSWORD la rulare
REMOTE_DIR = "/pub/data" # EN: the folder on the server | RO: directorul de pe server
RECURSIVE = True # EN: walk subfolders too | RO: parcurge subdirectoare
USE_TLS = True # EN: True = FTPS (FTP over TLS), recommended
# RO: True = FTPS (FTP peste TLS), recomandat
PASSIVE = True # EN: False = active mode | RO: False = mod activ
TIMEOUT = 30
DRY_RUN = False # EN: True = report only, don't download
# RO: True = doar raportează, nu descarcă
VERBOSE = False
# ============================================
# EN: In the Scheduler these come from the environment; outside it, they fall back below.
# RO: În Scheduler acestea vin din mediu; în afara lui, cad pe căile de mai jos.
OUTPUT_DIR = os.environ.get("OUTPUT_DIR") or "./out" # EN: downloaded files (published) | RO: fișierele descărcate (se publică)
STATE_DIR = os.environ.get("STATE_DIR") or OUTPUT_DIR # EN: the index (persists) | RO: indexul (persistă între rulări)
INDEX_PATH = os.path.join(STATE_DIR, ".ftp_index.json")
log = logging.getLogger("ftp_sync")
# ---------------------------------------------------------------- index
def load_index():
if not os.path.exists(INDEX_PATH):
return {"files": {}, "last_sync": None}
try:
with open(INDEX_PATH, "r", encoding="utf-8") as f:
data = json.load(f)
data.setdefault("files", {})
return data
except (json.JSONDecodeError, OSError) as e:
# EN: index is corrupt — rebuild from scratch (everything is re-downloaded once).
# RO: index corupt — se reconstruiește de la zero (se redescarcă tot o dată).
log.warning("Index corrupt (%s), rebuilding from scratch (everything re-downloads once).", e)
return {"files": {}, "last_sync": None}
def save_index(index):
os.makedirs(STATE_DIR, exist_ok=True)
# EN: write to a temp file then rename, so the index is never left half-written.
# RO: scrie într-un fișier temporar apoi redenumește, ca indexul să nu rămână scris pe jumătate.
tmp = INDEX_PATH + ".tmp"
with open(tmp, "w", encoding="utf-8") as f:
json.dump(index, f, indent=2, ensure_ascii=False)
os.replace(tmp, INDEX_PATH)
def sha256_file(path, chunk=1 << 20):
# EN: hash the file in chunks, so a big file costs no memory.
# RO: calculează hash-ul pe bucăți, ca un fișier mare să nu consume memorie.
h = hashlib.sha256()
with open(path, "rb") as f:
for block in iter(lambda: f.read(chunk), b""):
h.update(block)
return h.hexdigest()
# ---------------------------------------------------------------- ftp
def connect():
cls = ftplib.FTP_TLS if USE_TLS else ftplib.FTP
ftp = cls(timeout=TIMEOUT)
ftp.connect(HOST, PORT)
ftp.login(USER, PASSWORD)
if USE_TLS:
ftp.prot_p() # EN: encrypt the data channel too, not just the login
# RO: criptează și canalul de date, nu doar autentificarea
ftp.set_pasv(PASSIVE)
return ftp
def list_remote(ftp, remote_dir, _prefix=""):
"""EN: {relative_path: {"size": int, "mtime": str|None}} — MLSD if available, else SIZE+MDTM.
RO: {cale_relativă: {"size": int, "mtime": str|None}} — MLSD dacă există, altfel SIZE+MDTM."""
out = {}
ftp.cwd(remote_dir)
try:
for name, facts in list(ftp.mlsd()):
if name in (".", ".."):
continue
typ = facts.get("type", "")
rel = _prefix + name
if typ == "file":
out[rel] = {"size": int(facts.get("size", -1)), "mtime": facts.get("modify")}
elif typ == "dir" and RECURSIVE:
out.update(list_remote(ftp, name, rel + "/"))
ftp.cwd("..")
return out
except (ftplib.error_perm, ftplib.error_proto):
pass # EN: MLSD unavailable — fall back below | RO: MLSD indisponibil — fallback mai jos
for name in ftp.nlst():
if name in (".", ".."):
continue
size = mtime = None
try:
size = ftp.size(name)
except (ftplib.error_perm, ftplib.error_reply):
pass
if size is None:
continue # EN: probably a directory (no recursion on the fallback path)
# RO: probabil director (fără recursivitate pe fallback)
try:
resp = ftp.sendcmd("MDTM " + name)
if resp.startswith("213"):
mtime = resp[4:].strip()[:14]
except (ftplib.error_perm, ftplib.error_reply):
pass
out[_prefix + name] = {"size": size, "mtime": mtime}
return out
def download(ftp, rel_path, local_path):
os.makedirs(os.path.dirname(local_path) or ".", exist_ok=True)
# EN: download to a .part file, then rename — a half-file never looks complete.
# RO: descarcă într-un fișier .part, apoi redenumește — un fișier pe jumătate nu pare complet.
tmp = local_path + ".part"
ftp.cwd(REMOTE_DIR)
with open(tmp, "wb") as f:
ftp.retrbinary("RETR " + rel_path, f.write, blocksize=1 << 16)
os.replace(tmp, local_path)
# ---------------------------------------------------------------- logic / logica
def needs_download(meta, entry):
"""EN: The decision is made FROM THE INDEX (not from local files — OUTPUT_DIR is empty each run).
RO: Decizia se ia DUPĂ INDEX (nu după fișierele locale — OUTPUT_DIR e gol la fiecare rulare)."""
if entry is None:
return "new (not in the index)"
if entry.get("remote_size") != meta["size"]:
return "size changed on remote"
if meta.get("mtime") and entry.get("remote_mtime") != meta["mtime"]:
return "date changed on remote"
return None
def sync():
os.makedirs(OUTPUT_DIR, exist_ok=True)
os.makedirs(STATE_DIR, exist_ok=True)
index = load_index()
files = index["files"]
log.info("Connecting to %s:%s (%s) ...", HOST, PORT, "FTPS" if USE_TLS else "FTP")
ftp = connect()
try:
remote = list_remote(ftp, REMOTE_DIR)
log.info("Remote: %d files. Index: %d known.", len(remote), len(files))
new = updated = skipped = failed = 0
for rel in sorted(remote):
meta = remote[rel]
entry = files.get(rel)
reason = needs_download(meta, entry)
if reason is None:
skipped += 1
continue
local_path = os.path.join(OUTPUT_DIR, rel.replace("/", os.sep))
log.info("%s -> %s", rel, reason)
if DRY_RUN:
continue
try:
download(ftp, rel, local_path)
except (OSError, *ftplib.all_errors) as e:
log.error("Failed to download %s: %s", rel, e)
failed += 1
continue
files[rel] = {
"remote_size": meta["size"],
"remote_mtime": meta.get("mtime"),
"downloaded_size": os.path.getsize(local_path),
"downloaded_at": datetime.now(timezone.utc).isoformat(),
"sha256": sha256_file(local_path),
}
if entry is None:
new += 1
else:
updated += 1
# EN: files that no longer exist on the remote — we only flag them in the index.
# (We can't delete them from the published mirror — the real folder isn't mounted here.)
# RO: fișiere care nu mai există pe remote — doar le însemnăm în index.
# (Nu le putem șterge din oglinda publicată — folderul real nu e montat în sandbox.)
orphans = [r for r in list(files) if r not in remote]
for rel in orphans:
files[rel]["missing_on_remote"] = True
if orphans:
log.warning("%d file(s) no longer exist on the remote (kept in the mirror).", len(orphans))
index["last_sync"] = datetime.now(timezone.utc).isoformat()
if not DRY_RUN:
save_index(index)
log.info("Done. New: %d | Updated: %d | Unchanged: %d | Failed: %d | Orphaned: %d",
new, updated, skipped, failed, len(orphans))
if failed:
sys.exit(1) # EN: mark the run as failed if any download failed
# RO: marchează rularea ca eșuată dacă a picat vreo descărcare
finally:
try:
ftp.quit()
except ftplib.all_errors:
ftp.close()
def main():
global PASSWORD
logging.basicConfig(
level=logging.DEBUG if VERBOSE else logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
datefmt="%H:%M:%S",
stream=sys.stderr,
)
# EN: the password comes from the FTP_PASSWORD env var (preferred) or from CONFIG above.
# RO: parola vine din variabila de mediu FTP_PASSWORD (de preferat) sau din CONFIG de sus.
PASSWORD = os.environ.get("FTP_PASSWORD", PASSWORD)
if not PASSWORD:
log.error("No password. Set PASSWORD in CONFIG or the FTP_PASSWORD env var. / "
"Nicio parolă. Pune PASSWORD în CONFIG sau variabila de mediu FTP_PASSWORD.")
sys.exit(2)
log.info("Syncing %s:%s%s -> OUTPUT_DIR=%s (index in STATE_DIR=%s)",
HOST, PORT, REMOTE_DIR, OUTPUT_DIR, STATE_DIR)
try:
sync()
except ftplib.all_errors as e:
log.error("FTP error: %s", e)
sys.exit(1)
if __name__ == "__main__":
main()
Varianta recursivăThe recursive variant
ftp_sync.py — recursiv, tot arborelerecursive, whole tree
Aceeași idee, dar parcurge toate subdirectoarele de pe server și recreează structura identic, pornind din directorul curent (LOCAL_DIR = "."). Ține un index JSON (.ftp_index.json) cu folderul, dimensiunea și data fiecărui fișier — o imagine clară a structurii remote, pe care o poți reconstrui oriunde. Îl rulezi de mână sau din cron pe cutia B; editează doar blocul CONFIG. Parola o pune în variabila FTP_PASSWORD (dacă lipsește, o cere la rulare). Opțiuni: DELETE_ORPHANS, PRUNE_EMPTY, VERIFY_HASH, DRY_RUN.
Same idea, but it walks every subdirectory on the server and recreates the structure identically, starting from the current directory (LOCAL_DIR = "."). It keeps a JSON index (.ftp_index.json) with each file's folder, size and date — a clear picture of the remote structure you can rebuild anywhere. Run it by hand or from cron on box B; edit only the CONFIG block. Put the password in FTP_PASSWORD (if missing, it prompts at runtime). Options: DELETE_ORPHANS, PRUNE_EMPTY, VERIFY_HASH, DRY_RUN.
Descarcă ftp_sync.py (recursiv)Download ftp_sync.py (recursive)
#!/usr/bin/env python3
"""
ftp_sync.py
RO: Sincronizeaza recursiv directorul curent cu un director remote FTP.
Parcurge TOATE subdirectoarele de pe server si le recreeaza local identic,
pornind mereu din directorul din care rulezi scriptul (LOCAL_DIR = ".").
Tine un index JSON (.ftp_index.json) in care fiecare fisier are: folderul in
care se afla, dimensiune remote, mtime remote, data descarcarii locale si
optional hash. Astfel ai oricand o imagine clara a structurii remote si o
poti reconstrui exact in alta parte (foldere + fisiere).
La fiecare rulare: scaneaza recursiv serverul, recreeaza folderele local,
descarca ce lipseste sau ce nu corespunde (size / mtime / hash) si marcheaza
in index fisierele disparute de pe remote.
EN: Recursively syncs the current directory with a remote FTP directory.
Walks EVERY subdirectory on the server and recreates it locally, always
starting from the directory the script is run in (LOCAL_DIR = ".").
Keeps a JSON index (.ftp_index.json) where each file records: its folder,
remote size, remote mtime, local download timestamp and optional hash. This
gives a clear picture of the remote structure so it can be rebuilt exactly
elsewhere (folders + files).
Each run: recursively scans the server, recreates folders locally, downloads
what is missing or mismatched (size / mtime / hash) and flags files removed
from remote in the index.
RO: Editeaza blocul CONFIG de mai jos si ruleaza: python ftp_sync.py
EN: Edit the CONFIG block below and run: python ftp_sync.py
"""
import ftplib
import hashlib
import json
import logging
import os
import posixpath
import sys
from datetime import datetime, timezone
# ================== CONFIG ==================
# RO: host / IP server EN: server host / IP
HOST = "ftp.exemplu.ro"
PORT = 21
USER = "utilizator"
# RO: lasa "" ca sa o ceara la rulare EN: leave "" to be prompted at runtime
PASSWORD = "parola_aici"
# RO: directorul radacina de pe server EN: remote root directory
REMOTE_DIR = "/pub/date"
# RO: mereu directorul curent (de aici incepe oglinda)
# EN: always the current directory (mirror starts here)
LOCAL_DIR = "."
USE_TLS = False # RO: True = FTPS | EN: True = FTPS
PASSIVE = True # RO: False = mod activ | EN: False = active mode
VERIFY_HASH = False # RO: sha256 la fiecare fisier | EN: sha256 per file (slower)
DELETE_ORPHANS = False # RO: sterge local ce a disparut | EN: delete local files gone from remote
PRUNE_EMPTY = False # RO: sterge folderele goale | EN: remove leftover empty folders
DRY_RUN = False # RO: doar raporteaza, nu descarca| EN: report only, do not download
VERBOSE = False
TIMEOUT = 30
# ============================================
INDEX_NAME = ".ftp_index.json"
log = logging.getLogger("ftp_sync")
# ---------------------------------------------------------------- index
def load_index(path):
# RO: incarca indexul existent sau porneste unul gol.
# EN: load the existing index or start an empty one.
if not os.path.exists(path):
return {"root": REMOTE_DIR, "files": {}, "dirs": [], "last_sync": None}
try:
with open(path, "r", encoding="utf-8") as f:
data = json.load(f)
data.setdefault("files", {})
data.setdefault("dirs", [])
return data
except (json.JSONDecodeError, OSError) as e:
# RO: index corupt -> se reconstruieste de la zero.
# EN: corrupt index -> rebuild from scratch.
log.warning("Index corupt / corrupt index (%s), rebuilding.", e)
return {"root": REMOTE_DIR, "files": {}, "dirs": [], "last_sync": None}
def save_index(path, index):
# RO: scriere atomica (tmp + replace) ca sa nu corupem indexul.
# EN: atomic write (tmp + replace) to avoid corrupting the index.
tmp = path + ".tmp"
with open(tmp, "w", encoding="utf-8") as f:
json.dump(index, f, indent=2, ensure_ascii=False)
os.replace(tmp, path)
def sha256_file(path, chunk=1 << 20):
# RO: calculeaza sha256 pe bucati (fisiere mari).
# EN: compute sha256 in chunks (large files).
h = hashlib.sha256()
with open(path, "rb") as f:
for block in iter(lambda: f.read(chunk), b""):
h.update(block)
return h.hexdigest()
# ---------------------------------------------------------------- ftp
def connect():
# RO: deschide conexiunea (FTP sau FTPS) si autentifica.
# EN: open the connection (FTP or FTPS) and log in.
cls = ftplib.FTP_TLS if USE_TLS else ftplib.FTP
ftp = cls(timeout=TIMEOUT)
ftp.connect(HOST, PORT)
ftp.login(USER, PASSWORD)
if USE_TLS:
ftp.prot_p() # RO: cripteaza si canalul de date | EN: encrypt data channel too
ftp.set_pasv(PASSIVE)
return ftp
def walk_remote(ftp, root):
"""
RO: Parcurge recursiv 'root' pe server. Intoarce (files, dirs):
files = { cale_relativa_posix: {"dir": folder_rel, "name": nume,
"size": int, "mtime": str|None} }
dirs = [ cale_relativa_folder, ... ] (toate folderele gasite)
Caile relative sunt fata de 'root', cu separator '/'.
EN: Recursively walk 'root' on the server. Returns (files, dirs):
files = { posix_relative_path: {"dir": rel_folder, "name": name,
"size": int, "mtime": str|None} }
dirs = [ rel_folder_path, ... ] (all folders found)
Relative paths are against 'root', using '/' separator.
"""
files = {}
dirs = []
# RO: stiva (cale_absoluta_remote, cale_relativa) - evita recursia adanca.
# EN: stack (absolute_remote_path, relative_path) - avoids deep recursion.
stack = [(root, "")]
while stack:
abs_dir, rel_dir = stack.pop()
try:
ftp.cwd(abs_dir)
except ftplib.all_errors as e:
# RO: folder inaccesibil -> il sarim. EN: unreachable folder -> skip.
log.warning("Nu pot intra / cannot enter %s: %s", abs_dir, e)
continue
if rel_dir:
dirs.append(rel_dir)
for name, typ, size, mtime in _list_dir(ftp):
if name in (".", ".."):
continue
rel = posixpath.join(rel_dir, name) if rel_dir else name
abs_child = posixpath.join(abs_dir, name)
if typ == "dir":
# RO: pune folderul in stiva spre procesare.
# EN: push the folder onto the stack for processing.
stack.append((abs_child, rel))
elif typ == "file":
files[rel] = {
"dir": rel_dir, # RO: "" = radacina | EN: "" = root
"name": name,
"size": size if size is not None else -1,
"mtime": mtime,
}
return files, dirs
def _list_dir(ftp):
"""
RO: Listeaza directorul curent. Intoarce [(name, type, size, mtime), ...],
type in {"file","dir"}. Foloseste MLSD; daca lipseste, euristica NLST.
EN: List the current directory. Returns [(name, type, size, mtime), ...],
type in {"file","dir"}. Uses MLSD; falls back to NLST heuristics.
"""
out = []
try:
# RO: MLSD ofera tip + dimensiune + data direct (cand e suportat).
# EN: MLSD gives type + size + date directly (when supported).
for name, facts in ftp.mlsd():
typ = facts.get("type", "")
if typ == "file":
out.append((name, "file",
_to_int(facts.get("size")), facts.get("modify")))
elif typ == "dir":
out.append((name, "dir", None, None))
# RO: cdir/pdir/other ignorate | EN: cdir/pdir/other ignored
return out
except (ftplib.error_perm, ftplib.error_proto):
pass # RO: MLSD indisponibil | EN: MLSD unavailable
# RO: fallback NLST - deducem tipul din SIZE / posibilitatea de a intra.
# EN: NLST fallback - infer type from SIZE / ability to CWD in.
for entry in ftp.nlst():
base = posixpath.basename(entry.rstrip("/"))
if base in (".", ".."):
continue
size = None
try:
size = ftp.size(base) # RO: eroare/None -> posibil folder | EN: err/None -> maybe folder
except (ftplib.error_perm, ftplib.error_reply):
size = None
if size is None:
# RO: test - daca putem intra, e director.
# EN: test - if we can CWD into it, it is a directory.
cur = ftp.pwd()
try:
ftp.cwd(base)
ftp.cwd(cur)
out.append((base, "dir", None, None))
continue
except ftplib.all_errors:
try:
ftp.cwd(cur)
except ftplib.all_errors:
pass
# RO: fisier - incercam sa luam data prin MDTM.
# EN: file - try to get the timestamp via MDTM.
mtime = None
try:
resp = ftp.sendcmd("MDTM " + base)
if resp.startswith("213"):
mtime = resp[4:].strip()[:14]
except (ftplib.error_perm, ftplib.error_reply):
pass
out.append((base, "file", size, mtime))
return out
def _to_int(v):
# RO: conversie sigura la int (sau None). EN: safe int conversion (or None).
try:
return int(v)
except (TypeError, ValueError):
return None
def download(ftp, abs_remote_path, local_path):
# RO: descarca in .part apoi redenumeste (atomic).
# EN: download to .part then rename (atomic).
os.makedirs(os.path.dirname(local_path) or ".", exist_ok=True)
tmp = local_path + ".part"
with open(tmp, "wb") as f:
ftp.retrbinary("RETR " + abs_remote_path, f.write, blocksize=1 << 16)
os.replace(tmp, local_path)
# ---------------------------------------------------------------- logica / logic
def needs_download(meta, local_path, entry):
# RO: decide daca fisierul trebuie (re)descarcat si de ce.
# EN: decide whether the file must be (re)downloaded and why.
if not os.path.exists(local_path):
return "lipseste local / missing locally"
local_size = os.path.getsize(local_path)
if meta["size"] >= 0 and local_size != meta["size"]:
return f"size diferit / size mismatch (local {local_size} != remote {meta['size']})"
if entry is None:
return "lipseste din index / missing from index"
if entry.get("remote_size") != meta["size"]:
return "size din index difera de remote / index size differs from remote"
if meta.get("mtime") and entry.get("remote_mtime") != meta["mtime"]:
return "mtime remote schimbat / remote mtime changed"
if VERIFY_HASH and entry.get("sha256"):
if sha256_file(local_path) != entry["sha256"]:
return "hash local modificat / local hash changed"
return None
def local_path_for(local_root, rel):
# RO: transforma calea relativa posix in cale locala corecta pe OS.
# EN: turn a posix relative path into the correct OS-specific local path.
return os.path.join(local_root, *rel.split("/"))
def sync():
# RO: radacina locala = directorul curent, rezolvat absolut.
# EN: local root = current directory, resolved to absolute.
local_root = os.path.abspath(LOCAL_DIR)
os.makedirs(local_root, exist_ok=True)
index_path = os.path.join(local_root, INDEX_NAME)
index = load_index(index_path)
index["root"] = REMOTE_DIR
files = index["files"]
log.info("Radacina locala / local root: %s", local_root)
log.info("Conectare / connecting to %s:%s ...", HOST, PORT)
ftp = connect()
try:
log.info("Scanez recursiv / scanning recursively %s ...", REMOTE_DIR)
remote_files, remote_dirs = walk_remote(ftp, REMOTE_DIR)
log.info("Gasit / found: %d fisiere / files in %d subdirectoare / subdirs.",
len(remote_files), len(remote_dirs))
# RO: recreeaza structura de foldere local.
# EN: recreate the folder structure locally.
for d in remote_dirs:
path = local_path_for(local_root, d)
if not DRY_RUN:
os.makedirs(path, exist_ok=True)
index["dirs"] = sorted(remote_dirs)
new = updated = skipped = 0
for rel in sorted(remote_files):
meta = remote_files[rel]
local_path = local_path_for(local_root, rel)
abs_remote = posixpath.join(REMOTE_DIR, rel)
entry = files.get(rel)
reason = needs_download(meta, local_path, entry)
if reason is None:
skipped += 1
continue
existed = os.path.exists(local_path)
log.info("[%s] %s -> %s",
meta["dir"] or "(radacina/root)", meta["name"], reason)
if DRY_RUN:
continue
try:
download(ftp, abs_remote, local_path)
except (OSError, *ftplib.all_errors) as e:
log.error("Esec descarcare / download failed %s: %s", rel, e)
continue
# RO: notam in index folderul + toate detaliile fisierului.
# EN: record in the index the folder + all file details.
files[rel] = {
"dir": meta["dir"], # RO: folder rel. la root | EN: folder rel. to root
"name": meta["name"],
"remote_path": abs_remote,
"remote_size": meta["size"],
"remote_mtime": meta.get("mtime"),
"local_size": os.path.getsize(local_path),
"downloaded_at": datetime.now(timezone.utc).isoformat(),
"sha256": sha256_file(local_path) if VERIFY_HASH else None,
"reason": reason,
}
if existed:
updated += 1
else:
new += 1
# RO: fisiere care nu mai exista pe remote.
# EN: files that no longer exist on remote.
orphans = [r for r in files if r not in remote_files]
for rel in orphans:
local_path = local_path_for(local_root, rel)
if DELETE_ORPHANS and not DRY_RUN:
if os.path.exists(local_path):
os.remove(local_path)
del files[rel]
log.info("%s -> sters / deleted (nu mai e pe remote / gone from remote)", rel)
else:
files[rel]["missing_on_remote"] = True
log.warning("%s nu mai e pe remote / gone from remote (pastrat / kept)", rel)
# RO: optional, curata folderele locale ramase goale.
# EN: optionally, clean up leftover empty local folders.
if PRUNE_EMPTY and not DRY_RUN:
_prune_empty_dirs(local_root)
index["last_sync"] = datetime.now(timezone.utc).isoformat()
if not DRY_RUN:
save_index(index_path, index)
log.info("Gata / done. Noi/new: %d | Actualizate/updated: %d | "
"Neschimbate/unchanged: %d | Foldere/folders: %d | Orfane/orphans: %d",
new, updated, skipped, len(remote_dirs), len(orphans))
finally:
# RO: inchidem curat conexiunea. EN: close the connection cleanly.
try:
ftp.quit()
except ftplib.all_errors:
ftp.close()
def _prune_empty_dirs(root):
# RO: sterge recursiv folderele goale (fara a atinge radacina/indexul).
# EN: recursively remove empty folders (without touching the root/index).
root_abs = os.path.abspath(root)
for dirpath, dirnames, filenames in os.walk(root, topdown=False):
if os.path.abspath(dirpath) == root_abs:
continue
real = [f for f in filenames if f != INDEX_NAME]
if not real and not dirnames:
try:
os.rmdir(dirpath)
log.info("Folder gol sters / empty folder removed: %s", dirpath)
except OSError:
pass
def main():
global PASSWORD
logging.basicConfig(
level=logging.DEBUG if VERBOSE else logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
datefmt="%H:%M:%S",
)
# RO: variabila de mediu are prioritate; daca lipseste, cerem parola.
# EN: env variable takes priority; if missing, prompt for the password.
PASSWORD = os.environ.get("FTP_PASSWORD", PASSWORD)
if not PASSWORD:
import getpass
PASSWORD = getpass.getpass("Parola FTP / FTP password: ")
log.info("Sincronizez recursiv / recursive sync %s:%s%s -> %s (dir curent / current dir)",
HOST, PORT, REMOTE_DIR, os.path.abspath(LOCAL_DIR))
try:
sync()
except ftplib.all_errors as e:
log.error("Eroare FTP / FTP error: %s", e)
sys.exit(1)
except KeyboardInterrupt:
log.warning("Intrerupt / interrupted.")
sys.exit(130)
if __name__ == "__main__":
main()