Backing up our videos to Telegram, eight connections at a time
Our finished reels and films now go to a private Telegram group. One connection was far too slow, so each file goes up in parts over eight at once.

I make a lot of videos for the studio now. Reels for the games, trailers, and some longer films. A reel is 4 to 111 MB, and the 4K master of one film is 1,358 MB.
My laptop’s disk is small, so once a video is posted, its render gets deleted. The code that makes each video stays in git, so any of them can be made again. But a film can take hours to render again.
So now, before a render is deleted, it goes to a private Telegram group. The group has Topics turned on, with one topic for each series, like the music machine reels and the trailers. Telegram’s FAQ says you can share “an unlimited number of photos, videos and files” of up to 2 GB each. That is free storage for big files, as long as each one is under 2 GB.
Here is how our tool works, the one problem we hit, and how you can make your own.
The bot interface stops at 50 MB
The usual way to drive a Telegram bot is the Bot API, plain HTTP calls like sendDocument.
It is easy, and it takes uploads of up to 50 MB.
About a third of our 42 videos are bigger than that, and every film is.
The Bot API sits on top of Telegram’s own protocol, MTProto. Telethon’s docs explain it well in HTTP Bot API vs MTProto: the Bot API turns each HTTP request into MTProto calls on Telegram’s side. A bot can also skip that layer and speak MTProto itself, through a library like Telethon. Then a file can be up to 2,000 MB, the same as for a person: Telegram’s client configuration allows 4,000 parts of 512 KB each.
The Bot API has one more way past 50 MB. If you run your own Bot API server, the limit becomes 2,000 MB. Speaking MTProto directly needs no server, only a Python library.
To use MTProto you need an api_id and an api_hash of your own.
You get them at my.telegram.org, under API development tools, as Telegram’s guide to creating your application explains.
Telegram’s API terms are short and worth reading first, and they ask every app to use its own api_id.
One connection was too slow
The first real upload was a 107 MB reel, over one connection. It went up at about 0.05 MB/s. After 900 seconds the upload timed out. At that speed the reel alone needed more than half an hour.
Telethon’s own upload sends a file’s parts one after another, and waits for Telegram’s answer to each part before it sends the next. So one slow connection makes the whole file slow.
That is a real problem, not only a long wait. Telegram’s file docs say uploaded parts stay on its servers for “between several minutes and several hours”. At that speed our biggest file, the 1,358 MB film, would take about seven and a half hours.
Eight connections at once
The same file docs describe the fix. A client can keep several parts on their way at once, and use “multiple parallel call queues” on separate TCP connections. The page also says these extra connections for files use the login you already have, so nothing has to sign in again.
So our tool opens eight connections with the same login key. It puts every part number in a queue, and eight workers take numbers from it until it is empty. Tulir Asokan wrote this pattern for Telethon in his mautrix-telegram bridge, and ours works the same way.
Here is the upload, lightly trimmed from our tool:
PART = 512 * 1024
CONNECTIONS = 8
size = path.stat().st_size
total = -(-size // PART) # the number of parts, rounded up
file_id = random.getrandbits(63)
dc = await client._get_dc(client.session.dc_id)
senders = []
for _ in range(CONNECTIONS):
s = MTProtoSender(client.session.auth_key, loggers=client._log)
conn = client._connection(dc.ip_address, dc.port, dc.id,
loggers=client._log, proxy=client._proxy)
await s.connect(conn)
senders.append(s)
queue = asyncio.Queue()
for i in range(total):
queue.put_nowait(i)
fd = os.open(path, os.O_RDONLY)
async def worker(s):
while not queue.empty():
i = queue.get_nowait()
chunk = os.pread(fd, PART, i * PART)
for attempt in range(6):
try:
request = SaveBigFilePartRequest(file_id, i, total, chunk)
if await s.send(request):
break
except FloodWaitError as e:
await asyncio.sleep(e.seconds + 1)
except (ConnectionError, OSError, asyncio.TimeoutError):
await asyncio.sleep(2 ** attempt)
else:
raise SystemExit(f"part {i} failed six times")
await asyncio.gather(*(worker(s) for s in senders))
handle = InputFileBig(file_id, total, path.name)
After that it closes the file and the eight connections. A few things in it matter:
- Parts are 512 KB, the biggest Telegram allows and the size its docs recommend.
- Files over 10 MB must go up with
upload.saveBigFilePart, so that is the request here. Smaller files go up the plain way, with Telethon’supload_file. file_idis a random number the client picks for the file, and it tells Telegram which parts belong together.- Each worker reads its own part straight from the disk with
os.pread, so the whole file never sits in memory. - When Telegram asks the tool to wait, with a
FloodWaitError, the worker waits that long. A dropped connection is tried again after a longer pause each time, up to six tries. _get_dc,_connectionand_logare Telethon’s internal parts, the names that start with an underscore. They can change in any release, so pin the version you test with. Ours is Telethon 1.45.0.
That evening, over eight connections, the same 107 MB reel went up in 279 seconds. That is about 0.38 MB/s. Speeds change from hour to hour, so these are the numbers we saw, not a limit. In the full backup later, the four film masters, about 2,800 MB, went up in under 15 minutes.
Downloads were slow over one connection too, about 0.33 MB/s when we measured. So downloads work the same way, in 1 MB pieces over eight connections, each piece written straight into its place in the file. Telegram sends at most 1 MB for one request. The same reel came back in 37 seconds.
Send it as a document
Once the parts are up, one message puts them together in the group, under the right topic:
msg = await client.send_file(
group, handle, caption=caption, reply_to=topic_id, force_document=True,
attributes=[DocumentAttributeFilename(path.name)],
)
if msg.document.size != size:
raise SystemExit("Telegram has a different size. Nothing recorded.")
force_document=True is the important part.
Without it, Telethon’s send_file sends an image or a video “as such”, as a photo or a video.
A Telegram photo is not your file any more.
Telegram’s docs list the sizes it keeps of a photo, each one resized on its server, the biggest fitting in 2560 by 2560 pixels.
The Bot API also takes photos only up to 10 MB.
Each of our videos goes up with the pictures made for it, so this matters for us.
Sent as a document, a file keeps its name and its exact bytes, which is the whole point of a backup. And because every file is a document, Telegram’s answer always has the size it stored. If that is not the size on the disk, the tool stops and records nothing.
A fingerprint for every file
Before a file goes up, the tool reads it once and works out its SHA-256 fingerprint, with Python’s hashlib:
def sha256(path):
h = hashlib.sha256()
with path.open("rb") as f:
for block in iter(lambda: f.read(1 << 20), b""):
h.update(block)
return h.hexdigest()
The fingerprint goes in the message’s caption, and into an index file in git, one line per file, with its size, topic and message. If a fingerprint is already in the index, that file is skipped, so running a backup twice is safe. It also catches copies. Ten of our music machine reels sit in two folders on the laptop, and the second copy of each was skipped as “already in the vault”.
After a download, the tool works out the fingerprint again and compares:
await download(client, msg, out)
if sha256(out) != record["sha256"]:
raise SystemExit(f"{out} does not match its recorded fingerprint")
The 107 MB reel came back byte for byte the same.
The first full backup
On 18 November the tool backed up every finished video on the laptop: 42 videos and the 75 pictures that go with them, about 4.3 GB in all, the four film masters included. Now each new video is backed up before its render is deleted.
Make your own
You need:
- A private group with Topics turned on.
- A bot, made with Telegram’s @BotFather, added to the group as an admin who can manage topics.
- Your own
api_idandapi_hashfrom my.telegram.org. - Telethon, with
pip install telethon.
Keep the bot token, the api_id and the api_hash in files outside your repo, and never print them.
Telethon also saves the bot’s login in a .session file.
Anyone with that file can act as your bot, so keep it out of git too.
Then log in as the bot and find the group:
client = TelegramClient("backup-bot", api_id, api_hash)
await client.start(bot_token=bot_token)
group = await client.get_input_entity(<your group id>)
Put your group’s id where it says <your group id>.
From there, the upload, the message and the fingerprint check above are most of the tool.
Telegram is a messaging app, not a backup service. So keep a second way back to anything you cannot lose. For us that is the code in git, and the Telegram copy saves the hours it takes to render a video again.
Sources
- Telegram. Telegram FAQ.
- Telegram. Telegram Bot API: Sending files, and Using a Local Bot API Server.
- Lonami. Telethon’s Documentation: HTTP Bot API vs MTProto, and send_file in the Client Reference.
- Telegram. Client configuration: upload_max_fileparts_default.
- Telegram. Creating your Telegram Application.
- Telegram. Telegram API Terms of Service.
- Telegram. Uploading and Downloading Files.
- Tulir Asokan. parallel_file_transfer.py, in mautrix-telegram.
- Python Software Foundation. hashlib: Secure hashes and message digests.
