Download speeds, bandwidth, mirrors, hosting, and Bittorrent

We’ve had a couple of reports that downloads of the show are a bit slow at times. Here I’d like to kick off a discussion of what we might want to do about that, if anything.

I think there are the following options for ways we could make the show available:

  1. don’t do anything. It might be a little slow to download the show at times, but most people have it download in the background into their podcatcher software, so it doesn’t matter. This also requires zero technical effort :smile:
  2. set up a mirror network of people who are happy to mirror the show, and let people download from a mirror of their choice. LugRadio did this. It is a lot of technical effort to make it slick and to integrate with the RSS feed, and I’m pretty loath to do it for that reason.
  3. Move the mp3/ogg hosting somewhere else. I’m happy to hear suggestions for locations here: my biggest concern is cost, in that I would like there to not be any. Maybe somewhere like SoundCloud could help here? Are there dedicated free podcasting file hosts? (We’re happy with where the website is hosted, and we don’t want to move that.)
  4. Use Bittorrent Sync as an alternative way of distributing the show. We had a helpful offer from someone by email to the show who already runs a BTSync mirror of the show, and hopefully they will chip into this thread to explain how it works. I don’t get how a BTSync mirror will work, and to my mind torrents aren’t yet all that useful to the show because there won’t be sufficient people simultaneously downloading it to split the bandwidth between many peers. Perhaps I’m wrong in that? How would it work? Would we need two BTSync mirrors, one for mp3s and one for oggs? How does this interact with the RSS feed, or would we provide BTSync as a different and additional way of downloading the show? (So if someone says “I find the downloads slow”, we can say “you can use our BTSync mirror instead; that’s faster.” Would it even be faster? I have lots of questions.)

I’d love to hear people’s thoughts on this. Have you found downloads slow? Have you even noticed? Bear in mind that I have high standards for how easy such a thing is to use, for all of the file hosts, for listeners, and for me, and making things slick and easy normally requires lots of technical work, so if your proposed solution requires me to do a bunch of set up stuff in order for it to meet my high standards of ease-of-use, be prepared to justify why it’s better than the status quo :wink:

I don’t know what’s expensive, but libsyn gives you unlimited download bandwidth and only limits monthly uploads.
http://www.libsyn.com/plans-pricing/

Once @jonobacon reduces the audio file size, uploading 250MB a month would be enough. It would cost $15/month.

Torrent will not work with podcasts until a lot of things change.

  • There’s no podcatcher support for torrent except a few that are only for PCs.
  • Nobody will keep seeding podcasts forever so old episodes become harder to get as it happens with any old torrent file. This might be ok for news shows, but it depends on the show content if old episodes are relevant forever.
  • Nobody with a limited data plan on a phone will touch torrent and phones are how most access podcast (iphone is the exception because it is locked in on itunes)
  • Most podcatchers delete episodes after a while making it difficult to seed them for long.

It’s probably worth noting that a mirror network won’t necessarily speed up download times, either.

2 Likes

I am a little unclear on what problem we are trying to solve.

Is the issue with the download files being too big or that the server serving them is not transferring them fast enough? I would be surprised at the latter as our server has always had a pretty good connection to the net.

For former, and based on the discussion at Save bandwidth and money by better compressing the audio, I am going to investigate shrinking the files this week.

@sil we may want to automate the compression as part of our publishing script.

Most rrecent report, from Matthew Brennan Jones at https://plus.google.com/u/0/+StuartLangridge/posts/jAhFs7yPrKE: “Can you guys do something about the download speed? Every episode drips at like 14 KiB/s. I tried it on different connections and got the same result. That is to Northern California.”

I personally haven’t seen a particular problem, and I’d be perfectly happy if everyone says “I don’t think there’s a problem, so you should take option 1 do nothing because there’s nothing that needs doing”!

I can certainly do the compression stuff as long as someone gives me excruciating detail on what to do :wink:

So let’s ask the audience…how is the download speed for everyone else?

For the record, I personally haven’t had issue downloading from your UK server to my local town here in the U.S. of A., so I’m ok with keeping things as they stand. Though I could easily set you guys up with a free account on the CDN service my company provides if that’s of interest to you. Feel free to drop me a PM or email if so.

–Dan

1 Like

Thanks a lot, @dankles - we really appreciate the offer. :slight_smile:

We do indeed appreciate that offer, and I’d like to chat to you about it :smile:

I stream it via iTunes on an XBMC box and haven’t had any issues. I mention it because it any potential solution should take into account the fact that the existing method is working for one group of listeners.

1 Like

As a “one group of listeners” I’m fine with the way it is. Great sound quality, I get it in ogg, and use Gpodder to snag it. Never noticed any slowness here in Ohio.us.

1 Like

Setting up a mirror network of people willing to mirror the show isn’t as painful as you think. We have the following for uupc:-

  • A main central server which hosts the main archive of files.
  • A bunch of community maintained servers
  • A script we run at episode release which triggers a sync from all servers (or servers can just sync periodically)
  • A download php script which load balances across servers.
  • A script which checks all the mirrors periodically and emails us when one goes down or is out of sync

So basically we upload, run the sync script and forget about it. In the show notes we link to /download/uupc_s01e01.mp3 where there is a php script that does the redirect automatically. If a server goes down or is out of sync we get an email an can disable it or contact the hostmaster.

Listeners do nothing unusual, they just get a normal download link which has magic behind it. It’s worked fine for us for 6 years. Mirrors come and go, and there’s little impact. Indeed we could actually lose all our mirrors and the bitfolk vps would probably be fine, but its nice for us not to have to sustain all the hits.

hrm. That requires the mirrors to give you write access to their mirror, no? Which I didn’t want to do on LugRadio…

Out of curiosity, I checked the speed here in France (DSL) and on a fast pipe server in the central USA. In both cases the download speeds were up near the ceiling of what’s possible, DSL Europe 2+ Mbps and the server in DFW 5-6 Mbps. Unless something has changed, there shouldn’t be a problem with those speeds?

1 Like

Sorry it took me so long to reply here. I’m the bro who hosts the BtSync share for Bad Voltage. I’ve been hosting this share since Bad Voltage started last year.

Bittorrent Sync (BtSync) is an automatic distributed peer to peer content sharing application for Windows, Mac, Linux, Android, and iOS.

Basically BtSync functions on the basis of peers just like standard Bittorrent. However unlike standard torrents, BtSync shares are not static. As soon as the host provides new content, every peer syncs to that automatically. This makes it possible for listeners to connect to the share once and get new content as it gets pushed out.

The Technical Side

Technically speaking (and I know that’s what you want) all precipitants download the latest copy of BtSync from the official Bittorrent website. Windows and Mac users will get a graphical application to run on their desktop while Linux users will get a daemon that runs a web interface. I will have links below to a bash script I have written which Linux users can run to set up BtSync easily.

Once the host and peers have run BtSync, the host will generate a secret key (read+write or read-only permissions) which they will assign to a desired folder (this is done through the BtSync interface). Once this key has been generated and assigned to a folder, files can be placed into that folder and the secret key can be given to people so they can connect with their BtSync client. Those connecting will also create a folder but instead of assigning a newly generated secret key they will use for example: the Bad Voltage secret key. Once someone connects to the Bad Voltage secret key, any content that was placed into the shared folder by the host will sync to those connected (following me so far?).

Permissions work how you’d expect them to work:

  • Read+Write: Anyone connected to the share can edit files and add new content
  • Read-Only: Only the host can change the shared content.

For podcasts I’d recommend the Read-Only option for obvious reasons.

Whatever permissions you use, all newly added or changed content will automatically sync between everyone connected.

Bandwidth Benefits

There are many benefits to using BtSync for content distribution such as podcasts. The primary benefit may be bandwidth relief.

When a content provider places new content in a BtSync share, their machine will upload to the peers using full bandwidth capabilities but only until the first peer fully syncs. After the first peer fully syncs the new content, the content provider’s machine will upload at slower speeds. So the more peers that sync the new content, the less bandwidth will be requested from the content provider. Eventually with enough peers connected to the share, the content provider will be uploading nothing (or next to nothing). Everything balances out with more peers helping distribute bandwidth, a little bandwidth here and there from every peer. P2P FTW!

Another benefit is that BtSync can be hosted pretty much anywhere which includes cloud hosting services.

The Downside

Of course in order for BtSync to become a good choice as a content distribution method, enough peers are needed. In order for there to be peers, listeners/viewers etc…will need to be running a copy of BtSync. This is difficult if people don’t understand BtSync or care in general. Many people will simply use direct download links instead of something like BtSync. With that said however, BtSync is a great option to provide either way. There are probably other downsides on the technical side of BtSync but adoption is the largest in my humble opinion.

Strategy Suggestions

So what might be an efficient way to add BtSync to a workflow? For Bad Voltage this is pretty simple. Essentially the steps are:

  1. Setup BtSync on a remote host
  2. Mount the remote BtSync shared folder to a production machine
  3. Drop newly created episode into remote folder so it goes directly to BtSync share.
  4. Relax while everyone automatically gets the new episode
  5. ???
  6. Profit

Additionally a script can be created and setup on a scheduler (such as cron) to take care of this more automatically. Though that is beyond the scope of this comment.

Conclusion

While BtSync may not yet have large adoption, it’s a pretty good option to offer for content distribution. Spread your content, save some bandwidth, make people happy. BtSync is only going to improve in the future so those utilizing it now are on the front lines of what could be the next big thing.

Links

As promised, here is my BtSync install/update script for Linux users:

Mibsi - Mindless BtSync Installer:

Git Repo - Contribute, install and submit bugs

Unofficial Bad Voltage BtSync Secret Key

BOMZ7F2CBES6Z7ZOM4DSPXT2UNHLPYFDZ

Thanks for all this info! So, what do people think about BTSync? Are people using it now?

Fan of the No Agenda Podcast? They use BtSync as one of their download options. You can find it at the bottom of their front page. http://www.noagendashow.com/

I use BitTorrentSync to backup my personal files (photos, programming files, various docs, etc) across my machines. I would be happy to add a Read-Only folder to my setup to support this option.

That being said I get my podcast download through the PocketCasts app on android, so this would not be my primary method of obtaining the podcast. I am a big fan of BTSync though so I’m all for it as an option.

San Diego resident here. No issues noted with download speeds. In fact, when working on the iOS app, I’ve been hitting the server(s) fairly regularly with no noticeable slowdown.

1 Like

Does the time make any difference? I download on the morning after. Of course, I’m on a neck breaking 3 Mb/s line. And, I download another podcast at the same time. And I go off and do things while it’s doing. So, I guess I really don’t pay any attention to the time it takes.

The other option would be to have Jono hand deliver the show on some sort of media like a cd, mp3 player, 2TB hard drives, along with a bunch of organic kale. But I would see that really wouldn’t be practical, probably couldn’t find that much kale. :smiley: