I can provide some input here, at least from the Ubuntu/Debian side, but I think it maps to other distros too.
The major issue is that .deb and .rpm packages run as root (remember the infamous Shuttleworth comment about how we have root, this is what he is referring to).
When you create a .deb there are a set of maintainer scripts, and they run as root. These are scripts that are run pre and post installation that can modify your system. When you create these packages you want to ensure they don’t do anything naughty, and in a collaborative community where anyone can conceivably become a package maintainer, you have to have a very high bar before you give anyone direct access to upload packages. As such, becoming a core-dev or motu in Ubuntu or a dd in Debian requires a long and comprehensive journey.
So, from the outset, these packages have a fairly small number of people maintaining them and there is always far more work than people, particularly as many packages require very specialist experience - you don’t want someone who knows GNOME going around packaging the kernel for example.
The next issue here is policy, and this points to why you get old packages in the archive. When a new version of Ubuntu or Debian ship the archive is frozen. This means that no more updates or changes can go in to ensure the archive is as stable as possible.
So, as an example, imagine
FooPackage version 1.0 goes into Ubuntu ready for the 14.10 release, at this point we are all good…you can get the latest and greatest
FooPackage with a single click.
From that point onwards the archive is subject to the Stable Release Updates (SRU) policy. This basically means that the only updates to that package that can go in are security and vulnerability related. If
FooPackage 1.0 turns out to have a major security issue, the package can be replaced with a newer version, but in general only the bugfix will go in. Features are not allowed in, so that fancy 1.2 version won’t get in…you have to wait until Ubuntu 15.04 for that.
Thus, what happens is you get an archive that has generally very stable software but gets old quickly. This is one of the reasons why Ubuntu focused on the six-month release cycle - you only have old shit for six months and then you get a fresh batch. The problem is, since that model was invented the app store became and thing and now people want stable updates more often.
Ubuntu tried to alleviate this with PPAs in which you can run your own mini-archive, but these should be trusted about as much as downloading a random .exe off the Internet for Windows - someone could include a package that has a nasty maintainer script that hoses your system.
As such, app authors often ship their own packages with newer versions and don’t recommend the in-archive versions. This is more common with desktop software as older server software is often preferred as it is more stable.
The solution to this in Ubuntu was the creation of .click packages. We spent time building a complete security layer in Ubuntu in which anyone can ship a click package that is insulated by that security layer and can’t do naughty things, but they can update packages whenever you want. This significantly increases the speed of getting new software to users. The problem is, at least today, this is only focused on apps written for Ubuntu with the Ubuntu SDK and doesn’t currently cover non-Ubuntu SDK software. I know click will be extended to cover this when Ubuntu fully converges on the desktop, but this is a lot of work.
Hope this helps explain things.