Update Accelerator

Work in progress . . . Work in progress . . . Work in progress . . . Work in progress . . . Work in progress . . .
( questions and comments welcome!! bbitsch@ipfire.org)

Imagine this situation: Microsoft has just released another "patch Tuesday" update, the virus scan software has to be updated and Adobe and Symantec are releasing new updates at the same time. You have multiple systems to update and they all need to download the same files.

The Update Accelerator has been created especially for this type of scenario. It has been developed for networks with multiple clients. The Update Accelerator caches, similar to the Proxy server updates for the Operating System, Virus scanners and the like. The load on the Internet connection is reduced and it also prevents long waiting times when loading updates to multiple clients.

Unlike to the standard 'squid' proxy (which caches the most recently requested data in its internal cache), Update Accelerator works more as a File Server - the content defined by download usage on the LAN.
Update Accelerator saves and caches whole files only. The Update Accelerator caches files one first request, which starts two downloads. The first by the client and the second by Update Accelerator. All subsequent download requests for these files will be served by IPFire and processed at local network speed.
This means also, that only files requested by http:// URLs can be cached! With HTTPS the file request is encrypted inside the SSL tunnel between client and server.

Definitions

Update Accelerator process

This program is the 'heart' of the module.

It is implemented as a rewriter for Squid ( the proxy ). Each URL processed by Squid is sent to the rewriter chain ( URLFilter, Update Accelerator, Squid Clamav ) first. A rewriter decides, whether the URL is related to its task and probably changes the URL.

In case of Update Accelerator the URL is matched against the regular expressions describing the download links for the files managed.

If a match is found, the stored file is checked against the file to be downloaded. If the file in the repository is up-to-date, the URL is changed to a link to the repository. Otherwise the requesting URL isn't changed, but a download process is started in parallel.

Vendor

Also called Source. This name describes the source for the update. Examples are Microsoft ( commonly used for Windows updates ), Avira, AVG ( antivirus software ), Mozilla ( Firefox, Thunderbird ).
Vendor names start with an uppercase letter. Vendors are defined by the Update Accelerator process ( see Extensions below ).

Update Cache ( Repository )

Disk space for storage of the cached files. The path is /var/updatecache. This directory consists of subdirectories for each vendor.

The vendor directories contain subdirectories for each cached file, identified by a UUID for fast access. These (sub)directories also contain meta-data ( access time, source URL, ... ) for the cached files.

Performance Characteristic

  • smaller waiting times for updates (full LAN-speed)
  • user defined configuration
  • overview of all the files in cache
  • permanently growing list of supported apps
  • low CPU load (configurable)

In the default settings, the Update Accelerator has to be activated in the Webproxy page. The Accelerator itself is pre-configured and normally doesn't need any change. Only when there is little disk space (less than 5GB) available, should the settings have to be changed.

A configuration example:

None

Parameters:

  • Delivery only/no download mode: If enabled no new downloads are started. Requests are served from the existing update accelerator repository (if the file exists and is not outdated). If you are using the Update Accelerator for the first time you do not want to enable this.
  • Max. disk usage: Update Accelerator doesn't add files, if the usage of the file system holding the repository exceeds this number. (Default 75%)
  • Enable log: Update Accelerator logs it's activities to /var/log/updatexlrator

And a cache filled with updates:

The Cache-Statistic shows exact information of the used disk space and the number and sizes of the files.

Extensions

Here you can find some more extensions for miscellaneous software updates. All extensions needs to adjust to updxlrator which you can found in /usr/sbin/. Further it is necessary to store .gif files for display by the WUI in /srv/web/ipfire/html/images/updbooster.

These files must be named updxl-src-<vendor>.gif, where <vendor> is the vendor name in lowercase.

These examples are not really up-to-date. You should check the links.

Nero Software

filename = /usr/sbin/updxlrator

# -----------------------------------------------------------------------------
# Section: Nero Downloads (from www.csg-pc.de/updatebooster )
# -----------------------------------------------------------------------------

if ($source_url =~ m@^http://.*\.nero\.com/.*\.(7z|exe|msi|zip)$@i)
    {
    $xlrator_url = &check_cache($source_url,$hostaddr,$username,"Nero",$mirror);
    }

Intel Software

filename = /usr/sbin/updxlrator

# ------------------------------------------------------------------------------
# Section: Intel Downloads (from www.csg-pc.de/updatebooster )
# ------------------------------------------------------------------------------

if ($source_url =~ m@^http://.*\.intel\.com/.*\.(7z|exe|msi|zip)$@i)
    {
    $xlrator_url = &check_cache($source_url,$hostaddr,$username,"Intel",$mirror);
    }

ESET NOD32 - Antivirus Updates

filename = /usr/sbin/updxlrator

# ------------------------------------------------------------------------------
# Section: ESET Downloads (for Logo and Extensions PM @firefant | 10.05.2014)
# ------------------------------------------------------------------------------

if ($source_url =~ m@^http://(um01|um02|um03|um04|um05|um06|um21|um23|update|download)\.eset\.com/.*\.(ver|exe|msi|nup|cab)$@i)
    {
    $xlrator_url = &check_cache($source_url,$hostaddr,$username,"Eset",$mirror);
    }

Add your own!

To add a new set of updates to be cached, change the <vendor> and <vendorwebsite> placeholders as well as file URLs and types in this section of the file after copying this template:

filename = /usr/sbin/updxlrator

# -----------------------------------------------------------
#  Section: <vendor> Downloads
# -----------------------------------------------------------
#
if ($source_url =~ m@^http://download[\d]+\.<vendorwebsite>\.com/.*\.(exe|vpu)$@i)
{
    $xlrator_url = &check_cache($source_url,$hostaddr,$username,"<vendor>",$mirror);
}
Edit Page ‐ Yes, you can edit!

Older Revisions • October 17 at 7:33 pm • Bernhard Bitsch