Thousands of upstream communities make use of inumerable hosting sites..
What if one went away?
• Hardware Failure
• Remote Attack
• Natural Disaster
• Nation-State Censorship
We need to protect against that scenario.
is the only SCM maintained
by RCM. SysOps maintains and
provides the rest.
RCM forbids building product content from
SCMs we do not trust.
They must be
supported and backed up internally.
Everything but dist-git assumes exploded
That only includes patches,
spec files, and other small files.
Overwhelmingly git, some SVN, ancient
work on CVS.
RHEL 6 Host.
8 VCPUs, 6G RAM
Gitolite for Access Control
• hook to enforce LDAP groups
2 Volumes Mounted
• /srv/git (181G) - Sources
• /srv/cache/lookaside (1.5T) Tarballs
Metrics as of September 1st
Client Tool: rhpkg
• does support working with exploded
• integrates Koji, Errata Tool, OSBS,
Product Pages, and other services for
a single CLI experience.
• Total repositories: 18,301
• Total branches created: 79,712
• Average No. of branches: 4.244
• Oldest commit: GConf2 - 09-08-2004
• Biggest repository: kernel (3.1G)
• Most contributors: mondodb (37)
• Most commits: kernel (3569)
• Most Branches: kernel (1638)
Koji is the principal build system for all product content at Red Hat.
Understands how to get sources from Source Control and transform them into
binaries packaged in various formats.
Began exclusively for RPMs, but has seen many enhancements for Java
artifacts, VM images, containers, and others.
Groups built content into
for organizational purposes.
Except in special circumstances like an acquisition,
building all content in
Koji is a requirement to ship.
• 1400 repo regens per day
• 2400 repo regens (by arch)
• 260 rpm builds per day
• 25 maven builds per day
• 20 image builds per day
• 20 container builds per day
• 17.621 Tb of rpms
• 4.799 Tb of other build types
In use for over ten years, over half a million builds
Over 11 million tasks!
Currently producing about 2 TiB of builds per *quarter*,
though garbage collection eventually removes half or
more of that.
Single Build System Benefits
• Product Security can verify flaw reports quickly
• GSS can find updates or hotfixes easily
• Engineering, QE, and RCM only need to look in one place
• Product composition steps are streamlined
• Permanent archive of all things ever built and shipped
• Easier to share content between products
Upsteam Project Community
• Used in Fedora, Amazon, Scientific Linux, many
other RPM based distributions, and academia like Caltech
After enough of the product is built in the Build System, Engineers and
RCM need to copy that content out and arrange it the way a customer
would get it.
Not all products have this step, some are simple enough to skip it.
RCM supports two tools to perform a compose:
is a database that records what was included in
30Tb NetApp volume that holds all composes and Koji's content
storage. When you consider dedupe, we're consuming about
Recently had to migrate to C-DOT Infrastructure. This was a big deal!
Complicated because of extensive use of hard links.
Portions of Eng-Archive and other volumes are geographically
replicated around the world using Globalsync.
Recorded on Sept. 11-22nd, 2016
• Used for RHEL and a few Layered Products
• Produces yum repositories and installable media
• Run on per-product hosts: rcm-rhelX, rcm-sat, rcm-scl
• Upstream: "Pungi"
• Lighter-weight, used for many Layered Products
• Only produces yum repositories based on Koji tags or Advisories
• Run on rcm-dev or rcm-guest by Productization Engineers
• A collection of custom scripts generates Middleware composes, they
vary by product.
Release Check List
• product/update definition is recorded
• automated package testing
• functional testing
• meeting dependencies of other content
• meeting a release date
• meeting an embargo date (security flaws)
• delivery strategy is defined
All products go through these states in the order
shown except for DROPPED, which can interrupt at
Often there is a ping-pong effect of changing states
between NEW_FILES and QE. This indicates
Engineering handing off software that fails QE
Product content gets
between the QE and REL_PREP state.
RCM or PST
Release Check List 2
is a workflow engine that enforces the
checklist. Product content in the ET is grouped into
content built in the Build
System to advisories.
An advisory also has defects or features associated with
it which are expressed in
. This is how
RCM maintains a product or update
Each advisory has a
to indicate ownership of
moving the attached product content closer to release.
As the advisory transitions states, the ownership may
Several tools are hooked in the Errata Tool that are
kicked off on certain conditions.
basic packaging sanity
installation, upgrade, downgrade, removal,
and rebuild tests
ensure kernel symbols are still to spec
confirmation of content delivery
Depending on the workflow configuration the results
of these tools must pass or be waived for the advisory
When content is ready to be released, it is
through the Errata Tool to
. Pub keeps records of
who pushed what to where and when.
Pub has a
model like Koji.
Below are the
we deliver to.
Deliver RPMs and ISOs to RHN or RHSM
Containers to the Registry
SRPMs to ftp.redhat.com
Amazon Web Services EC2
Custom Service Portal
GCE and Azure
Partners rsync / FTP
Both of these distribution platforms are just dumb filesystems
that are mounted internally and block-sync'd to an external
There is some scripting for copying content around, but there
is no manager nor provider for these.
SRPMs in RHEL 7 and some other layered products are
pushed here instead of ftp.redhat.com.
A few Python scripts with regular expressions handle
the debranding automatically.
Some Distribution Platforms that RCM supports have
a service called a
. It is responsible for
to ensure it is accessible the way
client tooling expects.
: container tags/labels, yum repodata
Some Platforms lack a manager: ftp.redhat.com and
The big ones are RHN Classic itself, and Pulp, which
manages content in RHSM/CDN.
Middleware products that follow
are pushed to Nexus.
This is a collection of jars that make up said products so they
can be easily be built against by customers using
It is a necessary part of product deployment.
We do not push to
because communities can
easily supercede the jars we provide, making proper support
This IT service manages AMIs in Amazon EC2.
RCM uses it for 2 purposes: controlling access for customers
program, and recording released AMIs
that are available
Also used by Customer Service to manage custom account
access to the program.
Project Twilight is the effort to migrate customers off of RHN and
on RHSM/CDN instead. Involves targeted messages to customers.
Satellite 5 support is the main reason RHN still exists today. Long
struggle moving customers to Satellite 6 and/or RHSM. The
is a necessary enabler.
Anything on RHEL 7 and later can only be reached with
Satellite 5. On July 31st, 2017 that will be true for all products.
RHN has many limitations that become more apparent as
technology demands increase.
The content manager for RHSM is Pulp, and it supports
managing RPMs and Containers for the registry. (some
Consumes metadata from
, and exposes a
super set of it to
Unified Downloads, Download Manager
and other services.
Major component of
, and has an upstream.
Our primary content provider is
. We use their service
to geographically replicate content around the world so
customers have a consistent download experience.
This only applies to RHN Classic, RHSM/CDN, and parts of the
Customer Portal (Unified Downloads)
• billing is figured out like a data plan for your smart phone
• customers downloaded ~320Tb in September.
We use the
service. We have 3 hosts maintained
by Akamai that expose a filesystem. Simply writing to that
filesystem is all that is needed to geographically replicate
around the world.
Gotta protect the streams! They can be crossed. This only
applies to RHN Classic and RHSM/CDN.
Even though content has landed, customers still need to
A group in Red Hat called
Customer Data Operations
maps content to
(Stock Keeping Units), which are
associated with customer accounts when they buy something.
That mapping involves RHN
A service called
gives out certificates to customers
that grant access to Red Hat content in RHSM.
The Portal is our enterprise-grade website for all customer
Behind the scenes, Pulp manages content on the Portal, and
Akamai provides it.
Several independent services come together to provide the
web experience. The main one RCM interacts with is
Unified Downloads, which provides software content.
MetaXOR is a new service that will also inform parts of the
Portal, specifically the container catalog.
How software updates are downloaded vary as much
as the services that manage and provide them.
RHN - up2date or yum
RHSM (RPMs) - yum
RHSM (Containers) - docker
Maven-Hosted - maven
Direct downloads from the Portal for installation media
or other image types
Synchronize content with Satellite
ec2-run or the Web Console in AWS for AMIs
(Not covered: Resellers, ISVs, and other channels)