image/svg+xml Thousands of upstream communities make use of inumerable hosting sites.. What if one went away? • Hardware Failure• Remote Attack• Natural Disaster• Nation-State Censorship We need to protect against that scenario. UpstreamCommunities GNOME JBoss.org Gluster Ceph ManageIQ Fedora Spacewalk kernel KDE LibreOffice RDO OpenStack OpenShift Origin Apache Wildfly Fabric8 Hawkular Hibernate HornetQ Risk Mitigation Hosted Services GitHub SourceForge rubygems.org CodePlex Google Code Pagure Source Control dist-git is the only SCM maintainedby RCM. SysOps maintains andprovides the rest.RCM forbids building product content fromSCMs we do not trust. They must besupported and backed up internally.Everything but dist-git assumes explodedsources. That only includes patches,spec files, and other small files. Overwhelmingly git, some SVN, ancientwork on CVS. Dist-Git Anatomy RHEL 6 Host. 8 VCPUs, 6G RAMGitolite for Access Control • hook to enforce LDAP groups2 Volumes Mounted • /srv/git (181G) - Sources • /srv/cache/lookaside (1.5T) TarballsMetrics as of September 1st Client Tool: rhpkg • does support working with exploded sources! • integrates Koji, Errata Tool, OSBS, Product Pages, and other services for a single CLI experience. • Total repositories: 18,301 • Total branches created: 79,712 • Average No. of branches: 4.244 • Oldest commit: GConf2 - 09-08-2004 • Biggest repository: kernel (3.1G) • Most contributors: mondodb (37) • Most commits: kernel (3569) • Most Branches: kernel (1638) Build System Koji is the principal build system for all product content at Red Hat.Understands how to get sources from Source Control and transform them intobinaries packaged in various formats.Began exclusively for RPMs, but has seen many enhancements for Javaartifacts, VM images, containers, and others.Groups built content into Tags for organizational purposes.Except in special circumstances like an acquisition, building all content inKoji is a requirement to ship. kojid Kojira Web Hub kamid koji-vmd Client Statistics • 1400 repo regens per day• 2400 repo regens (by arch)• 260 rpm builds per day• 25 maven builds per day• 20 image builds per day• 20 container builds per day • 17.621 Tb of rpms• 4.799 Tb of other build types Storage In use for over ten years, over half a million buildsOver 11 million tasks! Averages Currently producing about 2 TiB of builds per *quarter*,though garbage collection eventually removes half ormore of that. Single Build System Benefits • Product Security can verify flaw reports quickly• GSS can find updates or hotfixes easily• Engineering, QE, and RCM only need to look in one place• Product composition steps are streamlined• Permanent archive of all things ever built and shipped• Easier to share content between products Upsteam Project Community • Used in Fedora, Amazon, Scientific Linux, many other RPM based distributions, and academia like Caltech RPMs RPM Yum Mock Koji Middleware Maven Mock Koji RPM Yum Images Anaconda Oz ImageFactory Koji Containers reactor OpenShift Koji Product Compose After enough of the product is built in the Build System, Engineers andRCM need to copy that content out and arrange it the way a customerwould get it.Not all products have this step, some are simple enough to skip it.RCM supports two tools to perform a compose: Distill and Puddle.The ComposeDB is a database that records what was included ina compose. Eng-Archive 30Tb NetApp volume that holds all composes and Koji's contentstorage. When you consider dedupe, we're consuming about 41 TbRecently had to migrate to C-DOT Infrastructure. This was a big deal!Complicated because of extensive use of hard links. Globalsync Portions of Eng-Archive and other volumes are geographicallyreplicated around the world using Globalsync. Supporting Services Recorded on Sept. 11-22nd, 2016Hoststatsglobalsync.bne.redhat.com (XEN)globalsync.eng.blr.redhat.com (KVM)globalsync.eng.tlv.redhat.com (KVM)globalsync.brq.redhat.com (XEN)globalsync.eng.pnq.redhat.com (KVM)globalsync.eng.pek2.redhat.com (XEN) Data700.0 GiB79.9 GiB857.0 GiB1.0 TiB307.4 GiB307.5 GiB Jobs212360342835 Link Speed400M100M200M1G100M100M Puddle Distill • Used for RHEL and a few Layered Products• Produces yum repositories and installable media• Run on per-product hosts: rcm-rhelX, rcm-sat, rcm-scl• Upstream: "Pungi" • Lighter-weight, used for many Layered Products• Only produces yum repositories based on Koji tags or Advisories• Run on rcm-dev or rcm-guest by Productization Engineers Middleware • A collection of custom scripts generates Middleware composes, they vary by product. Composing Tools Release Check List • product/update definition is recorded• automated package testing• functional testing• meeting dependencies of other content• documentation• meeting a release date• meeting an embargo date (security flaws)• delivery strategy is defined Advisory States All products go through these states in the ordershown except for DROPPED, which can interrupt atany time.Often there is a ping-pong effect of changing statesbetween NEW_FILES and QE. This indicatesEngineering handing off software that fails QEvalidationProduct content gets signed with a gold GPGbetween the QE and REL_PREP state. Advisory StateNEW_FILESQEREL_PREPPUSH_READYSHIPPED_LIVEDROPPED OwnerEngineeringQERCMRCM or PSTRCMN/A Release Check List 2 The Errata Tool is a workflow engine that enforces thechecklist. Product content in the ET is grouped intoAdvisories. Engineers attach content built in the BuildSystem to advisories.An advisory also has defects or features associated withit which are expressed in Bugzillas or JIRAs. This is howRCM maintains a product or update definition.Each advisory has a state to indicate ownership ofmoving the attached product content closer to release.As the advisory transitions states, the ownership maytransition too. Automated Testing Several tools are hooked in the Errata Tool that arekicked off on certain conditions. RPMDiff - basic packaging sanityTPS - installation, upgrade, downgrade, removal, and rebuild testsCovScan - static analysisClamAV - virus scannerABIDiff - ensure kernel symbols are still to specCCAT - confirmation of content delivery Depending on the workflow configuration the resultsof these tools must pass or be waived for the advisoryto proceed. Content Delivery When content is ready to be released, it is pushedthrough the Errata Tool to Pub. Pub keeps records ofwho pushed what to where and when.Pub has a Hub-and-Worker model like Koji.Below are the content types and distribution platformswe deliver to. Deliver RPMs and ISOs to RHN or RHSMContainers to the RegistrySRPMs to ftp.redhat.com Amazon Web Services EC2CentOS GitUnified DownloadsPartners' FTP/rsyncOSTreesCustom Service PortalMaven-Hosted RepositoriesGCE and Azure Partners rsync / FTP Both of these distribution platforms are just dumb filesystemsthat are mounted internally and block-sync'd to an externalhost.There is some scripting for copying content around, but thereis no manager nor provider for these. CentOS Git SRPMs in RHEL 7 and some other layered products arepushed here instead of ftp.redhat.com.A few Python scripts with regular expressions handlethe debranding automatically. Content Manager Some Distribution Platforms that RCM supports havea service called a Content Manager. It is responsible fortracking metadata to ensure it is accessible the wayclient tooling expects.Example: container tags/labels, yum repodataSome Platforms lack a manager: ftp.redhat.com andpartners' ftp/rsyncThe big ones are RHN Classic itself, and Pulp, whichmanages content in RHSM/CDN. Nexus Middleware products that follow Project Wolf specificationsare pushed to Nexus.This is a collection of jars that make up said products so theycan be easily be built against by customers using Maven.It is a necessary part of product deployment.We do not push to Maven Central because communities caneasily supercede the jars we provide, making proper supportvery difficult. Vaporizer This IT service manages AMIs in Amazon EC2.RCM uses it for 2 purposes: controlling access for customersin the Cloud Access program, and recording released AMIsthat are available On-Demand.Also used by Customer Service to manage custom accountaccess to the program. RHN Twilight Project Twilight is the effort to migrate customers off of RHN andon RHSM/CDN instead. Involves targeted messages to customers.Satellite 5 support is the main reason RHN still exists today. Longstruggle moving customers to Satellite 6 and/or RHSM. The Satellite Bridge is a necessary enabler.Anything on RHEL 7 and later can only be reached withSatellite 5. On July 31st, 2017 that will be true for all products.RHN has many limitations that become more apparent astechnology demands increase. Pulp The content manager for RHSM is Pulp, and it supportsmanaging RPMs and Containers for the registry. (someISO support)Consumes metadata from Product Proxy, and exposes asuper set of it to Unified Downloads, Download Manager,and other services.Major component of Satellite 6, and has an upstream. Content Provider Our primary content provider is Akamai. We use their serviceto geographically replicate content around the world socustomers have a consistent download experience.This only applies to RHN Classic, RHSM/CDN, and parts of theCustomer Portal (Unified Downloads)• billing is figured out like a data plan for your smart phone• customers downloaded ~320Tb in September. Akamai Anatomy We use the NetStorage service. We have 3 hosts maintainedby Akamai that expose a filesystem. Simply writing to thatfilesystem is all that is needed to geographically replicatearound the world. Entitlement Gotta protect the streams! They can be crossed. This onlyapplies to RHN Classic and RHSM/CDN.Even though content has landed, customers still need tosubscribe to channels or repositories.A group in Red Hat called Customer Data Operationsmaps content to SKUs (Stock Keeping Units), which areassociated with customer accounts when they buy something.That mapping involves RHN Channel Familes and RHSMEngineering Products.A service called Candlepin gives out certificates to customersthat grant access to Red Hat content in RHSM. Customer Portal The Portal is our enterprise-grade website for all customerinteractions.Behind the scenes, Pulp manages content on the Portal, andAkamai provides it.Several independent services come together to provide theweb experience. The main one RCM interacts with isUnified Downloads, which provides software content.MetaXOR is a new service that will also inform parts of thePortal, specifically the container catalog. Customer Interactions How software updates are downloaded vary as muchas the services that manage and provide them.RHN - up2date or yumRHSM (RPMs) - yumRHSM (Containers) - dockerMaven-Hosted - mavenDirect downloads from the Portal for installation mediaor other image typesSynchronize content with Satelliteec2-run or the Web Console in AWS for AMIs(Not covered: Resellers, ISVs, and other channels)