Lessons Learned Building
the Composer Internals


Jordi Boggiano
@seldaek

A Short History of Composer



History

Symfony & phpBB plugins

April 2011 - First Commit

September 2011 - Packagist.org

April 2012 - First 1'000 Packages

April 2013 - First 10'000 Packages

June 2014 - Toran Proxy

 

History

Symfony & phpBB plugins

April 2011 - First Commit

September 2011 - Packagist.org

April 2012 - First 1'000 Packages

April 2013 - First 10'000 Packages

June 2014 - Toran Proxy

December 2016 - Private Packagist

Big Project, Big Problems

PHP - "5 million" developers

Varied environments

// url-encode $ signs in URLs as bad proxies choke on them
if (($pos = strpos($filename, '$')) && preg_match('{^https?://.*}i', $filename)) {
    $filename = substr($filename, 0, $pos) . '%24' . substr($filename, $pos + 1);
}
                

Varied codebases

token_get_all vs PCRE

$contents = @php_strip_whitespace($path);
if (!preg_match('{\b(?:class|interface|trait)\s}i', $contents)) {
    return array();
}
// strip heredocs/nowdocs
$contents = preg_replace('{<<<\s*(\'?)(\w+)\\1(?:\r\n|\n|\r)(?:.*?)(?:\r\n|\n|\r)\\2(?=\r\n|\n|\r|;)}s', 'null', $contents);
// strip strings
$contents = preg_replace('{"[^"\\\\]*+(\\\\.[^"\\\\]*+)*+"|\'[^\'\\\\]*+(\\\\.[^\'\\\\]*+)*+\'}s', 'null', $contents);
// strip leading non-php code if needed
if (substr($contents, 0, 2) !== '<?') {
    $contents = preg_replace('{^.+?<\?}s', '<?', $contents, 1, $replacements);
    if ($replacements === 0) {
        return array();
    }
}
// strip non-php blocks in the file
$contents = preg_replace('{\?>.+<\?}s', '?><?', $contents);
// strip trailing non-php code if needed
$pos = strrpos($contents, '?>');
if (false !== $pos && false === strpos(substr($contents, $pos), '<?')) {
    $contents = substr($contents, 0, $pos);
}

preg_match_all('{
    (?:
         \b(?<![\$:>])(?P<type>class|interface|trait) \s++ (?P<name>[a-zA-Z_\x7f-\xff:][a-zA-Z0-9_\x7f-\xff:\-]*+)
       | \b(?<![\$:>])(?P<ns>namespace) (?P<nsname>\s++[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*+(?:\s*+\\\\\s*+[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*+)*+)? \s*+ [\{;]
    )
}ix', $contents, $matches);
                

Networking

Networking

PEAR & fsockopen

Networking in Composer..

composer/ca-bundle

Networking in Composer..

TLS certificates

manual redirect handling

$context = StreamContextFactory::getContext($url, $options, array('options' => array(
    'ssl' => array(
        'capture_peer_cert' => true,
        'verify_peer' => false,
    )
)));
// Ideally this would just use stream_socket_client() to avoid
// sending a HTTP request but that does not capture the certificate.
if (false === $handle = @fopen($url, 'rb', false, $context)) {
    return;
}
// Close non authenticated connection without reading any content.
fclose($handle);
$handle = null;
$params = stream_context_get_params($context);
if (!empty($params['options']['ssl']['peer_certificate'])) {
    $peerCertificate = $params['options']['ssl']['peer_certificate'];
    if (TlsHelper::checkCertificateHost($peerCertificate, parse_url($url, PHP_URL_HOST), $commonName)) {
        return array(
            'cn' => $commonName,
            'fp' => TlsHelper::getCertificateFingerprint($peerCertificate),
        );
    }
}
                    

Networking

gzip handling

if (PHP_VERSION_ID >= 50400) {
    $response = zlib_decode($response);
} else {
    // work around issue with gzuncompress & co that do not work with all gzip checksums
    $response = file_get_contents('compress.zlib://data:application/octet-stream;base64,'.base64_encode($response));
}
                

Networking

PHP 5.6+ fixes most TLS problems

Networking

Reliability issues: queues / background tasks

PHP Internals

Memory optimization

Fixing growing pains, do not prematurely optimize!

Memory optimization

const BITFIELD_TYPE = 0;
const BITFIELD_REASON = 8;
const BITFIELD_DISABLED = 16;

protected $bitfield;

public function getReason() {
    return ($this->bitfield & (255 << self::BITFIELD_REASON)) >> self::BITFIELD_REASON;
}
public function setType($type) {
    $this->bitfield = ($this->bitfield & ~(255 << self::BITFIELD_TYPE)) | ((255 & $type) << self::BITFIELD_TYPE);
}
public function getType() {
    return ($this->bitfield & (255 << self::BITFIELD_TYPE)) >> self::BITFIELD_TYPE;
}
public function disable() {
    $this->bitfield = ($this->bitfield & ~(255 << self::BITFIELD_DISABLED)) | (1 << self::BITFIELD_DISABLED);
}
public function enable() {
    $this->bitfield = $this->bitfield & ~(255 << self::BITFIELD_DISABLED);
}
                

Garbage Collection

gc_collect_cycles();
gc_disable();

// ... resolve dependencies ...

gc_enable();
                

Do NOT try this at home!

PHAR creation

seld/phar-utils

$util = new \Seld\PharUtils\Timestamps($pharFile);
$util->updateTimestamps($this->versionDate);
$util->save($pharFile, \Phar::SHA1);
                

Decoupling

Decoupling/Components

composer/ca-bundle

composer/semver

composer/spdx-licenses

seld/jsonlint

seld/cli-prompt

seld/phar-utils

Responsibility

Merging = Maintaining forever

Plugins / Extensibility

Shift responsibility for maintenance outside the project

Sustainability

Open Source Lottery

"More than 500 new projects get published to npm every day. In JS alone that’s 500+ projects hoping that, someday, they might be successful enough to be a maintenance burden for the creators."
- Mikeal Rogers

Popular OSS

"Popular OSS is a full time unpaid job that I’m afraid to be fired from and can’t quit"
- Isaac Schlueter

"In a sense, these GitHub notifications are a constant stream of negativity about your projects. Nobody opens an issue or a pull request when they’re satisfied with your work. They only do so when they’ve found something lacking. Even if you only spend a little bit of time reading through these notifications, it can be mentally and emotionally exhausting."
- Nolan Lawson

Fix bugs to reduce support load

UX problems are bugs too

// Check system temp folder for usability as it can cause weird runtime issues otherwise
$tempfile = sys_get_temp_dir() . '/temp-' . md5(microtime());
if (!(file_put_contents($tempfile, __FILE__) && (file_get_contents($tempfile) == __FILE__) && unlink($tempfile) && !file_exists($tempfile))) {
    $io->writeError(sprintf('<error>PHP temp directory (%s) does not exist or is not writable to Composer. Set sys_temp_dir in your php.ini</error>', sys_get_temp_dir()));
}
                    

Encourage self-debugging

$minSpaceFree = 1024 * 1024;
if ((($df = disk_free_space($dir = $config->get('home'))) !== false && $df < $minSpaceFree)
    || (($df = disk_free_space($dir = $config->get('vendor-dir'))) !== false && $df < $minSpaceFree)
    || (($df = disk_free_space($dir = sys_get_temp_dir())) !== false && $df < $minSpaceFree)
) {
    $io->writeError('<error>The disk hosting '.$dir.' is full, this may be the cause of the following exception</error>', true, IOInterface::QUIET);
}

if (Platform::isWindows() && false !== strpos($exception->getMessage(), 'The system cannot find the path specified')) {
    $io->writeError('<error>The following exception may be caused by a stale entry in your cmd.exe AutoRun</error>', true, IOInterface::QUIET);
    $io->writeError('<error>Check https://getcomposer.org/doc/articles/troubleshooting.md#-the-system-cannot-find-the-path-specified-windows- for details</error>', true, IOInterface::QUIET);
}

if (false !== strpos($exception->getMessage(), 'fork failed - Cannot allocate memory')) {
    $io->writeError('<error>The following exception is caused by a lack of memory or swap, or not having swap configured</error>', true, IOInterface::QUIET);
    $io->writeError('<error>Check https://getcomposer.org/doc/articles/troubleshooting.md#proc-open-fork-failed-errors for details</error>', true, IOInterface::QUIET);
}
                

Find the root problem
in user issues

Five whys

Understand the need behind what people ask

Keep the overview in mind, not everything belongs in your project

Maintenance never ends

Complex projects are never done

Feature requests / bug fixes

Needs to be sustainable

Private Packagist packagist.com

No Regrets

"Note that despite all the negativity expressed above, I still feel that open source has been a valuable addition to my life, and I don’t regret any of it."
- Nolan Lawson

Thank you.

@seldaek

slides.seld.be

joind.in/talk/0c3f1

After Party @ 18:30

Free Drinks & Snacks

Form Space @ Aleea Stadionului 2

Taxi or Bus 21 + 30