Lessons Learned Building
the Composer Internals


Jordi Boggiano
@seldaek

Short Project History

Code Learnings

Project Management Learnings

A Short History of Composer



History

Symfony & phpBB plugins

April 2011 - First Commit

September 2011 - Packagist.org

April 2012 - First 1'000 Packages

April 2013 - First 10'000 Packages

June 2014 - Toran Proxy

 

History

Symfony & phpBB plugins

April 2011 - First Commit

September 2011 - Packagist.org

April 2012 - First 1'000 Packages

April 2013 - First 10'000 Packages

June 2014 - Toran Proxy

December 2016 - Private Packagist

Code

token_get_all vs PCRE

$contents = @php_strip_whitespace($path);
if (!preg_match('{\b(?:class|interface|trait)\s}i', $contents)) {
    return array();
}
// strip heredocs/nowdocs
$contents = preg_replace('{<<<\s*(\'?)(\w+)\\1(?:\r\n|\n|\r)(?:.*?)(?:\r\n|\n|\r)\\2(?=\r\n|\n|\r|;)}s', 'null', $contents);
// strip strings
$contents = preg_replace('{"[^"\\\\]*+(\\\\.[^"\\\\]*+)*+"|\'[^\'\\\\]*+(\\\\.[^\'\\\\]*+)*+\'}s', 'null', $contents);
// strip leading non-php code if needed
if (substr($contents, 0, 2) !== '<?') {
    $contents = preg_replace('{^.+?<\?}s', '<?', $contents, 1, $replacements);
    if ($replacements === 0) {
        return array();
    }
}
// strip non-php blocks in the file
$contents = preg_replace('{\?>.+<\?}s', '?><?', $contents);
// strip trailing non-php code if needed
$pos = strrpos($contents, '?>');
if (false !== $pos && false === strpos(substr($contents, $pos), '<?')) {
    $contents = substr($contents, 0, $pos);
}

preg_match_all('{
    (?:
         \b(?<![\$:>])(?P<type>class|interface|trait) \s++ (?P<name>[a-zA-Z_\x7f-\xff:][a-zA-Z0-9_\x7f-\xff:\-]*+)
       | \b(?<![\$:>])(?P<ns>namespace) (?P<nsname>\s++[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*+(?:\s*+\\\\\s*+[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*+)*+)? \s*+ [\{;]
    )
}ix', $contents, $matches);
                

Memory optimization

const BITFIELD_TYPE = 0;
const BITFIELD_REASON = 8;
const BITFIELD_DISABLED = 16;

protected $bitfield;

public function getReason() {
    return ($this->bitfield & (255 << self::BITFIELD_REASON)) >> self::BITFIELD_REASON;
}
public function setType($type) {
    $this->bitfield = ($this->bitfield & ~(255 << self::BITFIELD_TYPE)) | ((255 & $type) << self::BITFIELD_TYPE);
}
public function getType() {
    return ($this->bitfield & (255 << self::BITFIELD_TYPE)) >> self::BITFIELD_TYPE;
}
public function disable() {
    $this->bitfield = ($this->bitfield & ~(255 << self::BITFIELD_DISABLED)) | (1 << self::BITFIELD_DISABLED);
}
public function enable() {
    $this->bitfield = $this->bitfield & ~(255 << self::BITFIELD_DISABLED);
}
                

Garbage Collection

gc_collect_cycles();
gc_disable();

// ... resolve dependencies ...

gc_enable();
                

Do NOT try this at home!

Networking

PEAR: fsockopen

Networking

composer/ca-bundle

TLS certificates

manual redirect handling

content-length checks for dropped connections

Networking

gzip handling

if (PHP_VERSION_ID >= 50400) {
    $response = zlib_decode($response);
} else {
    // work around issue with gzuncompress & co that do not work with all gzip checksums
    $response = file_get_contents('compress.zlib://data:application/octet-stream;base64,'.base64_encode($response));
}
                

Networking

bitbucket text/html for zips as login prompt

Networking

PHP 5.6+ fixes most TLS problems

Networking

Reliability issues: queues / background tasks

PHAR creation

seld/phar-utils

$util = new \Seld\PharUtils\Timestamps($pharFile);
$util->updateTimestamps($this->versionDate);
$util->save($pharFile, \Phar::SHA1);
                

Decoupling/Components

composer/ca-bundle

composer/semver

composer/spdx-licenses

seld/jsonlint

seld/cli-prompt

seld/phar-utils

Code re-use

Symfony components

justinrainbow/json-schema

Project Management

Merging = Maintaining forever

Plugins / Extensibility

Move responsibility for maintenance outside

Fix bugs to reduce support load

UX problems are bugs too

// Check system temp folder for usability as it can cause weird runtime issues otherwise
$tempfile = sys_get_temp_dir() . '/temp-' . md5(microtime());
if (!(file_put_contents($tempfile, __FILE__) && (file_get_contents($tempfile) == __FILE__) && unlink($tempfile) && !file_exists($tempfile))) {
    $io->writeError(sprintf('<error>PHP temp directory (%s) does not exist or is not writable to Composer. Set sys_temp_dir in your php.ini</error>', sys_get_temp_dir()));
}
                    

Encourage self-debugging

$minSpaceFree = 1024 * 1024;
if ((($df = disk_free_space($dir = $config->get('home'))) !== false && $df < $minSpaceFree)
    || (($df = disk_free_space($dir = $config->get('vendor-dir'))) !== false && $df < $minSpaceFree)
    || (($df = disk_free_space($dir = sys_get_temp_dir())) !== false && $df < $minSpaceFree)
) {
    $io->writeError('<error>The disk hosting '.$dir.' is full, this may be the cause of the following exception</error>', true, IOInterface::QUIET);
}

if (Platform::isWindows() && false !== strpos($exception->getMessage(), 'The system cannot find the path specified')) {
    $io->writeError('<error>The following exception may be caused by a stale entry in your cmd.exe AutoRun</error>', true, IOInterface::QUIET);
    $io->writeError('<error>Check https://getcomposer.org/doc/articles/troubleshooting.md#-the-system-cannot-find-the-path-specified-windows- for details</error>', true, IOInterface::QUIET);
}

if (false !== strpos($exception->getMessage(), 'fork failed - Cannot allocate memory')) {
    $io->writeError('<error>The following exception is caused by a lack of memory or swap, or not having swap configured</error>', true, IOInterface::QUIET);
    $io->writeError('<error>Check https://getcomposer.org/doc/articles/troubleshooting.md#proc-open-fork-failed-errors for details</error>', true, IOInterface::QUIET);
}
                

Find the root problem
in user issues

Five whys

Understand the need behind what people ask

Keep the overview in mind, not everything belongs in your project

Maintenance never ends

Complex projects are never done

Feature requests / bug fixes

Needs to be sustainable

Private Packagist packagist.com

Summary

Data imports are hard

Networking is harder

UX is essential

Code is for people

Thank you.

Questions?

@seldaek

slides.seld.be

Feedback

joind.in/talk/5ca17