nikic's blog

Cooperative multitasking using coroutines (in PHP!)

nikic@php.net (NikiC) — Sat, 22 Dec 2012 00:00:00 -0800

One of the large new features in PHP 5.5 will be support for generators and coroutines. Generators are already sufficiently covered by the documentation and various other blog posts (like this one or this one. Coroutines on the other hand have received relatively little attention. The reason is that coroutines are both a lot more powerful and a lot harder to understand and explain.

In this article I’d like to guide you through an implementation of a task scheduler using coroutines, so you can get a feeling for the stuff that they allow you to do. I’ll start off with a few introductory sections. If you feel like you already got a good grasp of the basics behind generators and coroutines, then you can jump straight to the “Cooperative multitasking” section.

Generators

The basic idea behind generators is that a function doesn’t return a single value, but returns a sequence of values instead, where every value is emitted one by one. Or in other words, generators allow you to implement iterators more easily. A very simple example of this concept is the xrange() function:

<?php

function xrange($start, $end, $step = 1) {
    for ($i = $start; $i <= $end; $i += $step) {
        yield $i;
    }
}

foreach (xrange(1, 1000000) as $num) {
    echo $num, "\n";
}

The xrange() function shown above provides the same functionality as the built-in range() function. The only difference is that range() will return an array with one million numbers in the above case, whereas xrange() returns an iterator that will emit these numbers, but never actually compute an array with all of them.

The advantages of this approach should be evident. It allows you to work with large datasets without loading them into memory all at once. You can even work with infinite data-streams.

All this can also be done without generators, by manually implementing the Iterator interface. Generators only make it (a lot) more convenient, because you no longer have to implement five different methods for every iterator.

Generators as interruptible functions

To go from generators to coroutines it’s important to understand how they work internally: Generators are interruptible functions, where the yield statements constitute the interruption points.

Sticking to the above example, if you call xrange(1, 1000000) no code in the xrange() function is actually run. Instead PHP just returns an instance of the Generator class which implements the Iterator interface:

<?php

$range = xrange(1, 1000000);
var_dump($range); // object(Generator)#1
var_dump($range instanceof Iterator); // bool(true)

The code is only run once you invoke one of the iterator methods on the object. E.g. if you call $range->rewind() the code in the xrange() function will be run until the first occurrence of yield in the control flow. In this case it means that $i = $start and then yield $i are run. Whatever was passed to the yield statement can then be fetched using $range->current().

To continue executing the code in the generator you need to call the $range->next() method. This will again resume the generator until a yield statement is hit. Thus, using a succession of ->next() and ->current() calls, you can get all values from the generator, until at some point no yield is hit anymore. For the xrange() this happens once $i exceeds $end. In this case control flow will reach the end of the function, thus leaving no more code to run. Once this happens the ->valid() method will return false and as such the iteration ends.

Coroutines

The main thing that coroutines add to the above functionality is the ability to send values back to the generator. This turns the one-way communication from the generator to the caller into a two-way channel between the two.

Values are passed into the coroutine by calling its ->send() method instead of ->next(). An example of how this works is the following logger() coroutine:

<?php

function logger($fileName) {
    $fileHandle = fopen($fileName, 'a');
    while (true) {
        fwrite($fileHandle, yield . "\n");
    }
}

$logger = logger(__DIR__ . '/log');
$logger->send('Foo');
$logger->send('Bar');

As you can see yield isn’t used as a statement here, but as an expression, i.e. it has a return value. The return value of yield is whatever was passed to ->send(). In this example yield will first return 'Foo' and then 'Bar'.

The above is an example where the yield acts as a mere receiver. It is possible to combine both usages, i.e. to both send and receive. Here is an example of how this works:

<?php

function gen() {
    $ret = (yield 'yield1');
    var_dump($ret);
    $ret = (yield 'yield2');
    var_dump($ret);
}

$gen = gen();
var_dump($gen->current());    // string(6) "yield1"
var_dump($gen->send('ret1')); // string(4) "ret1"   (the first var_dump in gen)
                              // string(6) "yield2" (the var_dump of the ->send() return value)
var_dump($gen->send('ret2')); // string(4) "ret2"   (again from within gen)
                              // NULL               (the return value of ->send())

The exact order of the outputs can be a bit hard to understand at first, so make sure that you get why it comes out in exactly this way. There are two things I’d like to especially point out: First, the use of parentheses around the yield expression is no accident. These parentheses are required for technical reasons (though I have been considering adding an exception for assignments, just like it exists in Python). Secondly, you may have noticed that ->current() is used without calling ->rewind() first. If this is done then the rewind operation is performed implicitly.

Cooperative multitasking

If reading the above logger() example you thought “Why would I use a coroutine for this? Why can’t I just use a normal class?”, then you were totally right. The example demonstrates the basic usage, but there aren’t really any advantages to using a coroutine in this context. This is the case for a lot of coroutine examples. As already mentioned in the introduction coroutines are a very powerful concept, but their applications are rare and often sufficiently complicated, making it hard to come up with simple and non-contrived examples.

What I decided to go for in this article is an implementation of cooperative multitasking using coroutines. The problem we’re trying to solve is that you want to run multiple tasks (or “programs”) concurrently. But a processor can only run one task at a time (not considering multi-core for the purposes of this post). Thus the processor needs to switch between the different tasks and always let one run “for a little while”.

The “cooperative” part of the term describes how this switching is done: It requires that the currently running task voluntarily passes back control to the scheduler, so it can run another task. This is in contrast to “preemptive” multitasking where the scheduler can interrupt the task after some time whether it likes it or not. Cooperative multitasking was used in early versions of Windows (pre Win95) and Mac OS, but they later switched to using preemption. The reason should be fairly obvious: If you rely on a program to pass back control voluntarily, badly-behaving software can easily occupy the whole CPU for itself, not leaving a share for other tasks.

At this point you should see the connection between coroutines and task scheduling: The yield instruction provides a way for a task to interrupt itself and pass control back to the scheduler, so it can run some other task. Furthermore the yield can be used for communication between the task and the scheduler.

For our purposes a “task” will be a thin wrapper around the coroutine function:

<?php

class Task {
    protected $taskId;
    protected $coroutine;
    protected $sendValue = null;
    protected $beforeFirstYield = true;

    public function __construct($taskId, Generator $coroutine) {
        $this->taskId = $taskId;
        $this->coroutine = $coroutine;
    }

    public function getTaskId() {
        return $this->taskId;
    }

    public function setSendValue($sendValue) {
        $this->sendValue = $sendValue;
    }

    public function run() {
        if ($this->beforeFirstYield) {
            $this->beforeFirstYield = false;
            return $this->coroutine->current();
        } else {
            $retval = $this->coroutine->send($this->sendValue);
            $this->sendValue = null;
            return $retval;
        }
    }

    public function isFinished() {
        return !$this->coroutine->valid();
    }
}

A task will be a coroutine tagged with a task ID. Using the setSendValue() method you can specify which value will be sent into it on the next resume (you’ll see what we need this for a bit later). The run() function really does nothing more than call the send() method on the coroutine. To understand why the additional beforeFirstYield flag is needed consider the following snippet:

<?php

function gen() {
    yield 'foo';
    yield 'bar';
}

$gen = gen();
var_dump($gen->send('something'));

// As the send() happens before the first yield there is an implicit rewind() call,
// so what really happens is this:
$gen->rewind();
var_dump($gen->send('something'));

// The rewind() will advance to the first yield (and ignore its value), the send() will
// advance to the second yield (and dump its value). Thus we loose the first yielded value!

By adding the additional beforeFirstYield condition we can ensure that the value of the first yield is also returned.

The scheduler now has to do little more than cycle through the tasks and run them:

<?php

class Scheduler {
    protected $maxTaskId = 0;
    protected $taskMap = []; // taskId => task
    protected $taskQueue;

    public function __construct() {
        $this->taskQueue = new SplQueue();
    }

    public function newTask(Generator $coroutine) {
        $tid = ++$this->maxTaskId;
        $task = new Task($tid, $coroutine);
        $this->taskMap[$tid] = $task;
        $this->schedule($task);
        return $tid;
    }

    public function schedule(Task $task) {
        $this->taskQueue->enqueue($task);
    }

    public function run() {
        while (!$this->taskQueue->isEmpty()) {
            $task = $this->taskQueue->dequeue();
            $task->run();

            if ($task->isFinished()) {
                unset($this->taskMap[$task->getTaskId()]);
            } else {
                $this->schedule($task);
            }
        }
    }
}

The newTask() method creates a new task (using the next free task id) and puts it in the task map. Furthermore it schedules the task by putting it in the task queue. The run() method then walks this task queue and runs the tasks. If a task is finished it is dropped, otherwise it is rescheduled at the end of the queue.

Lets try out the scheduler with two simple (and very pointless) tasks:

<?php

function task1() {
    for ($i = 1; $i <= 10; ++$i) {
        echo "This is task 1 iteration $i.\n";
        yield;
    }
}

function task2() {
    for ($i = 1; $i <= 5; ++$i) {
        echo "This is task 2 iteration $i.\n";
        yield;
    }
}

$scheduler = new Scheduler;

$scheduler->newTask(task1());
$scheduler->newTask(task2());

$scheduler->run();

Both tasks will just echo a message and then pass control back to the scheduler with yield. This is the resulting output:

This is task 1 iteration 1.
This is task 2 iteration 1.
This is task 1 iteration 2.
This is task 2 iteration 2.
This is task 1 iteration 3.
This is task 2 iteration 3.
This is task 1 iteration 4.
This is task 2 iteration 4.
This is task 1 iteration 5.
This is task 2 iteration 5.
This is task 1 iteration 6.
This is task 1 iteration 7.
This is task 1 iteration 8.
This is task 1 iteration 9.
This is task 1 iteration 10.

The output is exactly as expected: For the first 5 iterations the tasks alternate, then the second task finishes and only the first task continues to run.

Communicating with the scheduler

Now that the scheduler works we can turn to the next point on the agenda: Communication between the tasks and the scheduler. We will use the same method that processes use to talk to the operating system: Through system calls. The reason we need syscalls is that the operating system is on a different privilege level than the processes. So in order to perform privileged actions (like killing another process) there has to be some way to pass control back to the kernel, so it can perform said actions. Internally this is once again implemented using interruption instructions. Historically the generic int instruction was used, nowadays there are more specialized and faster syscall/sysenter instructions.

Our task scheduling system will reflect this design: Instead of simply passing the scheduler into the task (and thus allowing it to do whatever it wants) we will communicate via system calls passed through the yield expression. The yield here will act both as an interrupt and as a way to pass information to (and from) the scheduler.

To represent a system call I’ll use a small wrapper around a callable:

<?php

class SystemCall {
    protected $callback;

    public function __construct(callable $callback) {
        $this->callback = $callback;
    }

    public function __invoke(Task $task, Scheduler $scheduler) {
        $callback = $this->callback; // Can't call it directly in PHP :/
        return $callback($task, $scheduler);
    }
}

It will behave just like any callable (using __invoke), but tells the scheduler to pass the calling task and itself into the function. To handle it we have to slightly modify the scheduler’s run method:

<?php
public function run() {
    while (!$this->taskQueue->isEmpty()) {
        $task = $this->taskQueue->dequeue();
        $retval = $task->run();

        if ($retval instanceof SystemCall) {
            $retval($task, $this);
            continue;
        }

        if ($task->isFinished()) {
            unset($this->taskMap[$task->getTaskId()]);
        } else {
            $this->schedule($task);
        }
    }
}

The first system call will do nothing more than return the task ID:

<?php
function getTaskId() {
    return new SystemCall(function(Task $task, Scheduler $scheduler) {
        $task->setSendValue($task->getTaskId());
        $scheduler->schedule($task);
    });
}

It does so by setting the tid as next send value and rescheduling the task. For system calls the scheduler does not automatically reschedule the task, we need to do it manually (you’ll see why a bit later). Using this new syscall we can rewrite the previous example:

<?php

function task($max) {
    $tid = (yield getTaskId()); // <-- here's the syscall!
    for ($i = 1; $i <= $max; ++$i) {
        echo "This is task $tid iteration $i.\n";
        yield;
    }
}

$scheduler = new Scheduler;

$scheduler->newTask(task(10));
$scheduler->newTask(task(5));

$scheduler->run();

This will give the same output as with the previous example. Notice how the system call is basically done like any other call, but with a prepended yield. Two more syscalls for creating new tasks and killing them again:

<?php

function newTask(Generator $coroutine) {
    return new SystemCall(
        function(Task $task, Scheduler $scheduler) use ($coroutine) {
            $task->setSendValue($scheduler->newTask($coroutine));
            $scheduler->schedule($task);
        }
    );
}

function killTask($tid) {
    return new SystemCall(
        function(Task $task, Scheduler $scheduler) use ($tid) {
            $task->setSendValue($scheduler->killTask($tid));
            $scheduler->schedule($task);
        }
    );
}

The killTask function needs an additional method in the scheduler:

<?php

public function killTask($tid) {
    if (!isset($this->taskMap[$tid])) {
        return false;
    }

    unset($this->taskMap[$tid]);

    // This is a bit ugly and could be optimized so it does not have to walk the queue,
    // but assuming that killing tasks is rather rare I won't bother with it now
    foreach ($this->taskQueue as $i => $task) {
        if ($task->getTaskId() === $tid) {
            unset($this->taskQueue[$i]);
            break;
        }
    }

    return true;
}

A small script to test the new functionality:

<?php

function childTask() {
    $tid = (yield getTaskId());
    while (true) {
        echo "Child task $tid still alive!\n";
        yield;
    }
}

function task() {
    $tid = (yield getTaskId());
    $childTid = (yield newTask(childTask()));

    for ($i = 1; $i <= 6; ++$i) {
        echo "Parent task $tid iteration $i.\n";
        yield;

        if ($i == 3) yield killTask($childTid);
    }
}

$scheduler = new Scheduler;
$scheduler->newTask(task());
$scheduler->run();

This will print the following:

Parent task 1 iteration 1.
Child task 2 still alive!
Parent task 1 iteration 2.
Child task 2 still alive!
Parent task 1 iteration 3.
Child task 2 still alive!
Parent task 1 iteration 4.
Parent task 1 iteration 5.
Parent task 1 iteration 6.

The child is killed after three iterations, so that’s when the “Child is still alive” messages end. One should probably point about that this is not a real parent/child relationship, because the child can continue running even after the parent finished. Or the child could kill the parent. One could modify the scheduler to have a more hierarchic task structure, but I won’t implement that in this article.

There are many more process management calls one could implement, for example wait (which waits until a task has finished running), exec (which replaces the current task) and fork (which creates a clone of the current task). Forking is pretty cool and you can actually implement it with PHP’s coroutines, because they support cloning.

But I’ll leave these for the interested reader. Instead lets get to the next topic!

Non-Blocking IO

A really cool application of our task management system obviously is … a web server. There could be one task listening a socket for new connections and whenever a new connection is made it would create a new task handling that connection.

The hard part about this is that normally socket operations like reading data are blocking, i.e. PHP will wait until the client has finished sending. For a web-server that’s obviously not good at all: It would mean that it can only handle a single connection at a time.

The solution is to make sure that the socket is “ready” before actually reading/writing to it. To find out which sockets are ready to read from or write to the stream_select function can be used.

First, lets add two new syscalls, which will cause a task to wait until a certain socket is ready:

<?php

function waitForRead($socket) {
    return new SystemCall(
        function(Task $task, Scheduler $scheduler) use ($socket) {
            $scheduler->waitForRead($socket, $task);
        }
    );
}

function waitForWrite($socket) {
    return new SystemCall(
        function(Task $task, Scheduler $scheduler) use ($socket) {
            $scheduler->waitForWrite($socket, $task);
        }
    );
}

These syscalls are just proxies to the respective methods in the scheduler:

<?php

// resourceID => [socket, tasks]
protected $waitingForRead = [];
protected $waitingForWrite = [];

public function waitForRead($socket, Task $task) {
    if (isset($this->waitingForRead[(int) $socket])) {
        $this->waitingForRead[(int) $socket][1][] = $task;
    } else {
        $this->waitingForRead[(int) $socket] = [$socket, [$task]];
    }
}

public function waitForWrite($socket, Task $task) {
    if (isset($this->waitingForWrite[(int) $socket])) {
        $this->waitingForWrite[(int) $socket][1][] = $task;
    } else {
        $this->waitingForWrite[(int) $socket] = [$socket, [$task]];
    }
}

The waitingForRead and waitingForWrite properties are just arrays containing the sockets to wait for and the tasks that are waiting for them. The interesting part is the following method, which actually checks whether the sockets are ready and reschedules the respective tasks:

<?php

protected function ioPoll($timeout) {
    $rSocks = [];
    foreach ($this->waitingForRead as list($socket)) {
        $rSocks[] = $socket;
    }

    $wSocks = [];
    foreach ($this->waitingForWrite as list($socket)) {
        $wSocks[] = $socket;
    }

    $eSocks = []; // dummy

    if (!stream_select($rSocks, $wSocks, $eSocks, $timeout)) {
        return;
    }

    foreach ($rSocks as $socket) {
        list(, $tasks) = $this->waitingForRead[(int) $socket];
        unset($this->waitingForRead[(int) $socket]);

        foreach ($tasks as $task) {
            $this->schedule($task);
        }
    }

    foreach ($wSocks as $socket) {
        list(, $tasks) = $this->waitingForWrite[(int) $socket];
        unset($this->waitingForWrite[(int) $socket]);

        foreach ($tasks as $task) {
            $this->schedule($task);
        }
    }
}

The stream_select function takes arrays of read, write and except sockets to check (we’ll ignore that last category). The arrays are passed by reference and the function will only leave those elements in the arrays that changed state. We can then walk over those arrays and reschedule all tasks associated with them.

In order to regularly perform the above polling action we’ll add a special task in the scheduler:

<?php
protected function ioPollTask() {
    while (true) {
        if ($this->taskQueue->isEmpty()) {
            $this->ioPoll(null);
        } else {
            $this->ioPoll(0);
        }
        yield;
    }
}

This task needs to be registered at some point, e.g. one could add $this->newTask($this->ioPollTask()) to the start of the run() method. Then it will work just like any other task, performing the polling operation once every full task cycle (this isn’t necessarily the best way to handle it). The ioPollTask will call ioPoll with a 0 second timeout, which means that stream_select will return right away (rather than waiting).

Only if the task queue is empty we use a null timeout, which means that it will wait until some socket becomes ready. If we wouldn’t do this the polling task would just run again and again and again until a new connection is made. This would result in 100% CPU usage. It’s much more efficient to let the operating system do the waiting instead.

Writing the server is relatively easy now:

<?php

function server($port) {
    echo "Starting server at port $port...\n";

    $socket = @stream_socket_server("tcp://localhost:$port", $errNo, $errStr);
    if (!$socket) throw new Exception($errStr, $errNo);

    stream_set_blocking($socket, 0);

    while (true) {
        yield waitForRead($socket);
        $clientSocket = stream_socket_accept($socket, 0);
        yield newTask(handleClient($clientSocket));
    }
}

function handleClient($socket) {
    yield waitForRead($socket);
    $data = fread($socket, 8192);

    $msg = "Received following request:\n\n$data";
    $msgLength = strlen($msg);

    $response = <<<RES
HTTP/1.1 200 OK\r
Content-Type: text/plain\r
Content-Length: $msgLength\r
Connection: close\r
\r
$msg
RES;

    yield waitForWrite($socket);
    fwrite($socket, $response);

    fclose($socket);
}

$scheduler = new Scheduler;
$scheduler->newTask(server(8000));
$scheduler->run();

This will accept connections to localhost:8000 and just send back a HTTP response with whatever it was sent. Doing anything “real” would be a lot more complicated (properly handling HTTP requests is way outside the scope of this article). The above snippet just demos the general concept.

You can try the server out using something like ab -n 10000 -c 100 localhost:8000/. This will send 10000 requests to it with 100 of them arriving concurrently. Using these numbers I get a median response time of 10ms. But there is an issue with a few requests being handled really slowly (like 5 seconds), that’s why the total throughput is only 2000 reqs/s (with a 10ms response time it should be more like 10000 reqs/s). With higher concurrency count (e.g. -c 500) it mostly still works well, but some connections will throw a “Connection reset by peer” error. As I know very little about this low-level socket stuff I didn’t try to figure out what the issue is.

Stacked coroutines

If you would try to build some larger system using our scheduling system you would soon run into a problem: We are used to breaking up code into smaller functions and calling them. But with coroutines this is no longer possible. E.g. consider the following code:

<?php

function echoTimes($msg, $max) {
    for ($i = 1; $i <= $max; ++$i) {
        echo "$msg iteration $i\n";
        yield;
    }
}

function task() {
    echoTimes('foo', 10); // print foo ten times
    echo "---\n";
    echoTimes('bar', 5); // print bar five times
    yield; // force it to be a coroutine
}

$scheduler = new Scheduler;
$scheduler->newTask(task());
$scheduler->run();

This code tries to put the recurring “output n times” code into a separate coroutine and then invoke it from the main task. But this won’t work. As mentioned at the very beginning of this article calling a generator (or coroutine) will not actually do anything, it will only return an object. This also happens in the above case. The echoTimes calls won’t do anything than return an (unused) coroutine object.

In order to still allow this we need to write a small wrapper around the bare coroutines. I’ll call this a “stacked coroutine” because it will manage a stack of nested coroutine calls. It will be possible to call sub-coroutines by yielding them:

$retval = (yield someCoroutine($foo, $bar));

The subcoroutines will also be able to return a value, again by using yield:

yield retval("I'm a return value!");

The retval function does nothing more than returning a wrapper around the value which will signal that it’s a return value:

<?php

class CoroutineReturnValue {
    protected $value;

    public function __construct($value) {
        $this->value = $value;
    }

    public function getValue() {
        return $this->value;
    }
}

function retval($value) {
    return new CoroutineReturnValue($value);
}

In order to turn a coroutine into a stacked coroutine (which supports subcalls) we’ll have to write another function (which is obviously yet-another-coroutine):

<?php

function stackedCoroutine(Generator $gen) {
    $stack = new SplStack;

    for (;;) {
        $value = $gen->current();

        if ($value instanceof Generator) {
            $stack->push($gen);
            $gen = $value;
            continue;
        }

        $isReturnValue = $value instanceof CoroutineReturnValue;
        if (!$gen->valid() || $isReturnValue) {
            if ($stack->isEmpty()) {
                return;
            }

            $gen = $stack->pop();
            $gen->send($isReturnValue ? $value->getValue() : NULL);
            continue;
        }

        $gen->send(yield $gen->key() => $value);
    }
}

This function acts as a simple proxy between the caller and the currently running subcoroutine. This is handled in the $gen->send(yield $gen->key() => $value); line. Additionally it checks whether a return value is a generator, in which case it will start running it and pushes the previous coroutine on the stack. Once it gets a CoroutineReturnValue it will pop the stack again and continue executing the previous coroutine.

In order to make the stacked coroutines usable in tasks the $this->coroutine = $coroutine; line in the Task constructor needs to be replaced with $this->coroutine = stackedCoroutine($coroutine);.

Now we can improve the webserver example from above a bit by grouping the wait+read (and wait+write and wait+accept) actions into functions. To group the related functionality I’ll use a class:

<?php

class CoSocket {
    protected $socket;

    public function __construct($socket) {
        $this->socket = $socket;
    }

    public function accept() {
        yield waitForRead($this->socket);
        yield retval(new CoSocket(stream_socket_accept($this->socket, 0)));
    }

    public function read($size) {
        yield waitForRead($this->socket);
        yield retval(fread($this->socket, $size));
    }

    public function write($string) {
        yield waitForWrite($this->socket);
        fwrite($this->socket, $string);
    }

    public function close() {
        @fclose($this->socket);
    }
}

Now the server can be rewritten a bit cleaner:

<?php

function server($port) {
    echo "Starting server at port $port...\n";

    $socket = @stream_socket_server("tcp://localhost:$port", $errNo, $errStr);
    if (!$socket) throw new Exception($errStr, $errNo);

    stream_set_blocking($socket, 0);

    $socket = new CoSocket($socket);
    while (true) {
        yield newTask(
            handleClient(yield $socket->accept())
        );
    }
}

function handleClient($socket) {
    $data = (yield $socket->read(8192));

    $msg = "Received following request:\n\n$data";
    $msgLength = strlen($msg);

    $response = <<<RES
HTTP/1.1 200 OK\r
Content-Type: text/plain\r
Content-Length: $msgLength\r
Connection: close\r
\r
$msg
RES;

    yield $socket->write($response);
    yield $socket->close();
}

Error handling

As a good programmer you obviously noticed that the above examples all lack error handling. Pretty much every socket operation is fallible and can produce errors. I obviously did this because error handling is really tedious (especially for sockets!) and would easily blow up the code size by a few factors.

But still I’d like to cover how error handling for coroutines works in general: Coroutines provide the ability to throw exceptions inside them using the throw() method. As of this writing this method does not yet exist in PHP’s implementation, but I will commit it later today.

The throw() method takes an exception and throws it at the current suspension point in the coroutine. Consider this code:

<?php

function gen() {
    echo "Foo\n";
    try {
        yield;
    } catch (Exception $e) {
        echo "Exception: {$e->getMessage()}\n";
    }
    echo "Bar\n";
}

$gen = gen();
$gen->rewind();                     // echos "Foo"
$gen->throw(new Exception('Test')); // echos "Exception: Test"
                                    // and "Bar"

This is really awesome for our purposes, because we can make system calls and subcoroutine calls throw exceptions. For the system calls the Scheduler::run() method needs a small adjustment:

<?php

if ($retval instanceof SystemCall) {
    try {
        $retval($task, $this);
    } catch (Exception $e) {
        $task->setException($e);
        $this->schedule($task);
    }
    continue;
}

And the Task class needs to handle throw calls too:

<?php

class Task {
    // ...
    protected $exception = null;

    public function setException($exception) {
        $this->exception = $exception;
    }

    public function run() {
        if ($this->beforeFirstYield) {
            $this->beforeFirstYield = false;
            return $this->coroutine->current();
        } elseif ($this->exception) {
            $retval = $this->coroutine->throw($this->exception);
            $this->exception = null;
            return $retval;
        } else {
            $retval = $this->coroutine->send($this->sendValue);
            $this->sendValue = null;
            return $retval;
        }
    }

    // ...
}

Now we can start throwing exceptions from system calls! E.g. for the killTask call, lets throw an exception if the passed task ID is invalid:

<?php

function killTask($tid) {
    return new SystemCall(
        function(Task $task, Scheduler $scheduler) use ($tid) {
            if ($scheduler->killTask($tid)) {
                $scheduler->schedule($task);
            } else {
                throw new InvalidArgumentException('Invalid task ID!');
            }
        }
    );
}

Try it out:

<?php

function task() {
    try {
        yield killTask(500);
    } catch (Exception $e) {
        echo 'Tried to kill task 500 but failed: ', $e->getMessage(), "\n";
    }
}

Sadly this won’t work properly yet, because the stackedCoroutine function doesn’t handle the exception correctly. To fix it the function needs some modifications:

<?php

function stackedCoroutine(Generator $gen) {
    $stack = new SplStack;
    $exception = null;

    for (;;) {
        try {
            if ($exception) {
                $gen->throw($exception);
                $exception = null;
                continue;
            }

            $value = $gen->current();

            if ($value instanceof Generator) {
                $stack->push($gen);
                $gen = $value;
                continue;
            }

            $isReturnValue = $value instanceof CoroutineReturnValue;
            if (!$gen->valid() || $isReturnValue) {
                if ($stack->isEmpty()) {
                    return;
                }

                $gen = $stack->pop();
                $gen->send($isReturnValue ? $value->getValue() : NULL);
                continue;
            }

            try {
                $sendValue = (yield $gen->key() => $value);
            } catch (Exception $e) {
                $gen->throw($e);
                continue;
            }

            $gen->send($sendValue);
        } catch (Exception $e) {
            if ($stack->isEmpty()) {
                throw $e;
            }

            $gen = $stack->pop();
            $exception = $e;
        }
    }
}

Wrapping up

In this article we built a task scheduler using cooperative multitasking, including the ability to perform “system calls”, doing non-blocking IO operations and handling errors. The really cool thing about all this is that the resulting code for the tasks looks totally synchronous, even though it is performing a lot of asynchronous operations. If you want to read data from a socket you don’t have to pass some callback or register an event listener. Instead you write yield $socket->read(). Which is basically what you would normally do too, just with a yield in front of it.

When I first heard about all this I found this concept totally awesome and that’s what motivated me to implement it in PHP. At the same time I find coroutines really scary. There is a thin line between awesome code and a total mess and I think coroutines sit exactly on that line. It’s hard for me to say whether writing async code in the way outlined above is really beneficial.

In any case, I think it’s an interesting topic and I hope you found it interesting too. Comments welcome :)

Are PHP developers functophobic?

nikic@php.net (NikiC) — Fri, 10 Aug 2012 00:00:00 -0700

There is this one thing that I noticed recently and that concerns me: PHP devs don’t use functions.

Now, that was overly general, so let me clarify: PHP developers who have reached a certain degree of sophistication basically stop using plain functions - instead everything goes all classes and methods. At least that’s the observation I made when looking at various open-source libraries and frameworks. The only type of function you’ll find in any of the “high-quality” libs are anonymous functions. But that’s pretty much it.

If you look at other languages, the landscape is different. For example Python code will to a large part be class definitions, but there will always be normal functions in between.

So in PHP functions are basically used in two cases: If the programmer doesn’t yet use OOP or if the script isn’t worth using OOP for. The notion of “helper function” or something like that does not seem to exist.

This somewhat concerns me because I think that sticking everything into classes isn’t really the right thing to do. This kind of behavior usually leads to Class-Oriented Programming, where you are basically just using classes for the sake of using them. (Reminder: Using classes does not mean your code is object-oriented!)

I also think that the reluctance to define helper functions is what leads to the push for more and more functions in the PHP core. E.g. I recently have seen a few people asking for some kind of array_rand_value function. This function would basically be the same as array_rand, but it would return a value instead of a key. So it could be defined in userland code in two lines:

<?php
function array_rand_value(array $array) {
    if (empty($array)) return null;
    return $array[array_rand($array)];
}

This is a tiny small function that you can easily define yourself. But then arises what I think is the main problem: Where should you put it? How do I incorporate it with the autoloader?

PHP to large parts inherited the Java OOP model and one thing that came with it is having a one-to-one mapping between classes and files. PHP does not enforce this, but it is a very common convention. PHP’s autoloading support and the PSR-0 standard are emphasizing this further.

Other languages similar to PHP do not have this convention. For example in Python it is very normal to define multiple related classes and functions in one file. In Python a file is rather a module, i.e. a group of related components. In such a setup it is obviously much easier to define small functions in between.

Generally I think that the one-to-one file mapping so common in PHP is problematic. Apart from making it hard to use functions it also adds an additional cost to creating small classes:

Object oriented programming done right usually leads to a large number of small classes. This is great for code reuse, maintainability and testing.

But in PHP you have to create a new file for every one of those classes. And this is really getting in my way. I’d have no problem batch-defining ten tiny classes in one file. But I feel like creating ten distinct files for them is counterproductive (and hampers maintainability).

Another tangentially related “standard” practice I feel badly about is the excessive use of doccomments. In most cases phpdoc comments just comment the obvious - at the same increasing the code size by a factor of two or three. I have no problem with doccomments where necessary, but in most cases (at least with well designed code) the behavior should be obvious from the method and parameter names.

So, basically, what I’m thinking about here is being more “compact”: Instead of defining a file for every class, put related (smaller) classes into one file. Instead of defining a class for everything, just use a function. Instead of cluttering the code with lots of useless doccomments, just leave them out unless really necessary.

But it’s just a thought ;) Maybe I got it all wrong.

How to add new (syntactic) features to PHP

nikic@php.net (NikiC) — Fri, 27 Jul 2012 00:00:00 -0700

Several people have recently asked me where you should start if you want to add some new (syntactic) feature to PHP. As I’m not aware of any existing tutorials on that matter, I’ll try to illustrate the whole process in the following. At the same time this is a general introduction to the workings of the Zend Engine. So upfront: I apologize for this overly long post.

This post assumes that you already have some basic knowledge of C and also know the fundamental concepts of the PHP implementation (like zvals). If not, you should read up on them beforehand.

As an example I’ll use the addition of an in operator which you might already know from other languages like Python. It works as follows:

$words = ['hello', 'world', 'foo', 'bar'];
var_dump('hello' in $words); // true
var_dump('foo' in $words);   // true
var_dump('blub' in $words);  // false

$string = 'PHP is fun!';
var_dump('PHP' in $string);    // true
var_dump('Python' in $string); // false

So basically, for arrays the in operator is the same as the in_array function (but without the needle/haystack problem) and for strings it’s like doing a false !== strpos($str2, $str1).

Prerequisites

Before we can get going, you’ll have to first check out and compile PHP. To do so, you need a few tools. Most of them are probably already installed on your system, but you may need to install “re2c” and “bison” using the package manager of your choice. On Ubuntu you’d do this:

$ sudo apt-get install re2c
$ sudo apt-get install bison

Next, clone php-src from git and compile it:

// get source code
$ git clone http://git.php.net/repository/php-src.git
$ cd php-src
// create new branch for in operator
$ git checkout -b addInOperator
// build ./configure script
$ ./buildconf
// configure PHP in debug mode and with thread safety
$ ./configure --disable-all --enable-debug --enable-maintainer-zts
// compile (4 is the number of cores you have)
$ make -j4

The PHP binary should now be available in sapi/cli/php. You can try to do a few things:

$ sapi/cli/php -v
$ sapi/cli/php -r 'echo "Hallo World!";'

Now that you have a (hopefully) working PHP compile, we’ll take a look at what PHP actually does when it runs a script.

The life of a PHP script

To run a script PHP goes through three main phases:

Tokenization
Parsing & Compilation
Execution

In the following I’ll explain what exactly is done in each phase, how it is implemented and what we need to change in order to get the in operator working.

Tokenization

In the first phase PHP reads in the source code and breaks it down into smaller units called “tokens”. For example the PHP code <?php echo "Hello World!"; would be broken down to the following tokens:

T_OPEN_TAG (<?php )
T_ECHO (echo)
T_WHITESPACE ( )
T_CONSTANT_ENCAPSED_STRING ("Hello World!")
';'

As you can see the raw source code was broken down into semantically meaningful tokens. The process of doing so is referred to as tokenization, lexing or scanning and is implemented in the zend_language_scanner.l file of the Zend/ directory.

If you open the file and scroll down a bit (to somewhere around line 1000), you’ll find a large number of token definitions that look like this:

<ST_IN_SCRIPTING>"exit" {
    return T_EXIT;
}

The meaning should be rather obvious: If exit is encountered in the source code, the lexer should tag it as T_EXIT. The content between < and > is the state that the text should be matched in. ST_IN_SCRIPTING is the normal state for PHP code. Some examples of other states are ST_DOUBLE_QUOTE (in double quoted string), ST_HEREDOC (in heredoc string), etc.

Another thing that can be done in the scanning routines is specifying a “semantic” value (also called “lower value” or “lval” for short). Here is an example:

<ST_IN_SCRIPTING,ST_VAR_OFFSET>{LABEL} {
    zend_copy_value(zendlval, yytext, yyleng);
    zendlval->type = IS_STRING;
    return T_STRING;
}

{LABEL} matches a PHP identifier (it is defined as [a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*) and the code then returns the token T_STRING. Additionally it copies the text of the token into zendlval. So if the lexer encounters an identifier like FooBarClass it’ll set FooBarClass as the lval. The same is also done for strings, numbers, variable names, etc.

Luckily the in operator does not require in-depth knowledge of the lexer. We just have to add this code snippet somewhere in the file (analogous to the exit token above):

<ST_IN_SCRIPTING>"in" {
    return T_IN;
}

Furthermore we have to let the engine know that we added a new token. For this open the zend_language_parser.y and insert the following line somewhere among its peers:

%token T_IN "in (T_IN)"

Now you should compile PHP again using make -j4 (you have to run it in the top level php-src folder, not in Zend/). This will generate a new lexer using re2c and compile it. To test that it worked, you can try running something like this:

$ sapi/cli/php -r 'in'

This should give you a nice parse error:

Parse error: syntax error, unexpected 'in' (T_IN) in Command line code on line 1

A last thing we have to do is regenerate the data used by the tokenizer extension (which exposes the internal lexer to userland PHP code). For this you have to cd into the ext/tokenizer folder and execute ./tokenizer_data_gen.sh.

If you run git diff --stat now, you will see something like this:

Zend/zend_language_parser.y       |    1 +
Zend/zend_language_scanner.c      | 1765 +++++++++++++++++++------------------
Zend/zend_language_scanner.l      |    4 +
Zend/zend_language_scanner_defs.h |    2 +-
ext/tokenizer/tokenizer_data.c    |    4 +-
5 files changed, 904 insertions(+), 872 deletions(-)

The zend_language_scanner.c file is the actual lexer generated using re2c. Because it contains line number information every change to the lexer will cause a huge diff. So don’t worry about that ;)

Parsing & Compilation

Now that the source code is broken down in meaningful tokens PHP has to recognize larger structures like “this is an if block” or “you’re defining a function there”. This process is called parsing and is defined in the zend_language_parser.y file. Again this is just a definition file and the actual parser is generated using bison.

To understand how the parser definitions work, let’s look at an example:

class_statement:
        variable_modifiers { CG(access_type) = Z_LVAL($1.u.constant); } class_variable_declaration ';'
    |   class_constant_declaration ';'
    |   trait_use_statement
    |   method_modifiers function is_reference T_STRING { zend_do_begin_function_declaration(&$2, &$4, 1, $3.op_type, &$1 TSRMLS_CC); } '('
           parameter_list ')' method_body { zend_do_abstract_method(&$4, &$1, &$9 TSRMLS_CC); zend_do_end_function_declaration(&$2 TSRMLS_CC); }
;

For now, let’s leave the parts in curly braces out, so we’re left with the following:

class_statement:
        variable_modifiers class_variable_declaration ';'
    |   class_constant_declaration ';'
    |   trait_use_statement
    |   method_modifiers function is_reference T_STRING '(' parameter_list ')' method_body
;

You can read this as:

A class statement is
        a variable declaration (with access modifier)
    or  a class constant declaration
    or  a trait use statement
    or  a method (with method modifier, optional return-by-ref, method name, parameter list and method body)
.

To find out what exactly a “method modifier” is you’d then go to the method_modifier definition, etc. Should be fairly straightforward.

So in order to add in support to the parser, all you have to do is add a new expr T_IN expr rule to expr_without_variable:

expr_without_variable:
    ...
    |   expr T_IN expr
    ...
;

If you now run make -j4 again, bison will attempt to rebuild the parser, but it will fail with the following rather obscure error message:

conflicts: 87 shift/reduce
/some/path/php-src/Zend/zend_language_parser.y: expected 3 shift/reduce conflicts
make: *** [/some/path/php-src/Zend/zend_language_parser.c] Error 1

A shift/reduce conflict basically means that the parser doesn’t know what to do in some situation. The PHP grammar has three shift/reduce conflicts by itself (which is expected due to stuff like the dangling elseif/else ambiguity). The remaining 84 conflicts are caused by the new rule.

The reason is that we didn’t specify how in should behave around other operators. An example:

// if you write
$foo in $bar && $someOtherCond
// should PHP interpret this as
($foo in $bar) && $someOtherCond
// or as
$foo in ($bar && $someOtherCond)

The above is called “operator precedence”. A related concept is “operator associativity”, which determines what happens when you write $foo in $bar in $baz.

In order to fix the shift/reduce conflicts all you have to do is find the following line at the start of the parser and T_IN at the end of it:

%nonassoc '<' T_IS_SMALLER_OR_EQUAL '>' T_IS_GREATER_OR_EQUAL

What this means is that in has the same operator precedence as the <-style comparison operators and that the operator is non-associative. Here are some examples of what this does:

$foo in $bar && $someOtherCond
// is interpreted as
($foo in $bar) && $someOtherCond
// because `&&` has lower precedence than `in`

$foo in ['abc', 'def'] + ['ghi', 'jkl']
// is interpreted as
$foo in (['abc', 'def'] + ['ghi', 'jkl'])
// because `+` has a higher precedence than `in`

$foo in $bar in $baz
// will throw a parse error, because `in` is non-associative

If you rerun make -j4 now everything should work out fine. After that you can try out running something like sapi/cli/php -r '"foo" in "bar";'. This should do nothing, apart from printing a memory leak info:

[Thu Jul 26 22:33:14 2012]  Script:  '-'
Zend/zend_language_scanner.l(876) :  Freeing 0xB777E7AC (4 bytes), script=-
=== Total 1 memory leaks detected ===

This is expected because we haven’t told the parser yet what it should do when it matches an in operator. And this is where the curly braces come in. What you have to do is replace the existing expr T_IN expr rule with the following:

expr T_IN expr { zend_do_binary_op(ZEND_IN, &$$, &$1, &$3 TSRMLS_CC); }

The part in the curly braces is called a semantic action and is run whenever the parser matches a certain rule (or part of it). The strange looking $$, $1 and $3 variables in there are nodes. For example $1 refers to the first expr, $3 refers to the second expr ($3 because it is the third element in the rule) and $$ is the result node.

zend_do_binary_op is a compiler instruction. It tells the compiler to emit a ZEND_IN opcode that will take $1 and $3 as operands and put the result into $$.

Compiler instructions are defined in zend_compile.c (with a header entry in zend_compile.h). zend_do_binary_op for example looks like this:

void zend_do_binary_op(zend_uchar op, znode *result, const znode *op1, const znode *op2 TSRMLS_DC)
{
    zend_op *opline = get_next_op(CG(active_op_array) TSRMLS_CC);

    opline->opcode = op;
    opline->result_type = IS_TMP_VAR;
    opline->result.var = get_temporary_variable(CG(active_op_array));
    SET_NODE(opline->op1, op1);
    SET_NODE(opline->op2, op2);
    GET_NODE(result, opline->result);
}

The code should be easy to grasp and the next section should help putting it into context. One last thing to mention here is that in most cases you will have to add your own zend_do_* function when adding some new syntax. Adding a new binary operator is one of the few cases where you don’t have to. So if you have to add a new zend_do_* function, just have a look at the existing ones. Most of them are quite simple.

Execution

In the previous section I already mentioned that the compiler is emitting opcodes. Let’s look closer at how those opcodes look like (see zend_compile.h):

struct _zend_op {
    opcode_handler_t handler;
    znode_op op1;
    znode_op op2;
    znode_op result;
    ulong extended_value;
    uint lineno;
    zend_uchar opcode;
    zend_uchar op1_type;
    zend_uchar op2_type;
    zend_uchar result_type;
};

A short description of what the individual components mean:

opcode: This is the actual operation that should be executed. This could for example be ZEND_ADD or ZEND_SUB.
op1, op2, result: Every operation can have a maximum of two operands (it can obviously also use just one of them, or even none at all) and a result node. The type of the nodes is given by op1_type, op2_type and result_type. We’ll cover how nodes looks like and what types there are in a minute.
extended_value: The extended value is used to store flags or some other integer value. E.g. the variable fetching instruction uses it to store the variable kind (like ZEND_FETCH_LOCAL or ZEND_FETCH_GLOBAL).
handler: To optimize opcode execution this stores the handler function associated with the opcode and the operand types. This is determined automatically, so it doesn’t have to be set in the compilation code.
lineno: You know what that means…

There are five basic types that can go into the *_type properties:

IS_TMP_VAR: A temporary variable, usually the result of some expression like $foo + $bar. Temporary variables cannot be shared, so they don’t implement refcounting. They are usually very short lived, so one instruction creates them and the next one already frees them again. Temporary variables are usually written as ~n, so ~0 would be the first temporary variable, ~1 the second, etc.
IS_CV: A compiled variable. To save hash table lookups PHP caches the location of simple variables like $foo in an array (C array). Furthermore compiled variables allow PHP to optimize the hash table away altogether. CVs are denoted using !n (n here is the offset into the compiled-variable array.)
IS_VAR: Only simple variables can be turned into CVs. All other kinds of variable accesses, like $foo['bar'] or $foo->bar return an IS_VAR variable. It basically is just a normal zval (with refcounting and everything). Vars are written as $n.
IS_CONST: Constants are literals in the code. For example if you write "foo" or 3.141 in your code, those will be of type IS_CONST. Constants allow for some further optimizations, like reusing zvals for the same value, or precalculating hashes.
IS_UNUSED: Operand not used.

Related to this is how znode_op itself looks like:

typedef union _znode_op {
    zend_uint      constant;
    zend_uint      var;
    zend_uint      num;
    zend_ulong     hash;
    zend_uint      opline_num;
    zend_op       *jmp_addr;
    zval          *zv;
    zend_literal  *literal;
    void          *ptr;
} znode_op;

As you can see a node is a union, i.e. it can contain one of the elements above (just one!), depending on context. For example zv is used to store IS_CONST zvals, var is used to store the variable number for IS_CV, IS_VAR and IS_TMP_VAR variables. The rest is used in various special circumstances. E.g. jmp_addr is used with the JMP* instructions (which are required for conditions and loops). Others again are used only during compilation, not during execution (like constant).

So now that we know how individual ops look like, the only remaining question is where those are stored: For every function (and file) PHP creates a zend_op_array, which stores the opcodes as well as a lot of other information. I don’t want to go into detail what the individual components are for, you should just know that this structure exists.

Now let’s get back to the implementation of the in operator! We already instructed the compiler to emit a ZEND_IN opcode. Now we have to define what this opcode does.

This is done in the zend_vm_def.h file. If you look at it, you’ll find that it is full of definitions like this:

ZEND_VM_HANDLER(1, ZEND_ADD, CONST|TMP|VAR|CV, CONST|TMP|VAR|CV)
{
    USE_OPLINE
    zend_free_op free_op1, free_op2;

    SAVE_OPLINE();
    fast_add_function(&EX_T(opline->result.var).tmp_var,
        GET_OP1_ZVAL_PTR(BP_VAR_R),
        GET_OP2_ZVAL_PTR(BP_VAR_R) TSRMLS_CC);
    FREE_OP1();
    FREE_OP2();
    CHECK_EXCEPTION();
    ZEND_VM_NEXT_OPCODE();
}

The ZEND_IN opcode will look very similar to this, so it’s worth understanding what is going on there. I’ll go through it line by line:

// The header defines four things:
//   1. This is the opcode with ID 1
//   2. This opcode is called ZEND_ADD
//   3. This opcode accepts CONST, TMP, VAR and CV as the first operand
//   4. This opcode accepts CONST, TMP, VAR and CV as the second operand
ZEND_VM_HANDLER(1, ZEND_ADD, CONST|TMP|VAR|CV, CONST|TMP|VAR|CV)
{
    // USE_OPLINE means that we want to access the zend_op as `opline`.
    // This is required for all opcodes accessing operands or setting a return value.
    USE_OPLINE
    // For every operand that is accessed a free_op* variable has to be defined.
    // It is used to figure out whether the operand needs freeing.
    zend_free_op free_op1, free_op2;

    // SAVE_OPLINE() actually loads the zend_op into `opline`.
    // USE_OPLINE was only the declaration
    SAVE_OPLINE();
    // Call the fast add function
    fast_add_function(
        // And tell it to put the result into the temporary result variable
        // EX_T here accesses the temporary variable with ID opline->result.var.
        &EX_T(opline->result.var).tmp_var,
        // Fetch the first operand for reading (the R in BP_VAR_R)
        GET_OP1_ZVAL_PTR(BP_VAR_R),
        // Fetch the second operand for reading
        GET_OP2_ZVAL_PTR(BP_VAR_R) TSRMLS_CC);
    // Free both operands (if necessary)
    FREE_OP1();
    FREE_OP2();
    // Check for exceptions. Exceptions can occur virtually everywhere, so one has to check for them in nearly all
    // opcodes. If in doubt, add the check.
    CHECK_EXCEPTION();
    // Go to the next opcode
    ZEND_VM_NEXT_OPCODE();
}

As you probably noticed there is a lot UPPERCASE_STUFF in there. The reason is that zend_vm_def.h once again is only a definition file. The actual Zend VM is generated from it and stored in zend_vm_execute.h (biiig file). PHP has three different virtual machine kinds, namely CALL (default), GOTO and SWITCH. Because they all have different implementation details the definition file uses lots of pseudo-macros like USE_OPLINE that are later replaced by some concrete implementation.

Furthermore the generated VM creates specialized implementations from all possible operand-type permutations. So in the end there won’t be a single ZEND_ADD function, but rather different functions for ZEND_ADD_CONST_CONST, ZEND_ADD_CONST_TMP, ZEND_ADD_CONST_VAR…

Now, in order to implement the ZEND_IN opcode, you should add a new opcode definition skeleton at the end of the zend_vm_def.h file:

// 159 is the number of the next free opcode for me. You may need to choose a larger number
ZEND_VM_HANDLER(159, ZEND_IN, CONST|TMP|VAR|CV, CONST|TMP|VAR|CV)
{
    USE_OPLINE
    zend_free_op free_op1, free_op2;
    zval *op1, *op2;

    SAVE_OPLINE();
    op1 = GET_OP1_ZVAL_PTR(BP_VAR_R);
    op2 = GET_OP2_ZVAL_PTR(BP_VAR_R);

    /* TODO */

    FREE_OP1();
    FREE_OP2();
    CHECK_EXCEPTION();
    ZEND_VM_NEXT_OPCODE();
}

This does nothing more than fetching the operands and discarding them again right away.

In order to generate a new VM you now have to run php zend_vm_gen.php from within the Zend/ directory. (If it gives you lots of warnings about the /e modifier being deprecated, ignore those.) After that go into the top level directory again and run make -j4 to recompile.

If everything worked out fine you can now run sapi/cli/php -r '"foo" in "bar";' without getting any errors (but it still won’t do anything).

Now, finally, we can implement the actual logic. Let’s start with the string case:

if (Z_TYPE_P(op2) == IS_STRING) {
    zval op1_copy;
    int use_copy;

    /* Convert the needle into a string */
    zend_make_printable_zval(op1, &op1_copy, &use_copy);
    if (use_copy) {
        op1 = &op1_copy;
    }

    if (Z_STRLEN_P(op1) == 0) {
        /* For empty needles return true */
        ZVAL_TRUE(&EX_T(opline->result.var).tmp_var);
    } else {
        char *found = zend_memnstr(
            Z_STRVAL_P(op2),                  /* haystack */
            Z_STRVAL_P(op1),                  /* needle */
            Z_STRLEN_P(op1),                  /* needle length */
            Z_STRVAL_P(op2) + Z_STRLEN_P(op2) /* haystack end ptr */
        );

        ZVAL_BOOL(&EX_T(opline->result.var).tmp_var, found != NULL);
    }

    /* Free copy */
    if (use_copy) {
        zval_dtor(&op1_copy);
    }
}

The hardest part here is actually casting the needle into a string. This is done using zend_make_printable_zval. This function may either have to create a new zval, or not. That’s why we pass op1_copy and use_copy into it. If the function copied the value we just put it into the op1 variable (so we don’t have to deal with two different variables everywhere). Furthermore the copy has to be freed at the end (what the last three lines do).

After that everything is simple. We can find out whether the haystack contains the needle using zend_memnstr. If the needle is an empty string we just return true directly (because the empty string is part of every string).

Now, if you added the above code in place of the /* TODO */, reran zend_vm_gen.php and recompiled using make -j4, you will already have a half-working in operator:

$ sapi/cli/php -r 'var_dump("foo" in "bar");'
bool(false)
$ sapi/cli/php -r 'var_dump("foo" in "foobar");'
bool(true)
$ sapi/cli/php -r 'var_dump("foo" in "hallo foo world");'
bool(true)
$ sapi/cli/php -r 'var_dump(2 in "123");'
bool(true)
$ sapi/cli/php -r 'var_dump(5 in "123");'
bool(false)
$ sapi/cli/php -r 'var_dump("" in "test");'
bool(true)

Next, we have to implement the array behavior:

else if (Z_TYPE_P(op2) == IS_ARRAY) {
    HashPosition pos;
    zval **value;

    /* Start under the assumption that the value isn't contained */
    ZVAL_FALSE(&EX_T(opline->result.var).tmp_var);

    /* Iterate through the array */
    zend_hash_internal_pointer_reset_ex(Z_ARRVAL_P(op2), &pos);
    while (zend_hash_get_current_data_ex(Z_ARRVAL_P(op2), (void **) &value, &pos) == SUCCESS) {
        zval result;

        /* Compare values using == */
        if (is_equal_function(&result, op1, *value TSRMLS_CC) == SUCCESS && Z_LVAL(result)) {
            ZVAL_TRUE(&EX_T(opline->result.var).tmp_var);
            break;
        }

        zend_hash_move_forward_ex(Z_ARRVAL_P(op2), &pos);
    }
}

Here the haystack is simply traversed and every values is checked against the needle. We compare using ==. To compare using === one would have to replace is_equal_function with is_identical_function.

After rerunning zend_vm_gen.php and make -j4 the in operator should be fully operational:

$ sapi/cli/php -r 'var_dump("test" in []);'
bool(false)
$ sapi/cli/php -r 'var_dump("test" in ["foo", "bar"]);'
bool(false)
$ sapi/cli/php -r 'var_dump("test" in ["foo", "test", "bar"]);'
bool(true)
$ sapi/cli/php -r 'var_dump(0 in ["foo"]);'
bool(true) // because we're comparing using ==

One last thing to consider is what should happen when the second operator is neither array nor string. I’ll just take the easy way out for this: Throw a warning and return false:

else {
    zend_error(E_WARNING, "Right operand of in has to be either string or array");
    ZVAL_FALSE(&EX_T(opline->result.var).tmp_var);
}

After rebuilding and recompiling the VM:

$ sapi/cli/php -r 'var_dump("foo" in new stdClass);'

Warning: Right operand of in has to be either string or array in Command line code on line 1
bool(false)

This might not be the best behavior. E.g. one could allow to do things like 2 in 123 or 3.14 in 3.141. But I’m too lazy for that right now ;)

Finishing thoughts

I hope the above helped you understand how to add new features to PHP and what the Zend Engine does when it runs a PHP script. But even though the article is quite long I covered only small parts of the whole system. So when you want to do modifications to the ZE the largest part of the job will be reading the existing code. The cross-reference tool helps a lot when browsing through the PHP source code. Apart from that you can always ask questions in the #php.pecl room on efnet.

After you created an implementation for whatever feature you want, the next step is bringing it up on the internals mailing list. People will then look at your feature and decide whether or not it should go in.

Oh, and one last thing: The in operator here was just an example. I don’t plan on proposing it for inclusion ;)

As always, if you have any further comments or questions, please leave them below.

What PHP 5.5 might look like

nikic@php.net (NikiC) — Tue, 10 Jul 2012 00:00:00 -0700

PHP 5.4 was released just four months ago, so it probably is a bit too early to look at the next PHP version. Still I’d like to give all the people who aren’t following the internals mailing list a small sneak peek at what PHP 5.5 might look like.

But be sure to understand this: PHP 5.5 is in an early development stage, so nobody knows how the end result will look like. All I am talking about here are proposals. I’m pretty sure that not all of the things listed below will go into PHP 5.5, or at least not in their current form.

So, don’t get too excited about this :)

The list of new features / proposals is rather large and not sorted by significance. So if you don’t want to read through all of it, here are the four features I personally am most excited about:

A simple API for password hashing
Scalar typehinting
Getters and setters
Generators

Now, without further ado, the list of stuff being worked on for PHP 5.5:

Backwards compatibility breaks

We’ll start off with two changes that already landed in master and represent BC breaks (to some degree at least):

Windows XP and 2003 support dropped

Status: landed; Responsible: Pierre Joye

PHP 5.5 will no longer support Windows XP and 2003. Those systems are around a decade old, so PHP is pulling the plug on them.

/e modifier deprecated

Status: landed; Responsible: myself

The e modifier instructs the preg_replace function to evaluate the replacement string as PHP code instead of just doing a simple string replacement. Unsurprisingly this behavior is a constant source of problems and security issues. That’s why use of this modifier will throw a deprecation warning as of PHP 5.5. As a replacement you should use the preg_replace_callback function. You can find more information on this change in the corresponding RFC.

Function and class additions

Next we’ll look at some the planned function and class additions:

boolval()

Status: landed; Responsible: Jille Timmermans

PHP already implements the strval, intval and floatval functions. To be consistent the boolval function is now added, too. It does exactly the same thing as a (bool) cast, but can be used as a callback function.

hash_pbkdf2()

Status: landed; Responsible: Anthony Ferrara

PBKDF2 stands for “Password-Based Key Derivation Function 2” and is - as the name already says - an algorithm for deriving a cryptographic key from a password. This is required for encryption algorithms, but can also be used for password hashing. For a more extensive description and usage examples see the RFC.

Intl additions

Status: landed; Responsible: Gustavo André dos Santos Lopes

There have been many improvements to the intl extension. E.g. there will be new IntlCalendar, IntlGregorianCalendar, IntlTimeZone, IntlBreakIterator, IntlRuleBasedBreakIterator, IntlCodePointBreakIterator classes. I sadly don’t know much about the intl extension, so I will just direct you to the mailing list announcements for Calendar and BreakIterator, if you want to know more.

array_column()

Status: proposed; Responsible: Ben Ramsey

There is a proposal pending for a new array_column (or array_pluck) function that would behave as follows:

<?php

$userNames = array_column($users, 'name');
// is the same as
$userNames = [];
foreach ($users as $user) {
    $userNames[] = $user['name'];
}

So it would be like fetching a column from a database, but for arrays.

A simple API for password hashing

Status: proposed; Responsible: Anthony Ferrara

The recent password leaks (from LinkedIn etc) have shown that even large websites don’t get how to properly hash passwords. People have been advocating the use of bcrypt for years, but still most people seem to be using completely unsafe sha1 hashes.

We figured that the reason for this might be the really hard to use API of the crypt function. Thus we would like to introduce a new, simple API for secure password hashing:

<?php

$password = "foo";

// creating the hash
$hash = password_hash($password, PASSWORD_BCRYPT);

// verifying a password
if (password_verify($password, $hash)) {
    // password correct!
} else {
    // password wrong!
}

The new hashing API comes with a few more features, which are outlined in the RFC.

Language changes

Now comes the really interesting stuff: New language features and enhancements.

Constant dereferencing

Status: landed; Responsible: Xinchen Hui

“Constant dereferencing” means that array operations can be directly applied to string and array literals. Here two examples:

<?php

function randomHexString($length) {
    $str = '';
    for ($i = 0; $i < $length; ++$i) {
        $str .= "0123456789abcdef"[mt_rand(0, 15)]; // direct dereference of string
    }
}

function randomBool() {
    return [false, true][mt_rand(0, 1)]; // direct dereference of array
}

I don’t think that this feature is of much use in practice, but it makes the language a bit more consistent. See also the RFC.

empty() works with function calls (and other expressions)

Status: landed; Responsible: myself

Currently the empty() language construct can only be used on variables, not on other expressions. In particular code like empty($this->getFriends()) would throw an error. As of PHP 5.5 this becomes valid code. For more info see the RFC.

Getting the fully qualified class name

Status: proposed; Responsible: Ralph Schindler

PHP 5.3 introduced namespaces with the ability to alias classes and namespaces to shorter versions. This does not apply to string class names though:

<?php

use Some\Deeply\Nested\Namespace\FooBar;

// does not work, because this will try to use the global `FooBar` class
$reflection = new ReflectionClass('FooBar');

To solve this a new FooBar::class syntax is proposed, which returns the fully qualified name of the class:

<?php

use Some\Deeply\Nested\Namespace\FooBar;

// this works because FooBar::class is resolved to "Some\\Deeply\\Nested\\Namespace\\FooBar"
$reflection = new ReflectionClass(FooBar::class);

For more examples see the RFC.

Parameter skipping

Status: proposed; Responsible: Stas Malyshev

If you have a function accepting multiple optional parameters there is currently no way to change just the last one, leaving all others at their default.

Taking the example from the RFC, if you have a function like the following:

function create_query($where, $order_by, $join_type='', $execute = false, $report_errors = true) { ... }

Then there is no way to set $report_errors = false without replicating the other two default values. To solve this a way of skipping parameters is proposed:

create_query("deleted=0", "name", default, default, false);

Personally I’m not particular fond of this proposal. In my eyes code that needs this feature is just badly designed. Functions shouldn’t have 12 optional parameters.

Scalar typehinting

Status: proposed; Responsible: Anthony Ferrara

Scalar typehinting was originally planned to go into 5.4, but never made it due to lack of consensus. For more info on why scalar typehints haven’t made it into PHP yet, see: Scalar typehints are harder than you think.

For PHP 5.5 the discussion has come up again and I think there is a fairly decent proposal for scalar typehints with casting.

It would work by casting the input value to the specified type, but only if the cast can occur without data loss. E.g. 123, 123.0, "123" would all be valid inputs for an int parameter, but "hallo world" would not. This matches the behavior of internal functions.

function foo(int $i) { ... }

foo(1);      // $i = 1
foo(1.0);    // $i = 1
foo("1");    // $i = 1
foo("1abc"); // not yet clear, maybe $i = 1 with notice
foo(1.5);    // not yet clear, maybe $i = 1 with notice
foo([]);     // error
foo("abc");  // error

Getters and setters

Status: proposed; Responsible: Clint Priest

If you’ve never been a fan of writing all those getXYZ() and setXYZ($value) methods, then this should be a welcome change for you. The proposal adds a new syntax for defining what should happen when a property is set / read:

<?php

class TimePeriod {
    public $seconds;

    public $hours {
        get { return $this->seconds / 3600; }
        set { $this->seconds = $value * 3600; }
    }
}

$timePeriod = new TimePeriod;
$timePeriod->hours = 10;

var_dump($timePeriod->seconds); // int(36000)
var_dump($timePeriod->hours);   // int(10)

There are also some more features like read-only properties. If you want to know more, have a look at the RFC.

Generators

Status: proposed; Responsible: myself

Currently custom iterators are used only rarely, because their implementation requires lots of boilerplate code. Generators solve this issue by providing an easy and boilerplate-free way to create iterators.

For example, this is how you could define the range function, but as an iterator:

<?php

function *xrange($start, $end, $step = 1) {
    for ($i = $start; $i < $end; $i += $step) {
        yield $i;
    }
}

foreach (xrange(10, 20) as $i) {
    // ...
}

The above xrange function has the same behavior as the builtin range function with one difference: Instead of returning an array with all the values, it returns an iterator which generates the values on-the-fly.

For more in-depth introduction into the topic see the RFC.

List comprehensions and generator expressions

Status: proposed; Responsible: myself

List comprehensions provide a simple way to do small operations on arrays:

$firstNames = [foreach ($users as $user) yield $user->firstName];

The above list comprehension is equivalent to the following code:

$firstNames = [];
foreach ($users as $user) {
    $firstNames[] = $user->firstName;
}

It is also possible to filter arrays this way:

$underageUsers = [foreach ($users as $user) if ($user->age < 18) yield $user];

Generator expressions are similar, but return an iterator (which generates the values on-the-fly) instead of an array.

For more examples see the mailing list announcement.

Wrapping up

As you can see, there is a lot of awesome stuff being worked on for PHP 5.5. But as I already said, PHP 5.5 is still very young, so we don’t know for sure what will get in and what will not.

If you want to stay updated on the new features or want to help out in discussions and/or development, be sure to subscribe to the internals mailing list.

Comments welcome!

A plea for less (XML) configuration files

nikic@php.net (NikiC) — Mon, 09 Jul 2012 00:00:00 -0700

I recently tried using Phing (a PHP build system) to do some simple release automation. Just creating a PEAR package and doing a few string replacements here and there.

The result? After several wasted hours I ended up using Phing only for PEAR packaging and doing everything else in a custom PHP build script.

The reason? Phing uses XML files to configure what it should do during a build. So an excerpt from a build file might look like this:

<?xml version="1.0" encoding="UTF-8"?>
<project name="SomeName" default="package" basedir="..">
    <target name="package">
        <property file="build/build.properties" />
        <propertyprompt propertyName="project.version" promptText="Enter project version" />

        <delete dir="${path.results.lib}" />
        <mkdir dir="${path.results.lib}" />

        <copy todir="${path.results.lib}">
            <fileset dir="${path.lib}">
                <include name="**/**" />
            </fileset>
        </copy>

        <!-- do more stuff -->
    </target>

    <!-- more targets -->
</project>

Could you explain me, why you have to do this? Why do you have to specify everything in an XML configuration file? Why can’t you just do the same thing in the programming language you’re using? E.g., why can’t I write this instead:

<?php
function package(Phing $phing) {
    include 'properties.php';
    $project->version = $phing->prompt('Enter project version');

    $phing->deleteDir($path->results->lib);
    $phing->createDir($path->results->lib);
    $phing->copyDir($path->lib, $path->results->lib);

    // do more stuff, like replace version strings with $project->version
}

Don’t you think that this is much nicer? Some reasons why I prefer this over XML configs:

It uses an environment you’re already used to. You don’t have to learn how exactly the Phing XML files work. You don’t need to learn where you can use properties and where you can’t. You simply know already.
You can use the full power of the programming language. E.g. the main reason why I stopped using Phing was that I could not implement some slightly more complex string replacement in it without writing 200 lines of XML. I was able to implement those replacements in 5 lines of PHP on the other hand.
All the tools you normally use for programming will work. E.g. the IDE will be able to provide autocompletion and error analysis. One could use phpDocumentor to document the different build targets and options. Heck, you could even unit test the building process, if you really wanted to.

And obviously: XML is a rather verbose language, whereas most programming languages (short of Brainfuck) try to be concise and readable.

At this point you might say that the above isn’t really a configuration file, but rather a program written in XML. I’d agree with you there, but I think the the above also applies to “real” configuration files. They can also benefit from the power of the programming language (like putting repeating configuration “patterns” into functions or loops), etc.

Also, this isn’t specific to XML. I think that it is generally preferable to just use your usual programming language to write configuration, instead of using some special format.

But don’t get me wrong, there certainly are cases where using a special configuration format is the way to go:

If the configuration will be used by multiple programming languages. Obviously it doesn’t make sense to write PHP configs if they have to be used by Python too.
If the type of configuration is very specific and writing it would be much easier using a Domain Specific Language.

In my personal experience neither of those points apply for most of the config files I’ve encountered.

So, what I ask you for is: If you’re creating a config file, consider writing it in your normal programming language instead of XML or some DSL.

PHP solves problems. Oh, and you can program with it too!

nikic@php.net (NikiC) — Fri, 29 Jun 2012 00:00:00 -0700

I’d like to point your attention to one particular comment on Jeff Atwood’s new “PHP sucks!” blog post, a comment by Adam Victor Nazareth Brandizzi:

I am a Java programmer at work (could be no more corporate-y) and a Python developer in my projects (could be no more hipster) but I admire PHP and its ability of solving problems. It grows because, some times, some poor soul wants to create an online encyclopaedia, or some teacher needs an online teaching platform, or someone wants to write a blog. Those people do not want to learn to program, they want to solve problems.

I think this comment really nailed it. I think this is the most important reason for PHP’s success.

People come to PHP because they have some problem and they need to solve it. This is what PHP really shines at. You can simply take your static HTML website, add a simple <?php include 'counter.php'; ?> in there, and … be done!

From there you start writing simple scripts, learn how to process forms, how to talk to the database, etc. After some time you start using object oriented programming and maybe make use of some framework.

That’s actually pretty much how I got into programming.

With most other languages it is the other way around. With them you first study computer science for five years and then you go out into the world to find some problem you can solve. (You could say that PHP is a programmer-producing language, whereas most other languages are programmer-consuming.)

At this point you may ask: Well, once you learned programming, why not switch to another language? Simple: PHP has its quirks, but it is not that bad. Sure, people always try to tell you that, but it just isn’t true. Most of the things people criticize about PHP are really non-issues practically. Like the inconsistent needle/haystack order. That’s always at the top of things that are wrong with PHP. But in reality it really doesn’t matter. Sure, it would be nice if the order was consistent, but my IDE does a very good job at reminding me how to do it correctly.

To summarize:

PHP is a great language to start programming!
Once you started, PHP is also good for “real” programming (you know, object orientation and stuff).
PHP is not as bad as they say. There are issues, like with every language, but they rarely cause problems in practice.

To add to that, I also noticed that most of the PHP bashers have a ten-year-old image of the language. E.g. consider this quote from Jeff’s post:

What’s depressing is not that PHP is horribly designed. Does anyone even dispute that PHP is the worst designed mainstream “language” to blight our craft in decades? What’s truly depressing is that so little has changed.

This statement is outrageous. PHP has changed a lot in recent years, but most people seem to remember it as the shitty language with terrible OOP support that PHP 4 was. Well, I got news for you: That language is dead for nearly a decade now. PHP 5 has very good object orientation support, which is actually really similar to Java. PHP 5.3 additionally added namespace and lambda function support to the mix. (And PHP 5.5 will add a lot of awesome new features.) And you tell me nothing changed?

End of rant.

The true power of regular expressions

nikic@php.net (NikiC) — Fri, 15 Jun 2012 00:00:00 -0700

As someone who frequents the PHP tag on StackOverflow I pretty often see questions about how to parse some particular aspect of HTML using regular expressions. A common reply to such a question is:

You cannot parse HTML with regular expressions, because HTML isn’t regular. Use an XML parser instead.

This statement - in the context of the question - is somewhere between very misleading and outright wrong. What I’ll try to demonstrate in this article is how powerful modern regular expressions really are.

What does “regular” actually mean?

In the context of formal language theory, something is called “regular” when it has a grammar where all production rules have one of the following forms:

B -> a
B -> aC
B -> ε

You can read those -> rules as “The left hand side can be replaced with the right hand side”. So the first rule would be “B can be replaced with a”, the second one “B can be replaced with aC” and the third one “B can be replaced with the empty string” (ε is the symbol for the empty string).

So what are B, C and a? By convention, uppercase characters denote so called “non-terminals” - symbols which can be broken down further - and lowercase characters denote “terminals” - symbols which cannot be broken down any further.

All that probably sounds a bit abstract, so let’s look at an example: Defining the natural numbers as a grammar.

N -> 0
N -> 1
N -> 2
N -> 3
N -> 4
N -> 5
N -> 6
N -> 7
N -> 8
N -> 9
N -> 0N
N -> 1N
N -> 2N
N -> 3N
N -> 4N
N -> 5N
N -> 6N
N -> 7N
N -> 8N
N -> 9N

What this grammar says is:

A natural number (N) is
... one of the digits 0 to 9
or
... one of the digits 0 to 9 followed by another natural number (N)

In this example the digits 0 to 9 would be terminals (as they can’t be broken down any further) and N would be the only non-terminal (as it can be and is broken down further).

If you have another look at the rules and compare them to the definition of a regular grammar from above, you’ll see that they meet the criteria: The first ten rules are of the form B -> a and the second ten rules follow the form B -> aC. Thus the grammar defining the natural numbers is regular.

Another thing you might notice is that even though the above grammar defines such a simple thing, it is already quite bloated. Wouldn’t it be better if we could express the same concept in a more concise manner?

And that’s where regular expressions come in: The above grammar is equivalent to the regex [0-9]+ (which is a hell lot simpler). And this kind of transformation can be done with any regular grammar: Every regular grammar has a corresponding regular expression which defines all its valid strings.

What can regular expressions match?

Thus the question arises: Can regular expressions match only regular grammars, or can they also match more? The answer to this is both yes and no:

Regular expressions in the formal grammar sense can (pretty much by definition) only parse regular grammars and nothing more.

But when programmers talk about “regular expressions” they aren’t talking about formal grammars. They are talking about the regular expression derivative which their language implements. And those regex implementations are only very slightly related to the original notion of regularity.

Any modern regex flavor can match a lot more than just regular languages. How much exactly, that’s what the rest of the article is about.

To keep things simple, I’ll focus on the PCRE regex implementation in the following, simply because I know it best (as it’s used by PHP). Most other regex implementations are quite similar though, so most stuff should apply to them too.

The language hierarchy

In order to analyze what regular expressions can and cannot match, we first have to look at what other types of languages there are. A good starting point for this is the Chomsky hierarchy:

Chomsky hierarchy:

/-------------------------------------------\
|                                           |
|     Recursively enumerable languages      | Type 0
|                                           |
|   /-----------------------------------\   |
|   |                                   |   |
|   |    Context-sensitive languages    |   | Type 1
|   |                                   |   |
|   |   /---------------------------\   |   |
|   |   |                           |   |   |
|   |   |  Context-free languages   |   |   | Type 2
|   |   |                           |   |   |
|   |   |   /-------------------\   |   |   |
|   |   |   | Regular languages |   |   |   | Type 3
|   |   |   \-------------------/   |   |   |
|   |   \---------------------------/   |   |
|   \-----------------------------------/   |
\-------------------------------------------/

As you can see the Chomsky hierarchy divides formal languages into four types:

Regular languages (Type 3) are the least-powerful, followed by the context-free languages (Type 2), the context-sensitive languages (Type 1) and at last the all-mighty recursively enumerable languages (Type 0).

The Chomsky hierarchy is a containment hierarchy, so the smaller boxes in the above image are fully contained in the larger boxes. For example every regular language is also a context-free language (but not the other way around!)

So, let’s move one step up in that hierarchy: We already know that regular expressions can match any regular language. But can they also match context-free languages?

(Reminder: When I say “regular expression” here I obviously mean it in the programmer sense, not the formal language theory sense.)

Matching context-free languages

The answer to this is yes, they can!

Let’s take the classical example of a context-free language, namely {a^n b^n, n>0}, which means “A number of a characters followed by the same number of b characters”. The (PCRE) regex for this language is:

/^(a(?1)?b)$/

The regular expression is very simple: (?1) is a reference to the first subpattern, namely (a(?1)?b). So basically you could replace the (?1) by that subpattern, thus forming a recursive dependency:

/^(a(?1)?b)$/
/^(a(a(?1)?b)?b)$/
/^(a(a(a(?1)?b)?b)?b)$/
/^(a(a(a(a(?1)?b)?b)?b)?b)$/
# and so on

From the above expansions it should be clear that this expression can match any string with the same number of as and bs.

Thus regular expressions can match at least some non-regular, context-free grammars. But can they match all? To answer that, we first have to look at how context-free grammars are defined.

In a context-free grammar all production rules take the following form:

A -> β

Here A once again is a non-terminal symbol and β is an arbitrary string of terminals and non-terminals. Thus every production rule of a context-free grammar has a non-terminal on the left hand side and an arbitrary symbol string on the right hand side.

As an example, have a look at the following grammar:

function_declaration -> T_FUNCTION is_ref T_STRING '(' parameter_list ')' '{' inner_statement_list '}'

is_ref -> '&'
is_ref -> ε

parameter_list -> non_empty_parameter_list
parameter_list -> ε

non_empty_parameter_list -> parameter
non_empty_parameter_list -> non_empty_parameter_list ',' parameter

// ... ... ...

What you see there is an excerpt from the PHP grammar (just a few sample rules). The syntax is slightly different from what we used before, but should be easy to understand. One aspect worth mentioning is that the uppercase T_SOMETHING names here also are terminal symbols. These symbols which are usually called tokens encode more abstract concepts. E.g. T_FUNCTION represents the function keyword and T_STRING is a label token (like getUserById or some_other_name).

I’m using this example to show one thing: Context-free grammars are already powerful enough to encode quite complex languages. That’s why pretty much all programming languages have a context-free grammar. In particular this also includes well-formed HTML.

Now, back to the actual question: Can regular expressions match all context-free grammars? Once again, the answer is yes!

This is pretty easy to prove as regular expressions (at least PCRE and similar) provide a syntax very similar to the above for constructing grammars:

/
    (?(DEFINE)
        (?<addr_spec> (?&local_part) @ (?&domain) )
        (?<local_part> (?&dot_atom) | (?&quoted_string) | (?&obs_local_part) )
        (?<domain> (?&dot_atom) | (?&domain_literal) | (?&obs_domain) )
        (?<domain_literal> (?&CFWS)? \[ (?: (?&FWS)? (?&dtext) )* (?&FWS)? \] (?&CFWS)? )
        (?<dtext> [\x21-\x5a] | [\x5e-\x7e] | (?&obs_dtext) )
        (?<quoted_pair> \\ (?: (?&VCHAR) | (?&WSP) ) | (?&obs_qp) )
        (?<dot_atom> (?&CFWS)? (?&dot_atom_text) (?&CFWS)? )
        (?<dot_atom_text> (?&atext) (?: \. (?&atext) )* )
        (?<atext> [a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+ )
        (?<atom> (?&CFWS)? (?&atext) (?&CFWS)? )
        (?<word> (?&atom) | (?&quoted_string) )
        (?<quoted_string> (?&CFWS)? " (?: (?&FWS)? (?&qcontent) )* (?&FWS)? " (?&CFWS)? )
        (?<qcontent> (?&qtext) | (?&quoted_pair) )
        (?<qtext> \x21 | [\x23-\x5b] | [\x5d-\x7e] | (?&obs_qtext) )

        # comments and whitespace
        (?<FWS> (?: (?&WSP)* \r\n )? (?&WSP)+ | (?&obs_FWS) )
        (?<CFWS> (?: (?&FWS)? (?&comment) )+ (?&FWS)? | (?&FWS) )
        (?<comment> \( (?: (?&FWS)? (?&ccontent) )* (?&FWS)? \) )
        (?<ccontent> (?&ctext) | (?&quoted_pair) | (?&comment) )
        (?<ctext> [\x21-\x27] | [\x2a-\x5b] | [\x5d-\x7e] | (?&obs_ctext) )

        # obsolete tokens
        (?<obs_domain> (?&atom) (?: \. (?&atom) )* )
        (?<obs_local_part> (?&word) (?: \. (?&word) )* )
        (?<obs_dtext> (?&obs_NO_WS_CTL) | (?&quoted_pair) )
        (?<obs_qp> \\ (?: \x00 | (?&obs_NO_WS_CTL) | \n | \r ) )
        (?<obs_FWS> (?&WSP)+ (?: \r\n (?&WSP)+ )* )
        (?<obs_ctext> (?&obs_NO_WS_CTL) )
        (?<obs_qtext> (?&obs_NO_WS_CTL) )
        (?<obs_NO_WS_CTL> [\x01-\x08] | \x0b | \x0c | [\x0e-\x1f] | \x7f )

        # character class definitions
        (?<VCHAR> [\x21-\x7E] )
        (?<WSP> [ \t] )
    )
    ^(?&addr_spec)$
/x

What you see above is a regular expression for matching email addresses as per RFC 5322. It was constructed simply by transforming the BNF rules from the RFC into a notation that PCRE understands.

The syntax is quite simple:

All rules definitions are wrapped into a DEFINE assertion, which basically means that all those rules should not be directly matched against, they should just be defined. Only the ^(?&addr_spec)$ part at the end specifies what should be matched.

The rule definitions are actually not really “rules” but rather named subpatterns. In the previous (a(?1)?b) example the 1 referenced the first subpattern. With many subpatterns this obviously is impractical, thus they can be named. So (?<xyz> ...) defines a pattern with name xyz. (?&xyz) then references it.

Also, pay attention to another fact: The regular expression above uses the x modifier. This instructs the engine to ignore whitespace and to allow #-style comments. This way you can nicely format the regex, so that other people can actually understand it. (Much unlike this RFC 822 email address regex…)

The above syntax thus allows simple mappings from grammars to regular expressions:

A -> B C
A -> C D
// becomes
(?<A> (?&B) (?&C)
    | (?&C) (?&D)
)

The only catch is: Regular expressions don’t support left recursion. E.g. taking the above definition of a parameter list:

non_empty_parameter_list -> parameter
non_empty_parameter_list -> non_empty_parameter_list ',' parameter

You can’t directly convert it into a grammar based regex. The following will not work:

(?<non_empty_parameter_list>
    (?&parameter)
  | (?&non_empty_parameter_list) , (?&parameter)
)

The reason is that here non_empty_parameter_list appears as the leftmost part of it’s own rule definition. This is called left-recursion and is very common in grammar definitions. The reason is that the LALR(1) parsers which are usually used to parse them handle left-recursion much better than right-recursion.

But, no fear, this does not affect the power of regular expressions at all. Every left-recursive grammar can be transformed to a right-recursive one. In the above example it’s as simple as swapping the two parts:

non_empty_parameter_list -> parameter
non_empty_parameter_list -> parameter ',' non_empty_parameter_list

So now it should be clear that regular expressions can match any context-free language (and thus pretty much all languages which programmers are confronted with). Only problem is: Even though regular expressions can match context-free languages nicely, they can’t usually parse them. Parsing means converting some string into an abstract syntax tree. This is not possible using regular expressions, at least not with PCRE (sure, in Perl where you can embed arbitrary code into a regex you can do pretty much everything…).

Still, the above DEFINE based regex definition has proven to be very useful to me. Usually you don’t need full parsing support, but want to just match (e.g. email addresses) or extract small pieces of data (not the whole parse tree). Most complex string processing problems can be made much simpler using grammar based regexes :)

At this point, let me point out again what I already quickly mentioned earlier: Well-formed HTML is context-free. So you can match it using regular expressions, contrary to popular opinion. But don’t forget two things: Firstly, most HTML you see in the wild is not well-formed (usually not even close to it). And secondly, just because you can, doesn’t mean that you should. You could write your software in Brainfuck, still for some reason you don’t.

My opinion on the topic is: Whenever you need generic HTML processing, use a DOM library of your choice. It’ll gracefully handle malformed HTML and take the burden of parsing from you. On the other hand if you are dealing with specific situations a quick regular expression is often the way to go. And I have to admit: Even though I often tell people to not parse HTML with regular expressions I do it myself notoriously often. Simply because in most cases I deal with specific and contained situations in which using regex is just simpler.

Context-sensitive grammars

Now that we covered context-free languages extensively, let’s move up one step in the Chomsky hierarchy: Context-sensitive languages.

In a context-sensitive language all production rules have the following form:

αAβ → αγβ

This mix of characters might start to look more complicated, but it is actually quite simple. At it’s core you still have the pattern A → γ, which was how we defined context-free grammars. The new thing now is that you additionally have α and β on both sides. Those two form the context (which also gives this grammar class the name). So basically A can now only be replaced with γ if it has α to its left and β to its right.

To make this more clear, try to interpret the following rules:

a b A -> a b c
a B c -> a Q H c
H B -> H C

The English translations would be:

Replace `A` with `c`, but only if it has `a b` to its left.
Replace `B` with `Q H`, but only if it has `a` to its left and `c` to its right.
Replace `B` with `C`, but only if it has `H` to its left.

Context-sensitive languages are something that you will rarely encounter during “normal” programming. They are mostly important in the context of natural language processing (as natural languages are clearly not context-free. Words have different meaning depending on context). But even in natural language processing people usually work with so called “mildly context-sensitive languages”, as they are sufficient for modeling the language but can be parsed much faster.

To understand just how powerful context-sensitive grammars are let’s look at another grammar class, which has the exact same expressive power as the context-sensitive ones: Non-contracting grammars.

With non-contracting grammars every production rule has the form α -> β where both α and β are arbitrary symbol strings with just one restriction: The number of symbols on the right hand side is not less than on the left hand side. Formally this is expressed in the formula |α| <= |β| where |x| denotes the length of the symbol string.

So non-contracting grammars allow rules of any form as long as they don’t shorten the input. E.g. A B C -> H Q would be an invalid rule as the left hand side has three symbols and the right hand side only two. Thus this rule would be shortening (or “contracting”). The reverse rule H Q -> A B C on the other hand would be valid, as the right side has more symbols than the left, thus being lengthening.

This equivalence relationship of context-sensitive grammars and non-contracting grammars should make pretty clear that you can match near-everything with a context-sensitive grammar. Just don’t shorten :)

To get an impression of why both grammar kinds have the same expressive power look at the following transformation example:

// the non-contracting grammar
A B -> C D
// can be transformed to the following context-sensitive grammar
A B -> A X
A X -> Y X
Y X -> Y D
Y D -> C D

Anyways, back to regular expressions. Can they match context-sensitive languages too?

This time I can’t gave you definite answer. They certainly can match some context-sensitive languages, but I don’t know whether they can match all of them.

An example of a context-sensitive language that can be easily matched using regex is a modification of the context-free language {a^n b^n, n>0} mentioned above. When you change it into {a^n b^n c^n, n>0}, i.e. some number of as followed by the same number of bs and cs, it becomes context-sensitive.

The PCRE regex for this language is this:

/^
    (?=(a(?-1)?b)c)
     a+(b(?-1)?c)
$/x

If you ignore the (?=...) assertion for now you’re left with a+(b(?-1)?c). This checks that there is an arbitrary number of as, followed by the same number of bs and cs. The (?-1) is a relative subpattern reference and means “the last defined subpattern”, which is (b(?-1)?c) in this case.

The new thing now is the (?=...) which is a so called zero-width lookahead assertion. It checks that the following text matches the pattern, but it does not actually consume the text. Thus the text is basically checked against both patterns at the same time. The a+(b(?-1)?c) part verifies that the number of bs and cs is the same and the (a(?-1)?b)c part checks that the number of as and bs is the same. Both pattern together thus ensure that the number of all three characters is the same.

In the above regex you can already see how the concept of “context” is realized in regular expressions: Using assertions. If we get back to the definition of a context-sensitive grammar, you could now say that a production rule of type

αAβ → αγβ

can be converted into the following regex DEFINE rule:

(?<A> (?<= α ) γ (?= β ) )

This would then say that A is γ, but only if it has α to its left and β to its right.

Now, the above might look as if you can easily convert a context-sensitive grammar into a regular expression, but its not actually true. The reason is that lookbehind assertions ((?<= ... )) have one very significant limitation: They have to be fixed-width. This means that the length of the text matched by the assertion has to known in advance. E.g. you can write (?<= a(bc|cd) ), but you can’t write (?<= ab+). In the first case the assertion matches exactly three characters in any case, thus being fixed-width. In the second case on the other hand the assertion could match ab, abb, abbb etc. All of those have different lengths. Thus the engine can’t know when it should start to match them and as such they are simply disallowed.

This pretty much blows the easy conversion of context-sensitive grammars to regex. Pretty much all such grammars require variable-width lookbehind assertions.

But the fact that there is no direct context-sensitive grammar to regex conversion doesn’t by itself mean that regular expressions can’t match all of them. E.g. the above {a^n b^n c^n, n>0} language also has a grammar that would require variable-width lookbehind assertions. But we can still avoid using them as regex isn’t bound to specifying rules in a grammar. Maybe the same is possible for all other context-sensitive grammars too. I honestly don’t know.

So, what can we say here? Regex can match at least some context-sensitive languages, but it’s unknown whether it can match all of them.

Unrestricted grammars

The next grammar class in the Chomsky hierarchy are the unrestricted grammars. The language set which one can form using them is the set of all recursively enumerable languages.

There is little to say about unrestricted grammars as they are, well, unrestricted. Production rules for unrestricted grammars have the form α -> β, where α and β are symbol strings with no restrictions whatsoever.

So basically unrestricted grammars remove the “non-contracting” part of the non-contracting grammars. Thus for them A B C -> H Q would be a valid rule, even though previously it wasn’t.

How powerful are unrestricted grammars exactly? They are as powerful as it gets: They are Turing-complete. There even is a “programming language” which is based on unrestricted grammars: Thue. As it is Turing-complete it can do everything that other languages can do.

One implication of being Turing-complete is that checking whether a certain string adheres to some grammar is undecidable for the general case.

Sadly I can’t say anything whatsoever about how regular expressions and unrestricted grammars relate. Heck, I couldn’t even find an example of a meaningful unrestricted grammar (that wasn’t non-contracting).

But now that we started talking about Turing-completeness we get to another point:

Regular expressions with backreferences are NP-complete

There is another very powerful regular expression feature that I did not mentioned previously: backreferences.

E.g. consider this very simple regex:

/^(.+)\1$/

(.+) matches some arbitrary text and \1 matches the same text. In general \n means “whatever the nth subpattern matched”. E.g. if (.+) matched foo, then \1 will also match only foo and nothing else. Thus the expression (.+)\1 means “Some text followed by a copy of itself”.

What this simple regex matches is called the “copy language” and is another typical example of a context-sensitive language.

Similarly you can match the other example grammars from above using backreferences:

# {a^n b^n, n>0} (context-free)
/^ (?: a (?= a* (\1?+ b) ) )+ \1 $/x
# {a^n b^n c^n, n>0} (context-sensitive)
/^ (?: a (?= a* (\1?+ b) b* (\2?+ c) ) )+ \1 \2 $/x

Explaining how these work is outside the scope of this article, but you can read an excellent explanation on StackOverflow.

As you can see, the mere addition of backreference (without subpattern recursion support) already adds a lot of power to regular expressions. The addition is actually so powerful that it makes matching of regular expressions an NP-complete problem.

What does NP-complete mean? NP-complete is a computational complexity class for decision problems in which many “hard” problems fall. Some examples of NP-complete problems are the traveling salesman problem (TSP), the boolean satisfiability problem (SAT) and the knapsack problem (BKP).

One of the main conditions for a problem being NP-complete is that every other NP problem is reducible to it. Thus all NP-complete problems are basically interchangeable. If you find a fast solution to one of them, you got a fast solution to all of them.

So if somebody found a fast solution to a NP-complete problem, pretty much all of the computationally hard problems of humanity would be solved all in one strike. This would mean the end to civilisation as we know.

To prove that regular expressions with backreferences are indeed NP-complete one can simply take one of the known NP-complete problems and prove that it can be solved using regular expressions. As an example I choose the 3-CNF SAT problem:

3-CNF SAT stands for “3-conjunctive normal form boolean satisfiability problem” and is quite easy to understand. You get a boolean formula of the following form:

   (!$a ||  $b ||  $d)
&& ( $a || !$c ||  $d)
&& ( $a || !$b || !$d)
&& ( $b || !$c || !$d)
&& (!$a ||  $c || !$d)
&& ( $a ||  $b ||  $c)
&& (!$a || !$b || !$c)

Thus the boolean formula is made up of a number of clauses separated by ANDs. Each of those clauses consists of three variables (or their negations) separated by ORs. The 3-CNF SAT problem now asks whether there exists a solution to the given boolean formula (such that it is true).

The above boolean formula can be converted to the following regular expression:

<?php
$regex = '/^
    (x?)(x?)(x?)(x?) .* ;
    (?: x\1 | \2  | \4  ),
    (?: \1  | x\3 | \4  ),
    (?: \1  | x\2 | x\4 ),
    (?: \2  | x\3 | x\4 ),
    (?: x\1 | \3  | x\4 ),
    (?: \1  | \2  | \3  ),
    (?: x\1 | x\2 | x\3 ),
$/x';
$string = 'xxxx;x,x,x,x,x,x,x,';
var_dump(preg_match($regex, $string, $matches));
var_dump($matches);

If you run this code you’ll get the following $matches result:

array(5) {
  [0]=> string(19) "xxxx;x,x,x,x,x,x,x,"
  [1]=> string(1) "x"
  [2]=> string(1) "x"
  [3]=> string(0) ""
  [4]=> string(0) ""
}

This means that the above formula is satisfied if $a = true, $b = true, $c = false and $d = false.

The regular expression works with a very simple trick: For ever 3-clause the string contains a x, which has to be matched. So if you have something like (?: \1 | x\3 | \4 ), in the regex, then the string can be only matched if either \1 is x (true), \3 is the empty string (false) or \4 is x (true).

The rest is left up to the engine. It’ll try out different ways of matching the string until it either finds a solution or has to give up.

Wrapping up

As the article was quite long, here a summary of the main points:

The “regular expressions” used by programmers have very little in common with the original notion of regularity in the context of formal language theory.
Regular expressions (at least PCRE) can match all context-free languages. As such they can also match well-formed HTML and pretty much all other programming languages.
Regular expressions can match at least some context-sensitive languages.
Matching of regular expressions is NP-complete. As such you can solve any other NP problem using regular expressions.

But don’t forget: Just because you can, doesn’t mean that you should. Processing HTML with regular expressions is a really bad idea in some cases. In other cases it’s probably the best thing to do.

Just check what the easiest solution to your particular problem is and use it. If you choose to solve a problem using regular expressions, don’t forget about the x modifier, which allows you to nicely format your regex. For complex regular expressions also don’t forget to make use of DEFINE assertions and named subpatterns to keep your code clean and readable.

That’s it.

Understanding PHP's internal array implementation (PHP's Source Code for PHP Developers - Part 4)

nikic@php.net (NikiC) — Wed, 28 Mar 2012 00:00:00 -0700

Welcome back to the fourth part of the “PHP’s Source Code for PHP Developers” series, in which we’ll cover how PHP arrays are internally represented and used throughout the code base.

In case you missed them, here are the previous parts of this series:

Everything is a hash table!

Basically, everything in PHP is a hash table. Not only are hash tables used in the underlying implementation of PHP arrays, they are also used to store object properties and methods, functions, variables and pretty much everything else.

And because the hash table is so fundamental to PHP, it is worth having a deeper look into how it works.

So, what is a hash table?

Remember that in C arrays are basically chunks of memory, which you can access by index. Thus arrays in C only have integer keys and have to be continuous (i.e. you can’t have a key 0 and the next key is 1332423442). There is no such thing as an associative array.

And this is where hash tables come in: They convert string keys into normal integer keys using a hash function. The result can then be used as an index into a normal C array (aka chunk of memory). The problem here obviously is that the hash function can have collisions, i.e. multiple string keys can yield the same hash. For example in a PHP array with up to 64 elements the strings "foo" and "oof" would have the same hash.

This problem is solved by not storing the value directly at the generated index, but storing a linked list of possible values instead.

HashTable and Bucket

So, after the basic concept of hash tables is clear, let’s have a look at the structures actually used in PHP’s hash table implementation:

The first one is the HashTable:

typedef struct _hashtable {
    uint nTableSize;
    uint nTableMask;
    uint nNumOfElements;
    ulong nNextFreeElement;
    Bucket *pInternalPointer;
    Bucket *pListHead;
    Bucket *pListTail;
    Bucket **arBuckets;
    dtor_func_t pDestructor;
    zend_bool persistent;
    unsigned char nApplyCount;
    zend_bool bApplyProtection;
#if ZEND_DEBUG
    int inconsistent;
#endif
} HashTable;

Let’s quickly go through it:

nNumOfElements specifies how many values are currently stored in the array. This is also the number that count($array) returns.
nTableSize specifies the size of the internal C array. It is always the next power of 2 greater or equal to nNumOfElements. E.g. if an array stores 32 elements, the internal C array also has a size of 32. But if one more element is added, i.e. the array then contains 33 elements, the internal C array is resized to 64 elements.

This is done to always keep the hash table efficient in space and time. It is clear that if the internal array is too small there will be many collisions and the performance will degrade. If the internal array is too big on the other hand, we’d be wasting memory. The power-of-2 size is a good compromise.
nTableMask is the table size minus one. This mask is used to adjust the generated hashes for the current table size. For example the actual hash for "foo" (through the DJBX33A hashing function) is 193491849. If we currently have a table size of 64, we obviously can’t use that as an index into the array. Instead we only take the lower bits of the hash by applying the table mask:
```
   hash  |   193491849 |   0b1011100010000111001110001001
 & mask  | &        63 | & 0b0000000000000000000000111111
---------------------------------------------------------
 = index | =         9 | = 0b0000000000000000000000001001
```
nNextFreeElement is the next free integer key, which is used when you append to an array using $array[] = xyz.
pInternalPointer stores the current position in the array. This is used for foreach iteration and can be accessed using the reset(), current(), key(), next(), prev() and end() functions.
pListHead and pListTail specify the first and last element of the array. Remember: PHP arrays have an order. E.g. ['foo' => 'bar', 'bar' => 'foo'] and ['bar' => 'foo', 'foo' => 'bar'] contain the same elements, but have a different order.
arBuckets is the “internal C array” we were always talking about. It’s defined as a Bucket **, so it can be seen as an array of bucket pointers (we’ll get to what exactly a Bucket is in a minute).
pDestructor is the destructor for the values. If a value is removed from the HT this function will be called on it. For a normal array the destructor function is zval_ptr_dtor. zval_ptr_dtor will reduce the reference count of the zval and, if it reaches 0, destroy and free it.

The last four properties aren’t really of interest to us. So let’s just say that persistent specifies that the hash table can live between multiple requests, nApplyCount and bApplyProtection are used to prevent infinite recursion in some places and inconsistent is used to catch incorrect uses of hash tables in debug mode.

Let’s move on to the second important structure: Bucket:

typedef struct bucket {
    ulong h;
    uint nKeyLength;
    void *pData;
    void *pDataPtr;
    struct bucket *pListNext;
    struct bucket *pListLast;
    struct bucket *pNext;
    struct bucket *pLast;
    const char *arKey;
} Bucket;

h is the hash (without the table mask applied).
arKey is used to save string keys. nKeyLength is the corresponding length. For integer keys those two aren’t used.
pData or pDataPtr is used to store the actual value. For PHP arrays that value is a zval (but it’s also used for other things internally.) Don’t bother with the fact that there are two properties for this. The difference between them is who is responsible for freeing the value.
pListNext and pListLast specify the order of the array elements. If PHP wants to traverse the array it starts off at the pListHead bucket (specified in the HashTable struct) and then always takes the pListNext bucket. The same works in reverse, by starting at pListTail and always following pListLast. (You can do this in userland by calling end() and then always calling prev().)
pNext and pLast form the “linked list of possible values” I mentioned above. The arBuckets array stores a pointer to the first possible bucket. If that bucket hasn’t the right key, PHP will look at the bucket which pNext points to. This is done until the right bucket is found. pLast can be used to do the same in reverse.

As you can see PHP’s hash table implementation is fairly complex. This is the price one has to pay for its ultra-flexible array type.

How are hash tables used?

The Zend Engine defines a large number of API functions for the work with hash tables. An overview of low-level hash table functions can be found in zend_hash.h. Additionally the ZE defines a set of slightly higher level APIs in zend_API.h.

We don’t have time to go through all of these, but we can look at a sample function to see at least some of them in action. We’ll use array_fill_keys as that sample function.

Using the technique outlined in the second part you should be able to find the function definition in ext/standard/array.c easily. Let’s quickly walk through it now.

As always there’s a set of variable declarations and a zend_parse_parameters call at the top:

zval *keys, *val, **entry;
HashPosition pos;

if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "az", &keys, &val) == FAILURE) {
    return;
}

The az obviously means that the first parameter is an array (fetched into the keys variable) and the second one is an arbitrary zval (fetched into the val variable).

After the parameters are parsed the array to be returned is initialized:

/* Initialize return array */
array_init_size(return_value, zend_hash_num_elements(Z_ARRVAL_P(keys)));

This line contains already three important parts of the array API:

The Z_ARRVAL_P macro fetches the hash table from a zval.
zend_hash_num_elements fetches the number of elements in a hash table (the nNumOfElements property).
array_init_size initializes an array with a size hint.

So this line initializes an array into return_value with the same size as the keys array.

The size hint here is just an optimization. The function could have also called just array_init(return_value), in which case PHP would have to do multiple resizes as more and more elements are added to the array. By specifying an explicit size, PHP allocates the right amount of memory from the start.

After the return array initialization the function loops through the keys array using a while loop with roughly this structure:

zend_hash_internal_pointer_reset_ex(Z_ARRVAL_P(keys), &pos);
while (zend_hash_get_current_data_ex(Z_ARRVAL_P(keys), (void **)&entry, &pos) == SUCCESS) {
    // some code

    zend_hash_move_forward_ex(Z_ARRVAL_P(keys), &pos);
}

This can be translated to PHP code easily:

reset($keys);
while (null !== $entry = current($keys)) {
    // some code

    next($keys);
}

Which is the same as:

foreach ($keys as $entry) {
    // some code
}

The only real difference is that the C iteration doesn’t use the internal array pointer, but uses it’s own pos variable to store the current position.

The code within the loop has two branches: One for integer keys and one for other keys. The integer key branch contains only two lines:

zval_add_ref(&val);
zend_hash_index_update(Z_ARRVAL_P(return_value), Z_LVAL_PP(entry), &val, sizeof(zval *), NULL);

This is pretty straightforward: First the refcount of the value is increased (adding the value to the hash table means adding another reference to it) and then the value is actually inserted into the hash table. The arguments of the zend_hash_index_update macro are the hash table to update Z_ARRVAL_P(return_value), the integer index Z_LVAL_PP(entry), the value &val, the size of the value sizeof(zval *) and the destination pointer (which we don’t care about, thus NULL).

The non-integer-key branch is slightly more complicated:

zval key, *key_ptr = *entry;

if (Z_TYPE_PP(entry) != IS_STRING) {
    key = **entry;
    zval_copy_ctor(&key);
    convert_to_string(&key);
    key_ptr = &key;
}

zval_add_ref(&val);
zend_symtable_update(Z_ARRVAL_P(return_value), Z_STRVAL_P(key_ptr), Z_STRLEN_P(key_ptr) + 1, &val, sizeof(zval *), NULL);

if (key_ptr != *entry) {
    zval_dtor(&key);
}

First the key is converted to a string (unless it already is one) using convert_to_string. But before this can be done the entry has to be copied into a new key variable. The key = **entry line takes care of that. Additionally zval_copy_ctor has to be called, otherwise complex structures (like strings or arrays) won’t be copied correctly.

The copy is necessary to ensure that the cast doesn’t change the original array. Without the copy the cast would modify not only our local variable, but also the element in the keys array (which obviously would be quite unexpected to the user).

Obviously the copy has to be removed again after the loop, which is what the zval_dtor(&key) line does. The difference between zval_ptr_dtor and zval_dtor is that zval_ptr_dtor only destroys the zval if the refcount reaches 0, whereas zval_dtor destroys it always, regardless of the refcount. That’s why you’ll find zval_ptr_dtor used with “normal” values and zval_dtor with temporary variables, which aren’t used anywhere else anyways. Also zval_ptr_dtor frees the zval after destroying it, while zval_dtor does not. As we never malloc()d anything, we also don’t have to free(), so zval_dtor is the right choice in this respect too.

Now, let’s look at the last two lines left (the important ones ^^):

zval_add_ref(&val);
zend_symtable_update(Z_ARRVAL_P(return_value), Z_STRVAL_P(key_ptr), Z_STRLEN_P(key_ptr) + 1, &val, sizeof(zval *), NULL);

Those are very similar to what was done in the integer-key branch. The difference is that now zend_symtable_update is called instead of zend_hash_index_update and the string key and it’s length are passed in.

The Symtable

The “normal” function for inserting string keys into a hash table is zend_hash_update, but here zend_symtable_update is used instead. What’s the difference?

A symtable basically is a special type of hash table, which is used for arrays. The difference to the ordinary hash table is how it handles numeric string keys: In a symtable the keys "123" and 123 are considered identical. So if you store a value in $array["123"], you can then retrieve it using $array[123].

The underlying implementation could use two ways: Either save both 123 and "123" using the "123" key or save both using the 123 key. PHP obviously chooses the latter option (as integers are smaller and faster than strings).

You can have a little bit of fun with symtables, if you manage to somehow insert the "123" key without it being cast to 123. One way is to exploit the array to object cast:

<?php
$obj = new stdClass;
$obj->{123} = "foo";
$arr = (array) $obj;
var_dump($arr[123]);   // Undefined offset: 123
var_dump($arr["123"]); // Undefined offset: 123

Object properties are always saved under string keys, even if they are numbers. So the $obj->{123} = 'foo' line actually saves "foo" under the "123" index, not the 123 index. When doing the array cast this is not changed. But as both $arr[123] and $arr["123"] try to access the 123 index (not the "123" index that actually exists), both throw an error. So, congratulations, you’ve created a hidden array element!

In the next part

The next part will again be published on ircmaxell’s blog. It will analyze how objects and classes work internally.

PHP's Source Code for PHP Developers - Part 3 - Variables

nikic@php.net (NikiC) — Wed, 21 Mar 2012 00:00:00 -0700

The third part of the PHP’s Source Code for PHP Developers series can be found on ircmaxell’s blog. It introduces how PHP values are internally represented and how they are used in the source code.

The fourth part, which will again be published on this blog, will cover how arrays are implemented in PHP.

Understanding PHP's internal function definitions (PHP's Source Code for PHP Developers - Part 2)

nikic@php.net (NikiC) — Fri, 16 Mar 2012 00:00:00 -0700

Welcome to the second part of the “PHP’s Source Code For PHP Developers” series.

In the previous part ircmaxell explained where you can find the PHP source code and how it is basically structured and also gave a small introduction to C (as that’s the language PHP is written in). If you missed that post, you probably should read it before starting with this one.

What we’ll cover in this article is locating the definitions of internal functions in the PHP codebase, as well as understanding them.

How to find function definitions

For a start, let’s try to find out how the strpos function is defined.

The first thing to try, is to go to the PHP 5.4 source code root and type strpos into the search box at the top of the page. The result will be a huge listing of strpos occurrences in the PHP source code.

As this doesn’t really help us much, we use a little trick: Instead of searching for just strpos, we search for "PHP_FUNCTION strpos" instead (don’t forget the quotes, they are important).

Now we are left with only two entries:

/PHP_5_4/ext/standard/
    php_string.h 48   PHP_FUNCTION(strpos);
    string.c     1789 PHP_FUNCTION(strpos)

First thing to notice is that both occurrences are in the ext/standard folder. This is exactly where one would expect to find them, as the strpos function (together with pretty much all other string, array and file functions) is part of the standard extension.

Now open both links in new tabs and see what code hides behind them.

You’ll find that the first link leads you to the php_string.h file, which is full of code looking like this:

// ...
PHP_FUNCTION(strpos);
PHP_FUNCTION(stripos);
PHP_FUNCTION(strrpos);
PHP_FUNCTION(strripos);
PHP_FUNCTION(strrchr);
PHP_FUNCTION(substr);
// ...

This is exactly how a typical header file (a file ending in .h) looks like: A plain list of functions which are defined elsewhere. We aren’t really interested in this, as we already know what we’re looking for.

The second link is much more interesting: It leads to the string.c file, which contains the actual source code of the function.

Before I’ll walk you through the code step by step, I’d recommend you to try and understand the function by yourself. It’s a really simple function and most things should be clear even if you don’t know the exact details.

The skeleton of a PHP function

All PHP functions share the same basic structure. At the top there are a few variable declarations, then there is a zend_parse_parameters call, then comes the main logic, with RETURN_*** and php_error_docref calls intermixed.

So, let’s start with the variable declarations:

zval *needle;
char *haystack;
char *found = NULL;
char  needle_char[2];
long  offset = 0;
int   haystack_len;

The first line declares needle as being a pointer to a zval. A zval is PHP’s internal representation of an arbitrary PHP value. How exactly it looks will be the subject of the next post.

The second line declares haystack as a pointer to a character. At this point you’ll have to remember that in C, arrays are represented by pointers to their first value. I.e. the haystack will point to the first character of the $haystack string you passed in. Then haystack + 1 will point to the second character, haystack + 2 to the third, and so on. So one could read in the whole string by always incrementing the pointer by one.

The problem arising here is that PHP has to know when the string ends. Otherwise it would always keep incrementing the pointer without ever stopping. In order to deal with this, PHP also stores an explicit length, here in the haystack_len variable.

The last declaration of interest to us at this point is the offset variable, which will be used to store the third parameter of the function: the offset to start searching at. It is declared as a long, which is an integer datatype, just like int. The difference between those two is not of importance here, but you should know that PHP integers are stored in longs and string lengths are stored in ints.

Now let’s look at the next three lines:

if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "sz|l", &haystack, &haystack_len, &needle, &offset) == FAILURE) {
    return;
}

What these lines basically do, is take the parameters that were passed to the function and put them into the variables which were declared above.

The first argument to the function is the number of arguments passed. This number is provided by the ZEND_NUM_ARGS() macro.

The next argument is the TSRMLS_CC macro, which is kind of an idiosyncrasy of PHP. You’ll find this strange macro scattered across pretty much the whole PHP code base. It is part of the Thread Safe Resource Mananger (TSRM), which ensures that PHP doesn’t mix up variables between multiple threads. This is unimportant to us, so whenever you see TSRMLS_CC (or TSRMLS_DC) in the code, just ignore it. (A strangeness which you might have noticed, is that there is no comma before this “argument”. This has to do with the fact that depending on whether or not you are using a thread-safe build, the macro will either evaluate to nothing or to , tsrm_ls. So basically the comma is part of the macro.)

Now comes the important stuff: The "sz|l" string specifies which parameters the function accepts:

s  // first parameter is a *s*tring
z  // second parameter is a *z*val (an arbitrary value)
|  // the following parameters (here just one) are optional
l  // third parameter is a *l*ong (an integer)

There are more type specifiers than s, z and l, but most should be clear from the character. For example b is a boolean, d is a double (floating point number), a is an array, f is a callback (function) and o is an object.

The remaining arguments &haystack, &haystack_len, &needle, &offset specify the variables to put the values of the arguments into. As you can see, they are all passed by reference (&), which means that not the variables themselves are passed, but pointers to them.

After this call haystack will contain the haystack string, haystack_len the length of that string, needle the needle value and offset the starting offset.

Additionally the function is checked for FAILURE (which happens if you try to pass invalid arguments to the function, e.g an array to a string parameter). In this case zend_parse_parameters will throw a warning and the code of the function just returns (which will eventually return null to the userland PHP code).

So after the parameters are parsed, the main function body starts:

if (offset < 0 || offset > haystack_len) {
    php_error_docref(NULL TSRMLS_CC, E_WARNING, "Offset not contained in string");
    RETURN_FALSE;
}

What this code does is pretty obvious. If the offset is out of bounds an E_WARNING level error is thrown through php_error_docref and then false is returned using the RETURN_FALSE macro.

php_error_docref is the error function you’ll mainly find in extensions (i.e. the ext folder). The name comes from the fact that it emits a reference to the documentation in the error message (you know, the one that never works…). Additionally there is the zend_error function, which is mainly used by the Zend Engine, but also occurs in extension code from time to time.

Both functions use sprintf-like formatting, thus error messages can contain placeholders, which are then filled using the following arguments. Here is an example:

php_error_docref(NULL TSRMLS_CC, E_WARNING, "Failed to write %d bytes to %s", Z_STRLEN_PP(tmp), filename);
// %d is filled with Z_STRLEN_PP(tmp)
// %s is filled with filename

Let’s proceed in the code:

if (Z_TYPE_P(needle) == IS_STRING) {
    if (!Z_STRLEN_P(needle)) {
        php_error_docref(NULL TSRMLS_CC, E_WARNING, "Empty delimiter");
        RETURN_FALSE;
    }

    found = php_memnstr(haystack + offset,
                        Z_STRVAL_P(needle),
                        Z_STRLEN_P(needle),
                        haystack + haystack_len);
}

The first five lines should be clear: This branch is only executed if the needle is a string and an error is thrown if it is empty. Then comes the interesting part: php_memnstr is called, which is the function doing the main work. As always you can click on the function name to see its source code.

php_memnstr returns the pointer to the first occurrence of the needle in the haystack (that’s why the found variable is declared as char *, i.e. a pointer to character). From this the offset can be easily computed by subtracting the two pointers, as can be seen at the end of the function:

 RETURN_LONG(found - haystack);

Finally, let’s look at the branch which is taken when the needle is not a string:

else {
    if (php_needle_char(needle, needle_char TSRMLS_CC) != SUCCESS) {
        RETURN_FALSE;
    }
    needle_char[1] = 0;

    found = php_memnstr(haystack + offset,
                        needle_char,
                        1,
                        haystack + haystack_len);
}

I’ll just quote what this does from the manual: “If needle is not a string, it is converted to an integer and applied as the ordinal value of a character.” This basically means that instead of writing strpos($str, 'A') you could also write strpos($str, 65), because the ordinal value of A is 65.

If you look up at the variable declarations, you’ll see that needle_char is declared as char needle_char[2], i.e. a string with two characters. php_needle_char will put the actual character (in our example the A) into needle_char[0]. Then the strpos code will set needle_char[1] to 0. The reason behind this is that in C, strings are zero-terminated, i.e. the last character is set to NUL (the character with the ordinal value 0). In the context of PHP this doesn’t make much sense, as PHP stores an explicit length for all strings (so it does not need zero-termination to find the end of a string), but this still is done in order to ensure compatibility with the C functions used internally by PHP.

Zend functions

I’m getting tired of strpos, so lets try to find another function: strlen. We’ll do this using our usual approach:

Starting from the PHP 5.4 source code root try to search for strlen.

You’ll see lots of unrelated uses of the function, so instead search for "PHP_FUNCTION strlen". While doing so, you’ll notice something strange though: There won’t be any results.

The reason is that strlen is one of the few functions, which is not defined by an extension, but by the Zend Engine itself. In such cases the function is not defined as PHP_FUNCTION(strlen), but as ZEND_FUNCTION(strlen). Thus we also have to search for "ZEND_FUNCTION strlen" instead.

As we already know, we have to click on the entry without a semicolon ; at the end to get to the source code. This leads us to the following definition in Zend/zend_builtin_functions.c:

ZEND_FUNCTION(strlen)
{
    char *s1;
    int s1_len;

    if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "s", &s1, &s1_len) == FAILURE) {
        return;
    }

    RETVAL_LONG(s1_len);
}

I don’t think that I have to further comment on this, as the function is so simple.

Methods

We’ll cover how classes and objects work in more detail in a different post, but as a small peek ahead: You can search for class methods by typing ClassName::methodName into the search. As an example, try to search for SplFixedArray::getSize.

In the next part

The next part will again be published on ircmaxell’s blog. It will cover what zvals are, how they work and how they are used in the source code (all those Z_*** macros…)

Scalar type hinting is harder than you think

nikic@php.net (NikiC) — Tue, 06 Mar 2012 00:00:00 -0800

One of the features originally planned for PHP 5.4 was scalar type hinting. But as you know, they weren’t included in the release.

Recently the topic has come up again on the mailing list and there has been a hell lot of discussion about it. Yesterday ircmaxell published a blog post about his particular proposals.

The reactions on reddit were mixed. On one hand it is clear that people do really want scalar type hints, on the other hand they didn’t seem to like that particular proposal.

One comment particularly caught my interest:

Why can’t the PHP dev’s just get this thing right? Why must they always choose the worst, more useless, most convoluted way to implement every new feature! It’s disheartening. Especially when there aren’t good backwards-compatibility reasons for doing it wrong.

In a way, this is a pretty important comment: It shows what people think about how PHP is being developed. They think that it’s a bunch of people that somehow, magically, manage to always do the wrong decisions.

That’s not how it works. Really. You see, most people, when confronted with scalar type hinting, think “Hey, this is so simple, just do XYZ”, and then wonder why PHP is having such a hard time implementing this most trivial of all features.

The thing is: It’s just not that simple. Most proposals seem so easy and straightforward on first sight, but when it comes to the details they are not.

In the following I want to introduce you to some of the various proposals that were made in the past and the problems associated with them. I hope this way people will get a little bit more insight into why this is so hard.

Strict type hinting

I’ll start with the proposal that I personally dislike most: Strict type hinting, i.e. allowing only the hinted type to be passed and not any of the types that PHP would normally consider equivalent. See this example:

<?php
function foo(int $i) { /* ... */ }

foo(1);   // works
foo(1.0); // fatal error: int expected, float given
foo("1"); // fatal error: int expected, string given

I think it is evident that this is not an option. One of PHP’s greatest strengths is being a weakly typed language and this proposal would turn it into a strictly typed one. This goes against the PHP philosophy and also differs from how pretty much everything else works in PHP.

Unenforced type hinting

Another proposal that was made, is type hinting that isn’t enforced by the engine. WTF? What would this be good for? Basically, it would work just like doc comments (which aren’t enforced either ^^), but with nicer syntax.

<?php
function foo(int $i) { /* ... */ }

foo(1);          // works
foo(1.0);        // works
foo("1");        // works
foo(array(123));  // works
foo($iAmObject); // works
// everything works...

I dislike this proposal, too, for obvious reasons. For me it just doesn’t make any sense to have type hints that are ignored. Doc comments do that already well enough…

Casting weak type hinting

A proposal that came up recently (this is the one introduced by ircmaxell) is type hinting based on casts.

<?php
function foo((int) $i) {
    var_dump($i);
}
foo(1);       // int(1)
foo(1.0);     // int(1)
foo("1");     // int(1)

foo(1.5);     // int(1)
foo(array()); // int(0)
foo("hi");    // int(0)

This is one of the more interesting proposals, for several reasons:

Firstly, it is the only proposal that can be implemented without breaking backwards compatibility at all. All other proposals would at least break existing type hints on classes called “Int” or “String” (believe it or not, but some people really do use classes like this…)

Secondly, the syntax makes very clear that it will cast the value. It also reuses PHP’s normal cast semantics, thus making it consistent.

But it also has some not insignificant issues:

The first one is that the syntax is simply ugly. Sure, you don’t have to tell me, that’s subjective and stuff like that, but honestly, this is an important aspect to consider. Scalar type hinting will be used a lot and we shouldn’t make it ugly.

The second, by far more important issue is that PHP’s cast semantics simply suck. In the above example, the first three casts seem logical, but the three last ones are clearly broken. At least I don’t want "foobar" or array() to be valid arguments to an integer parameter. It just doesn’t make sense. That’s why this proposal would go hand in hand with a change of implicit cast semantics: Whenever such a “strange” cast (one with data loss) occurs, it should throw a notice. This again would be a controversial change, as it would also change stuff in other places (which is probably a good thing though ^^). Also I would argue that just a notice is not enough and it should instead throw a recoverable fatal error.

Also I’m not really sure as to whether we really want incoming parameters to be cast, instead of just being validated. One argument against it is, that this approach is somewhat different from the existing array, callable and Class type hints. Especially if you consider that you would now have both an (array) and an array type hint, the issue might become more clear: When would you use one over another? When would you want the argument to be cast and when just checked? Additionally the casts would prevent type hinted parameters to be used by reference in any reasonable manner.

Strict weak type hinting

This leads us to another possibility, which is also my favorite: Doing weak type hints, but with stricter input validation (and without casts).

<?php
function foo(int $i) {
    var_dump($i);
}
foo(1);       // int(1)
foo(1.0);     // float(1.0)
foo("1");     // string(1) "1"

foo(1.5);     // fatal error: int expected, float given
foo(array()); // fatal error: int expected, array given
foo("hi");    // fatal error: int expected, string given

I like this proposal most, for several reasons: Firstly, it uses the “normal” type hinting syntax, not the strange cast syntax. Secondly it has stricter validation semantics as it only lets those values through which are representable in the hinted type without data loss. And thirdly it does not do casts, which is in line with previous type hints and also allows references.

But it obviously also comes with problems: Again, this won’t be backwards compatible as it at least will break existing Int or String class type hints. Additionally, depending on the exact implementation, it would also make stuff like int and string reserved keywords, thus (maybe) breaking even more stuff. Realistically though I’d say that this will only affect a small fraction of scripts.

By the way, this proposal would be very similar to how parameters for internal functions are parsed. E.g. an int argument of an internal function will not accept "foobar" and instead throw a warning.

Boxing based type hinting

Another very interesting idea that also came from ircmaxell is type hinting using boxing. Basically this would add magic methods for casting objects to scalars and the other way around. Something like this (just example names):

public function __toScalar()
public static function __fromScalar($value)

Now see how this could be useful:

<?php

class Int {
    protected $value;

    public function __construct($value) {
        if (false === $this->value = filter_var($value, FILTER_VALIDATE_INT)) {
            throw new Exception('Malformed integer'));
        }
    }

    public function __toScalar() {
        return $this->value;
    }

    public function __fromScalar($value) {
        return new static($value);
    }
}

foo(10); // foo() is expecting an Int, so call Int::__fromScalar(10),
         // which returns an Int(10). This is then passed to foo().

foo("hi"); // Int::__fromScalar("hi") is called, which tries to create
           // an Int("hi"), but this throws an Exception.

function foo(Int $i) {
    return str_repeat('*', $i); // $i is Int(10) here, but str_repeat expects an integer,
                                // thus call $i->__toScalar() [or ->__toInt() or whatever],
                                // which returns 10. This is then passed to str_repeat().
}

This basically allows you to implement type hinting in userland code. It also allows to create strict type hinting:

<?php

class StrictInt {
    protected $value;

    public function __construct($value) {
        if (!is_int($value)) {
            throw new Exception('Not an integer'));
        }

        $this->value = $value;
    }

    public function __toScalar() {
        return $this->value;
    }

    public function __fromScalar($value) {
        return new static($value);
    }
}

function foo(StrictInt $i) { /* ... */ }

foo(10); // foo() is expecting a StrictInt, so call StrictInt::__fromScalar(10),
         // which returns a StrictInt(10). This is then passed to foo().

foo("10"); // StrictInt::__fromScalar("10") is called, which tries to create
           // StrictInt("10"), but this raises an exception.

As you can see, this proposal is very powerful as it allows users to define their own type hinting rules (PHP could obviously still provide some default Int, etc classes.)

My main issue with this proposal is that it requires the value to be boxed and unboxed on every type hint. This seems like overkill and like a performance issue. But obviously the internally defined type hinting classes could optimize this.

Generally this proposal is not yet that mature, so it could well have some more flaws. Still, I think that this is one of the more interesting approaches, especially as it is so powerful.

We need you!

I hope that I could give you a small overview of the different possible approaches.

Now it’s your turn! PHP needs your valuable feedback on the different proposals, so a good decision can be made.

You can give feedback directly on the internals mailing list, but if you don’t want to do that, you can also leave a comment or ping ircmaxell or me in the StackOverflow PHP chatroom.

Pointer magic for efficient dynamic value representations

nikic@php.net (NikiC) — Thu, 02 Feb 2012 00:00:00 -0800

What always intrigued me were the various little tricks that JavaScript implementations use to be as fast as they are. One of those tricks is how JS implementations represent values to be more time and space efficient.

I think it’s worth the time to look at those neat little tricks for two reasons: Firstly, some of them are also applicable in other situations (though most of them admittedly are not). And secondly, because they remind you of how various data is represented at the low level (stuff you may have learned in your CS courses at some point - or probably not).

So, let’s start!

The trivial approach: Tagged unions

JavaScript is a dynamically typed language and as such needs some kind of “value” data structure, which stores the current type and the value in that type. The trivial approach to this problem is to use a tagged union. This could look roughly like this:

#include <stdint.h>

class Value {
public:
    enum TypeTag {
        IntType, DoubleType, StringType, ObjectType
    }

    // constructors, methods, ...
private:
    union {
        int32_t  asInt32;
        double   asDouble;
        String  *asString;
        Object  *asObject;
    } payload;
    TypeTag type;
};

As you can see the idea is very simple. You have a type tag, which specifies which type the value currently has. And a union that’s used to store the various types. Examples to store integers and objects would look like that:

inline Value::Value(int32_t number) {
    type = IntType;
    payload = number;
}
inline Value::Value(Object *object) {
    type = ObjectType;
    payload = object;
}

This straightforward approach has several problems: The size of the above object would be 16 bytes = 128 bits (on a 64 bit machine). This is a size that you wouldn’t just pass around by value. So you need to allocate it on the heap and use a pointer to it. Allocating stuff on the heap is slow and incurs memory overhead. When using the value later it has to be fetched from the heap, which is again slow.

So what we want is to make values smaller and move them off the heap. Before we get to that, let’s first have a look at a technique called pointer tagging:

An interlude: Pointer tagging

Have a look at the above structure again. It consists of a union, which has a size of 8 bytes, and an integer which fits into a single byte (a char). So theoretically the total size should be 9 bytes. But it’s not - it’s 16 bytes. This is because the compiler adds padding to the data structure in order to align it in memory.

This is done for performance reasons: The CPU can access the memory only in word sized chunks. So if our data always starts at a word it can be fetched efficiently. If it were to start somewhere in the middle of a word the CPU would have to apply masks and shifts in order to get the data - which would be too slow.

A word on a 64 bit machine is 64 bit = 8 bytes (obviously ^^). So if all pointers to our object will be aligned to 8 bytes it means that they will be multiples of 8. A pointer thus can be 8, 16, 24, 109144, etc. But it can not be 7 or 13. Or speaking in binary: 0b1000 (=8), 0b10000 (=16), 0b11000 (=24), 0b11010101001011000 (=109144).

As you can see the lower three bits are always zero. So those three bits are basically free to use. We can use them to store a “tag” in where, which is an integer between 0 (0b000) and 7 (0b111).

pppppppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppTTT
^- actual pointer                                   three tag bits -^

Here is a sample implementation of how to do so:

#include <cassert>
#include <stdint.h>

template <typename T, int alignedTo>
class TaggedPointer {
private:
    static_assert(
        alignedTo != 0 && ((alignedTo & (alignedTo - 1)) == 0),
        "Alignment parameter must be power of two"
    );

    // for 8 byte alignment tagMask = alignedTo - 1 = 8 - 1 = 7 = 0b111
    // i.e. the lowest three bits are set, which is where the tag is stored
    static const intptr_t tagMask = alignedTo - 1;

    // pointerMask is the exact contrary: 0b...11111000
    // i.e. all bits apart from the three lowest are set, which is where the pointer is stored
    static const intptr_t pointerMask = ~tagMask;

    // save us some reinterpret_casts with a union
    union {
        T *asPointer;
        intptr_t asBits;
    };

public:
    inline TaggedPointer(T *pointer = 0, int tag = 0) {
        set(pointer, tag);
    }

    inline void set(T *pointer, int tag = 0) {
        // make sure that the pointer really is aligned
        assert((reinterpret_cast<intptr_t>(pointer) & tagMask) == 0);
        // make sure that the tag isn't too large
        assert((tag & pointerMask) == 0);

        asPointer = pointer;
        asBits |= tag;
    }

    inline T *getPointer() {
        return reinterpret_cast<T *>(asBits & pointerMask);
    }
    inline int getTag() {
        return asBits & tagMask;
    }
};

The code is fairly straightforward I think. You could then use it like this:

double number = 17.0;
TaggedPointer<double, 8> taggedPointer(&number, 5);
taggedPointer.getPointer(); // == &number
taggedPointer.getTag(); // == 5

The technique described above is useful in many contexts. Basically it’s applicable to any situation where you want to add a small bit of info to an aligned pointer. And this is also the situation we are in :)

But before we get to that let’s first have a look at another, very similar trick:

Storing integers in the pointer

JavaScript does not have an integer type per se, it only has IEEE 754 doubles. Still people mainly work with small integral numbers (think of loops, array indexes, etc). Thus it makes sense to store those small integers in a real integer variable instead of a double, because many operations are much faster this way. That’s also the reason why in the Value class above I have a distinct int32 type and not just a double type.

Especially for integers the above Value approach seems like overkill. One has to allocate 16 bytes of memory (+ overhead) and has to maintain a pointer, just to store 4 bytes of data. That’s inefficient both in terms of memory and performance.

The solution obviously is to use something similar to the tagged pointers introduced above. You can say: “If the last bit of the pointer is 1 then it actually isn’t a pointer at all, but it’s an integer.”

Here is how a pointer would look:

pppppppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppTT0
^- actual pointer                                     two tag bits -^ ^- last bit 0

And this is how the integer would look like (note that as we need one bit for distinguishing the type the integer has only 63 bits):

iiiiiiii|iiiiiiii|iiiiiiii|iiiiiiii|iiiiiiii|iiiiiiii|iiiiiiii|iiiiiii1
^- actual integer                                                     ^- last bit 1

To get the actual integer value we just need to shift off the 1 bit using integer >> 1.

Here again a sample implementation:

#include <cassert>
#include <stdint.h>

template <typename T, int alignedTo>
class TaggedPointerOrInt {
private:
    static_assert(
        alignedTo != 0 && ((alignedTo & (alignedTo - 1)) == 0),
        "Alignment parameter must be power of two"
    );
    static_assert(
        alignedTo > 1,
        "Pointer must be at least 2-byte aligned in order to store an int"
    );

    // for 8 byte alignment tagMask = alignedTo - 1 = 8 - 1 = 7 = 0b111
    // i.e. the lowest three bits are set, which is where the tag is stored
    static const intptr_t tagMask = alignedTo - 1;

    // pointerMask is the exact contrary: 0b...11111000
    // i.e. all bits apart from the three lowest are set, which is where the pointer is stored
    static const intptr_t pointerMask = ~tagMask;

    // save us some reinterpret_casts with a union
    union {
        T *asPointer;
        intptr_t asBits;
    };

public:
    inline TaggedPointerOrInt(T *pointer = 0, int tag = 0) {
        setPointer(pointer, tag);
    }
    inline TaggedPointerOrInt(intptr_t number) {
        setInt(number);
    }

    inline void setPointer(T *pointer, int tag = 0) {
        // make sure that the pointer really is aligned
        assert((reinterpret_cast<intptr_t>(pointer) & tagMask) == 0);
        // make sure that the tag isn't too large
        assert(((tag << 1) & pointerMask) == 0);

        // last bit isn't part of tag anymore, but just zero, thus the << 1
        asPointer = pointer;
        asBits |= tag << 1;
    }
    inline void setInt(intptr_t number) {
        // make sure that when we << 1 there will be no data loss
        // i.e. make sure that it's a 31 bit / 63 bit integer
        assert(((number << 1) >> 1) == number);

        // shift the number to the left and set lowest bit to 1
        asBits = (number << 1) | 1;
    }

    inline T *getPointer() {
        assert(isPointer());

        return reinterpret_cast<T *>(asBits & pointerMask);
    }
    inline int getTag() {
        assert(isPointer());

        return (asBits & tagMask) >> 1;
    }
    inline intptr_t getInt() {
        assert(isInt());

        return asBits >> 1;
    }

    inline bool isPointer() {
        return (asBits & 1) == 0;
    }
    inline bool isInt() {
        return (asBits & 1) == 1;
    }
};

Usage example:

// either a pointer to a double (with a two bit tag) or a 63 bit integer
double number = 17.0;
TaggedPointerOrInt<double, 8> taggedPointerOrInt(&number, 3);
taggedPointerOrInt.isPointer(); // == true;
taggedPointerOrInt.getPointer(); // == &number
taggedPointerOrInt.getTag(); // == 3

taggedPointerOrInt.setInt(123456789);
taggedPointerOrInt.isInt(); // == true
taggedPointerOrInt.getInt(); // == 123456789

Now using the above technique we have a powerful way to optimize the original Value class. Integers will be stored directly in the pointer; doubles, strings and objects will be stored as a pointer with a tag identifying their type (e.g 0 = 0b00 = object, 1 = 0b01 = string, 2 = 0b10 = double). Booleans could be stored similarly to ints (and fetched using a simple bool >> 3):

00000000|00000000|00000000|00000000|00000000|00000000|00000000|0000b110
                                                actual bool value -^  ^- last bit 0 (as it isn't an int)
                       tag = 3 = 0b11 to identify that it's a bool -^^

So what have we gained overall? Quite a bit! Integers (and bools) aren’t allocated on the heap anymore. That saves memory and probably more importantly improves performance (because the code doesn’t have to access the heap). Also the pointers to strings and objects can now be dereferenced directly, without first fetching a value (which stores a pointer to the real string/object).

But we aren’t done yet: The double still requires a pointer; wouldn’t it be nice to get rid of that too?

Pointers in the NaN-Space

Recap: How do doubles look like?

Doubles in JS are following the IEEE 754 Standard for Floating-Point Arithmetic. In C++ doubles aren’t specified to follow any standard, but we assume that they use IEEE 754, too (which, practically speaking is a pretty good assumption. In general this whole article is nothing more than a large set of evil assumptions that any self-respecting C++ developer would murder you for.)

As you probably know IEEE 754 doubles are internally represented using the following bit sequence:

seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
 ^- exponent ^- mantissa
^- sign bit

The resulting float is basically (-1)^s * m * 2^e (speaking oversimplifyingly).

What is much more important to us are some special values that doubles can have:

Positive zero (0.0):
seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
00000000|00000000|00000000|00000000|00000000|00000000|00000000|00000000
^- sign bit 0 (= +), all other bits also 0

Negative zero (-0.0):
seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
10000000|00000000|00000000|00000000|00000000|00000000|00000000|00000000
^- sign bit 1 (= -), all other 0

IEEE defines two zeros: +0 and -0. Both have exponent and mantissa zeroed out and are distinguished by the value of the sign bit. This may seem pointless at first - zero is zero after all - but it allows for some subtle distinctions. For example 1.0 / 0.0 is +INF whereas 1.0 / -0.0 is -INF. There are also other operations where the sign of the zero makes a difference.

One interesting behavior of the zeros is that they are considered equal in comparison. I.e. 0.0 == -0.0. The only way to find out whether a number is negative zero is to check whether the sign bit is set:

inline bool isNegativeZero(double number) {
    return number == 0 && *reinterpret_cast<int64_t *>(&number) != 0;
}

Positive infinity (+INF):
seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
01111111|11110000|00000000|00000000|00000000|00000000|00000000|00000000
 ^- exponent bits all 1                          mantissa bits all 0 -^
^- sign bit 0 (= +)

Negative infinity (-INF):
seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
11111111|11110000|00000000|00000000|00000000|00000000|00000000|00000000
 ^- exponent bits all 1                          mantissa bits all 0 -^
^- sign bit 1 (= -)

Infinity has all exponent bits set and a zero mantissa. Again there is +INF and -INF, distinguished by the sign bit.

Signaling NaN:
seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
s1111111|11110ppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp
             ^- first mantissa bit 0    everything else is "payload" -^
 ^- exponent bits all 1
^- any sign bit

Quiet NaN:
seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
s1111111|11111ppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp
             ^- first mantissa bit 1    everything else is "payload" -^
 ^- exponent bits all 1                 and mustn't be all-zero (as it
^- any sign bit                         would be INF then)

NaNs represent values that are Not a Number. E.g. 0.0/0.0 = NaN. NaNs also have interesting comparison semantics, as any comparison with NaN will be false (including NaN == NaN).

The representations of NaNs also have all exponent bits set (like infinity) but have a non-zero mantissa. The sign bit is irrelevant for NaNs.

There are two types of NaNs: Signaling NaNs (sNaN) and quiet NaNs (qNaN). They are distinguished by the first bit after the exponent: If it is 1 then the NaN is quiet; if it is 0 it is signaling. Signaling NaNs - in theory - should throw an invalid operation exception (EM_INVALID) when they are operated upon, whereas quiet NaNs should just be left alone. Practically this doesn’t seem to be used much, at least in MSVC this is disabled by default and needs to be enabled by compiler option + __controlfp() call.

Where it starts to get interesting for us is that NaNs additionally encode a 51 bit “payload” in the mantissa. This payload was originally designed to contain error information.

But deceitful as we are we will misuse that NaN payload to stuff other things in it - like integers and pointers.

64 bit is a lie

On 64 bit architectures pointers have a size of 64 bits, obviously. But think about that again: How much is 64 bit actually? That’s 2^64 = 1.84467441 * 10^19 addressable memory bytes. That’s 16 EiB (Exabytes). Or talking in more convenient terms that’s 17179869184 Gigabytes. I don’t know about you but I only have 4GB of memory. I’ve heard of exotic server setups that have around 800 GB of memory. But even those abominations costing hundreds of thousands of dollars still would only need a 40 bit pointer space (2^40 = 1024 GB).

Unsurprisingly we aren’t the first to notice this: the x86-64 architecture utilizes only the lower 48 bits (which still allows 256 TiB) of a pointer. Additionally bits 63 through 48 must be copies of bit 47. Pointers that follow this pattern are called canonical.

This leads to a strange looking address space (you can find a nicer version of this image on Wikipedia):

0xFFFFFFFF.FFFFFFFF
         ...        <- Canonical upper "half"
0xFFFF8000.00000000
         ...
         ...        <- Noncanonical
         ...
0x00007FFF.FFFFFFFF
         ...        <- Canonical lower "half"
0x00000000.00000000

The operating system normally assigns only pointers from the lower half to applications, reserving the upper half for itself. As always there are outliers - e.g. Solaris assigns addresses from the upper half under some circumstances (mmap). But we’ll just ignore that and assume that only the lower half from 0x00000000.00000000 to 0x00007FFF.FFFFFFFF is used. (Again, all these evil assumptions…)

At this point you should see what I am driving at: If pointers are really only 48 bits large they fit perfectly into our 51 bit payload space.

Sample implementation

Here is an implementation stub for doing so (just implements doubles, ints and void pointers; a real implementation would look similar, just with more types):

#include <stdint.h>
#include <cassert>

inline bool isNegativeZero(double number) {
    return number == 0 && *reinterpret_cast<int64_t *>(&number) != 0;
}

class Value {
private:
    union {
        double asDouble;
        uint64_t asBits;
    };

    static const uint64_t MaxDouble = 0xfff8000000000000;
    static const uint64_t Int32Tag  = 0xfff9000000000000;
    static const uint64_t PtrTag    = 0xfffa000000000000;
public:
    inline Value(const double number) {
        int32_t asInt32 = static_cast<int32_t>(number);

        // if the double can be losslessly stored as an int32 do so
        // (int32 doesn't have -0, so check for that too)
        if (number == asInt32 && !isNegativeZero(number)) {
            *this = Value(asInt32);
            return;
        }

        asDouble = number;
    }

    inline Value(const int32_t number) {
        asBits = number | Int32Tag;
    }

    inline Value(void *pointer) {
        // ensure that the pointer really is only 48 bit
        assert((reinterpret_cast<uint64_t>(pointer) & PtrTag) == 0);

        asBits = reinterpret_cast<uint64_t>(pointer) | PtrTag;
    }

    inline bool isDouble() {
        return asBits < MaxDouble;
    }
    inline bool isInt32() {
        return (asBits & Int32Tag) == Int32Tag;
    }
    inline bool isPointer() {
        return (asBits & PtrTag) == PtrTag;
    }

    inline double getDouble() {
        assert(isDouble());

        return asDouble;
    }
    inline int32_t getInt32() {
        assert(isInt32());

        return static_cast<int32_t>(asBits & ~Int32Tag);
    }
    inline void *getPointer() {
        assert(isPointer());

        return reinterpret_cast<void *>(asBits & ~PtrTag);
    }
};

Again, the code should be easy enough to understand. As an aid, here are the three consts from the top in binary:

Maximum double (qNaN with sign bit set without payload):
                    seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
0xfff8000000000000: 11111111|11111000|00000000|00000000|00000000|00000000|00000000|00000000

32 bit integer:
                    seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
0xfff9000000000000: 11111111|11111001|00000000|00000000|iiiiiiii|iiiiiiii|iiiiiiii|iiiiiiii
                                                        ^- integer value

Pointer:
                    seeeeeee|eeeemmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm|mmmmmmmm
0xfffa000000000000: 11111111|11111010|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp|pppppppp
                                      ^- 48 bit pointer value

A few more considerations

The above code makes doubles accessible directly, without further operations and accesses pointers using a bit mask (that’s what Mozilla does). An alternative implementation could do it the other way around and make pointers accessible directly and doubles using an offset (i.e. you would add 0x0001000000000000 on inserting a double and subtract it when retrieving it). The latter is what WebKit does.

On 32 bit machines one can access the lower 32 bits of the 64 bit value directly. So both pointers and integers can be accessed directly in any case. On the other hand, whereas the tagged pointer with embedded integer approach would need only a 32 bit value on 32 bit machines, the NaN approach will need a 64 bit value in both 32 bit and 64 bit environments. So on 32 bit two words need to be passed around, instead of just one (that’s why Mozilla calls this approach fatvals).

Still, considering the overall wins it is worth it: Both integers and doubles (and for that matter also bools and all other “small” values) can be stored directly in the value, without accessing the heap.

htmlspecialchars() improvements in PHP 5.4

nikic@php.net (NikiC) — Sat, 28 Jan 2012 00:00:00 -0800

There has been lots of buzz about many of the new features in PHP 5.4, like the traits support, the short array syntax and all those other syntax improvements.

But one set of changes that I think is particularly important was largely overlooked: For PHP 5.4 cataphract (Artefacto on StackOverflow) heroically rewrote large parts of htmlspecialchars thus fixing various quirks and adding some really nice new features.

(The changes discussed here apply not only to htmlspecialchars, but also to the related htmlentities and in parts to htmlspecialchars_decode, html_entity_decode and get_html_translation_table.)

Here a quick summary of the most important changes:

UTF-8 as the default charset
Improved error handling (ENT_SUBSTITUTE)
Doctype handling (ENT_HTML401, …)

UTF-8 as the default charset

As you hopefully know the third argument for htmlspecialchars is the character set. Thing is: Most people just leave that argument out, thus falling back to the default charset. This default charset was ISO-8859-1 before PHP 5.4 and as such did not match the UTF-8 encoding most people use. PHP 5.4 fixes this by making UTF-8 the default.

Improved error handling

Error handling in htmlspecialchars before PHP 5.4 was … uhm, let’s call it “unintuitive”:

If you passed a string containing an “invalid code unit sequence” (which is Unicode slang for “not encoded correctly”) htmlspecialchars would return an empty string. Well, okay, so far so good. The funny thing was that it additionally would throw an error, but only if error display was disabled. So it would only error if errors are hidden. Nice, innit?

This basically meant that on your development machine you wouldn’t see any errors, but on your production machine the error log would be flooded with them. Awesome.

So, as of PHP 5.4 thankfully this behavior is gone. The error will not be generated anymore.

Additionally there are two options that allow you to specify an alternative to just returning an empty string:

ENT_IGNORE: This option (which isn’t actually new, it was there in PHP 5.3 already) will just drop all invalid code unit sequences. This is bad for two reasons: First, you won’t notice invalid encoding because it’ll be simply dropped. Second, this imposes a certain security risk (for more info see the Unicode Security Considerations).
ENT_SUBSTITUTE: This new alternative option takes a much more sensible approach at the problem: Instead of just dropping the code units they will be replaced by a Unicode Replacement Character (U+FFFD). So invalid code unit sequences will be replaced by � characters.

Let’s have a look at the different behaviors (demo):

<?php // "\80" is invalid UTF-8 in this context
var_dump(htmlspecialchars("a\x80b"));                 // string(0) ""
var_dump(htmlspecialchars("a\x80b", ENT_IGNORE));     // string(2) "ab"
var_dump(htmlspecialchars("a\x80b", ENT_SUBSTITUTE)); // string(5) "a�b"

Clearly, you want the last behavior. In your real code it will probably look like this:

<?php

// this goes into the bootstrap (or where appropriate) to make the code
// not throw a notice on PHP 5.3
if (!defined('ENT_SUBSTITUTE')) {
    define('ENT_SUBSTITUTE', 0);          // if you want the empty string behavior on 5.3
    // or
    define('ENT_SUBSTITUTE', ENT_IGNORE); // if you want the char removal behavior on 5.3
                                          // (don't forget about the security issues though!)
}

// don't forget to specify the charset! Otherwise you'll get the old default charset on 5.3.
$escaped = htmlspecialchars($string, ENT_QUOTES | ENT_SUBSTITUTE, 'UTF-8');

Doctype handling

In PHP 5.4 there are four additional flags for specifying the used doctype:

ENT_HTML401 (HTML 4.01) => this is the default
ENT_HTML5 (HTML 5)
ENT_XML1 (XML 1)
ENT_XHTML (XHTML)

Depending on which doctype you specify htmlspecialchars (and the other related functions) will use different entity tables.

You can see this in the following example (demo):

<?php
var_dump(htmlspecialchars("'", ENT_HTML401)); // string(6) "&#039;"
var_dump(htmlspecialchars("'", ENT_HTML5));   // string(6) "&apos;"

So for HTML 5 an ' entity will be generated, whereas for HTML 4.01 - which does not yet support ' - a numerical ' entity is returned.

The difference becomes more evident when using htmlentities, because the differences are larger there. You can easily see this by having a look at the raw translation tables:

To do this, we can use the get_html_translation_table function. Here first an example for the XML 1 doctype (demo):

<?php
var_dump(get_html_translation_table(HTML_ENTITIES, ENT_QUOTES | ENT_XML1));

The result will look like this:

array(5) {
  ["""]=>
  string(6) "&quot;"
  ["&"]=>
  string(5) "&amp;"
  ["'"]=>
  string(6) "&apos;"
  ["<"]=>
  string(4) "&lt;"
  [">"]=>
  string(4) "&gt;"
}

This matches our expectations: XML by itself defines only the five basic entities.

Now try the same thing for HTML 5 (demo) and you’ll see something like this:

array(1510) {
  ["	"]=>
  string(5) "&Tab;"
  ["
"]=>
  string(9) "&NewLine;"
  ["!"]=>
  string(6) "&excl;"
  ["""]=>
  string(6) "&quot;"
  ["#"]=>
  string(5) "&num;"
  ["$"]=>
  string(8) "&dollar;"
  ["%"]=>
  string(8) "&percnt;"
  ["&"]=>
  string(5) "&amp;"
  ["'"]=>
  string(6) "&apos;"
  // ...
}

So HTML 5 defines a vast number of entities - 1510 to be precise. You can also try HTML 4.01 and XHTML; they both define 253 entities.

Also affected by the chosen doctype is another new error handling flag which I did not mention above: ENT_DISALLOWED. This flag will replace characters with a Unicode Replacement Character, which formally are a valid code unit sequences, but are invalid in the given doctype.

This way you can ensure that the returned string is always well formed regarding encoding (in the given doctype). I’m not sure though how much sense it makes to use this flag. The browser will handle invalid characters gracefully anyways, so this seems unnecessary to me (though I’m probably wrong).

There is other stuff too…

… but I don’t want to list everything here. I think the three changes mentioned above are the most important improvements.

<?php
htmlspecialchars("<\x80The End\xef\xbf\xbf>", ENT_QUOTES | ENT_HTML5 | ENT_DISALLOWED | ENT_SUBSTITUTE, 'UTF-8');

Careful: XDebug can skew your performance numbers

nikic@php.net (NikiC) — Thu, 19 Jan 2012 00:00:00 -0800

XDebug is a great tool and a really big aid in debugging.

But running it incurs a certain overhead on many common operations - like calling functions. Due to this having XDebug enabled can skew your benchmarking and profiling numbers. (“Enabled” here means just enabled, no debugging going on or such.)

This has an important implication: If you optimize a script while XDebug is enabled it can actually turn out that it gets slower on your production server (as you hopefully don’t have XDebug enabled there).

An example: “Optimizing” a lexer wrapper

For my PHP parser I need a smaller wrapper around PHP’s token_get_all function. This wrapper (called PHPParser_Lexer) only does nothing more than a little bit of normalization, filtering and mapping on the tokens.

There are basically two ways how this wrapper can work:

For every source code to be parsed a new Lexer instance is created and passed into the parser, which then does multiple calls to ->lex() to fetch one token at a time.
A Lexer is creates once and reused for all source codes. Always only a single call to ->lex() is made, which then returns all the tokens at once.

So, what I did is implement both approaches and see how much time is spent lexing the whole Symfony tree. I got the following results:

Scenario 1 on 5.3.8 with XDebug enabled: 16.939509868622 seconds
Scenario 2 on 5.3.8 with XDebug enabled:  6.104107856751 seconds

That’s an impressive saving of 10 seconds (aka 60%) right there! is what I though.

But then I tried the same thing without XDebug:

Scenario 1 on 5.3.8 with XDebug disabled: 3.656718969345 seconds
Scenario 2 on 5.3.8 with XDebug disabled: 3.195748090744 seconds

Now that XDebug is disabled the numbers are drastically lower and also much closer. The 17 seconds from above are only 3.7 seconds now. The time for the 6 seconds case halved.

Why is there such a drastic change on the first number, and a less pronounced one on the second? The first scenario needs approximately 70000 ->lex() calls, whereas the second one only needs 2000. XDebug adds a quite large overhead to function calls, so the scenario with 70000 calls is impacted much more.

Finally, let’s look at the numbers for PHP 5.4.0 without XDebug:

Scenario 1 on 5.4.0 with XDebug disabled: 2.8629820346832 seconds
Scenario 2 on 5.4.0 with XDebug disabled: 3.0883010864258 seconds

And here the numbers are now actually turned around. PHP 5.4 got some optimizations that made the seconds variant slower.

Another example: Micro benchmarking

As pointless as they are, people still love micro benchmarks, I do too. You’ll find lots of blog posts comparing the performance of something vs. something else. The problem is: Most of these are done on development machines, with XDebug enabled.

As an example this recent blog post about the performance of exceptions will serve. Here are the numbers the author measured for exceptions on PHP 5.3/5.4 for a certain script:

PHP 5.3: 1.1479668617249
PHP 5.4: 0.1864490332

Looking at those you might say: Damn, that’s a pretty impressive improvement!

Well, not really: Seeing these numbers I immediately had the suspicion that the author actually measured two different things: 5.3 with XDebug and 5.4 without it.

So I tested both versions without XDebug and got 0.14 seconds for PHP 5.3 and 0.09 seconds for PHP 5.4, i.e. a much smaller difference. You can find more detailed results in edorian’s blog post on the topic.

I think many micro benchmarking posts you’ll find on the net are affected by this. I was also trapped by this multiple times (e.g. see this question I asked about function call performance).

Conclusion

The conclusion from the above is: If you do benchmarking, benchmark on a machine that’s configured like your production machine. Otherwise you might actually be anti-optimizing.

Also: Low level optimizations like function inlining may actually measurably improve performance on your development machine (like in my case), but have little effect in the production environment. So, you can safely go for clean code with small functions and not fear about performance degradation.

Disproving the Single Quotes Performance Myth

nikic@php.net (NikiC) — Mon, 09 Jan 2012 00:00:00 -0800

If there is one PHP related thing that I really hate, then it is definitely the Single Quotes Performance Myth. Let’s do a random Google search for “PHP single quotes performance”: You will get many results telling you that single quotes are faster than double quotes and that string interpolation is much slower than string concatenation. Most of them advise to use single quotes and concatenation to improve the performance of your application.

Let’s be clear here: This is pointless.

I hold the opinion that microoptimization like that is just pointless. I doubt that string parsing performance is the bottleneck of even a single PHP application out there. This comment nails it pretty well: “And, on top of that, by using less pixels, you reduce greenhouse emissions.” I think that everybody will agree that using single quotes won’t save any significant greenhouse emissions. Still everybody seems to believe that it saves significant execution time. Strange.

But okay, some say “Well, it doesn’t cost me anything to write single quotes, so why not just do it?”

Sounds plausible. So I decided to look into how big the difference really is. And it turned out: There is none.

Strings at runtime

What many people don’t realize is that strings are parsed during lexing, not during execution. When a PHP script runs all strings are already parsed.

A simple proof:

<?php
$x = 'Test';
$y = "\x54\x65\x73\x74";

If you have a look at the opcodes that PHP generates for this they will look like this (online demo):

compiled vars:  !0 = $x, !1 = $y
line     # *  op                           fetch          ext  return  operands
---------------------------------------------------------------------------------
   3     0  >   ASSIGN                                                 !0, 'Test'
   4     1      ASSIGN                                                 !1, 'Test'
         2    > RETURN                                                 1

You don’t need to understand the complete output, but you should still see the main point: Both 'Test' and "\x54\x65\x73\x74" compiled down to the very same opcode.

What does this mean? That the used quote type and the use of escape sequences does not affect runtime, not at all. You obviously can still try but any difference you see there is just measurement error.

Strings at compile time

Now some may argue: PHP is an interpreted language, so compile time actually also happens at runtime. My response to that would normally be: Use APC. APC will cache the generated opcodes so they don’t need to be created on every single request. This can greatly improve page loading time and CPU load, so if you aren’t using APC yet you might try as well now ;)

But, hypothetically, let use assume that we are not using APC. So let’s see whether single quoted strings are actually lexed faster than double quoted ones.

For this we will use a handy function called token_get_all: It lexes a PHP string into a token array. The function will return the token texts plainly and not preparsed, but the internal string parsing routines still will get called, so we can use this as a good approximation to the real numbers.

For testing I use the following script:

<?php echo '<pre>';

const NUM = 10000000;

$singleQuotedStringCode = "<?php '" . str_repeat('x', NUM) . "';";
$doubleQuotedStringCode = '<?php "' . str_repeat('x', NUM) . '";';

$startTime = microtime(true);
token_get_all($singleQuotedStringCode);
$endTime = microtime(true);

echo 'Single quotes: ', $endTime - $startTime, ' seconds', "\n";

$startTime = microtime(true);
token_get_all($doubleQuotedStringCode);
$endTime = microtime(true);

echo 'Double quotes: ', $endTime - $startTime, ' seconds', "\n";

It creates two strings with ten million x characters and lexes them.

You can try it yourself. If you reload the page multiples the numbers will vary, but the general image will be somewhere like this:

Single quotes: 0.061846971511841 seconds
Double quotes: 0.061599016189575 seconds

Namely: No difference. The Single Quotes Performance Myth thus is just a big lie: Single quotes are neither faster at runtime nor at compile time.

String interpolation

So the next question is, what is faster: String concatenation or string interpolation?

Let’s consider this simple case:

<?php
$world = 'World';
'Hallo ' . $world;
"Hallo $world";

If you try running the corresponding benchmark you will get numbers similar to this:

Concatenation: 0.015104055404663 seconds
Interpolation: 0.016894817352295 seconds

So interpolation seems slightly slower than concatenation, doesn’t it? Well, not exactly, but more on that later. Let’s look at the opcodes first:

compiled vars:  !0 = $world
line     # *  op                           fetch          ext  return  operands
---------------------------------------------------------------------------------
   2     0  >   ASSIGN                                                   !0, 'World'
   3     1      CONCAT                                           ~1      'Hallo+', !0
         2      FREE                                                     ~1
   4     3      ADD_STRING                                       ~2      'Hallo+'
         4      ADD_VAR                                          ~2      ~2, !0
         5      FREE                                                     ~2
         6    > RETURN                                                   1

Here you see the reason why concatenation is slightly faster than interpolation in the above example:

Concatenation uses a single CONCAT opcode with 'Hallo+', !0 (!0 is $world here). All CONCAT basically does is create a new string buffer with the new length, copy 'Hallo+' into it and then copy $world into it.

Interpolation on the other hand needs two opcodes: First ADD_STRING is called, which in this case basically copies 'Hallo+' into a new string buffer. Then ADD_VAR is called with ~2, !0 (~2 is the buffer that 'Hallo+' was copied to and !0 is still $world), which basically does the same as CONCAT.

If you read closely the difference is clear: Interpolation copies the 'Hallo+' part twice, concatenation only copies it one time. (Disclaimer: I’m actually not perfectly sure that this is really the reason, but it would be my guess. It could also be just the overhead of another opcode.)

So, interpolation is slower after all, isn’t it? Well, not really, let’s consider this more realistic example:

<?php
$name  = 'Anonymous';
$age   = 123;
$hobby = 'nothing';

'Hi! My name is ' . $name . ' and I am ' . $age . ' years old! I love doing ' . $hobby . '!';
"Hi! My name is $name and I am $age years old! I love doing $hobby!";

If you run the benchmark you will get numbers looking approximately like this:

Concatenation: 0.053942918777466 seconds
Interpolation: 0.049801111221313 seconds

And… interpolation is faster now! Why so? Let’s have a look at the new opcodes:

compiled vars:  !0 = $name, !1 = $age, !2 = $hobby
line     # *  op                           fetch          ext  return  operands
---------------------------------------------------------------------------------
   2     0  >   ASSIGN                                                   !0, 'Anonymous'
   3     1      ASSIGN                                                   !1, 123
   4     2      ASSIGN                                                   !2, 'nothing'
   6     3      CONCAT                                           ~3      'Hi%21+My+name+is+', !0
         4      CONCAT                                           ~4      ~3, '+and+I+am+'
         5      CONCAT                                           ~5      ~4, !1
         6      CONCAT                                           ~6      ~5, '+years+old%21+I+love+doing+'
         7      CONCAT                                           ~7      ~6, !2
         8      CONCAT                                           ~8      ~7, '%21'
         9      FREE                                                     ~8
   7    10      ADD_STRING                                       ~9      'Hi%21+My+name+is+'
        11      ADD_VAR                                          ~9      ~9, !0
        12      ADD_STRING                                       ~9      ~9, '+and+I+am+'
        13      ADD_VAR                                          ~9      ~9, !1
        14      ADD_STRING                                       ~9      ~9, '+years+old%21+I+love+doing+'
        15      ADD_VAR                                          ~9      ~9, !2
        16      ADD_CHAR                                         ~9      ~9, 33
        17      FREE                                                     ~9
        18    > RETURN                                                   1

As you can see the interpolation opcodes are more optimal here: For once interpolation distinguishes between adding a single character, a string and a variable. Also it reuses the same temporary variable for the whole interpolation (~9), whereas concatenation uses six of them (~3, ~4, ~5, ~6, ~7, ~8).

So interpolation becomes more efficient the more is interpolated. For the simple string + var case it is slower, but for more realistic cases with multiple variables it becomes faster.

Lesson learned

At this point I want to emphasize again that the above has no practical impact. All above demos use high iteration counts and any differences in execution time are completely negligible.

Does this still teach us something? Yes, it does: “Never trust a statistic you didn’t forge yourself.” Amen.

Supercolliding a PHP array

nikic@php.net (NikiC) — Wed, 28 Dec 2011 00:00:00 -0800

Did you know that inserting 2^16 = 65536 specially crafted values into a normal PHP array can take 30 seconds? Normally this would take only 0.01 seconds.

This is the code to reproduce it:

<?php echo '<pre>';

$size = pow(2, 16); // 16 is just an example, could also be 15 or 17

$startTime = microtime(true);

$array = array();
for ($key = 0, $maxKey = ($size - 1) * $size; $key <= $maxKey; $key += $size) {
    $array[$key] = 0;
}

$endTime = microtime(true);

echo 'Inserting ', $size, ' evil elements took ', $endTime - $startTime, ' seconds', "\n";

$startTime = microtime(true);

$array = array();
for ($key = 0, $maxKey = $size - 1; $key <= $maxKey; ++$key) {
    $array[$key] = 0;
}

$endTime = microtime(true);

echo 'Inserting ', $size, ' good elements took ', $endTime - $startTime, ' seconds', "\n";

Try it yourself! You may need to adjust the number 16 in the $size = pow(2, 16); line based on what hardware you have. I would start with 14 and increase it one at a time. By the way, I intentionally don’t provide an online demo at the amazing Viper-7 codepad this time so that it doesn’t get overloaded.

Here is a sample output from my machine:

Inserting 65536 evil elements took 32.726480007172 seconds
Inserting 65536 good elements took 0.014460802078247 seconds

How does this work?

PHP internally uses hashtables to store arrays. The above creates a hashtable with 100% collisions (i.e. all keys will have the same hash).

For those who don’t know how hashtables work: When you write $array[$key] in PHP the $key is run through a fast hash function that yields an integer. This integer is then used as an offset into a “real” C array (here “array” means “chunk of memory”).

Because every hash function has collisions this C array doesn’t actually store the value we want, but a linked list of possible values. PHP then walks these values one by one and does a full key comparison until it finds the right element with our $key.

Normally there will be only a small number of collisions, so in most cases the linked list will only have one value.

But the above script creates a hash where all elements collide.

How can you make all of them collide?

In PHP it’s very simple. If the key is an integer the hash function is just a no-op: The hash is the integer itself. All PHP does is apply a table mask on top of it: hash & tableMask.

This table mask ensures that the resulting hash is within the bounds of the underlying real C array. This underlying C array has always a size which is a power of 2 (this way the hashtable is always reasonably efficient in both space and time). So if you store 10 elements the real size will be 16. If you store 33 it will be 64. If you store 63 it will also be 64. The table mask is the size minus one. So if the size is 64, i.e. 1000000 in binary the table mask will be 63, i.e. 0111111 in binary.

So basically the table mask removes all bits that are greater than the hashtable size. And this is what we are exploiting: 0 & 0111111 = 0, but 1000000 & 0111111 = 0 too, so is 10000000 & 0111111 = 0 and 11000000 & 0111111 = 0. As long as we keep those lower bits the same the result of the hash will also stay the same.

So if we insert a total of 64 elements, the first one 0, the second one 64, the third one 128, the fourth one 192, etc., all of those elements will have the same hash (namely 0) and all will be put into the same linked list. And that’s what the code does. Only not with 64 elements, but with a few more.

Why is that so abysmally slow?

Well, for every insertion PHP has to traverse the whole linked list, element for element. On the first insertion it needs to traverse 0 elements (there is nothing there yet). On the second one it traverses 1 element. On the third one 2, on the fourth 3 and on the 64th one 63. Those who know a little bit of math probably know that 0+1+2+3+...+(n-1) = (n-1)*(n-2)/2. So the number of elements to traverse is quadratic. For 64 elements it’s 62*63/2 = 1953 traversals. For 2^16 = 65536 it’s 65534*65535/2=2147385345. As you see, the numbers grow fast. And with the number of iteration grows the execution time.

Hashtable collisions as DOS attack

At this point you may wonder what the above is actually useful for. For the casual user: Not useful at all. But the “bad guys” can easily exploit behavior like the above to perform a DOS (Denial of Service) attack on a server. Remember that $_GET and $_POST and $_REQUEST are just normal arrays and suffer from the same problems. So by sending a specially crafted POST request you can easily take a server down.

PHP is not the only language vulnerable to this. Actually pretty much all other languages used for creating websites have similar problems, as was presented at the 28C3 conference.

But there is hope! PHP already landed a change (which will ship with PHP 5.3.9) which will add a max_input_vars ini setting which defaults to 1000. This setting determines the maximum number of POST/GET variables that are accepted, so now only a maximum of 1000 collisions can be created. If you run the above script with 2^10 = 1024 elements you will get runtimes in the order of 0.003 seconds, which obviously is far less critical than 30 seconds. (Note though that above I am demonstrating an integer key collision. You can also collide string keys, in which case the traversal will be a good bit slower.)

Don't be STUPID: GRASP SOLID!

nikic@php.net (NikiC) — Tue, 27 Dec 2011 00:00:00 -0800

Ever heard of SOLID code? Probably: It is a term describing a collection of design principles for “good code” that was coined by Robert C. Martin (aka “uncle bob”), our beloved evangelist of clean code.

Programming is full of acronyms like this. Other examples are DRY (Don’t Repeat Yourself!) and KISS (Keep It Simple, Stupid!). But there obviously are many, many more…

So, why not approach the problem from the other side for once? Looking at what makes up bad code.

Sorry, but your code is STUPID!

Nobody likes to hear that their code is stupid. It is offending. Don’t say it. But honestly: Most of the code being written around the globe is an unmaintainable, unreusable mess.

What characterizes such code? What makes code STUPID?

Singleton
Tight coupling
Untestability
Premature Optimization
Indescriptive Naming
Duplication

Do you agree with this list? Yes? Great. No? I’ll explain the individual points in greater detail in the following, so you can better understand why exactly those patterns were chosen.

Singleton

<?php
class DB {
    private static $instance;

    public static function getInstance() {
        if (!isset(self::$instance)) {
            self::$instance = new self;
        }

        return self::$instance;
    }

    final private function __construct() { /* something */ }
    final private function __clone() { }

    /* actual methods here */
}

The above is the typical database access implementation you will find in pretty much any PHP tutorial. I actually used something similar to this myself not too long ago.

Now you wonder: What’s wrong with that? You can easily access the DB from anywhere using DB::getInstance() and the code also ensures that you only have one database connection open at a time. What could be bad about that?

Well, yeah, I thought that too ^^ “I only need one connection.” When the application grew larger it turned out that I actually needed a second connection to a different database. And that’s where the mess began. I changed the singleton to have a ->getSecondInstance(), thus converting the singleton to a .. erm .. dupleton? Instead I should have realized that the database connection simply isn’t a singleton and thus can’t be sanely implemented as one. The same also applies to any other use of singletons that you will find. The request object surely is a singleton! Ever heard of subrequests? But the logger definitely is! Never wanted to log different things differently?

And that was only one issue. Another important issue is that by using DB::getInstance() in your code you are binding your code to the DB classname. This means: You can’t extend the DB class. At some point I wanted to optionally log query performance data to APC. But the tight coupling to the class name did not allow me to. If my application had used dependency injection at that time I could have easily extended DB and passed the new instance. But the singleton prevented. What I did instead was somethink looking roughly like this:

<?php
// original DB class
class _DB { /* ... */ }

// extending class
class DB extends _DB { /* ... */ }

One word: Ugly. One could add some other words like Hacky, Unmaintainable, Crap. Or STUPID.

One last point to consider: Remember when I said “You can easily access the DB from anywhere using DB::getInstance()”. Well, actually that’s a bad thing too. Read “from anywhere” as “globally” and translate to “A singleton is a global variable with a fancy name.” When you learned PHP you were probably told that it’s evil to use the global keyword. But by using a singleton you are doing just that: creating global state. This creates non-obvious dependencies and thus makes your app hard to reuse and test.

Tight coupling

You can actually generalize the Singleton issue to static methods and properties in general. Whenever you are writing Foo::bar() in your code you are tightly coupling your code to the Foo class. This makes extending Foos functionality impossible and thus makes your code hard to reuse and hard to test.

Similarly most other plain uses of class names are code smell too. This also applies to the new operator:

<?php
class House {
    public function __construct() {
        $this->door   = new Door;
        $this->window = new Window;
    }
}

How would you replace the door or the window in your house now? Simple: You can’t. As a good developer you will obviously find some dirty hack that does allow you to somehow replace the door or a window. But why not simply write this instead:

<?php
class House {
    public function __construct(Door $door, Window $window) { // Door, Window are interfaces
        $this->door   = $door;
        $this->window = $window;
    }
}

This way one can easily create houses with different doors and windows. The code is easy to extend, easy to reuse and easy to test. What could you want more?

The above code is actually “dependency injection” in a nutshell. Many people associate DI with some Dependency Injection Container (DIC) like Symfony’s, but the actual concept of DI is so much simpler.

Untestability

Unit testing is important. If you don’t test your code, you are bound to ship broken code. But still most people don’t properly cover their code with tests. Why? Mostly because their code is hard to test. What makes code hard to test? Mainly the previous point: Tight coupling. Unit testing - it might seem obvious - tests units of code (usually classes). But how can you test individual classes if they are tightly coupled? You probably can somehow, with even more dirty hacks. But people usually don’t do those efforts and just leave their code untested and broken.

Whenever you don’t write unit tests because you “don’t have time” the real cause probably is that your code is bad. If your code is good you can test it in no time. Only with bad code unit testing becomes a burden.

Premature Optimization

Here is a code snippet from an old version of a website I wrote:

<?php
if (isset($frm['title_german'][strcspn($frm['title_german'], '<>')])) {
    // ...
}

Guess what it does!

How long did it take you to realize that this code does nothing more than check whether the German title contains the < or > character? Did you realize it at all?

Let me explain: If < and > are both not contained, strcspn will return the length of the string. So the code will basically be isset($str[strlen($str)]). As the maximum offset set in a string is the length minus one, this will return false. If one of the characters is contained the function will return a number smaller than the length of the string and thus the whole expression will be true.

So why did I write this unintelligible piece of code? Why didn’t I write this instead:

<?php
if (strlen($frm['title_german']) == strcspn($frm['title_german'], '<>'))) {
    // ...
}

Because some days ago I read that isset is so much faster than strlen… But that code still is not particularly intelligible, because you need to know the exact semantics of the strcspn function (which probably most PHP programmers do not). So why not just write this:

<?php
if (preg_match('(<|>)', $frm['title_german'])) {
    // ...
}

Because I read that regex is slow… (Which by the way is mostly a lie: regular expressions are much faster and much more powerful than you think.)

So, what did I gain from these “optimizations”, apart from unreadable code? Nothing. Even now, when this site attracts approximately fourty million pageviews every month (at the time I wrote this code it was faaar less), even now this micro-optimization won’t make any difference. Simply because it is not the bottleneck of the application. The actual bottleneck is a tripple JOIN in the most-frequently accessed controller (the bottleneck of your application probably is similar).

You will find many more micro-optimization tips like that on the internet. Like “use single quotes, they are faster”. Don’t listen. Most of the advice is plain wrong and even if it isn’t, it won’t make your code measurably faster, it’ll only waste your time.

Indescriptive Naming

By the way, do you know what the strpbrk PHP function does? No? You didn’t even know it existed? I’m not surprised. Nobody who wants to search a string for a list of characters looks for a function named strpbrk. Where did that name come from anyways? This function was inherited from C and its name stands for “string pointer break”. Yeah, really nice, especially in a language that doesn’t actually have pointers (I mean PHP, not C).

Oh, and reading the code snippet in the above section, did you know what the strcspn function does off the top of your head? No? Again, I’m not surprised. It’s short for “string complement span”, just in case you didn’t know.

The lesson from this: Please, name your classes, methods, variables properly, so that people actually know what you mean. I’m not arguing about variables like $i, those are short, but still self-explanatory. The problem is functions like the ones named above. Functions like strpbrk or variables like $yysstk may be obvious the author, but also only to the author.

Duplication

I think everybody agrees that code that is particularly short, concise and pertinent is considered especially elegant (oh, and I don’t mean the Perl/Ruby style “short”). On the other hand most people would consider long and redundant code rather ugly. This is what the aforementioned DRY (Don’t Repeat Yourself!) and KISS (Keep It Simple, Stupid!) design principle want to teach you.

So where does code duplication come from? Programmers are lazy animals, so it would be only natural to type as little code as possible. Still duplication prevails.

I think the most common reason for duplication is following the second STUPID principle: Tight Coupling. If your code is tightly coupled, you just can’t reuse it. And here comes your duplication.

Don’t be STUPID: GRASP SOLID!

So what are the alternatives to writing STUPID code? Simple, SOLID and GRASP.

SOLID deciphers to:

Single responsibility principle
Open/closed principle
Liskov substitution principle
Interface segregation principle
Dependency inversion principle

GRASP stands for General Responsibility Assignment Software Principles, which are the following:

Information Expert
Creator
Controller
Low Coupling
High Cohesion
Polymorphism
Pure Fabrication
Indirection
Protected Variations

Happy coding and a happy new year to you!

PS: If you wonder where STUPID comes from: The idea came up on the PHP chatroom on StackOverlow and the individual components were formed by edorian (StackOverflow, Github, Twitter), James and me. Gordon (StackOverflow, Github, Twitter) came up with the title of this blog post.

How big are PHP arrays (and values) really? (Hint: BIG!)

nikic@php.net (NikiC) — Mon, 12 Dec 2011 00:00:00 -0800

Upfront I want to thank Johannes and Tyrael for their help in finding some of the more hidden memory usage.

In this post I want to investigate the memory usage of PHP arrays (and values in general) using the following script as an example, which creates 100000 unique integer array elements and measures the resulting memory usage:

<?php
$startMemory = memory_get_usage();
$array = range(1, 100000);
echo memory_get_usage() - $startMemory, ' bytes';

How much would you expect it to be? Simple, one integer is 8 bytes (on a 64 bit unix machine and using the long type) and you got 100000 integers, so you obviously will need 800000 bytes. That’s something like 0.76 MBs.

Now try and run the above code. You can do it online if you want. This gives me 14649024 bytes. Yes, you heard right, that’s 13.97 MB - eightteen times more than we estimated.

So, where does that extra factor of 18 come from?

Summary

For those who don’t want to know the full story, here is a quick summary of the memory usage of the different components involved:

                             |  64 bit   | 32 bit
---------------------------------------------------
zval                         |  24 bytes | 16 bytes
+ cyclic GC info             |   8 bytes |  4 bytes
+ allocation header          |  16 bytes |  8 bytes
===================================================
zval (value) total           |  48 bytes | 28 bytes
===================================================
bucket                       |  72 bytes | 36 bytes
+ allocation header          |  16 bytes |  8 bytes
+ pointer                    |   8 bytes |  4 bytes
===================================================
bucket (array element) total |  96 bytes | 48 bytes
===================================================
total total                  | 144 bytes | 76 bytes

The above numbers will vary depending on your operating system, your compiler and your compile options. E.g. if you compile PHP with debug or with thread-safety, you will get different numbers. But I think that the sizes given above are what you will see on an average 64-bit production build of PHP 5.3 on Linux.

If you multiply those 144 bytes by our 100000 elements you get 14400000 bytes, which is 13.73 MB. That’s pretty close to the real number - the rest is mostly pointers for uninitialized buckets, but I’ll cover that later.

Now, if you want to have a more detailed analysis of the values mentioned above, read on :)

The `zvalue_value` union

First have a look at how PHP stores values. As you know PHP is a weakly typed language, so it needs some way to switch between the various types fast. PHP uses a union for this, which is defined as follows in zend.h#307 (comments mine):

typedef union _zvalue_value {
    long lval;                // For integers and booleans
    double dval;              // For floats (doubles)
    struct {                  // For strings
        char *val;            //     consisting of the string itself
        int len;              //     and its length
    } str;
    HashTable *ht;            // For arrays (hash tables)
    zend_object_value obj;    // For objects
} zvalue_value;

If you don’t know C, that isn’t a problem as the code is pretty straightforward: A union is a means to make some value accessible as various types. For example if you do a zvalue_value->lval you’ll get the value interpreted as an integer. If you use zvalue_value->ht on the other hand the value will be interpreted as a pointer to a hashtable (aka array).

But let’s not get too much into this here. Important for us only is that the size of a union equals the size of its largest component. The largest component here is the string struct (the zend_object_value struct has the same size as the str struct, but I’ll leave that out for simplicity). The string struct stores a pointer (8 bytes) and an integer (4 bytes), which is 12 bytes in total. Due to memory alignment (structs with 12 bytes aren’t cool because they aren’t a multiple of 64 bits / 8 bytes) the total size of the struct will be 16 bytes though and that will also be the size of the union as a whole.

So now we know that we don’t need 8 bytes for every value, but 16 - due to PHP’s dynamic typing. Multiplying by 100000 values gives us 1600000 bytes, i.e. 1.53 MB. But the real value is 13.97 MB, so we can’t be there yet.

The `zval` struct

And this is quite logical - the union only stores the value itself, but PHP obviously also needs to store its type and some garbage collection information. The structure holding this information is called a zval and you may have already have heard of it. For more information on why PHP needs it I’d recommend to read an article by Sara Golemon. Anyways, this struct is defined as follows:

struct _zval_struct {
    zvalue_value value;     // The value
    zend_uint refcount__gc; // The number of references to this value (for GC)
    zend_uchar type;        // The type
    zend_uchar is_ref__gc;  // Whether this value is a reference (&)
};

The size of a struct is determined by the sum of the sizes of its components: The zvalue_value is 16 bytes (as computed above), the zend_uint is 4 bytes and the zend_uchars are 1 byte each. That’s a total of 22 bytes. Again due to memory alignment the real size will be 24 bytes though.

So if we store 100000 elements á 24 bytes that would be 2400000 in total, which is 2.29 MB. The gap is closing, but the real value is still more than six times larger.

The cycles collector (as of PHP 5.3)

PHP 5.3 introduced a new garbage collector for cyclic references. For this to work PHP has to store some additional data. I don’t want to explain how the algorithm works here, you can read that up on the linked page from the manual. Important for our size calculations is that PHP will wrap every zval into a zval_gc_info:

typedef struct _zval_gc_info {
    zval z;
    union {
        gc_root_buffer       *buffered;
        struct _zval_gc_info *next;
    } u;
} zval_gc_info;

As you can see Zend only adds a union on top of it, which consists of two pointers. As you hopefully remember the size of a union is the size of its largest component: Both union components are pointers, thus both have a size of 8 bytes. So the size of the union is 8 bytes too.

If we add that on top of the 24 bytes we already have we get 32 bytes. Multiply that by the 100000 elements and we get a memory usage of 3.05 MB.

The Zend MM allocator

C unlike PHP does not manage memory for you. You need to keep track of your allocations yourself. For this purpose PHP uses a custom memory manager that is optimized specifially for its needs: The Zend Memory Manager. The Zend MM is based on Doug Lea’s malloc and adds some PHP specific optimizations and features (like memory limit, cleaning up after each request and stuff like that).

What is important for us here is that the MM adds an allocation header to every allocation done through it. It is defined as follows:

typedef struct _zend_mm_block {
    zend_mm_block_info info;
#if ZEND_DEBUG
    unsigned int magic;
# ifdef ZTS
    THREAD_T thread_id;
# endif
    zend_mm_debug_info debug;
#elif ZEND_MM_HEAP_PROTECTION
    zend_mm_debug_info debug;
#endif
} zend_mm_block;

typedef struct _zend_mm_block_info {
#if ZEND_MM_COOKIES
    size_t _cookie;
#endif
    size_t _size; // size of the allocation
    size_t _prev; // previous block (not sure what exactly this is)
} zend_mm_block_info;

As you can see the definitions are cluttered with lots of compile option checks. So if one of those options is set the allocation header will be bigger and will be largest if you build PHP with heap protection, multi-threading, debug and MM cookies.

For this example though we will assume that all those options are disabled. In that case the only thing left are the two size_ts _size and _prev. A size_t has 8 bytes (on 64 bit), so the allocation header has a total size of 16 bytes - and that header is added on every allocation.

So now we need to adjust our zval size again. In reality it isn’t 32 bytes, but it’s 48, due to that allocation header. Multiplied by our 100000 elements that’s 4.58 MB. The real value is 13.97 MB, so we already got approximately a third covered.

Buckets

Until now we have only considered single values. But array structures in PHP also take up lots of space: “Array” actually is a badly chosen term here. PHP arrays are hash tables / dictionaries in reality. So how do hash tables work? Basically for every key a hash is generated and that hash is used as an offset into a “real” C array. As the hashes can clash, all elements that have the same hash are stored in a linked list. When accessing an element PHP first computes the hash, looks for the right bucket and the traverses the link list, comparing the exact key, element by element. A bucket is defined as follows (see zend_hash.h#54):

typedef struct bucket {
    ulong h;                  // The hash (or for int keys the key)
    uint nKeyLength;          // The length of the key (for string keys)
    void *pData;              // The actual data
    void *pDataPtr;           // ??? What's this ???
    struct bucket *pListNext; // PHP arrays are ordered. This gives the next element in that order
    struct bucket *pListLast; // and this gives the previous element
    struct bucket *pNext;     // The next element in this (doubly) linked list
    struct bucket *pLast;     // The previous element in this (doubly) linked list
    const char *arKey;        // The key (for string keys)
} Bucket;

As you can see one needs to store loads of data to get the kind of abstract array data structure that PHP uses (PHP arrays are arrays, dictionaries and linked lists at the same time, that sure needs much info). The sizes of the individual components are 8 bytes for the unsigned long, 4 bytes for the unsigned int and 7 times 8 bytes for the pointers. That’s a total of 68. Add alignment and you get 72 bytes.

Buckets like zvals need to be allocated on the head, so we need to add the 16 bytes for the allocation header again, giving us 88 bytes. Also we need to store pointers to those buckets in the “real” C array (Bucket **arBuckets;) I mentioned above, which adds another 8 bytes per element. So all in all every bucket needs 96 bytes of storage.

So if we need a bucket for every value, that’s 96 bytes for the bucket and 48 bytes for the zval, which is 144 bytes in total. For 100000 elements that’s 14400000 bytes aka 13.73 MB.

Mystery solved.

Wait, there’s another 0.24 MB left!

Those last 0.24 MB are due to uninitialized buckets: The size of the real C array storing the buckets should ideally be approximately the same as the number of array elements stored. This way you have the least number of collisions (unless you want to waste lots of memory.) But PHP obviously can’t reallocate the whole array every time an element is added - that would be reeeally slow. Instead PHP always doubles the size of the internal bucket array if it hits the limit. So the size of the array is always a power of two.

In our case it is 2^17 = 131072. But we need only 100000 of those buckets, so we are leaving 31072 buckets unused. Those buckets will not be allocated (so we don’t need to spend the full 96 bytes), but the memory for the bucket pointer (the one stored in the internal bucket array) still needs to be allocated. So we additionally use 8 bytes (a pointer) * 31072 elements. This is 248576 bytes or 0.23 MB. That matches the missing memory. (Sure, there are still a few bytes missing, but I don’t really want to cover there. They are things like the hash table structure itself, variables, etc.)

Mystery really solved.

What does this tell us?

PHP ain’t C. That’s all this should tell us. You can’t expect that a super dynamic language like PHP has the same highly efficient memory usage that C has. You just can’t.

But if you do want to save memory you could consider using an SplFixedArray for large, static arrays.

Have a look a this modified script:

<?php
$startMemory = memory_get_usage();
$array = new SplFixedArray(100000);
for ($i = 0; $i < 100000; ++$i) {
    $array[$i] = $i;
}
echo memory_get_usage() - $startMemory, ' bytes';

It basically does the same thing, but if you run it, you’ll notice that it uses “only” 5600640 bytes. That’s 56 bytes per element and thus much less than the 144 bytes per element a normal array uses. This is because a fixed array doesn’t need the bucket structure: So it only requires one zval (48 bytes) and one pointer (8 bytes) for each element, giving us the observed 56 bytes.

PCRE and newlines

nikic@php.net (NikiC) — Sat, 10 Dec 2011 00:00:00 -0800

PCRE (this is what you use when you do a preg_* call in PHP) has a plethora of newline related escape sequences and options, but only few know about those. I want to shed light on some of those options in this post.

`\r\n`, `\n` and `\r`

Okay, everyone knows about those escape sequences:

\r\n is the CRLF newline Windows uses
\n is the LF newline Unix uses
\r is the CR newline that old versions of Mac OS used

A common combination is (?>\r\n|\n|\r), which matches any CRLF type newline.

Meet `\R`!

But apart from \r and \n PCRE also has another character group matching newlines: \R. By default \R matches Unicode newlines sequences, but it can be configured using several options:

/(*BSR_ANYCRLF)\R/: Matches any CRLF type newline sequence and thus is equivalent to the /(?>\r\n|\n|\r)/ regex
/\R/: Matches any Unicode newline sequence that is in the ASCII range. I.e. it is equivalent to /(?>\r\n|\n|\r|\f|\x0b|\x85)/. So additionally to the CRLF type newlines it also matches FF formfeeds (\f), VT vertical tabs (\x0b) and the NEL next line character (\x85).
/\R/u (u means UTF-8 mode): Matches any Unicode newline sequence including newline characters outside the ASCII range, which is equivalent to /(?>\r\n|\n|\r|\f|\x0b|\x85|\x{2028}|\x{2029})/. This means only two additonal characters are added: The LS line separator (\x{2028}) and the PS paragraph separator (\x{2029}).

Note that \R is not special in character classes. So /[^\R]/u for example will not match any non-newline character. Instead it will simply match any character which is not R.

The magic `.` dot

. is usally said to be “any character”. In a default configuration this is not quite true. . is “any character which is not a newline”. What exactly a “newline” means here can also be configured similarly to \R:

/(*CRLF)./: . matches everything apart from CRLF (\r\n) characters
/(*LF)./: . matches everything apart from LF (\n) characters
/(*CR)./: . matches everything apart from CR (\r) characters
/(*ANYCRLF)./: . matches everything apart from CRLF style newlines, i.e. everything apart from CRLF, LF or CR.
/(*ANY)./: . matches everything apart from Unicode newline sequences (ASCII)
/(*ANY)./u: . matches everything apart from Unicode newline sequences (any)
/./: What this matches depends on how PCRE was compiled, but the default is to behave as if (*LF) was specified. I.e. unless you compiled PCRE with some other newline configuration . will match everything apart from \n.

To sum up, have a look at the following code:

<?php
var_dump(preg_match('/^a.+b$/',        "a\r\nb"));  // 0 (Newline \n is     contained by \r\n)
var_dump(preg_match('/^a.+b$/',        "a\nb"));    // 0 (Newline \n is     contained by \n)
var_dump(preg_match('/^a.+b$/',        "a\rb"));    // 1 (Newline \n is not contained by \r)
var_dump(preg_match('/(*CR)^a.+b$/',   "a\nb"));    // 1 (Newline \r is not contained by \n)
var_dump(preg_match('/(*CR)^a.+b$/',   "a\rb"));    // 0 (Newline \r is     contained by \r)
$LS = "\xE2\x80\xA8"; // Line Separator in UTF-8
var_dump(preg_match('/(*ANY)^a.+b$/u', "a{$LS}b")); // 0 (Newline LS is     contained by LS)
var_dump(preg_match('/(*ANY)^a.+b$/',  "a{$LS}b")); // 1 (u modifier was not specified, so LS isn't a newline anymore)

`PCRE_DOTALL` and `\N`

PCRE also provides an option which instructs . to really match any character. This option is called PCRE_DOTALL and can be specified using the s modifier in PHP. So /./s will match absolutely any character, including newlines.

But even in DOTALL mode you can get the behavior of the “normal” dot: The \N escape sequence behaves the same as ., but is not affected by the s modifier. So \N will always match any character which is not a newline (where “newline” is again defined by the above options).

\N, just like \R, looses it’s special meaning within character groups.

Whitespace character groups

As a small addendum I would also like to point out what the different whitespace character groups contain, as this isn’t quite clear to most people:

The commonly used \s group matches LF (\n), CR (\r), HT (tab), FF (form feed) and space characters. So it does not contain the VT vertical tab character. The POSIX character group [:space:] on the other hand includes the vertical tab, too.

The \pZ Unicode character property for separators does not contain the “classic” newlines. But there are two special, PCRE specific, character properties for that purpose: p{Xsp} contains \pZ as well as LF, CR and FF. \p{Xps} additionally contains VT.

Those two Unicode properties are also internally used in UCP mode (UCP mode makes the normal \s style character groups behave like the Unicode character properties). I.e. (*UCP)\s is equivalent to \p{Xsp} and (*UCP)[:space:] is equivalent to \p{Xps}.

There are also two more character groups for whitespace matching, namely \v for vertical whitespace and \h for horizontal. Contrary to the other \s style character groups these two match non-ASCII characters in UTF-8 mode even when not in UCP mode. The reason for this is that they were added only quite late, whereas the others existed pretty much from the beginning. \v matches CR, LF, VT, FF, NEL, LS, PS. \h matches HT, space and 17 other horizontal spaces which you wouldn’t normally know. Those contain things like “Six-per-em space” or “Medium mathematical space”.

Manually installing PEAR on Windows

nikic@php.net (NikiC) — Sat, 03 Dec 2011 00:00:00 -0800

So, you tried installing PEAR on Windows using their go-pear.phar installer and it - for some unknown reason - didn’t work? Then you still have one options left: Installing it by hand. This is slightly more cumbersome than just running an installer, but at least it works.

Download PEAR

First download the PEAR package, extract the .tgz file (e.g. using 7-Zip) so you get a .tar file. Extract that file again into some folder (e.g. PEAR-files/). If you enter that folder you should see three entries:

PEAR-x.y.z/
package.xml
package2.xml

Go into the PEAR-x.y.z/ folder, where x.y.z is the PEAR version you downloaded and copy all the files to the location you want to install PEAR in (I’ll use the D:\htdocs\PEAR path in this example).

Write a pear.bat

Unless you happen to be the rare case where the pear.bat just works you will now need to create a custom pear.bat file (it’s located in the script/ directory). Open the file in some text editor (e.g. Notepad++) and insert the following:

@ECHO OFF

REM The directory we want to put everything in
SET "PREFIX=D:\htdocs\PEAR"

REM The PEAR environment variables (using the %PREFIX% defined above)
SET "PHP_PEAR_INSTALL_DIR=%PREFIX%"
SET "PHP_PEAR_SYSCONF_DIR=%PREFIX%"
SET "PHP_PEAR_EXTENSION_DIR=%PREFIX%\ext"
SET "PHP_PEAR_DOC_DIR=%PREFIX%\docs"
SET "PHP_PEAR_BIN_DIR=%PREFIX%\scripts"
SET "PHP_PEAR_DATA_DIR=%PREFIX%\data"
SET "PHP_PEAR_CFG_DIR=%PREFIX%\cfg"
SET "PHP_PEAR_WWW_DIR=%PREFIX%\www"
SET "PHP_PEAR_TEST_DIR=%PREFIX%\tests"
SET "PHP_PEAR_TEMP_DIR=%PREFIX%\temp"

REM The PHP binary to use (here a XAMPP binary, yours might be elsewhere)
SET "PHP_PEAR_PHP_BIN=C:\xampp\php\php.exe"

"%PHP_PEAR_PHP_BIN%" -C -d date.timezone=UTC -d output_buffering=1 -d safe_mode=0 -d open_basedir="" -d auto_prepend_file="" -d auto_append_file="" -d variables_order=EGPCS -d register_argc_argv="On" -d "include_path='%PHP_PEAR_INSTALL_DIR%'" -f "%PHP_PEAR_BIN_DIR%\pearcmd.php" -- %1 %2 %3 %4 %5 %6 %7 %8 %9

@ECHO ON

Change the PREFIX variable to your PEAR installation directory (i.e. the location you moved the files to). Also change the PHP_PEAR_PHP_BIN variable to the PHP binary you want to use. You can also adjust the other env variables, though changing some of them will require you to take further actions. E.g. if you want to change the BIN_DIR to %PREFIX%\bin you’ll need to rename the scripts/ folder too.

Download additional packages

If you now open the command line, cd into the scripts/ dir and run pear all you will get is a bunch of include errors. This is because PEAR requires some more packages to function. You’ll need to install those manually too:

Download the Console_Getopt package and extract the .tgz file just like you extracted the main PEAR package. Now go into the folder that you have extracted the files into. You’ll find a package.xml there and a folder named Console_Getopt-x.y.z/. Enter that folder. It’ll contain a yet another folder named Console/. Copy this folder into your PEAR installation (i.e. D:\htdocs\PEAR in this example).

If you now run pear again it should work fine. Run some commands, like pear update-channels and pear config-set auto_discover 1. Both should work fine. But if you now run pear install --alldeps pear.phpunit.de/PHPUnit you’ll again get a bunch of errors. This is because PEAR requires two further packages: Archive_Tar and Structures_Graph. You can install them in the same manner as the Console_Getopt package. Only difference is that these packages may also contain docs/ and tests/ directories. Just copy them, too.

If you now try pear install --alldeps pear.phpunit.de/PHPUnit again everything should work fine. PEAR should have created a PHPUnit/ directory in your installation path and there should be a phpunit.bat in your scripts/ directory.

Adjusting your include_path

The last step is to add your PEAR installation to the include_path. Just open your php.ini and find the line specifying the include_path. Replace it with:

include_path = ".;D:\htdocs\PEAR" ; or whatever your install dir might be

Now you’re done :)

PHP internals: When does foreach copy?

nikic@php.net (NikiC) — Fri, 11 Nov 2011 00:00:00 -0800

Important: This post requires some knowledge of PHP’s internal workings, namely zvals, refcounts and copy-on-write behavior. If those terms don’t mean anything to you, you will need to read up on them in oder to understand this post. I would recommend an article by Sara Golemon.

PHP’s foreach is a very neat and to-the-point language construct. Still some people don’t like to use it, because they think it is slow. One reason usually named is that foreach copies the array it iterates. Thus some people recommend to write:

<?php
$keys = array_keys($array);
$size = count($array);
for ($i = 0; $i < $size; $i++) {
    $key   = $keys[$i];
    $value = $array[$key];

    // ...
}

Instead of the much more intuitive and straightforward:

<?php
foreach ($array as $key => $value) {
    // ...
}

There are two problems with this:

Microoptimization is evil. Usually it only wastes your time and doesn’t give any measurable performance improvements.
The copying behavior of foreach is somewhat more complicated than most people think. Often the “optimized” variant happens to be slower than the original.

When does foreach copy?

Whether or not foreach copies the array and how much of it depends on three things: Whether the iterated array is referenced, how high its refcount is and whether the iteration is done by reference.

Not referenced, refcount == 1

In the below code $array is not referenced and has a refcount of 1. In this case foreach will not copy the array (proof) - contrary to the popular belief that foreach always copies the iterated array if it isn’t referenced.

<?php
test();
function test() {
    $array = range(0, 100000);
    foreach ($array as $key => $value) {
        // ...
    }
}

The reason is simple: Why should it? The only thing that foreach modifies about $array is it’s internal array pointer. This is expected behavior and thus doesn’t need to be prevented.

Not referenced, refcount > 1

The following code looks very similar to the previous one. The only difference is that the array is now passed as an argument. This seems like an insignificant difference, but it does change the behavior of foreach: It now will copy the array structure, but not the values (proof; if you want to see that this is really only the structure being copied compare this and that script. The first only copies the structure, the second copies both).

<?php
$array = range(0, 100000);
test($array);
function test($array) {
    foreach ($array as $key => $value) {
        // ...
    }
}

This might seem odd at first: Why would it copy when the array is passed through an argument, but not if it is defined in the function? The reason is that the array zval is now shared between multiple variables: The $array variable outside the function and the $array variable inside it. If foreach would iterate the array without copying its structure it would not only change the array pointer of the $array variable in the function, but also the pointer of the $array variable outside the function. Thus foreach needs to copy the array structure (i.e. the hash table). The values on the other hand still can share zvals and thus don’t need to be copied.

Referenced

The next case is very similar to the previous one. The only difference is that the array is passed by reference. In this case the array again will not be copied (proof).

<?php
$array = range(0, 100000);
test($array);
function test(&$array) {
    foreach ($array as $key => $value) {
        // ...
    }
}

In this case the same reasoning applies as with the previous case: The outer $array and the inner $array share zvals. The difference is that they now are references (isref == 1). Thus in this case it is expected that any change to the inner array will also be done to the outer array. So if the array pointer of the inner array is changed, the array pointer of the outer array should change, too. That’s why foreach doesn’t need to copy.

Iterated by reference

The above examples were all iterating by value. For iteration by reference the same rules apply, but the additional value reference changes copying behavior of the array values (the behavior about structure copying stays).

The case “Not referenced, refcount == 1” doesn’t change. By reference iteration means we want to change the original array if there are any changes to the $value, so the array isn’t copied (proof).

The case “Referenced” also stays the same, as in this case a change to $value should change all variables referencing the iterated array (proof).

Only the “Not referenced, refcount > 1” case changes, as now both the array structure and its values need are be copied. The array structure because otherwise the array pointer of the $array variable outside the function would change and the values because a change to $value would also change the outside $array values (proof).

Summary

To summarize:

foreach will copy the array structure if and only if the iterated array is not referenced and has a refcount > 1
foreach will additionally copy the array values if and only if the previous point applies and the iteration is done by reference

Improving lexing performance in PHP

nikic@php.net (NikiC) — Sun, 23 Oct 2011 00:00:00 -0700

Lexing isn’t something you would normally do in PHP, simply because other languages like C can easily outperform PHP by several orders of magnitude. Still it sometimes is desirable to write lexers in PHP, e.g. for tokenizing templates, doccomments and other DSLs. So I want to share some thoughts on how to write fast lexers in PHP.

Lexing CSV files

PHP already provides functions for parsing CSV files and strings. I will use CSV lexing as an example anyways, simply because it’s so simple. Here an example of a CSV line (as PHP implements it):

Field,Another Field,"comma -> , <- comma","quote -> \" <- quote"

If we define the tokens in terms of regular expressions, they would look roughly like this:

<?php
$tokenMap = array(
    '~[^",\r\n]+~A'                     => T_PLAIN_FIELD,
    '~"[^"\\\\]*(?:\\\\.[^"\\\\]*)*"~A' => T_QUOTED_FIELD,
    '~,~A'                              => T_FIELD_SEPARATOR,
    '~\r\n?|\n~A'                       => T_LINE_SEPARATOR,
);

Looping through the regexes

The most obvious approach to lexing in PHP is to simply loop through the regexes and match them against the current position, until there are no characters left:

<?php
function lex($string, array $tokenMap) {
    $tokens = array();

    $offset = 0; // current offset in string
    while (isset($string[$offset])) { // loop as long as we aren't at the end of the string
        foreach ($tokenMap as $regex => $token) {
            if (preg_match($regex, $string, $matches, null, $offset)) {
                $tokens[] = array(
                    $token,      // token ID      (e.g. T_FIELD_SEPARATOR)
                    $matches[0], // token content (e.g. ,)
                );
                $offset += strlen($matches[0]);
                continue 2; // continue the outer while loop
            }
        }

        throw new LexingException(sprintf('Unexpected character "%s"', $string[$offset]));
    }

    return $tokens;
}

The drawback of this code is fairly obvious: On every new offset one needs to iterate through all regexes and match them one at the time. This maybe isn’t such a problem in this case as we only have four regexes, but when lexing some real language one usually deals with hundreds of them.

Compiling into a single regex

The solution is to compile all regexes into a single big one. Our above regex would be converted to:

<?php
$regex = '~
    ([^",\r\n]+)
  | ("[^"\\\\]*(?:\\\\.[^"\\\\]*)*")
  | (,)
  | (\r\n?|\n)
~xA';

But how can we use this? How can we know which of the four subregexes matched the string? Simple: As all the subregexes are enclosed in parenthesis they are capturing groups. So our $matches array will contain the matched string at position [0] and at the position of the matched subregex. All other groups will be empty.

An example: If the regex matched the , subregex, we would get this $matches array:

<?php
array(
    0 => ',',
    1 => '',
    2 => '',
    3 => ',',
    4 => '',
);

From this we know that the 3rd subregex matched, as it is the one which has a value.

The resulting code for an abstract lexer class would be:

<?php
class Lexer {
    protected $regex;
    protected $offsetToToken;

    public function __construct(array $tokenMap) {
        $this->regex = '((' . implode(')|(', array_keys($tokenMap)) . '))A';
        $this->offsetToToken = array_values($tokenMap);
    }

    public function lex($string) {
        $tokens = array();

        $offset = 0;
        while (isset($string[$offset])) {
            if (!preg_match($this->regex, $string, $matches, null, $offset)) {
                throw new LexingException(sprintf('Unexpected character "%s"', $string[$offset]));
            }

            // find the first non-empty element (but skipping $matches[0]) using a quick for loop
            for ($i = 1; '' === $matches[$i]; ++$i);

            $tokens[] = array($matches[0], $this->offsetToToken[$i - 1]);

            $offset += strlen($matches[0]);
        }

        return $tokens;
    }
}

How much does this really change?

I am seeing approximately 30% performance improvement for the average case (online demo). But this value varies with different regexes and input. If the number of regexes increases the performance improvement is bigger. Additionally in the compiled-regex variant the order of the regexed has less influence on the execution time (online demo). I.e. it is not that important to put the more probable regexes first and the less probable ones last. (But it still is important, only less important!)