Blog RSS Feed Subscribe

Jordi Boggiano

Jordi Boggiano Passionate web developer, specialized in web performance and php. Partner at Nelmio, information junkie and speaker.

Categories

Toran Proxy and the future of Composer

TL;DR: New shiny product to support Composer development: Toran Proxy

A bit of history

I have spent quite a large part of the last three years working on both Composer and Packagist. This has been great fun for the most part, I learned tons, met a gazillion people both online and offline, got to travel places to talk about it at conferences. One question I get asked frequently is how I find the time to work on this for free, and my answer until recently was along the lines of: "I run my own business, which affords me quite some flexibility".

The problem is that I can not use this answer anymore. We have changed Nelmio's business model a few months ago to focus more on consulting and contracting. While this does not change the amount of time I can decide to spend on open source, it means that if I keep working the way I did in the past I will have to move under a bridge sooner or later.

Open-Source or a Salary?

I have spent quite some time over the last few months evaluating options to get out of the current situation. Having to choose between working on open source or earning a living is not great, and I feel horrible for ignoring GitHub issues and such. So the one thing I settled on is to work on an product around Composer that helps businesses using it but does not take anything away from the regular users.

Introducing Toran Proxy

Toran Proxy does mainly two things at the moment: it proxies Packagist, GitHub and others to make Composer faster and more reliable, and it allows you to host private packages easily. This was already sort of possible with Satis and many people used various amounts of hackery to get there. Toran makes it easier and faster. It integrates better with Composer since it acts as a real proxy. It can fetch/build packages lazily or can build them ahead of time, enabling you to choose between a lower disk usage or more safety against GitHub outages and such.

It comes with a yearly license fee, which includes updates and will hopefully allow me to work full time on improving Composer & Packagist. There is quite a big backlog of issues to look at and pull requests to review, finalize and merge. Packagist also has tons of potential for improvement. I have a million ideas and I really hope I get the chance to work on them. For example improvements in the discoverability of packages alone would benefit most everyone in the PHP community.

Of course Composer and Packagist remain free to use and fully open source. There is no way that will change. I just hope I can continue to work on them, for the community and supported by the community. Check it out!

June 19, 2014 // News, PHP // Post a comment

Authentication management in Composer

Up until today if you run a home-grown package repository serving private packages it was quite a pain to use with Composer. You did not have efficient way to password-protect the repository except by inlining the password in the composer.json or by typing the username/password every single time.

With the merge of PR#1862 and some further improvements you can now remove credentials from your composer.json! The first time Composer needs to authenticate against some domain it will prompt you for a username/password and then you will be asked whether you want to store it. The storage can be done either globally in the COMPOSER_HOME/auth.json file (COMPOSER_HOME defaults to ~/.composer or %APPDATA%/Composer on Windows) or also in the project directory directly sitting besides your composer.json.

You can also configure these by hand using the config command if you need to configure a production machine to be able to run non-interactive installs. For example to enter credentials for example.org one could type:

composer config http-basic.example.org username password

That will store it in the current directory's auth.json, but if you want it available globally you can use the --global (-g) flag.

The advantage of having it in a separate file is that you can easily add this auth.json to .gitignore and let every developer in your company have their own credentials in there.

And I did not forget the security-minded folks that do not want to store anything on disk and do not want to be prompted every time! You can use composer config -g store-auths false

Altogether these small improvements should make some use cases much easier so that is great news.

May 27, 2014 // News, PHP // Post a comment

PSR-4 autoloading support in Composer

As of today and thanks to a pull request by Andreas Hennings who did the bulk of the work, we have PSR-4 autoloading support in Composer. It is a feature that can have a serious impact on users of your packages so I wanted to detail what it means for everyone.

First of all if you are not familiar with PSR-4 but know about PSR-0 the main difference and benefit is that it allows for flatter directory structures in your git repositories. While you typically had src/Vendor/Lib/Class.php in libraries it is now possible to use a simpler src/Class.php while retaining the namespacing at the code level. The other small difference is there no more support for PEAR-style namespacing of classes using underscores (e.g. Vendor_Lib_Class) so those packages should keep using PSR-0 and not migrate.

So we can use it in Composer now, and that's cool. I am sure some of you already stopped reading and are pushing changes to your repos to be using the new shiny, but please don't. Not just yet.

The issue is that Composer support for PSR-4 is needed for your packages to autoload properly. If you push this out now, people are going to freak out as soon as they update your package and classes can not be autoloaded anymore. In addition we will for sure have a ton of bogus support requests, and perhaps you too. Therefore I would like to urge everyone to wait at least until February before using this in any semi-popular package. I will tag a new release of Composer soon so that homebrew users for example also get the PSR-4 support in a timely manner.

For the gory details you can head to the Composer documentation, but it's fairly straightforward to upgrade. To only upgrade at the Composer level only you can just update mappings like this:

// before
{
    "autoload": {
        "psr-0": {
            "Foo\\Bar\\": "src/"
        }
    }
}

// after
{
    "autoload": {
        "psr-4": {
            "Foo\\Bar\\": "src/Foo/Bar/"
        }
    }
}

While if you want to benefit from the shorter paths it is even more simple, you turn the psr-0 into a psr-4 and move all the files from src/Foo/Bar/* into src/*

Finally, a quick note for users of the target-dir property (mostly that is Symfony2 bundle authors as far as I know). The property is now deprecated and using it together with PSR-4 is forbidden since was essentially a hack to support PSR-0 autoloading in non-standard repositories. If you were using that you can easily get rid of it by updating the composer.json as such:

// before
{
    "autoload": {
        "psr-0": {
            "Foo\\BarBundle\\": ""
        }
    },
    "target-dir": "Foo/BarBundle"
}

// after
{
    "autoload": {
        "psr-4": {
            "Foo\\BarBundle\\": ""
        }
    }
}

Thanks for your attention, and again please refrain from jumping on this too quickly to save everyone some trouble. And while I am at it: Happy new year!

January 03, 2014 // PHP // Post a comment

Upcoming Conferences

Here is a short update on the conferences I will attend in the coming months, in case you want to meet or if you are just looking for a good one to attend:

Next month I am very happy to have the chance to visit Tunisia for the first time and go speak about Composer at the Symfony Tunisia event. It will also be the first time I give a talk in French, which freaks me out a bit since I am so used to speaking about technical matters in English. You can therefore expect a bastardized language known as Frenglish.

Right after that is the first SymfonyCon happening in Europe, which will be in Warsaw, Poland. And I hear it is almost sold out so in case you were thinking of going hurry up and grab a ticket now.

In January I will head back to my mother nation of Belgium (insert "while it is still a country" joke) for two events: first I will speak at PHPBenelux about web application monitoring with Heka. Then the weekend after there is FOSDEM where I hope I can also meet a few people from outside the PHP community.

November 19, 2013 // News, PHP // Post a comment

The pyramid of cluelessness

I met this developer employed by a big company, he seems to be happily sitting at his desk watching time pass by. Enjoying what David Graeber recently described as a Bullshit Job. Cashing in big bucks in a race to the bottom of counter-productivity.

Now I do not want to be judgmental, it is his choice and right to piss his life away in an environment that sounds like intellectual death to me. I can not understand but I will accept. What I want to talk about however is something else he mentioned.

He says he could be doing open source work in the evenings but that he is not good enough to produce meaningful work that would affect the world. And I want to call bullshit on this. I have no pretense that my work is affecting the world at large directly, but one thing I am convinced of is that there is enough world-changing work to be done and not enough people doing it.

I visualize this as a pyramid of cluelessness. At the bottom of the pyramid are many people having no clue how they can help move things forward, and further up those few that do have their mind set on specific goals.

The time people have is a finite resource, but by taking care of the small boring things you can in a way donate time to the system and enable people higher up the pyramid to do more meaningful work. Fix a small issue here and there. Do the crap work so that nobody else has to. If everyone at the bottom did their share the ones on top would have a ton more time to direct real change.

I look at this from the angle of open-source because I am knee-deep in that world. However I imagine it applies to any sort of labor that is not made for personal gain but rather the greater good. So go out there and help someone help the world!

August 23, 2013 // Random // Post a comment

Composer: Installing require-dev by default

Jeremy Kendall started a small twitter shitstorm last night by asking why Composer's install command now installs the require-dev dependencies by default. Indeed until a few months ago the only way to install dev requirements was to run composer commands with the --dev flag. This was changed when the require-dev handling was fixed to be a lot more reliable, and the update command started installing dev requirements by default.

A couple months ago when releasing alpha7 I took care to note in the changelog that the install command would also start installing dev requirements by default in the next release. I did that change some weeks ago and now people started to notice.

The rationale behind the change is fairly simple, it's about consistency and ease of use. Consistency between the various commands which now all default to have require-dev enabled. Ease of use because in 99% of the cases, when you type a composer command by hand you should be doing so on a dev machine where it makes sense to have dev requirements enabled. The only case where you want them disabled is when deploying to production or other similar environments. Since those deployments should be scripted, adding --no-dev to your script vs having to type --dev every single time you run composer makes sense. I understand it may create some pain in the short run - although having dev requirements installed in prod is usually harmless - but I truly believe it is the right thing to do if you look at the big picture.

Jeremy also said that install is meant for prod, and while this is not a wrong statement per se, I would like to take the chance to clarify that install is not only meant for prod. Install should be used for prod for sure, because you don't want the prod server to run newer packages than those you last tested on your dev machines. But in many cases developers should also run install to just sync up with the current dependencies of the project when pulling in new code, or when switching to an older feature branch or older release to do a hotfix for example. Developers also might need to run install in some larger teams where only a few select devs are responsible to update the dependencies and test that things still work, while the other devs just run install to sync up with those changes.

And for those that are still not committing their composer.lock file, note that the above paragraph only applies if you have a lock file available in the project's git repository. If you are not sure what this file does please read more about it in the docs.

July 11, 2013 // PHP // Post a comment

Composer: an update on require-dev

Update: the install command now also defaults to --dev, read more about the rationale

Using require-dev in Composer you can declare the dependencies you need for development/testing. It works in most simple cases, but when the dev dependencies overlap with the regular ones, it can get tricky to handle. In too many cases it also tends to just fail at resolving dependencies with quite strange error messages.

Since this was quite unreliable, I set out to rework the whole feature this week-end. The patch has been merged, and it fixes six open issues which is great. The short story there is that it now does things in one pass instead of two before, so it should be faster and a lot more reliable. Also dev dependencies can now impact the non-dev ones without problems since it's all resolved at once.

Workflow changes

I took the chance to change another thing while I was at it. The update command now installs dev requirements by default. This makes sense since you should only run it on dev environments. No more update --dev, the dev flag is now implicit and if you really don't want these packages installed you can use update --no-dev instead.

The install command on the other hand remains the same. It does not install dev dependencies by default, and it will actually remove them if they were previously installed and you run it without --dev. Again this makes sense since in production you should only run install to get the last verified state (stored in composer.lock) of your dependencies installed.

I think this minor change in workflow will simplify things for most people, and I really hope it doesn't break any assumptions that were made in third party tools.

March 04, 2013 // PHP // Post a comment

One logger to rule them all

I called the vote on the Logger Interface proposal last week. When the vote ends next week it will become PSR-3 (since it already collected a majority). The fourth recommendation from the PHP-FIG group, and the first one actually including interfaces/code, which is a great milestone.

You can read the proposal if you have not done so yet, but I wanted to discuss the goal and long term hopes I have in more details here.

Where we come from

Most PHP frameworks and larger applications have in the past implemented their own logging solutions and this makes sense since I think everyone recognizes the usefulness of logs. Traditionally most of those did not have many external dependencies, established libraries were few and far between. Having no logging capability in those was not such a hindrance.

Libraries deserve logs too

Yet in the last couple years, thanks to GitHub allowing easier sharing, composer allowing more reusability, and mentalities shifting slowly to a less-NIH approach, we are seeing more and more libraries used in applications and even by frameworks themselves. This is great, but as soon as you call a library you enter a black box and if you want anything to show up in your logs you have to log yourself.

The availability of the PSR-3 interface means that libraries can optionally accept a Psr\Log\LoggerInterface instance, and if it is given to them they can log to it. That opens up a whole lot of possibilities for tighter integration of libraries with the framework/application loggers. I really hope library developers will jump on this and start logging more things so that when things go south it is easier to identify problems by looking at your application logs.

Take a deep breath

I am sure people will have questions or complaints regarding details of the interface itself, but I hope this helped you see the broader benefits it brings.

December 13, 2012 // PHP // Post a comment

Writing games for fun

GitHub organized a Game Off - or a game competition - back in November. The contest ran for a month and its only limitations were that the game runs in a browser and relate somewhat to forking/cloning/pushing/pulling. And I am here to tell you that no matter what kind of programmer you are, you should take part in such contests!

I am hardly a game developer. I do spend most of my time writing web things in both PHP and JS, and I had not worked on a game in at least 5 years. This sounded like a good opportunity for a little change. It is easy to get stuck in what you do once you do it well, but just like I enjoy playing with other languages every now and then, working on a different product in a familiar language also offers interesting challenges. Game mechanics, canvas rendering and more visual programming are all things that I am not used to work with.

Long story short, I came up with this small simple Split game that I invite you to try out. It was a lot of fun to write, and given the 1 month deadline you do not have time to mess around and let feature creep take over. You have to get things done fast. It is a great exercise both for programming skills and time management/prioritization.

If you are interested in seeing other entries to the contest, there is a full list available but it does contain quite a lot of incomplete and barely playable games. Having gone through most of the list, I can recommend those few games that I enjoyed, mostly because they went out of the beaten path and are trying something new: Echo, Radiance, Mazeoid and Release Cycle.

Mozilla is running a similar contest until February, so there is your chance to get your hands dirty over the Christmas holidays. You have nothing to lose, and building games is both fun and challenging!

December 02, 2012 // Web, JavaScript // Post a comment

Encouraging contributions with the Easy Pick label

One of the barriers to convert users into contributors in an open-source projects is that many people have no idea where to start. They are usually scared to take on large tasks because they are not comfortable enough with the code-base. Yet I think there are ways you can help them as a project maintainer.

One good way that I found to fix this is to tag specific issues that are a good starting point for new contributors. However I think the practice would be even more effective if more projects did the same way, so that people know to look for it.

The way I do it is using a custom Easy Pick label to indicate issues on GitHub that are just that. Easy to pick up tasks, either because they are small in scope, or just don't involve much in-depth knowledge of the project.

The result of this is a much clearer view of issues. On the Composer project for example if you go in the issues tab and then filter by "Easy Pick", you end up with 14 issues listed instead of 170. A much more manageable amount to look at and pick from, and you are empowered by the knowledge that those should all be reasonably easy to work out.

I have also created this label in the Symfony2 issues a while back. As you see both use the same wording and the same yellow that's one of the default colors on GitHub.

I would love to see this spread because I have already seen it bring in a few new contributors. So if you feel like encouraging people to join in on your project give it a try. And if you feel like giving back this week-end, browse the issues of the projects you use and enjoy. See if you find anything you can help them with.

November 16, 2012 // PHP // Post a comment

[1] 2 3 4 Older entries > Last page