Automated Deployment of PHP Applications using git

Posted on March 5, 2011 by Jon

Heroku is sweet hosting platform which offers a pure git deployment workflow. I love using it to deploy my Rails projects. I miss using when I deploy a PHP application.

Lets fix that…. Here is a quick heroku-ish git deployment workflow for your PHP applications. We will be using WordPress for this example. The same steps can be adapted for Drupal or any other PHP application.

Our goal is to development and test new features, themes, and plugins for WordPress locally on our development machine. When changes are working properly, we want to deploy them to our live site with one command:

git push origin master

#1: Set up password-less login to your webserver with SSH keys

To run deployments we will need to access our live webserver without a password. Follow the steps in Setting up ssh public keys to login to webserver without a password.

For each user or machine from which you want to allow deploys you will need to install its SSH key using the steps above.

#2: Setup Code Repository on the Webserver

I connect to the webserver via SSH. My home directory is /home/saintsjd.

I have a /home/saintsjd/www folder for my website and PHP files. I also have a /home/saintsjd/code folder for my git repositories.

cd into /home/saintsjd/code and create a folder called blog.git

cd /home/saintsjd/code
mkdir blog.git

Next, initialize a bare git repository

cd blog.git
git init --bare blog.git

This gives me a git repository to store my code at /home/saintsjd/code/blog.git. I then disconnect from the SSH connection to the webserver.

#3: Setup Development Box

I have php, mysql, and apache running locally on my laptop. I create my WordPress themes, plugins, and updates on my laptop. After I test them, I deploy them to the live webserver for the world to see.

My local website folder is /home/jon/Website. To get started I clone my git repo locally:

git clone ssh://saintsjd@saintsjd.com/home/saintsjd/code/blog.git ./blogdev

This give me:

/home/jon/Website/blogdev

Now i can start filling in code!

Into the /home/jon/Website/blogdev/ folder I copy in all the files for WordPress and run the installer.

Next, I need to add a gitignore file to the project. I use the one here. https://github.com/github/gitignore/blob/master/Wordpress.gitignore. I copy the contents of that file into a .gitignore file in /home/jon/Website/blogdev/.gitignore. I then:

git add .
git commit -m "inital commit"

To send my changes to the server I type:

git push origin master

#4: Setup Live Webserver

Now I install WordPress on my server. I SSH to my server. There my website directory is /home/saintsjd/www. I need to remove this folder so I backup everything in this directory. Then I remove it and recreate it using git clone. I type:

rm -rf /home/saintsjd/www
git clone /home/saintsjd/code/blog.git /home/saintsjd/www

I then browse to my website http://www.saintsjd.com and run the wordpress installer so the database gets installed on the server.

IMPORTANT: Protect your .git repository from prying eyes

I don’t really want others reading my source code from the git repository via http://www.saintsjd.com/.git/

I prevent this by running:

chmod -R og-rx /home/saintsjd/www/.git

I also add a .htaccess file to the .git folder containing:

Deny from all

#5: Automated Deployment Script

Now that wordpress is installed on both my local machine and my live webserver, I want to be able to deploy any changes I make on my local copy with one git push command. I can do this using a Git hook.

I SSH to my webserver and create a file /home/saintsjd/code/blog.git/hooks/post-receive

touch /home/saintsjd/code/blog.git/hooks/post-receive

I make the file executable

chmod u+x /home/saintsjd/code/blog.git/hooks/post-receive

Then I edit the file

nano -w /home/saintsjd/code/blog.git/hooks/post-receive

and place this code in it (adjusting the config variables to match your server settings)

#!/bin/bash
#CONFIG
LIVE="/home/saintsjd/www"

read oldrev newrev refname
if [ $refname = "refs/heads/master" ]; then  
  echo "===== DEPLOYING TO LIVE SITE ====="  
  unset GIT_DIR  
  cd $LIVE  
  git pull origin master  
  echo "===== DONE ====="
fi

#6 Let the (heroku-like) Fun Begin: Automated Deployment for WordPress

Now that everything is setup. I am free to develop new themes, install new plugins and update on my local development box. Once everything is working properly locally commit and push. The post-receive hook automatically deploys the code changes to my live webserver for me.

Here are my steps:

Make changes to my local site’s code: new theme, new plug-in, etc
test, test and test so more!
When I am happy I commit to the local git repository:
```
git add .
git commit -m "new theme css for header"
```
I push and deploy to the server using one command:
```
git push origin master
```
After the post-receive hook moves the new code to the live website. I open http://www.saintsjd.com/wp-admin on the live server to trigger and database updates that need to run

That’s it… heroku-ish git deployment for WordPress websites.

Continuous deployment for WordPress using git and fabric

Posted on January 30, 2011 by Jon

——

UPDATE: This post is out of date. Please see the new and improved version of this page Automated Deployment of WordPress using git

——-

Continuous Deployment is a strategy you’ll want to implement for any successful web project. According to Eric Reis, “It’s a process whereby all code that is written for an application is immediately deployed into production. The result is a dramatic lowering of cycle time and freeing up of individual initiative. It has enabled companies I’ve worked with to deploy new code to production as often as fifty times every day.”

This article describes one method for configuring a continuous deployment infrastructure using git, fabric, and SSH. Its fairly lightweight and should work with basic hosting systems.

Our goal here is to create new features locally on a development machine, commit happy changes to a code repository, and finally deploy the changes to a live web server with ease using server side scripts. It makes deploying new changes to a live wordpress website so easy you’ll want to do it 50 times a day!

Install Deployment Tools on your Dev Machine:

I use fabric for all of my non-rails applications (PHP, Python, etc). Fabric allows you to execute command on remote servers by running a local script on your machine.

Its available at http://docs.fabfile.org/

To install on my local Ubuntu Linux development machine I just:

sudo apt-get install fabric

Set up SSH keys

To run deployments we will need to access our remote SSH server without a password. Follow the steps in Setting up ssh public keys to login to SSH servers without a password.

For each user or machine from which you want to allow deploys you will need to install its SSH key using the steps above.

Use git to Setup Code Repositories

I use two separate git repositories: one for my wordpress website and another to track changes to my fabric deployment scripts. I store my repositories on my web server in the folder

/home/jon/code/saintsjd.com
/home/jon/code/saintsjd.com/www.git (is my wordpress website git repo)
/home/jon/code/saintsjd.com/deploy.git (is my repo for deployment scripts)

So I cd into /home/jon/code/saintsjd then:

git init --bare www.git
git init --bare deploy.git

Setup a Local Development Space

I have php, mysql, and apache running locally on my development machine. My local website folder is /home/jon/Website. There I create a folder /home/jon/Website/saintsjd.com. I then clone my git repos locally:

git clone ssh://jon@saintsjd.com/home/jon/code/saintsjd.com/deploy.git ./deploy
git clone ssh://jon@saintsjd.com/home/jon/code/saintsjd.com/www.git ./www

This give me:

/home/jon/Website/saintsjd.com/www
/home/jon/Website/saintsjd.com/deploy

Now i can start filling in code!

Install WordPress in Dev Space

Into the /home/jon/Website/saintsjd.com/www/ folder I copy in all the files for wordpress and run the wordpress installer. After WP is installed properly on my local machine I initialize a git repository to start tracking my code.

First, I need to add a gitignore file to the project. I use the one here. https://github.com/github/gitignore/blob/master/Wordpress.gitignore. I copy the contents of that file into a .gitignore file in /home/jon/Website/saintsjd.com/www/.gitignore. I then:

git add .
git commit -m "inital commit"

To send my changes to the server I type:

git push origin master

Install WordPress on the server

Now I install wordpress on my server. I ssh to my server. There my website directory is /home/jon/public_html. I backup everything in this directory then I remove this directory and recreate it using git clone. I type:

rm -rf public_html
git clone /home/jon/code/saintsjd.com/www.git public_html

I then run the wordpress installer.

Writing the deploy scripts

With wordpress installed locally and on my server. I am ready to write a fabric deploy script to easily synchronize changes I make locally to my live version of wordpress on the web server.

On my development machine I:

cd /home/jon/Website/saintsjd.com/deploy

There I create fabfile.py and fill the file with the code below:

from fabric.api import *
env.hosts = ['jon@saintsjd.com']
WEBSITE_PATH = "/home/jon/public_html"

def deploy():
    with cd(WEBSITE_PATH):
        run('git pull')

You will need to change the env.hosts value to use your ssh login for your server. You will also need to change WEBSITE_PATH to reflect the path to your website on the server.

Now when ever we want to deploy changes to the live website we can just

commit our changes with git and push
go to the command line and
```
fab deploy
```

If you want to (I do), you can get fancy and make a backup before each deployment. This is great in case something goes wrong. My fabfile.py with backups looks like:

from fabric.api import *

env.hosts = ['jon@saintsjd.com']
LIVE = "/home/jon/www/saintsjd.com"
BACKUPS = "/home/jon/backups/saintsjd.com"
MYSQL_USER = ""
MYSQL_PASSWORD = ""
MYSQL_DATABASE = ""

def deploy():
    backup()
    with cd(LIVE):
        run('git pull origin master')

def backup():
    run('mysqldump -u %s -p %s --add-drop-table --password=%s  > %s/saintsjd.com-database-current.mysql' % (MYSQL_USER, MYSQL_DATABASE, MYSQL_PASSWORD, BACKUPS) )
    run('/usr/bin/nice tar -czf %s/saintsjd-`date +%%Y%%m%%d%%H%%M%%S`.tar.gz %s/saintsjd.com-database-current.mysql %s' % (BACKUPS,BACKUPS,LIVE) )

With fabfile.py written and working I add it to my git repo and push to the server for safe keeping:

git commit -am “initial commit”
git push origin master

Protect your .git folder on the live website using .htaccess

I don’t really want others reading my git repository via http://www.saintsjd.com/.git/ I prevent this by running:

chmod -R og-rx /home/public_html/.git

I also add a .htaccess file to the .git folder containing:

Deny from all

Continuous deployment of WordPress using git and fabric

My continuous deployment for the website is all set. Now I

make changes to my local site’s code: new theme, new plug-in, etc
test, test test
when I am happy I commit to the local git repository
I push to the server
I then change directory into my ../deploy folder and run
```
fab deploy
```
on the command line to send the change to the live website
finally I open wp-admin on the live server to trigger and database updates that need to run

What is a bare git repository?

Posted on January 26, 2011 by Jon

What is the difference between a repository created using the “git init” command and the “git init –bare” command?

Repositories created with “git init” command contain 2 things: the .git folder repository (or code history) and also real life working copies of your source code files. You can think of this type of repository as a working directory. Its a folder with code history stored in .git folder and all the source files as well. You can work in this directory changing source files and save your changes using the “git add” and “git commit” commands.

Bare repositories contain only the .git folder and no working copies of your source files. If you “cd” into a bare repository you find only the .git folder and nothing else. A bare repository strictly contains the version history of your code.

Why use one or the other?

Well, a working repository is for just that… working. Its where you will actually edit files, add files, delete files to create your project saving your changes to the local .git repo along the way. If you are starting a project in a folder which will contain your code as well as a git repo for tracking changes use “git init”. Also if you “git clone” a repository from some one else you will be given a working repository with the .git folder and copies of the working files.

A bare repository is for… sharing. Bare repositories are shared repos where developers can send their local changes from their working copies of projects to the world or to the rest of their team. By using the “git push” command, you are able to send changes in your local working copies of a project to a centralized shared bare repository. Other developers can then “git pull” to receive the changes that you made. If you are collaborating with a team of developers or you need to work on a project from multiple computers , then a bare git repository is what you need in between to coordinate the distributed development.

To summarize: the working directory created with “git init” or “git clone” is my local copy of a project. Its where I add my changes to a project’s code files and test. After I am happy I “git add” and “git commit”. Then I “git push” to a bare repository, usually on another server, so that other developers can access my changes. When I want to update my local working copy I “git pull” to receive the changes that other developers have made.

2010 Reading Off to a Great Start

Posted on March 20, 2010 by Jon

I have greatly enjoyed each of the books I have read so far in 2010. Here is the list so far, and my few sentence take-away from each.

Siddhartha by Herman Hess
Every time I read this book I feel inspired and love it. ”What other path was there?”

Thousand Splendid Suns by Khaled Hosseini
Wonderfully written characters immerse you in the recent history of Afghanistan.

The 4 Hour Work Week by Timothy Ferris
As an entrepreneur, make it your goal to decrease the amount of time you need to run your business. If you start working 12 hour days make it the point of your business to find efficiencies and ways to do the same in 4 hours. Its a great mentality that can lead to profitability, new ideas, and a healthy life style.

Born to Run by Christopher McDougall
The more I think about this book, the more I see running as a metaphor for life. We are endurance animals meant to persistently peruse solutions to our problems wearing them down not with bursts of sped or luck but with persistence and presence over time.

I love when the book talks about building endurance… getting to know the Beast (fatigue). Fatigue is not something you can beat in one training run or out sprint or out luck. As you train for endurance races, each time, its a chance to get to know the Beast to spend a little bit long time in its presence to wear it down just a little bit and build up your tolerance/comfort to it. So many things in life are endurance races.

The final thought from the book: “We do not get old because we run. We get old because we stop running” Find ways… any way… to stay patiently and persistently active.

Rework by Jason Fried
The best summary of this wonderful take on business is a review in the introduction: “This book’s assumption is that an organization is like a piece of software. Editable. Malleable. Sharable. Fault-tolerant.” Businesses, the smart ones, stay nimble and simple. Organizations have versions and features and milestones. When I stumbled into business, the only proven way I knew to build things was following the software development model. I have said, though not as clearly as this book, that the principles of good software development often apply to good business development. Release early and often. Create and Celebrate bite-sized milestones. Under do the competition. Planning is Guessing. Half but not Half Assed. This book presents an eloquent philosophy for business.

Next on the reading list is Food Rules: An Eater’s Manual by Michael Pollan.

The Magento (rocky road) Upgrade to 1.4

Posted on March 13, 2010 by Jon

Magento released version 1.4 sometime ago. Before upgrading I watched the forums… sure enough, I saw upgrades failing and sites crashing left and right. Almost one month later, today I tried the upgrade from 1.3 to 1.4 using magento connect on a development installation of our shop.

The first upgrade failed. With errors in the Connect console. I found that I needed to disable the Blank theme. I switched to the default theme and removed the Blank theme using Magento Connect. After that upgrade proceeded without error… almost.

When I logged back into admin after the upgrade I experienced a common problem many other users have had. I saw only the Dashboard and the System menus in the admin section. Fortunately, http://www.magentocommerce.com/boards/viewreply/216653/ helped me solve the problem. After that, I could see all the links in Admin.

Next, an error appeared, saying that indexes where out of date. Clicking the link, I was able to update indexes without trouble.

Next I tried to enable my old custom theme for magento 1.3. The file structure of the new themes were just too different. And I notice that Magento Connect had not copied in all the new theme files compared to a fresh install of 1.4. So, back to square one I went:

Switched to the Default Default theme
Disable cache
Backed up everything!
http://www.magentocommerce.com/boards/viewthread/79499/
Update product indexes System -> Index management

At this point all the store data seemed to be working properly. From what I read in the forum, its best to re-create your old custom theme from scratch. So I did:

Create folder /skin/frontend/myPackage/default
Create folder /app/design/frontend/myPackage/default
Copy /app/design/frontend/default/blank/etc and /app/design/frontend/default/blank/locale to /app/design/frontend/myPackage/default
Copy /skin/frontend/default/blank/css and /skin/frontend/default/blank/images and skin/frontend/default/blank/favicon.ico to /skin/frontend/myPackage/default
Enable my theme and test /admin/system_config/edit/section/design/
From there I slowly copied in the CSS and images from the theme I had designed from magento 1.3

Upgrading Magento to 1.4 was painful and time consuming even with all the help of the poor souls who had crashed their sites before me. That said, using magento still seems to be faster and less time consuming that building my own ecommerce solution from scratch. Given the time it takes to make Magento perform well and maintain its updates, Shopify’s pricing does look more attractive and reasonable.

Run Drupal from the Command Line

Posted on September 9, 2009 by Jon

Often I need to perform mass operations on Drupal content (for example: importing content from an old CMS database system or making automated changes to a large set of nodes).

To do this I use the script below to run Drupal and access the API from the command line.

Create a file called update-node-12-title.cli and put inside of it:

<?php

// variables
$drupal_home = '/path/to/your/drupal';

// set some server variables so Drupal doesn't freak out
$_SERVER['SCRIPT_NAME'] = '/script.php';
$_SERVER['SCRIPT_FILENAME'] = '/script.php';
$_SERVER['HTTP_HOST'] = 'example.com';
$_SERVER['REMOTE_ADDR'] = '127.0.0.1';
$_SERVER['REQUEST_METHOD'] = 'POST';

// act as the first user
global $user;
$user->uid = 1;

// change to the Drupal directory
chdir($drupal_home);

// run the initial Drupal bootstrap process
require_once('includes/bootstrap.inc');
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

//now we have full access to the Drupal API
$node = node_load(12);
$node->title = "Hello, this is the new title";
node_save($node);
And from the command line I can run the script:
/path/to/php update-node-12-title.cli

Magento Ecommerce, “Please Wait”… and Wait… and Wait

Posted on September 1, 2009 by Jon

I installed Magento (http://www.magentocommerce.com) on a dedicated server with 2 Xeon processors and 4 GB of RAM. Often when adding products, editing them, or choosing a category in the admin Magento takes 5-10minutes to complete the Save operation. A “Please Wait” modal dialog appears for long periods of time while Magento is thinking. The operation always completes without error, but It often takes 5-10 minutes just to save a simple product.

I asked for a solution in the Magento Forums and found one…

Following the the instructions in the DYI tuning section here
http://www.magentocommerce.com/blog/comments/performance-is-key-notes-on…

I installed each optimization individually to see which made the difference. I found the following:
- Apache KeepAlives enabled. This seems to help the category selection pages load much much faster.
- Modify the configuration for your MySQL server. No noticeable gain on performance. Products still take 5 minutes to save.
- APC or XCache. This one fixed the problem.

Installing xcache seems to have fixed the problem. Products now save in about 2 seconds.

Edit on November 15th, 2009:

I found this in the Magento System Reqs:
“Memory_limit no less than 256Mb (preferably 512)”

Also please see:

The Lean Startup Talk by Eric Ries

Posted on August 20, 2009 by Jon

Talk by Eric Ries at the Hotel Boulderado last night on The Lean Startup.

The talk was framed as a tale of two startups that he worked for. One was a failure the other was a success.

Startup #1 the Failure

On the surface the failure startup did everything right. They had a great business plan, the best team of people he had ever seen assembled, a well engineered product, and perfect launch publicity. The companies $40M failure was blamed on a set of “secret beliefs” that everyone at the company had but no one recognized outloud.

The “seceret beliefs” where:

That we know what customers will want: they spent 4 years building one big product that was going to change the way people used the internet. The product was perfect from an engineering standpoint. But they never got feedback from the market along the way. This may sound obvious now, but at the time 2001 this is how things where done. They had one release and not “pivot points” along the way to adjust their ideas to the demands of the market.
That we can predict the future: The team on startup #1 wrote a perfect business plan. They also executed it well. They judged their success through the 4 years of development based on how well they were sticking to the plan. Measuring success by sticking to the plan, assumes that the authors of the plan can magically see into the future and predict a path to success. No one can do this. Remember that just sticking to a business plan does not mean progress. The market may have moved somewhere else from when you wrote your perfect plan. Without feedback from it you will never know where it went.

What is real progress? We will talk about that later.

Summary: Startup #1 launched with perfect product, team, and publicity. But, they burned through their money before they had a chance to pivot the business to market feedback.

Startup #2 The success:

Very Simillar product to Startup #1 but… 10M in revenue 2007… much more now

Had a business plan (that went 6 months into the future instead of 4 years). Planned to release a Minimum Viable Product, no matter what state it was in, in six months of starting to write code. Good Team. No Publicity for first buggy beta terrible release. A few… 2 or three people actually downloaded and played with the release that was more likely to crash your computer than to work properly. They received invaluable feedback from these users. One thing they found was that the code Eric Reis had been working on for six months was a total waste. They were able to refocus his energy to other matters quickly. Now days, he recommends this feedback happen within a month or two, weekly, or even daily. 6 months is way to long now.

Another great success of Startup #2 was that they did not divide the team into departments. There was no support team, dev team, management team, HR, etc. Instead everyone was cross discipline. You could divide the focus of their work into two categories:

Those focused on delivering the product to customers and the market and bringing feedback back to their work
And those focused on taking the feedback and engineering solutions to it.

Startup #2 success also came from Continuous Deployment. He says they have created test coverage that allows everyone on the team to deploy to production servers new small incremental features up to 50 times per day. They try to have newbies deploy to production on their first day. This continual deployment process evolved over time, learning from mistakes one by one… but you have to make mistakes to learn from them. They do not do weekly or monthly releases. Small deploys allow problems to be isolated quickly instead of wasting a whole release on something that might need to be reverted anyway. Startup #2 practices regularly reverting to be sure their revert process is fast, as quick as deploy, and always works. Every time a mistake is made a test is written to address it and a the human cause of it is addressed in person, possibly with a group. They use SimpleTest, Selenium, and Nagios to build up a product immune system that detects problems before they go live, and that can heal and revert quickly.

Through startup #2, Eric learned his definition from progress in a startup company. Progress is “Validated Learning” on behalf of an employee. This should happen each day for each employee. The best way to achieve progress is to setup systems, processes, and culture that allows for validated learning to take place regularly, rapidly.

Finally Eric asked us to close our eyes and think of one thing we would like to improve at our current workplaces. That one thing he said should be done in one day. Think of it… plan it… do it tomorrow.

Here is Eric’s Blog: Startup Lessons Learned

saintsjd

The blog of Jon Saints

Tag Archives: open source