Jon Saints

A Better Way to Visualize Your Organization - Trees instead of Squids

Sun, 18 Apr 2021 00:00:00 +0000

Companies often represent their organization structure in a hierarchical org chart. These charts start at the top with the President/CEO. From there, they sprawl down and out through the ranks to those doers and newbies at the “bottom” of the org. Charts like this look like a squid.

I am not a fan of drawing things this way.

An alternative to the hierarchical org chart is the org tree. The org tree flips the org chart upside down. It starts with the President/CEO at the bottom, like the trunk of a tree. From there branches are formed which support smaller and smaller branches. Finally, the leaves represent those at the “top” - the experienced doers and up-and-coming newbies of the organization.

The tree view tells a powerful organizational story which I like much more:

Those at the bottom of this org carry weight. They support. They empower. They are experienced anchors in times of change. Their goal is to ensure the thriving of the people in their branch.
The doers and newbies at the top of the org are valued. They are vital. They are the most supported and empowered to do the work that is critical to the org’s success.
We all matter. We all depend on each other. We co-exist to serve our customer.
Power and influence come from doing your part really well and supporting more and more people with your work

Which organization would you want to be a part of? Which leader’s vision would you want to follow?

Lucas Clay Saints

Tue, 12 May 2015 00:00:00 +0000

On Saturday, May 9, 2015 at 9:19pm we welcomed our second baby boy into the family. Lucas Clay Saints was born in the water at the Birth Center of Boulder to a happy healthy Mom and an excited Dad. He was born after nearly two weeks of cloudy spring rains, capped off by a May snow storm on the Eve of Mother’s Day.

As the second-born, baby Luca joins our family mid-stream, representing new growth and bringing us new love, new perspective and new joy. The name Lucas means bearer of light, light giving or illumination. Our home and our hearts are already filled with new light.

His middle name, Clay, is a shout-out to art and to creation. It’s inspired by his mother and the deep colorful way that she lives. It is a reference to vibrant colors of clay we have seen in our travels around the world. It is a reminder of our roots in the dust and clay, our connection to the earth and our God that created it all. That life was blown into all of us as a gift. You are dust and to dust you shall return, but your spirit endures forever. By giving the name Clay, we hope it grounds him in the knowledge that he is shaped by Potters hands, and that he can be a channel for this source of creativity bringing beauty into this world.

We cherish the bond that Rio and Luca will build as brothers. Rio is already such a sweet, proud, big brother to baby Luca.

The birth experience with Luca was quite different from Rio. The labor was swift and intense lasting only 5 hours. We rushed to the Birth Center in Boulder as soon as Marissa noticed her first contractions. They were already close together and getting stronger by the minute. While Rio’s birth was private and personal, Luca’s birth can best be described as a party. The birth was attended by his three Grandmas, our good family friend Cinda, big brother Rio, our wonderful friend and doula Lara, the midwife and nurse. Plus, as a bonus, Grandpa and Uncle Andrew were eagerly waiting and entertaining Rio just outside the birthing room.

We are astounded already by the immense capacity of the heart to love and expand as our family grows. Welcome to the Saints family, precious Luca.

Love Jon and Marissa

Export Postgres Query to CSV

Thu, 20 Nov 2014 00:00:00 +0000

Postgres makes it easy to export a query a remote server and save the results to CSV on your local machine.

First, connect to the remote database server:

psql -h REMOTE_SERVER_ADDRESS -U DB_USER -W

Then, export a query to CSV… \copy makes the file go to your local machine rather than the database server file system

\copy (select * from my_table limit 10) TO '~/Downloads/export.csv' CSV HEADER

Simple Postgresql Database Connections with Python

Wed, 15 Oct 2014 00:00:00 +0000

UPDATE:

I’ve spent some time using the method that I outlined below and found some problems with it. I now have a new prefered way of querying postgres from python and getting results as a dictionary. You should use the new way in most cases.

Here are the problems with the previous method:

The previous method requires sqlalchemy which is a large dependency
The previous method was using a sqlalchemy connection pool and leaving connections open too long. This was a problem when I tried to us this code in background worker tasks that needed to control when connections are closed (so they could open a connection, read some data, disconnect, run a calculation for a while, then re-connect and write back to the database). Leaving the connections open in a pool in sqlalchemy on a large set of workers was consuming all the available connections to the databse.

My new suggestion is to use psycopg2 and the with statement to ensure that connections are efficient and handled properly by garbage collection when they are no longer in use.

http://initd.org/psycopg/docs/usage.html#with-statement

See also the psycopg2 list of Best Practices:

http://initd.org/psycopg/docs/faq.html#best-practices

There is also a dictcursor which will give you results as python dictionaries:

http://initd.org/psycopg/docs/extras.html#dictionary-like-cursor

OUTDATED: The ideas below are out of date… see above

Deep in the SQLAlchemy docs, I found this gem: a simple way of querying a database from Python that returns query results as a python dictionary.

First, setup a virtualenv for your project and install SQLAlchemy and psycopg2

cd my_app
virtualenv venv
. venv/bin/activate
pip install SQlAlchemy
pip install psycopg2

Now, with just a few lines you can run queries from your python scripts. Results are returned as python dictionaries:

from sqlalchemy import create_engine
engine = create_engine('postgresql://scott:tiger@localhost/mydatabase')

result = engine.execute("select * from users where fname=%s", ('Jon',) )
for row in result:
    print "username:", row['username']
    print "email:", row['email']
    print "First Name:", row['fname']

There are a few advantages to doing things this way:

Easier syntax than the raw psycopg2 driver
Uses SQLAlchemy connection pooling by default
It appears that SQLAlchemy does the right thing and closes connections if an exception is thrown or the script exits successfully.

Note: To install psycopg2 on my Mac I had to first install Postgres.app and set my PATH in .bash_profile to PATH="/Applications/Postgres.app/Contents/Versions/9.3/bin:$PATH"

Note: If you are on Ubuntu you will need to sudo apt-get install libpq-dev python-dev before running pip install psycopg2

Howto install Postgresql and PostGIS on Ubuntu Trusty 14.04 with secure tunnel for connections

Wed, 13 Aug 2014 00:00:00 +0000

Install Ubuntu 14.04

Start with a fresh install of Ubuntu 14.04 Trusty. (AWS or vagrant might be a good way.)

Install Postgresql

On the server run the commands below.

sudo apt-get update
sudo apt-get install -y postgresql postgresql-contrib

Create a database and a user to access it

I prefer to create one user per database for security.

Replace DATABASE_NAME_HERE and USER_NAME_HERE with the values that you want to use

# this will prompt you for a database password
sudo -u postgres createuser -P USER_NAME_HERE
sudo -u postgres createdb -O USER_NAME_HERE DATABASE_NAME_HERE

Test connecting to Postgresql

psql -h localhost -U USER_NAME_HERE DATABASE_NAME_HERE

Postgresql will ask you for your password. Then you should see:

psql (9.3.5)
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)
Type "help" for help.

DATABASE_NAME_HERE=>

To exit type:

\q

Optional: Add PostGIS support to the database

sudo apt-get install -y postgis postgresql-9.3-postgis-2.1
sudo -u postgres psql -c "CREATE EXTENSION postgis; CREATE EXTENSION postgis_topology;" DATABASE_NAME_HERE

Connect to the database securely from PgAdmin3

Now, on your desktop computer install pgAdmin3 and connect securely to the database via an SSH tunnel.

First, open a terminal on your Desktop computer (these instructions are for Mac or Linux)

Open a tunnel using an ssh password or PEM file. You will need to know the username you use to SSH to the server and the server address or ip address.

ssh -L 127.0.0.1:5433:127.0.0.1:5432 SSH_USERNAME_HERE@SERVER_HOST_ADDRESS -N

If you are using a PEM file, then you can add -i option to the SSH command

-i PEM_FILE_PATH_HERE.pem

This will open a secure tunnel between your desktop and the database server. To exit the tunnel, just press Cntrl+C.

Next, open pgAdmin3 and create a new connection using these settings:

Name: (describe what this connection is for here)
Host: 127.0.0.1
Port: 5433
Username: USERNAME_HERE
Password: PASS_WORD_HERE

When you are done using the database close PgAdmin3 and in the terminal where the tunnel is running press Cntrl+c to exit the tunnel.

Howto install libgeos with PHP5 bindings on Ubuntu Trusty 14.04 LTS

Thu, 05 Jun 2014 00:00:00 +0000

Recently I needed to install libgeos with PHP5 bindings on Ubuntu 14.04 LTS. The PHP5 bindings are not available as a package in Ubuntu. Here is how I installed it from source.

apt-get install -y apache2 php5 libapache2-mod-php5 php5-dev phpunit

wget http://download.osgeo.org/geos/geos-3.4.2.tar.bz2
tar -xjvf geos-3.4.2.tar.bz2
cd geos-3.4.2/
./configure --enable-php
make
make install

cat > /etc/php5/mods-available/geos.ini << EOF
; configuration for php geos module
; priority=50
extension=geos.so
EOF

php5enmod geos
service apache2 restart

# note: you might need to run ldconfig here if you see permissions issues. I don't see it on my setup, but users in the comments have
ldconfig

You can test this in a test.php file with

<?php

echo GEOSVersion();

Run Vendored Binaries on Heroku

Mon, 12 May 2014 00:00:00 +0000

Summary

Often on Heroku your app might depend on a system package that is not installed in the base Heroku system. Including an external binary dependency in your Heroku app is easy. These binaries are called vendored binaries.

Here is a quick summary of how to do this:

Compile the binary on the heroku platform using a one-off dyno bash prompt heroku run /bin/bash
Copy the compiled package to your local git repo and place it in a folder .heroku/vendor

On your next push to heroku, the system path will recognize the binary dependencies in your path LD_LIBRARY_PATH.

Background

Recently I needed to install the python Shapely library on a Heroku app. Shapely has a dependency on the C Library GEOS which is not part of the base Heroku system.

Here is how I complied and vendored the GEOS binary for Heroku. You could use the same process to vendor a different binary.

Step #1: Setup your git repository and heroku app locally

I assume you have read and followed the appropriate Heroku Getting Started Guide for your language. I used Getting Started with Python on Heroku

Step #2: Compile binaries on Heroku

With your application configured, run bash on a Heroku process

heroku run /bin/bash

From the Heroku bash command prompt, download and extract the source code for the binary package you need to run on Heroku. In my case it was:

curl -O http://download.osgeo.org/geos/geos-3.4.2.tar.bz2
tar -xjvf geos-3.4.2.tar.bz2

Open the README or the INSTALL doc for your source code and follow the compile instructions with one change. Change the prefix for the install to be something easy to find in your /app folder.

cd geos-3.4.2
./configure --prefix=/app/.heroku/vendor
make
make install

When the compiling is finished you will have a working binary in the /app/scratch-space folder. In my case geos made three folders in the scratch-space folder: bin, lib, and include.

Create an zip archive of the binaries.

tar -czvf /app/geos-3.4.2-heroku.tar.gz .heroku/vendor

Step #3: Add the vendor library to your local source control

Next copy the geos-3.4.2-heroku.tar.gz to your local machine. I did this in two steps. You might have a better method. I used scp to copy the file to another server and then scp again copy the geos-3.4.2-heroku.tar.gz to my local machine.

With the geos-3.4.2-heroku.tar.gz on your local machine extract it into a folder .heroku/vendor in the root of your app. Heroku will automatically add the .heroku/vendor folder to your LD_LIBRARY_PATH.

# from my app root folder
mkdir .heroku
mkdir .heroku/vendor
tar -xzvf ~/Downloads/geos-3.4.2-heroku.tar.gz

Make sure you have the same folder structure as scratch-space in .heroku/vendor. In my case I have three folders: bin, lib, and include.

ls .heroku/vendor
bin include lib

Add the binaries to your git repo and deploy to Heroku. The binaries will be available to your app.

git add .
git commit -m "vendored binary for lib geos"
git push heroku master

Optional Step #4: Use buildpacks instead of adding binaries to local git repo

Instead of Step #3 you can also use the Multi Buildpack and the Vendored Libraries Buildpack build if you prefer not to have vendored binaries in your git repo.

You will want to modify your tar.gz file from Step #2 so that it extracts its contents to .heroku/vendor/BINARY_FILES_HERE.

Then copy your binary from Step #2 to amazon s3. Make the URL public.

Install Multi build pack support. See https://github.com/ddollar/heroku-buildpack-multi

Install Vendored Binary buildpack support and create a .vendor_urls file and add your S3 url to the tar.gz to it.

Note on Installing libgeos PHP bindings on Heroku

Use this command to find where php .so extentions are installed:

php-config --extension-dir

look for geos.so in the folder listed and include this in your zip file

tar -czvf /app/geos-3.4.2-heroku.tar.gz .heroku/vendor/ `php-config --extension-dir`/geos.so

Then add .heroku/php/etc/php/conf.d/geos.ini with

; GEOS extension
extension=geos.so

Commit all of this to your local source and push to heroku. NOTE: Everytime PHP is upgraded on heroku you will need to recompile the geos extension. The php-config --extension-dir will change with each upgrade as well.

Links that helped me figure all of this out:

What to Optimize

Thu, 17 Apr 2014 00:00:00 +0000

If you hang out with developers enough, you will hear this refrain:

“Premature optimization is the root of all evil”

The idea is don’t spend time fixing problems that you don’t have because you are more likely to make your code clumsy and more complicated than it needs to be.

In my recent work on www.tomnod.com we were hit with a huge amount of traffic during the search for missing Malaysia airlines flight MH370. From that experience, I can tell you that if premature optimization is evil…

“Optimizing under fire is only slightly more fun”

The question becomes what design decisions, which foundations can I lay today, that will let me scale my app to 100x or 1000x the load it handles today. Here is what we learned scaling Tomnod in March of 2014.

It is never premature to optimize these things:

Create a Productive Error Page The page should be static html and javascript. Make sure it never includes hits to the database. This static page lets you provide useful information to the hungry crowd banging on your app’s front door and gives you breathing room. While that page is displayed you are free to do updates and scale the system behind the scenes. Consider it one of the top priority features of your system. Test it in your releases. Error pages can be useful. An email signup form on our error page (which posted to our SendGrid mailing list) collected email addresses and twitter followers who wanted to be updated as soon as the site was back up and usable. The amount of people who sign up on the error page when our site was “down” amounted to 3x what our entire database of contacts had been before. This made for highly productive downtime and kept our users informed.
Create a Method for Sharding your Users During our traffic onslaught, an engineer from Facebook suggested the #1 thing we should do is “hash or shard our users”. By sharding we mean the ability to show a different version of your app to a fraction of users. In our case, needed to bring a bigger database online to support load. We knew the database needed time to warm up its in memory cache. This could only be done with queries to the data. If we had allowed all of our users to hammer the database before the in memory cache was ready, it would have crashed the system. So, we sharded our users. We let 5% of users access the app while the other 95% saw our productive error page. The cache warmed a bit. We then let in 10%… 20%… 50%… 75%… and finally 100% of users. Turns out sharding is useful in other ways. You can use it to launch and load test new features. Show new feature to x% of users and see if you get bug reports or unexpected load spikes. When things look good with your sample users, then roll the feature out to everyone.
Know Your Outs For every component in the system, know and test the next size up. In some cases this might be testing that migrations to the next bigger instance size actually work. In others cases, it might be testing that adding nodes to horizontal worker or web farms works well. Test this frequently. Always make sure you know your next step for each component of your app. Easy to do. Dev environment might be small. Production a bit bigger with more nodes. Moving between them should be push button no crazy deployment hoops to go to larger system.

In sum: with a productive error page, ability to shard users, and by knowing your outs, you should be ready to scale your app to the next order of magnitude. Doing much beyond that might be premature optimization. Doing less might mean a huge lost opportunity when the hungry crowd comes knocking.

Lafayette Teen Tech Week 2014 Hackathon

Thu, 20 Mar 2014 00:00:00 +0000

I helped teach the hackathon at Lafayette Teen Tech Week.

The students learned to encrypt and decrypt Ceasar ciphers. The decrypted messages sent them on a scavenger hunt around the Lafayette Public Library.

Here are some photos of Lafayette Teen Tech Week 2014!