Michael's musings

Thu, 08 May 2008

Unit testing with model security

As explained in: http://www.embracingchaos.com/2007/05/model_security_.html the problem with model security is that it requires that you have a controller for your unit tests to work.

We solved this with a mock object we called "AlwaysAdmin"

class AlwaysAdmin

  # this routine is most useful when using script/console!
  def self.fake_out_login
    Thread.current[:session] = Hash.new
    User.current=AlwaysAdmin.new
  end

  def logged_in_as_admin?
    true
  end

  def logged_in_as_role1?
    true
  end

  def logged_in?
    true
  end
  def admin?
    true
  end
end

In our unit tests where we don't care at all about model security, we do:

class MyTest < Test::Unit::TestCase
  fixtures :myfixtures

  def setup
    AlwaysAdmin.fake_out_login
  end
end

Bruce Perens says:

I'm rewriting it right now, as part of the software development of new blog software at http://new.technocrat.net . There is a source link at the bottom of the page there that returns source.tgz containing a tar of the running software version created every time I restart the application. Only the User and Session classes are done so far, not ModelSecurity. They are under vendor/plugins/user in the source.tgz archive. User and Session are RESTful now, and are an Engine so they can have views and routes.

converting mov to mp4

Today, I converted some MOV (Quicktime) files from the railscasts.com webcast site to mp4, which my Neuros OSD can read.

I couldn't do this with ffmpeg from debian stable, because it doesn't have the aac codec. I did it with ffmpeg from http://debian-multimedia.org/

I used:

gimli-[/ssw/cadillac/movies/railscasts] mcr 1075 %ffmpeg -i 004_move_find_into_model.mov 004_move_find_into_model.mp4
FFmpeg version SVN-rUNKNOWN, Copyright (c) 2000-2007 Fabrice Bellard, et al.
  configuration: --prefix=/usr --libdir=${prefix}/lib --shlibdir=${prefix}/lib --incdir=${prefix}/include/ffmpeg --enable-shared --enable-libmp3lame --enable-gpl --enable-libfaad --mandir=${prefix}/share/man --enable-libvorbis --enable-pthreads --enable-libfaac --enable-xvid --enable-libdts --enable-amr_nb --enable-amr_wb --enable-pp --enable-libogg --enable-libgsm --enable-x264 --enable-liba52 --enable-libtheora --extra-cflags=-Wall -g -fPIC -DPIC --cc=ccache cc --enable-swscaler
  libavutil version: 49.4.0
  libavcodec version: 51.40.2
  libavformat version: 51.11.0
  built on Feb  4 2008 14:45:57, gcc: 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '004_move_find_into_model.mov':
  Duration: 00:02:02.8, start: 0.000000, bitrate: 328 kb/s
  Stream #0.0(eng): Audio: aac, 44100 Hz, stereo
  Stream #0.1(eng): Video: qtrle, rgb24, 800x600, 29.97 fps(r)
File '004_move_find_into_model.mp4' already exists. Overwrite ? [y/N] y
Output #0, mp4, to '004_move_find_into_model.mp4':
  Stream #0.0: Video: mpeg4, yuv420p, 800x600, q=2-31, 200 kb/s, 29.97 fps(c)
  Stream #0.1: Audio: aac, 44100 Hz, stereo, 64 kb/s
Stream mapping:
  Stream #0.1 -> #0.0
  Stream #0.0 -> #0.1

Tue, 06 May 2008

VMware server 2.0 beta

I installed VMware server 2.0 beta (p-84186) on a machine. It doesn't use the vmware-server-console application to talk to it.

It uses a web interface.. I found the web interface uninteresting. I tried to open an existing VM that I had built on it and it failed.

Do I really want to run a VNC client in my web browser to access the consoles? No. Do I think that that they will drop Linux support and become IE7 only? Yes.

Alas, this machine doesn't have VT extensions (or at least, doesn't have a BIOS that supports that for the XEONs inside), so my original plan to run XEN with HVM will fail. Unfortunately, I need to support some native kernels due to customized patches to the kernels, so I can't use paravirtualization, as much as I'd prefer to.

Summary: I don't like VMware server 2.

Thu, 01 May 2008

testing with authenticated resources

So, you add new code to authenticate various access, and your functional tests stop working.. You need to authenticate!

Add this to your test_helper.rb:

  def user1_login_details
    'Basic ' + Base64.encode64('user1'+':'+'user1pw')
  end

  def user1_login
    @request.env['HTTP_AUTHORIZATION'] = user1_login_details
  end

(the details are split out because we used them in other places too)

then add a call to "user1_login" to your functional tests, or if all of them need it, to the "def setup".

Remember to write tests for not being logged in, and for situations where the logged in user should not have access to some protected resource.

embrace and extend of php/mysql application

We had a prototype for a user-facing service written in PHP.

After some discussion, we decided that we really wanted the Test Driven Development that is so easy in RoR, and even thought we could do it in PHP, as our PHP developers had no experience using any kind of object-oriented or template driven PHP development (the code was totally raw PHP), we decided that we would gradually/incrementally replace the code with Ruby-on-Rails.

The first task was to create an empty RoR application. We set it all up with Capistrano, SVN, etc. and deployed it to a test application server. We setup Apache and Mongrel as normal, and made sure to enable PHP in Apache.

We then copied all of the existing web site into Rails' public directory, and checked it into SVN. We then deployed again, and took a look at the web server, and lo-and-behold, there was our legacy PHP application.

On our laptops, we want to develop, so we need Apache to get the PHP code running. As described in:

http://www.sandelman.ca/mcr/blog/2008/04/30#starting_local_mysql_database

we had some code that started Apache+Postgresql, and this time we run just the Apache code. I decided that test/cluster/etc was needlessly deep, and I installed things into test/etc instead:

See http://www.sandelman.ca/mcr/ruby-files/embrace/ for files: apache2.conf.in, php.ini.in (a snippet), portnum.sh, runweb.sh.in, shutit.sh.in, and apache.rake.

Place apache.rake into lib/tasks, it has:

namespace :apache do

  desc "Start web server and local server"
  task :start do
    puts "Starting up Apache"
    system "test/etc/runweb.sh"
    system "sh -c 'script/server -p 9000 &'"
  end

  desc "Setup web server"
  task :setup do
    puts "Setting up Apache"
    system "test/etc/fixup.sh"
  end
end

Start out (after a raw checkout) with:

rake apache:setup

This will process the .in files into proper config files for apache. It will find the right modules directories for apache, as this sometimes varies between distros.

Then run:

rake apache:start

This will start apache on port 8000 + your numerical userid. This picks a consistent port, but lets' multiple users run simultaneously if they happen to be logging into a common work machine.

Point your browser at http://localhost:8xxx/ , and you should see your application running. A copy of mongrel can be started under the normal load balancer, setup to run in development mode. We commented that line out as actually, we prefer to have it in the foreground in a window, in case we want to do debugging.

Using ssh+svn URLs for capistrano

We use SVN (we'd like to use git, but we haven't convinced TracGitPlugin to work for us yet) with SSH access to deploy our application.

Our application servers have SSH access to our master repository, but they must login with public keys only, and we prefer that they do so with the user id of the person doing the deploying.

So we have in our config/deploy.rb:

set :repository,  "svn+ssh://"+ENV['USER']+"@svn.example.com/path/to/proj/repo//trunk"

Each of our application servers have a special login accessed by SSH public key authentication, which holds our source code. We don't use sudo, but we probably should. Eventually, with git, we would expect to do a git push to our application server, and then it could do a local checkout. That would permit our application servers to operate with access to the SVN master system.

To get the public key authentication from application server to SVN master we needed to have SSH authentication agent forwarding. With Rails 1.2.3, we had to hack the file:

/var/lib/gems/1.8/gems/capistrano-2.0.0/lib/capistrano/ssh.rb

and every works the way we wanted.

strange complains about class being obsolete

We have some ActiveRecord (and ActiveResource) models that are subclassed using the "type" field. We would attempt to make multiple selection boxes using collection_select for them, and we would get errors about the #type method being obsolete, or worse, we'd get complaints that we couldn't turn a class into a string.

The problem is that Rails 1.2.3 (at least, we haven't seen this with 2.0.2) uses the 'type' field for subclassing, but ruby's Object used to have a .type method, now renamed .class.

Since the ActiveRecord would synthesize the attribute accessor methods, the type method wasn't made until runtime, and since it already exists in Object, one wasn't synthesized.

To get around this, we added the following to our models (to the parent class):

  def type
    self[:type]
  end

I tried removing this with rails 2.0.2 to generate the precise error, but it appears to be fixed. I may update this entry with the exact error by checking out some older code that uses 1.2.3.

starting local mysql database

We transitioned from using sqlite on devel laptops (with inherent very low hassle) to having to run mysql everywhere to as our migrations and some of our complex database operations just didn't work with sqlite3.

Setting up all the database junk in mysql in annoying. Some developers have laptops with other mysql programs running on them too, and some development machines are shared by multiple projects and multiple developers.

It's much nicer to just run a "local" mysql. This is relatively easy to do with postgresql, but we are locked into mysql due to corporate desire for someone to buy support from.

Building local clusters for mysql is annoying, but not ultimately that hard. We use several shell scripts, but we think that these could be turned into rake tasks easily, and we intend to do that soon.

These scripts were inheirited from a PHP/Postgresql project call "ITERation" (a TBS.gc.ca project).

There is a Makefile in test/cluster/Makefile

SRCDIR=$(shell cd ../..; pwd)
...
%.sh:	%.sh.in Makefile
	sed \
		-e 's,@SRCDIR@,'${CortlandSRCDIR}',' \
		$< >$@
	chmod +x $@

dbflush: etc/initdb.sh etc/rundb.sh etc/shutdb.sh
	-[ -f run/mysqld/mysqld.pid ] && etc/shutdb.sh
	rm -rf ${SRCDIR}/db/mysql
	etc/initdb.sh
	etc/rundb.sh

dbinit: etc/initdb.sh
	etc/initdb.sh

dbstart: etc/rundb.sh
	etc/rundb.sh

The first is test/cluster/etc/initdb.sh.in.

#!/bin/sh

SRCDIR=@SRCDIR@
USER=${USER}
RUNDIR=${SRCDIR}/test/cluster/run
LOGDIR=${SRCDIR}/test/cluster/log
PIDFILE=${RUNDIR}/mysqld/mysqld.pid
ROOTPW=therootpw
mkdir -p ${RUNDIR}/mysqld
mkdir -p ${SRCDIR}/db/mysql
mkdir -p ${LOGDIR}/mysql
/usr/bin/mysql_install_db --basedir=/usr --datadir=${SRCDIR}/db/mysql --pid-file=${PIDFILE} --skip-external-locking --socket=${RUNDIR}/mysqld/mysqld.sock --log_bin=${LOGDIR}/mysql/mysql-bin.log

# now start the DB.
# have to start up mysql with TCP networking enabled!
/usr/sbin/mysqld --basedir=/usr --datadir=${SRCDIR}/db/mysql --pid-file=${RUNDIR}/mysqld/mysqld.pid --skip-external-locking --socket=${RUNDIR}/mysqld/mysqld.sock --port=3307 --log_bin=${LOGDIR}/mysql/mysql-bin.log &
sleep 10

/usr/bin/mysqladmin -h 127.0.0.1 --port=3307 -u root password $ROOTPW
echo "update user set host='%' where host='localhost';" | mysql -h 127.0.0.1 -u root --password=$ROOTPW mysql

(
echo "CREATE DATABASE application_test;"
echo "CREATE DATABASE application_development;"

echo "CREATE USER application;"
echo "SET PASSWORD FOR application = PASSWORD('nonprivpw');"
echo "GRANT SELECT,INDEX,INSERT,UPDATE,DELETE,ALTER,CREATE,DROP ON application_test.* TO 'application'@'%';"
echo "GRANT SELECT,INDEX,INSERT,UPDATE,DELETE ON application_development.* TO 'application'@'%';"

echo "CREATE USER webuser;"
echo "SET PASSWORD FOR webuser = PASSWORD('phppw');"

echo "FLUSH PRIVILEGES;"
) | mysql --protocol=socket --socket=${RUNDIR}/mysqld/mysqld.sock -u root --password=$ROOTPW mysql

/usr/bin/mysqladmin --protocol=socket --socket=${RUNDIR}/mysqld/mysqld.sock  -u root --password=$ROOTPW shutdown

And the script which starts things up for normal things:

#!/bin/sh

SRCDIR=@SRCDIR@
USER=${USER}
RUNDIR=${SRCDIR}/test/cluster/run
LOGDIR=${SRCDIR}/test/cluster/log
SOCKET=${RUNDIR}/mysqld/mysqld.sock
mkdir -p ${RUNDIR}/mysqld
mkdir -p ${SRCDIR}/db/mysql
mkdir -p ${LOGDIR}/mysql

ln -s  ${SOCKET} ${SRCDIR}/../application.sock

/usr/sbin/mysqld --basedir=/usr --datadir=${SRCDIR}/db/mysql --pid-file=${RUNDIR}/mysqld/mysqld.pid --skip-external-locking --socket=${SOCKET} --skip-networking --log_bin=${LOGDIR}/mysql/mysql-bin.log &

The symlink is placed in the dir above because in our case, we have multiple RoR applications that want to read from that database. Slowly we are converting them to RESTful/ActiveResource mechanism.

To access the symlink, the database.yml looks like:

development:
  adapter: mysql
  database: application_development
  username: application
  password: nonprivpw
  host: localhost
  socket: <%= RAILS_ROOT %>/test/cluster/run/mysqld/mysqld.sock

Running migrations with a different database ID

As we developed our virtual desktop provisioning application, we had to integrate against some PHP code that wanted direct access to the database. (See next tip about that)

We started to maintain the properties of that access including low-priviledge database login, and associated limited access views for it. To do this, we initially gave our normal RoR database login the right permissions. We got less and less enamored of this:

1. it gave our RoR application power that it really didn't need.
2. it made our migrations MySQL specific, and we liked to use sqlite on our laptops for development.

We couldn't do a lot about the second point because RoR doesn't abstract some of things we wanted to do, and sqlite doesn't even support them. Initially, we just avoiding running those migrations that edited views, or GRANTed permissions by wrapping the code itself in:

if ENV['RAILS_ENV'] == 'production'
    execute "alter view webview_table as select uuid,firstname,lastname,username from logins;"
    execute "GRANT SELECT (`id`, `filesrv_ip`, `title`, `code`, `baynum`) ON `locations` TO 'webuser'@'localhost';"
end

What we decided to do was to make a new database stanza in database.yml:

development_migration:
  adapter: mysql
  database: application_development
  username: root
  password: mypasswd
  host: localhost
  socket: <%= RAILS_ROOT %>/test/cluster/run/mysqld/mysqld.sock

We run this with:

RAILS_ENV=development_migration rake db:migrate

We are looking for a way to make that the default for db:migrate, but our rake-fu isn't high enough yet.

To make this work during capistrano deployments, we add to config/deploy.rb:

set :rails_env, "production_migration"

default schemas for active resource

A problem with the RESTful ActiveResource class is that it doesn't get any kind of schema from the server. For GET/Retrieve of CRUD, it's not a problem. For POST/CREATE it is.

We put code like this in our ActiveResource method's initialize:

class MyRestfulResourcet < ActiveResource::Base
  self.site = "http://mysite/"
  self.collection_name = "myrecords"
  self.element_name = "myrecord"

  Schema = [ :name, :type, :firstboot, :subscriber_id ]

  def self.find_single(x, options)
    return nil if x.nil?
    return nil if x.to_i < 1
    super(x, options)
  end

  def initialize(attributes = {})
    super(attributes)

    if @attributes["type"].nil?
      @attributes["type"] = "TheSubclass"
    end

    Schema.each { |attr|
      attr = attr.to_s
      if @attributes[attr].nil?
	@attributes[attr] = nil
      end
    }
  end
end

This initializes the @attributes hash to contain the things that we need, and so appropriate attribute accessor methods will be created.

Maybe there is a better way.