Michael's musings

Thu, 01 Dec 2011

Active Scaffold obscures internal errors

In a newly scaffold'ed model and controller, created with ActiveScaffold 3.0.5, on rails 3.0.9, I was getting errors from the default created rspec code that I could not diagnose:

  1) Admin::ConnectionsController POST create with valid params creates a new Connection
     Failure/Error: post :create, :connection => valid_attributes
     NoMethodError:
       You have a nil object when you didn't expect it!
       You might have expected an instance of Array.
       The error occurred while evaluating nil.each
     # ./spec/controllers/admin/connections_controller_spec.rb:54

Worse, these things were working just fine in RAILS_ENV=development.

Well, of course, it is failing on the line where the :create is invoked. But, where is the nil.each occuring?

I ran things with:

bundle exec rspec -d spec/controllers/admin/connections_controller_spec.rb \
   -e "POST create with valid params creates a new Connection"

after putting "debugger" in before the test case:

  describe "POST create" do
    describe "with valid params" do
      it "creates a new Connection" do
        # expect {
          debugger
          post :create, :connection => valid_attributes
        #}.to change(Connection, :count).by(1)
      end

(I'm still looking for a good ruby-debug mode that works like gdb-mode in Emacs works, that can show me the code around where I am...)

One winds up in the rescue in:

/var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/rescue.rb

on line 19.

So, stick a breakpoint on the super there:

break /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/rescue.rb:17

This lets you see the exception:

(rdb:1) p exception
#<NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.each>

The annoying part is that the action is invoked at /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/instrumentation.rb:29

ActiveSupport::Notifications.instrument("process_action.action_controller", raw_payload) do |payload|

so, it evaluates code, and there are in fact one block passed to another block, and it seems really hard (a major ruby-debug limitation), that I can not put a breakpoint easily into the beginning of a block passed in.

I had to resort to editing that file, and sticking "debugger" in there!

Finally, one gets to:

/var/lib/gems/1.8/gems/actionpack-3.0.9/lib/abstract_controller/base.rb:150
send_action(method_name, *args)

In the debugger, the right thing to do is:

catch NoMethodError

This finally shows me that the failure is at:

/corp/projects/credil/hydra/t3041/vendor/plugins/active_scaffold/lib/active_scaffold/attribute_params.rb:42

Why? Because attributes is nil.

Why, because the generated controllers spec file says:

    describe "with valid params" do
      it "creates a new Connection" do
        expect {
          post :create, :connection => valid_attributes
        }.to change(Connection, :count).by(1)
      end

should have been generated as:

    describe "with valid params" do
      it "creates a new Connection" do
        expect {
          post :create, :record => valid_attributes
        }.to change(Connection, :count).by(1)
      end

Mon, 09 May 2011

Problems (insecurities) in ActiveResource

I have an application that talks to Redmine/Chiliproject using its API with results in JSON. I use ActiveResource to make these calls, and it suddendly started failing after an upgrade from redmine to chiliproject:

test_retrieve_the_thomas_watson_project_by_id(ProjectTest):
ActiveRecord::UnknownAttributeError: unknown attribute: created_on

The fact that I was getting an error from ActiveRecord and not ActiveResource was puzzling. My ActiveResource class was called ProjectResource. The thing that I was retrieving was a "project", and yes, I happened to have a model called "Project", which was a subclass of ActiveRecord.

Looking at the JSON results using curl:

marajade-[~/C/dracula/hourbank3] mcr 10293 %curl 'http://localhost:3100/projects/show/16?format=json&key=abcdAPIKEY09123456789'
{"project":{"description":"Voice and Video softphone system for Android, with SIP support.","updated_on":"2010/10/08 10:10:24-0400","identifier":"thomas-watson","homepage":"","name":"Thomas-Watson","created_on":"2009/08/23 12:21:38 -0400","id":16}}

and also in the debugger, at

(rdb:1) c
Breakpoint 1 at /var/lib/gems/1.8/gems/activeresource-3.0.4/lib/active_resource/base.rb:889
/var/lib/gems/1.8/gems/activeresource-3.0.4/lib/active_resource/base.rb:889
new(record).tap do |resource|
(rdb:1) p record
{"project"=>{"name"=>"Thomas-Watson", "created_on"=>"2009/08/23 12:21:38 -0400", "id"=>16, "updated_on"=>"2010/10/08 10:10:24 -0400", "homepage"=>"", "description"=>"Voice and Video softphone system for Android, with SIP support.", "identifier"=>"thomas-watson"}}

what happens next is that the word "project" is passed to

find_or_create_resource_for(key)

and this finds and returns the "Project" class which is in my model. My model does not have a field created_on, thus the error.

So there three problems with this behaviour:

additions to the API should not break my old code, I should just ignore them.

there is no guarantee that the class that was found, "Project" has any of the behaviour that I need in the thing returned from ActiveResource.

worst, since the word "project" came from the remote system, the remote system could pick any class it wanted and invoke code on it. It's a reverse attack by a server on a client, but it's wrong to assume that the server is fully trusted by the client.

I'm not sure what the easiest way to fix this, but it's certainly wrong, and it's been there awhile in ActiveResource.

Wed, 02 Mar 2011

Deploying Django applications with Capistrano

Yesterday, I cooked up a deploy.rb so that Capistrano can deploy a Django application. While there is a Python app called http://docs.fabfile.org/0.9.0/ from what I could tell, it was very general to running commands on multiple servers, and not really specific to checking out a web framework and deploying it to one or more servers.

First, my deploy.rb, and then my notes about how I used it. I have changed only one or two things from my real code. My application is called "clientportal" and the host running it is called "clientportal.isp.example.net". On the server, it runs as a user called "clientportal".

This code does not yet invoke the Django database migrations, which it ought to, and I'll do another blog post once I figure out that part.

set :application, "clientportal"
set :me, "#{ENV['LOGNAME']}"
set :repository,  "git+ssh://#{me}@code.credil.org/git/path/to/repo/clientportal"

set :scm, :git
set :user, :clientportal

set :ssh_options, { :forward_agent => true }
set :use_sudo, false
set :git_enable_submodules, true
set :deploy_to, "/home/#{user}/#{application}"

role :web, "clientportal.isp.example.net"     # Your HTTP server, Apache/etc
role :app, "clientportal.isp.example.net"

# This is where Rails migrations will run
role :db,  "clientdb.isp.example.net", :primary => true

namespace :deploy do
  task :start do ; end
  task :stop do ; end

  # this overrides a rails specific thing.
  task :finalize_update do ; end
  task :migrate         do ; end

  task :restart, :roles => :app, :except => { :no_release => true } do
    # something to restart django.
    run "sudo /usr/sbin/apache2ctl graceful"

  end
  task :update_database_yml, :roles => [:app,:web] do
    db_config = "/home/#{user}/settings.py"
    run "cp #{db_config}   #{release_path}/settings.py"
    run "ln -f -s #{release_path} /home/clientportal/clientportal/clientportal"
    puts "Ran update database settings"
  end

end

after "deploy:update_code", "deploy:update_database_yml"

Some details. First, I put my settings.py file into my /home/clientportal directory. I do not check this file into my repo, because it always specific to the installation (it's different on your laptop than on the devel server or the production server). Also see my:

Like http://blog.perplexedlabs.com/2010/02/08/deployment-using-capistrano-and-webistrano-via-rails-and-phusion-passenger/ I had to adjust my django.wsgi file as well. I wound up with:

import site
site.addsitedir('/usr/local/pythonenv/CLIENTPORTAL/lib/python2.5/site-packages')

import os, sys

sys.path.append('/home/clientportal/clientportal')
sys.path.append('/home/clientportal/clientportal/current')
os.environ['DJANGO_SETTINGS_MODULE'] = 'clientportal.settings'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

The important changes were to the path that was added. It used to add $HOME/clientportal and $HOME to the path, but now it is one directory deeper, and you will notice above in the update_database_yml task that it creates a symlink in $HOME/clientportal with the name "clientportal" that is essentially the same as "current".

This is necessary because the settings are loaded as "clientportal.settings", and python basically turns the . into a / when looking for the file. I could have just changed the name of the settings file, but we had other modules that were loaded using the clientportal. namespace.

Note that the server already had it's apache configured to do what was needed. I would normally package these config files up into a .deb file, but I haven't done that yet for this project, it being my first django project.

I am not sure if I actually have to restart apache. I added that for good luck, and and I added:

clientportal ALL=NOPASSWD: /usr/sbin/apache2ctl graceful

My apache config looks like:

<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName clientportal.isp.example.net
        ServerAlias portal1.isp.example.net
        ServerAlias portal.example.net

        DocumentRoot /home/clientportal/clientportal/current
        <Directory "/home/clientportal/clientportal/current">
                Options Indexes FollowSymLinks
                Options -MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

        ErrorLog /var/log/apache2/error.log

        Alias /media/ /home/clientportal/clientportal/current/media/
        WSGIScriptAlias / /home/clientportal/clientportal/current/wsgi/django.wsgi
        <Directory /home/clientportal/clientportal/current/apache/>
                Order allow,deny
                Allow from all
        </Directory>
...

Some other links I found, but I didn't use much: http://groups.google.com/group/django-developers/browse_thread/thread/f34e59275e04f9c5?pli=1 http://gnuvince.wordpress.com/2008/01/10/deploying-django/

Mon, 06 Jul 2009

Rails 2.3.x requires new test_helper

I was getting weird errors like:

.../test/test_helper.rb:11: undefined method `use_instantiated_fixtures=' for Test::Unit::TestCase:Class (NoMethodError)

When trying to run "rake test" on an application that actually had no tests defined, (my goal is to add a few) that was built with rails 2.3.2.

It turns out that in rails 2.3 (?- maybe 2.2, my other applications are at

2.1), the unit test cases are now subclassed from ActiveSupport::TestCase

instead of Test::Unit::TestCase.

This matters for both test_helper.rb and the *_test.rb files.

Thu, 04 Sep 2008

Foxy Fixtures in Ruby

Not obvious to me at first is that Foxy Fixtures in RoR, where you can specify a foreign key relationship by name, as in:

   ### in pirates.yml

   reginald:
     name: Reginald the Pirate
     monkey: george

   ### in monkeys.yml

   george:
     name: George the Monkey
     pirate: reginald

Depends upon the fact that you didn't include id: in the fixtures. That is, RoR is not looking into the pirates.yml file to find the "reginald" fixture, and then inserting the "id" from it in.

Rather, it's applying a hash of the string "reginald" to get the id, and so the id: of reginald had better be derived in the same way.

Mon, 23 Jun 2008

mongrel_cluster vs cap:deploy

Many people (including me), get:

Couldn't find any pid file in '/data/deploy/cortland/releases/20080623202133/tmp/pids' matching 'dispatch.[0-9]*.pid'

This is because you've (correctly) started your mongrel with mongrel_cluster, and it does not create the PIDs with the same name.

To fix this, change your mongrel_cluster.yaml file to say: pid_file: tmp/pids/dispatch.pid

Thu, 08 May 2008

Unit testing with model security

As explained in: http://www.embracingchaos.com/2007/05/model_security_.html the problem with model security is that it requires that you have a controller for your unit tests to work.

We solved this with a mock object we called "AlwaysAdmin"

class AlwaysAdmin

  # this routine is most useful when using script/console!
  def self.fake_out_login
    Thread.current[:session] = Hash.new
    User.current=AlwaysAdmin.new
  end

  def logged_in_as_admin?
    true
  end

  def logged_in_as_role1?
    true
  end

  def logged_in?
    true
  end
  def admin?
    true
  end
end

In our unit tests where we don't care at all about model security, we do:

class MyTest < Test::Unit::TestCase
  fixtures :myfixtures

  def setup
    AlwaysAdmin.fake_out_login
  end
end

Bruce Perens says:

I'm rewriting it right now, as part of the software development of new blog software at http://new.technocrat.net . There is a source link at the bottom of the page there that returns source.tgz containing a tar of the running software version created every time I restart the application. Only the User and Session classes are done so far, not ModelSecurity. They are under vendor/plugins/user in the source.tgz archive. User and Session are RESTful now, and are an Engine so they can have views and routes.

Thu, 01 May 2008

testing with authenticated resources

So, you add new code to authenticate various access, and your functional tests stop working.. You need to authenticate!

Add this to your test_helper.rb:

  def user1_login_details
    'Basic ' + Base64.encode64('user1'+':'+'user1pw')
  end

  def user1_login
    @request.env['HTTP_AUTHORIZATION'] = user1_login_details
  end

(the details are split out because we used them in other places too)

then add a call to "user1_login" to your functional tests, or if all of them need it, to the "def setup".

Remember to write tests for not being logged in, and for situations where the logged in user should not have access to some protected resource.

embrace and extend of php/mysql application

We had a prototype for a user-facing service written in PHP.

After some discussion, we decided that we really wanted the Test Driven Development that is so easy in RoR, and even thought we could do it in PHP, as our PHP developers had no experience using any kind of object-oriented or template driven PHP development (the code was totally raw PHP), we decided that we would gradually/incrementally replace the code with Ruby-on-Rails.

The first task was to create an empty RoR application. We set it all up with Capistrano, SVN, etc. and deployed it to a test application server. We setup Apache and Mongrel as normal, and made sure to enable PHP in Apache.

We then copied all of the existing web site into Rails' public directory, and checked it into SVN. We then deployed again, and took a look at the web server, and lo-and-behold, there was our legacy PHP application.

On our laptops, we want to develop, so we need Apache to get the PHP code running. As described in:

http://www.sandelman.ca/mcr/blog/2008/04/30#starting_local_mysql_database

we had some code that started Apache+Postgresql, and this time we run just the Apache code. I decided that test/cluster/etc was needlessly deep, and I installed things into test/etc instead:

See http://www.sandelman.ca/mcr/ruby-files/embrace/ for files: apache2.conf.in, php.ini.in (a snippet), portnum.sh, runweb.sh.in, shutit.sh.in, and apache.rake.

Place apache.rake into lib/tasks, it has:

namespace :apache do

  desc "Start web server and local server"
  task :start do
    puts "Starting up Apache"
    system "test/etc/runweb.sh"
    system "sh -c 'script/server -p 9000 &'"
  end

  desc "Setup web server"
  task :setup do
    puts "Setting up Apache"
    system "test/etc/fixup.sh"
  end
end

Start out (after a raw checkout) with:

rake apache:setup

This will process the .in files into proper config files for apache. It will find the right modules directories for apache, as this sometimes varies between distros.

Then run:

rake apache:start

This will start apache on port 8000 + your numerical userid. This picks a consistent port, but lets' multiple users run simultaneously if they happen to be logging into a common work machine.

Point your browser at http://localhost:8xxx/ , and you should see your application running. A copy of mongrel can be started under the normal load balancer, setup to run in development mode. We commented that line out as actually, we prefer to have it in the foreground in a window, in case we want to do debugging.

Using ssh+svn URLs for capistrano

We use SVN (we'd like to use git, but we haven't convinced TracGitPlugin to work for us yet) with SSH access to deploy our application.

Our application servers have SSH access to our master repository, but they must login with public keys only, and we prefer that they do so with the user id of the person doing the deploying.

So we have in our config/deploy.rb:

set :repository,  "svn+ssh://"+ENV['USER']+"@svn.example.com/path/to/proj/repo//trunk"

Each of our application servers have a special login accessed by SSH public key authentication, which holds our source code. We don't use sudo, but we probably should. Eventually, with git, we would expect to do a git push to our application server, and then it could do a local checkout. That would permit our application servers to operate with access to the SVN master system.

To get the public key authentication from application server to SVN master we needed to have SSH authentication agent forwarding. With Rails 1.2.3, we had to hack the file:

/var/lib/gems/1.8/gems/capistrano-2.0.0/lib/capistrano/ssh.rb

and every works the way we wanted.

strange complains about class being obsolete

We have some ActiveRecord (and ActiveResource) models that are subclassed using the "type" field. We would attempt to make multiple selection boxes using collection_select for them, and we would get errors about the #type method being obsolete, or worse, we'd get complaints that we couldn't turn a class into a string.

The problem is that Rails 1.2.3 (at least, we haven't seen this with 2.0.2) uses the 'type' field for subclassing, but ruby's Object used to have a .type method, now renamed .class.

Since the ActiveRecord would synthesize the attribute accessor methods, the type method wasn't made until runtime, and since it already exists in Object, one wasn't synthesized.

To get around this, we added the following to our models (to the parent class):

  def type
    self[:type]
  end

I tried removing this with rails 2.0.2 to generate the precise error, but it appears to be fixed. I may update this entry with the exact error by checking out some older code that uses 1.2.3.

starting local mysql database

We transitioned from using sqlite on devel laptops (with inherent very low hassle) to having to run mysql everywhere to as our migrations and some of our complex database operations just didn't work with sqlite3.

Setting up all the database junk in mysql in annoying. Some developers have laptops with other mysql programs running on them too, and some development machines are shared by multiple projects and multiple developers.

It's much nicer to just run a "local" mysql. This is relatively easy to do with postgresql, but we are locked into mysql due to corporate desire for someone to buy support from.

Building local clusters for mysql is annoying, but not ultimately that hard. We use several shell scripts, but we think that these could be turned into rake tasks easily, and we intend to do that soon.

These scripts were inheirited from a PHP/Postgresql project call "ITERation" (a TBS.gc.ca project).

There is a Makefile in test/cluster/Makefile

SRCDIR=$(shell cd ../..; pwd)
...
%.sh:	%.sh.in Makefile
	sed \
		-e 's,@SRCDIR@,'${CortlandSRCDIR}',' \
		$< >$@
	chmod +x $@

dbflush: etc/initdb.sh etc/rundb.sh etc/shutdb.sh
	-[ -f run/mysqld/mysqld.pid ] && etc/shutdb.sh
	rm -rf ${SRCDIR}/db/mysql
	etc/initdb.sh
	etc/rundb.sh

dbinit: etc/initdb.sh
	etc/initdb.sh

dbstart: etc/rundb.sh
	etc/rundb.sh

The first is test/cluster/etc/initdb.sh.in.

#!/bin/sh

SRCDIR=@SRCDIR@
USER=${USER}
RUNDIR=${SRCDIR}/test/cluster/run
LOGDIR=${SRCDIR}/test/cluster/log
PIDFILE=${RUNDIR}/mysqld/mysqld.pid
ROOTPW=therootpw
mkdir -p ${RUNDIR}/mysqld
mkdir -p ${SRCDIR}/db/mysql
mkdir -p ${LOGDIR}/mysql
/usr/bin/mysql_install_db --basedir=/usr --datadir=${SRCDIR}/db/mysql --pid-file=${PIDFILE} --skip-external-locking --socket=${RUNDIR}/mysqld/mysqld.sock --log_bin=${LOGDIR}/mysql/mysql-bin.log

# now start the DB.
# have to start up mysql with TCP networking enabled!
/usr/sbin/mysqld --basedir=/usr --datadir=${SRCDIR}/db/mysql --pid-file=${RUNDIR}/mysqld/mysqld.pid --skip-external-locking --socket=${RUNDIR}/mysqld/mysqld.sock --port=3307 --log_bin=${LOGDIR}/mysql/mysql-bin.log &
sleep 10

/usr/bin/mysqladmin -h 127.0.0.1 --port=3307 -u root password $ROOTPW
echo "update user set host='%' where host='localhost';" | mysql -h 127.0.0.1 -u root --password=$ROOTPW mysql

(
echo "CREATE DATABASE application_test;"
echo "CREATE DATABASE application_development;"

echo "CREATE USER application;"
echo "SET PASSWORD FOR application = PASSWORD('nonprivpw');"
echo "GRANT SELECT,INDEX,INSERT,UPDATE,DELETE,ALTER,CREATE,DROP ON application_test.* TO 'application'@'%';"
echo "GRANT SELECT,INDEX,INSERT,UPDATE,DELETE ON application_development.* TO 'application'@'%';"

echo "CREATE USER webuser;"
echo "SET PASSWORD FOR webuser = PASSWORD('phppw');"

echo "FLUSH PRIVILEGES;"
) | mysql --protocol=socket --socket=${RUNDIR}/mysqld/mysqld.sock -u root --password=$ROOTPW mysql

/usr/bin/mysqladmin --protocol=socket --socket=${RUNDIR}/mysqld/mysqld.sock  -u root --password=$ROOTPW shutdown

And the script which starts things up for normal things:

#!/bin/sh

SRCDIR=@SRCDIR@
USER=${USER}
RUNDIR=${SRCDIR}/test/cluster/run
LOGDIR=${SRCDIR}/test/cluster/log
SOCKET=${RUNDIR}/mysqld/mysqld.sock
mkdir -p ${RUNDIR}/mysqld
mkdir -p ${SRCDIR}/db/mysql
mkdir -p ${LOGDIR}/mysql

ln -s  ${SOCKET} ${SRCDIR}/../application.sock

/usr/sbin/mysqld --basedir=/usr --datadir=${SRCDIR}/db/mysql --pid-file=${RUNDIR}/mysqld/mysqld.pid --skip-external-locking --socket=${SOCKET} --skip-networking --log_bin=${LOGDIR}/mysql/mysql-bin.log &

The symlink is placed in the dir above because in our case, we have multiple RoR applications that want to read from that database. Slowly we are converting them to RESTful/ActiveResource mechanism.

To access the symlink, the database.yml looks like:

development:
  adapter: mysql
  database: application_development
  username: application
  password: nonprivpw
  host: localhost
  socket: <%= RAILS_ROOT %>/test/cluster/run/mysqld/mysqld.sock