kevinharvey.net

… makes it work.

HTC Incredible Not Receiving Text Messages?

October 21st, 2011

After a recent Android update, your HTC Incredible may have stopped receiving text messages. After an hour on the phone with Verizon Wireless customer service got me no where, I turned to the interwebs. Check out this message thread. The long and the short of is that you need to download and install a hotfix. Go to this URL with your phone’s web browser:

http://dl3.htc.com/misc/inc8049.apk

After the download completes, slide down the notification bar and click the file. You’ll need to enable installation of non-market apps for the hotfix to work, and you’ll be prompted to do so when you click the file. If you’re not, do Menu -> Settings – > Applications -> Unknown Sources and check it.

After you run the file all the messages you missed will download.

Here are the details of my phone, you might want to check your own before trying this hotfix on yours:

  • Hardware Version: 0003
  • Android Version: 2.3.4
  • Baseband Version: 2.15.10.07.07
  • Kernel Version: 2.6.35.13-g55a06e5 / htc-kernel@and18-2 #1 / Thu Aug 16 16:33:55 CST 2011
  • Build Number: 4.06.605.3 CL140944 release-keys
  • Software Number: 4.06.605.3
  • Browser Version: WebKit/533.1
  • PRI Version: 1.28_002
  • PRL Version: 52501
  • ERI Version: 5

Fix the dreaded “Not an image file” MySQL Django error: use VirtualEnv

May 9th, 2011

If you develop Django using MySQL on a Mac, you’ve undoubtedly run into the “Not an image file” error for models with image fields. The problem has to do with JPEG support for Mac OS and how it interacts with Python Imaging Library. Diabolically, once you’ve installed PIL with bad JPEG support it seems impossible to fix.

My solution is to use a virtualenv. virtualenv is a python tool to create an isolated Python interpreter on your system, which allows you to control the installed packages without mussing up your main Python installation. It’s good practice to use a separate virtualenv for each project, and if you use Eclipse you usually want one for each workspace

There are, of course, other solutions to this problem (using PostgreSQL is probably one of them). The instructions here are based on this post on DjangoDays.com.

Prereqs

You’ll need to have xCode installed. Also, if you want to use Eclipse, vlku.com has an excellent screencast that covers everything.

Create a “workspace”

The best place to keep the virtualenv for your Django project is right next to your Django code. Create a directory to hold both. This can be just a plain directory, but if you’re using Eclipse make a new workspace first and complete the following steps in that workspace. If you have existing code, drop it in the root of this folder. For this tutorial let’s call the folder “myworkspace” and your project “myproject”.

Set up a Virtual Environment with –no-site-packages

Use easy_install to get virtualenv.

easy_install virtualenv

Change into the ‘myworkspace’ directory and create a new virtualenv. The key here is to use the –no-site-packages argument, which creates a Python that ignores the system packages already installed. We don’t want anything to come over from the system Python in case you’ve already installed a useless PIL.

python virtualenv.py python --no-site-packages

You now have a fresh, Django- and PIL-less Python interpreter at myworkspace/python/bin/python. The directory structure should now be as follows:

- myworkspace
+ - myproject
+ - python

Install libjpeg (JPEG Support) system wide

Here’s the key to fixing the problem, and where the DjangoDays.com post really saved my butt. Forget the virtualenv for a second, we’re installing JPEG support system wide.

  1. Download JPEG support. This little sucker keeps moving around the internet. I used version 6, which is still available here. The project is maintained my the Independent JPEG Group, who are showing version 8 as of this posting.
  2. Extract the tarball, cd into the resulting folder (in my case, it was called jpeg-6b, and install it with these changes.
    cd jpeg-6b
    cp /usr/share/libtool/config.sub .
    cp /usr/share/libtool/config.guess .
    ./configure –enable-shared
    make
    sudo mkdir -p /usr/local/include
    sudo mkdir -p /usr/local/bin
    sudo mkdir -p /usr/local/lib
    sudo mkdir -p /usr/local/man/man1
    sudo make install

Thanks DjangoDays.com.

Install PIL

Now we’ll install PIL and point it to the correctly installed JPEG Support. This is again lifted from the DjangoDays.com post.

  1. Download the PIL tarball here.
  2. Extract it, cd into the resulting folder, and edit setup.py to change these lines:
    JPEG_ROOT = none
    ZLIB_ROOT = none

    to this:

    JPEG_ROOT = “/usr/local/include”
    ZLIB_ROOT = “/usr/local/include”
  3. Install PIL with these commands:
    python setup.py build_ext -i
    python selftest.py
    sudo python setup.py install

I got 9 errors when I ran selftest.py, but kept going and haven’t seen any indication that the errors are giving me any trouble. Again, thanks DjangoDays.com.

Install Django, MySQL-python, etc.

Here’s a post a did a last year detailing Django installation on Mac OS X. That is, again, not the only way of doing it. The key here is to use the Python interpreter you just created to install Django and all it’s prereqs. You also have a virtualenv specific easy_install. Use it to install any other dependencies.

myworkspace/python/bin/python
myworkspace/python/bin/easy_install

Run your project

I’m skipping a lot of steps (and debugging) here, but assuming you’ve got your project set up correctly you should now be able to cd into the ‘myproject’ directory and run the server with the following command:

../python/bin/python manage.py runserver

As a shortcut, you can set an environment variable in the ‘myworkspace’ directory to map the command ‘python’ to /myworkspace/python/bin/python. Better yet, if you’re using Eclipse, use that Python as you’re interpreter, and configure the debugger to run the command for you.

Can’t start the dev server for a Google App Engine Java project on Snow Leopard?

March 16th, 2011

This one was ruining my afternoon until I came across this post on the Google App Engine Issues page. I was running the command to start the dev server from the Google App Engine Java tutorial in Terminal on OS X 10.6. The process looked like it would start fine, but would then immediately stop (as evidenced by the new terminal line after the success message):

Kevins-iMac:com.google.appengine.eclipse.sdkbundle kevin$ ./appengine-java-sdk/bin/dev_appserver.sh appengine-java-sdk/demos/guestbook/war
2011-03-16 17:06:59.443 java[8217:a07] [Java CocoaComponent compatibility mode]: Enabled
2011-03-16 17:06:59.444 java[8217:a07] [Java CocoaComponent compatibility mode]: Setting timeout for SWT to 0.100000
Mar 16, 2011 9:07:00 PM com.google.apphosting.utils.jetty.JettyLogger info
INFO: Logging to JettyLogger(null) via com.google.apphosting.utils.jetty.JettyLogger
Mar 16, 2011 9:07:00 PM com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml
INFO: Successfully processed /Applications/eclipse/plugins/com.google.appengine.eclipse.sdkbundle/appengine-java-sdk/demos/guestbook/war/WEB-INF/appengine-web.xml
Mar 16, 2011 9:07:00 PM com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
INFO: Successfully processed /Applications/eclipse/plugins/com.google.appengine.eclipse.sdkbundle/appengine-java-sdk-1.4.2/demos/guestbook/war/WEB-INF/web.xml
Mar 16, 2011 5:07:00 PM com.google.appengine.tools.development.DevAppServerImpl start
INFO: The server is running at http://localhost:8080/
Kevins-iMac:com.google.appengine.eclipse.sdkbundle kevin$      # this is where things start to suck

Long story short, Apple decided to stop maintaining Java on their operating systems. To fix the problem, install the latest version of OpenJDK.

  1. Download this and install it: http://openjdk-osx-build.googlecode.com/files/OpenJDK-1.7-x86_64-20110221.dmg (most recent as of 3/16/11)
  2. Open your Java Preferences by going to Applications -> Utilities -> Java Preferences
  3. Drag the new Java version (Java SE 7) up above any other Java versions listed

After that you should be able to run the server again.

If you want to fix the problem from within Eclipse, check out Comment 14.

Create and Download a CSV (or any other file type) with a Plone View

February 28th, 2011

Do you have a content type that you’d like to be able to offer as Comma Separated Value (CSV)/Excel download? Python can create both formats pretty easily (using either the standard csv library, or xlwt) and you can write a view for your content type that will create the file and return it to the user as a file download.

At Plone.org, Martin Aspeli covers this process as a part of the five.grok documentation. I’d already created my product without using five.grok, and I didn’t figure this one use case was enough reason to switch. So here’s what I came up with:

First, register the view in browser/configure.zcml

...
    <browser:view
      for="my.product.interfaces.IMyContentType"
      name="csv_view"
      class=".mycontenttypeview.MyContentTypeCsvView"
      permission="zope.Public"
      />
...

I added the view to browser/myContentType.py under the default view.

...
class MyContentTypeCsvView(BrowserView):
    """
    Download the content type as a CSV file.
    """
    
    def __call__(self):
        """
	Build a CSV from the content type.
	"""
	out = StringIO()
	writer = csv.writer(out)

        ...
        # make the CSV file
        ...

	filename = "%s.csv" % context.id

	self.request.response.setHeader('Content-Type', 'text/csv')
	self.request.response.setHeader('Content-Disposition', 'attachment; filename="%s"' % filename)

        return out.getvalue()

Now you should be able to link to http://mysite.com/mycontenttype/csv_view and initiate a file download.

How to access the SD card from the Android Emulator on Mac OS

February 2nd, 2011

Here’s another one in the “I’m writing this down so I won’t forget it” category. When you’re working with the Android Emulator and you want to add files to the virtual SD card on the AVD, you have to mount it. To do so, hdid the the SD card image:

hdid ~/.android/avd/my-avd.avd/sdcard.img

The guts of the AVD live in a hidden folder in your user folder. Change the ‘my-avd.avd’ bit to match the name of the AVD whose SD card you want to manipulate. After that you’ll have a drive called ‘SDCARD’ on your Desktop that you can open and drop files in.

Unforunately, with this method you have to restart the Emulator every time you want to change the file. That’s not a big deal for me right now, but I’m sure it will be soon. I’ll update this post when/if I figure out how to change the files without restarting the device.

How to view the contents of the browser in ZopeTestCase

January 20th, 2011

In first writing tests for Plone products, it was difficult for me to figure out what was in the browser when a failure occurred. Adding the two following lines just after the offending code will create a file ‘test.html’ that you can view in a real broswer:

>>> f = open('/Users/kevin/Desktop/test.html','w')
>>> f.write(browser.contents)

Change the path as necessary to point to your desktop, run the test, and click the file on your Desktop to figure out what the heck is going on.

Adding a new user in a ZopeTestCase doctest

January 14th, 2011

One of the content types in a Plone product I’m working on needed to refer to one of the users in the site. Therefore I needed to get some users in the system in the doctest. At first I tried .click()ing through the interface to get to add a new user through Site Setup. After much wailing and gnashing of teeth, I realized I could do this in straight Python:

>>> pr = self.portal.portal_registration
>>> pr.addMember(id = 'ddastardly',
...              password = 'password',
...              roles = ["Member",],
...              properties = {'fullname': 'Dick Dastardly',
...                            'username': 'ddastardly',
...                            'email': 'ddastardly@example.com',}
...              )

This is cool because I don’t need to test the Plone “Add a New User” interface as a part of my product. I only need a few users available to test my content type. Later, if you need to do anything as this user, you can log in like this.

>>> browser.getLink('Log in').click()
>>> browser.getControl(name='__ac_name').value = 'swhiplash'
>>> browser.getControl(name='__ac_password').value = 'password'
>>> browser.getControl(name='submit').click()
>>> 'You are now logged in' in browser.contents
True

Accessing radio buttons & checkboxes in the ZopeTestCase browser

January 14th, 2011

This definitely falls into the ‘reminding-myself-for-next-time’ blog category, but maybe this will help someone else out. I found myself needing to test radio buttons that didn’t have labels (don’t ask why). Those radio buttons all shared the same name attribute. Trying to access them by the name got the larger ListControl, not any of the radio buttons themselves:

>>> # here the radio buttons have the name "age_group"
>>> from Products.Five.testbrowser import Browser
>>> browser = Browser()
>>> browser.getControl(name='age_group').click()
>>> .... (continued)
>>> AttributeError: 'ListControl' object has no attribute 'click'

You can, however, access the individual radio buttons from the ListControl using the controls attribute on the ListControl. controls returns a list of the ListControl’s child controls, which you can access by index.

>>> # here the second radio button's value attribute is '20-25'
>>> browser.getControl(name='age_group').controls[1].selected = True
>>> browser.getControl(name='age_group').value
['20-25']

Date Constraint for XForms in Open Data Kit

December 15th, 2010

Had some trouble Googling this little nugget, so I thought I might share it here.

If you want to constrain a date field in Open Data Kit by a specific date (i.e. “The date must be before 12/15/2000″), here’s the syntax:

<bind nodeset="/demographics/dob"
      type="date"
      constraint=". &lt; date('2000-12-15')"
      jr:constraintMsg="Date of Birth must be before 12/15/2000"
      />

Don’t forget the quotes inside the parentheses.

Benchmarking Plone 3.3.5 with ApacheBench

July 13th, 2010

Ever wondered what a good caching policy can do for your site’s performance? I’m deep in the throws of a ground-up Plone installation behind Apache, and I’ll be using CacheFu and Varnish to speed up the site. As I go along I’ll run ApacheBench at major milestones to see what difference they’re making.

The Command

ApacheBench comes installed with Apache, so no need to add anything new. To benchmark my site, I’ll run the following command:

ab -n 100 -c 3 http://127.0.0.1:8080/Plone/front-page

That’s 100 total requests, no more than 3 at the same time.

Initial Data

Here are the results from the “Welcome to Plone” page on a freshly created Plone site.

Server Software:        Zope/(unreleased
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /Plone/front-page
Document Length:        21120 bytes

Concurrency Level:      3
Time taken for tests:   34.529235 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      2140200 bytes
HTML transferred:       2112000 bytes
Requests per second:    2.90 [#/sec] (mean)
Time per request:       1035.877 [ms] (mean)
Time per request:       345.292 [ms] (mean, across all concurrent requests)
Transfer rate:          60.53 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   4.4      0      32
Processing:   263 1021 3556.2    528   34526
Waiting:      261 1019 3553.4    523   34494
Total:        263 1022 3557.4    528   34526

Percentage of the requests served within a certain time (ms)
50%    528
66%    553
75%    564
80%    575
90%    667
95%    967
98%   8281
99%  34526
100% 34526 (longest request)

Next, let’s see what happens when we add content.

With Data, Running on the Bare Zope Server

Since the site is running on ZEO, I decided to temporarily leave one client running on the bare Zope application server and set up one behind Apache. Here’re the stats for the non-Apache client.

Server Software:        Zope/(unreleased
Server Hostname:        127.0.0.1
Server Port:            8081

Document Path:          /Plone/
Document Length:        26136 bytes

Concurrency Level:      3
Time taken for tests:   42.539567 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      2641800 bytes
HTML transferred:       2613600 bytes
Requests per second:    2.35 [#/sec] (mean)
Time per request:       1276.187 [ms] (mean)
Time per request:       425.396 [ms] (mean, across all concurrent requests)
Transfer rate:          60.63 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   15 115.0      0     991
Processing:   408 1247 221.9   1221    2181
Waiting:       10 1235 253.2   1219    2179
Total:        820 1263 189.2   1221    2181

Percentage of the requests served within a certain time (ms)
 50%   1221
 66%   1225
 75%   1227
 80%   1231
 90%   1409
 95%   1596
 98%   2181
 99%   2181
 100%   2181 (longest request)

With Data, Running Behind Apache

… and here’s what I got on the client running behind Apache. I hadn’t spent any time on caching yet.

Server Software:        Zope/(unreleased
Server Hostname:        **************************
Server Port:            80

Document Path:          /
Document Length:        25745 bytes

Concurrency Level:      3
Time taken for tests:   46.875144 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      2602800 bytes
HTML transferred:       2574500 bytes
Requests per second:    2.13 [#/sec] (mean)
Time per request:       1406.254 [ms] (mean)
Time per request:       468.751 [ms] (mean, across all concurrent requests)
Transfer rate:          54.21 [Kbytes/sec] received

Connection Times (ms)
 min  mean[+/-sd] median   max
Connect:        0   12  87.6      0     740
Processing:   719 1385 2949.9    899   24272
Waiting:      264 1377 2935.4    899   24271
Total:        719 1397 2978.6    901   24272

Percentage of the requests served within a certain time (ms)
 50%    901
 66%    934
 75%    988
 80%   1031
 90%   1453
 95%   1954
 98%  19548
 99%  24272
 100%  24272 (longest request)

With Data, from my local machine

This is what it looks like from my local machine.

Server Software:        Apache/2.2.3
Server Hostname:        ***********************
Server Port:            80

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      3
Time taken for tests:   3.004 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Non-2xx responses:      100
Total transferred:      25500 bytes
HTML transferred:       0 bytes
Requests per second:    33.29 [#/sec] (mean)
Time per request:       90.124 [ms] (mean)
Time per request:       30.041 [ms] (mean, across all concurrent requests)
Transfer rate:          8.29 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       26   44  13.6     39     101
Processing:    27   45  14.8     40     100
Waiting:       26   45  14.9     40     100
Total:         54   89  22.1     85     161

Percentage of the requests served within a certain time (ms)
  50%     85
  66%     95
  75%    101
  80%    108
  90%    122
  95%    137
  98%    143
  99%    161
 100%    161 (longest request)