Django

Change / modify app labels urls in Django Admin

26 Mar , 2013  

We can update app labels / application name and urls in Django Admin without change the application name folder. For instance:

1
2
App name : student_insurance
Admin : /student_insurance

What we want:

1
2
App name : student_insurance
Admin : /student-insurance

By:

1
2
3
4
5
6
class YourModel(models.Model):
    ....

    class Meta:
        app_label = string_with_title("student-insurance", "Student Insurance")
        db_table = "stundent_insurance_yourmodel"

,

Web

Solve php-fpm consume high CPU usage 100% 200%

26 Mar , 2013  

Suddenly my server crashed and I see that php5-fpm usage was the culprit and it’s consume 100% of CPU usage. Fyi, I use NGINX and WordPress on this case. I tried several thing to solve php-fpm issue like change listener into socket instead of ports. Also, I use APC as well, but it’s doesn’t works.

Until I found solution was using Varnish. Here is a quick step to install Varnish for WordPress on your server ( I use debian ):

1. Install Varnish on Debian

1
apt-get install varnish

More…

Django

Django set urls conf on parents of apps with include and slug

21 Mar , 2013  

For instance, we have urls.py like this:

1
(r'^insurances/', include('insurances.urls', namespace="insurances")),

Then we want to have slug on parent of apps by :

1
/insurances/slug

Then in our urls.py in apps:

1
url(r'^(?P<slug>[-w]+)/$', 'details', name="details"),

Done! 😀

,

Django

Scroll down pages into bottom in Django Lettuce Selenium

18 Mar , 2013  

The most common things happen when we have long pages is the element’s is not found or not clickable.

1
2
3
4
5
6
7
8
Traceback (most recent call last):
      File "/home/yodi/.virtualenvs/insurance/lib/python2.7/site-packages/lettuce/core.py", line 143, in __call__
        ret = self.function(self.step, *args, **kw)
      File "/home/yodi/web/insurance/../insurance/tests/step_definitions/web_steps.py", line 63, in i_press
        world.browser.find_by_xpath('//button[text()="%s"]' % value).click()
      File "/home/yodi/.virtualenvs/insurance/lib/python2.7/site-packages/splinter/element_list.py", line 67, in __getattr__
        raise AttributeError("'%s' object has no attribute '%s'" % (self.__class__.__name__, name))
    AttributeError: 'ElementList' object has no attribute 'click'

To solve this problem, we can execute scroll down by :

1
2
3
@step("I scroll down")
def i_scroll_down(step):
    world.browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")

,

Django

Set readonly fields only for view details on Django Admin

14 Mar , 2013  

In some condition we want to make fields became read-only on “view details” in Django Admin. That’s mean, for adding new records, this fields must be not in read-only mode. To solve this problems, we can use get_readonly_fields().

Here is an example:

1
2
3
4
5
6
7
class InsuranceAdmin(admin.ModelAdmin):
    list_display = ('id', 'email', 'brand', 'website')

    def get_readonly_fields(self, request, obj=None):
        if obj:
            return ('email')
        return ()

This code will give email fields editable on “Add” pages but become read-only on “View” pages.

,

Django

Django south access orm another app

11 Mar , 2013  

To access another application in django south data migration fowards() or backwards(), we just use :

1
orm["app_name.Model_name"]

For instance, my app name is “insurance” and my model name is “Records”:

1
orm['insurance.Records'].objects.all()

Django

Django Mod WSGI Apache and Virtualhost on Fedora 17

7 Mar , 2013  

Here is a steps for setup Django with modwsgi + httpd + Virtualhost in Fedora 17.

1. Install Apache and Mod WSGI

1
sudo yum install -u httpd* mod_wsgi

2. Load modwsgi
To avoid error :

1
invalid command 'WSGIScriptAlias', perhaps misspelled

We need to check whether this line exists in “/etc/httpd/conf/httpd.conf”:

1
LoadModule wsgi_module modules/mod_wsgi.so

3. Create wsgi
We need to create WSGI handler which contain configuration of our web path, python interpreter and settings. In this example, I will point into virtualenvironment for python interpreter.

django_wsgi.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import os, sys
import site

sys.stdout = sys.stderr # for mod_wsgi versions < 3.0, logging to stdout will trigger an IOError exception

ALLDIRS = ['/home/yodi/.virtualenvs/django/lib/python2.7/site-packages'] # path to virtualenv lib folder on server

# Remember original sys.path.
prev_sys_path = list(sys.path)

# Add each new site-packages directory.
for directory in ALLDIRS:
    site.addsitedir(directory)

site.addsitedir("/home/yodi/htdocs/web/") # django project folder

sys.path.append(os.path.dirname(os.path.abspath(__file__)) + '/..')
os.environ['DJANGO_SETTINGS_MODULE'] = 'web.settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

4. Create Django Virtualhost on Apache
Open “/etc/httpd/conf/httpd.conf” and add this configuration. Please change the value with yours django path.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
NameVirtualHost *:8001
Listen 8001

<virtualhost *:8001>
    ServerAdmin yodi@localhost
    ServerName yodi

    # Django settings
    WSGIScriptAlias / /home/yodi/htdocs/web/django_wsgi.py
    <Directory /home/yodi/htdocs/web>
        Order deny,allow
        Allow from all
    </Directory>
    WSGIDaemonProcess tripvillas user=yodi group=yodi processes=10 threads=5 python-path=/home/yodi/.virtualenv/django/lib/python2.7/site-packages/
    WSGIProcessGroup yodi
</virtualhost>

WSGISocketPrefix /var/run/wsgi

5. Check permission and SELINUX problem in Fedora
If you’re want to setup this for development, you can remove “SELINUX” that might give you headache because several permission problem in httpd:

1
sudo yum remove -y selinux*

The last step you need is just restart httpd service and check on http://localhost:8001

If you got this errors:

1
[Thu Mar 07 12:01:50 2013] [error] [client 127.0.0.1] (13)Permission denied: access to / denied (filesystem path '/home/yodi/htdocs/web') because search permissions are missing on a component of the path

Maybe your httpd.conf is already modified before. Just save it into somewhere and re-install apache.

1
2
sudo yum remove -y httpd*
sudo yum install -y httpd* mod_wsgi

6. Add monitor to restart Apache whenever code changes
Add monitor.py in same level with your projects :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
import os
import sys
import time
import signal
import threading
import atexit
import Queue

_interval = 1.0
_times = {}
_files = []

_running = False
_queue = Queue.Queue()
_lock = threading.Lock()

def _restart(path):
    _queue.put(True)
    prefix = 'monitor (pid=%d):' % os.getpid()
    print >> sys.stderr, '%s Change detected to '%s'.' % (prefix, path)
    print >> sys.stderr, '%s Triggering process restart.' % prefix
    os.kill(os.getpid(), signal.SIGINT)

def _modified(path):
    try:
        # If path doesn't denote a file and were previously
        # tracking it, then it has been removed or the file type
        # has changed so force a restart. If not previously
        # tracking the file then we can ignore it as probably
        # pseudo reference such as when file extracted from a
        # collection of modules contained in a zip file.

        if not os.path.isfile(path):
            return path in _times

        # Check for when file last modified.

        mtime = os.stat(path).st_mtime
        if path not in _times:
            _times[path] = mtime

        # Force restart when modification time has changed, even
        # if time now older, as that could indicate older file
        # has been restored.

        if mtime != _times[path]:
            return True
    except:
        # If any exception occured, likely that file has been
        # been removed just before stat(), so force a restart.

        return True

    return False

def _monitor():
    while 1:
        # Check modification times on all files in sys.modules.

        for module in sys.modules.values():
            if not hasattr(module, '__file__'):
                continue
            path = getattr(module, '__file__')
            if not path:
                continue
            if os.path.splitext(path)[1] in ['.pyc', '.pyo', '.pyd']:
                path = path[:-1]
            if _modified(path):
                return _restart(path)

        # Check modification times on files which have
        # specifically been registered for monitoring.

        for path in _files:
            if _modified(path):
                return _restart(path)

        # Go to sleep for specified interval.

        try:
            return _queue.get(timeout=_interval)
        except:
            pass

_thread = threading.Thread(target=_monitor)
_thread.setDaemon(True)

def _exiting():
    try:
        _queue.put(True)
    except:
        pass
    _thread.join()

atexit.register(_exiting)

def track(path):
    if not path in _files:
        _files.append(path)

def start(interval=1.0):
    global _interval
    if interval < _interval:
        _interval = interval

    global _running
    _lock.acquire()
    if not _running:
        prefix = 'monitor (pid=%d):' % os.getpid()
        print >> sys.stderr, '%s Starting change monitor.' % prefix
        _running = True
        _thread.start()
    _lock.release()

And add this into last line of django_wsgi.py:

1
2
import monitor
monitor.start(interval=1.0)

Restart your apache to apply this monitor settings.

,

OS

Fix slow update query MySQL in Django Session table

5 Mar , 2013  

After using new relic, i found there is query that took more than long time to finihed. After investigate, I found that “update query” on django session is the most expensive and slowest query. Here is the query:

1
UPDATE `django_session` SET `session_data` = %s, `expire_date` = %s WHERE `django_session`.`session_key` = %s

I’m wondering what the reason that queries need a lot of time to be finished. Fyi, there is a million records under this table. Since i don’t have deep understanding on how MySQL works, then I start reading a few stuff that related with this problem.

1. Internal Locking
There are 2 action of locking, read and write locking. First case, we want to make select query on a table which mean “read”. MySQL will check if there is lock on selected table or not. If no lock found, then MySQL will put “read lock” on this table. But, ff there is a lock on this table, then this “read lock” will put into queue.

Same thing with “write”, for instance we want to update records. MySQL will looking for lock on table, if no lock, then it will put “write lock” on this table or put into queue if lock found. The difference here that “write lock” have higher priorities than “read lock”.

Detail : http://dev.mysql.com/doc/refman/5.0/en/internal-locking.html

2. Primary Indexes and Data Type
In django session, it will match the “session_key” and update it’s value. Fyi, “session_key” is the primary key in django session table which that’s mean this column is being used as pointer. Primary key can be INT, CHAR and VARCHAR. Is that the data type of this primary key is matter? My answer is yes it is. In this cases, i’ll refering for MyISAM engine.

When we make query, then MyISAM will retrieve the primary keys, put this list of primary keys into index blocks (to minimize disk I/O) and arrange them into Binary tree. For instance, 1 index block have 1024 byte. If our primary keys is INT (4 bytes) (http://dev.mysql.com/doc/refman/5.0/en/integer-types.html), that’s mean each single index can hold 256 items. Meanwhile, for varchar that have 11 character, it’s have size about 12-13 bytes which need 1-2 bytes as prefix that contain length of value. That means 1 index block only can hold about 85 items.

Based on this two documentation about how MySQL works, I’m understand why “update” query on session table takes long time. Then I start to thinking about solution :

1. Reduce records
“Session” table might have the most requests in Django Apps. Because mostly each action need to be validated and that’s mean requests on this session table. Since I see more than million records on this table, reduce it into 100K records will give a big difference. Better run clean-up scripts from session that expired since one days ago. Some documentation related with this clean-up:
https://docs.djangoproject.com/en/dev/topics/http/sessions/#s-clearing-the-session-store

2. Optimize MySQL
MySQL configuration have options that we can tune and increasing performance by configure it correctly. Also use MySQL query cache wisely ( http://www.mysqlperformanceblog.com/2006/07/27/mysql-query-cache/ )

Anyway, if our session table is too big, then we can delete it manually from MySQL-client by :

1
delete from django_session where expire_date <= NOW();

,

Uncategorized

Solve Django OperationalError: (2013, ‘Lost connection to MySQL server during query’)

4 Mar , 2013  

I got this errors when running my scripts that using multiprocessing to speed-up the task.

1
OperationalError: (2013, 'Lost connection to MySQL server during query')

Apparently, this problems occurs because another process try to open database connection that already opened by Django.
To solve this problem, just close django database connection in your target worker function with:

1
2
from django import db
db.close_connection()

Uncategorized

Setup Debian Server on HPCloud for production

2 Mar , 2013  

Here are quick steps to setup Debian server for production in HPCloud. I use Debian 6 on this example.

1. Install basic software

1
2
3
apt-get update && apt-get upgrade
apt-get install nmap vim bind9 mysql-server mysql-client apache2  php5 php-pear php5-suhosin php5-mysql python-dev nginx php5-common phpmyadmin apache2 zip unzip unrar-free imagemagick rsync python-pip dnsutils libxml2 libxml2-dev libxslt1-dev libbz2-dev libssl-dev p7zip-full rar lha unrar unzip unace unp bzip2 gzip patch htop screen
a2enmod rewrite

2. Install php5-fpm on Debian
First, edit “/etc/apt/sources.list” and add :

1
deb http://packages.dotdeb.org stable all

Then start adding GPG Key and install software by :

1
2
3
4
5
wget http://www.dotdeb.org/dotdeb.gpg
cat dotdeb.gpg | apt-key add -
rm dotdeb.gpg
apt-get update
apt-get install php5 php5-fpm php-pear php5-common php5-mcrypt php5-mysql php5-cli php5-gd

3. Install NodeJS in Debian

1
2
3
4
5
6
7
8
9
apt-get update && sudo apt-get install git-core curl build-essential openssl libssl-dev
git clone https://github.com/joyent/node.git
cd node
git checkout v0.9.5 (or the version you want to build)
./configure --openssl-libpath=/usr/lib/ssl
make
sudo make install
node -v
npm -v

OS

Solve Permission denied (publickey) EC2 to New Server

2 Mar , 2013  

I use Debian AMI for this example. Suddenly, I can’t SSH into new server that created. Even worse, i can’t made SSH connection from one server to another server in Amazon EC2. Apparently, the problem is the way of make SSH connection.

This is what Amazon EC2 suggestion:

1
ssh -i my.pem root@<my-ip-address>

And it give me result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Found key in /root/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: east-micro.pem
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

How about :

1
ssh -l root -i my.pem <my-ip-address>

And it works! I say that it’s weird!

,

OS

Solve MySQL Incorrect key file for table /tmp repair it in EC2

2 Mar , 2013  

Today i got this errors while installing MySQL in my Debian :

1
ERROR 126 (HY000) at line 1: Incorrect key file for table '/tmp/#sql_45cf_0.MYI'; try to repair it

This errors usually refering for out of space in our server. I just replace server with new EBS which have more space. Then after doing “df -hT”, i found something wrong here :

1
2
3
4
5
6
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/xvda1    ext3    5.0G  1.1G  3.7G  23% /
tmpfs        tmpfs    306M   12K  306M   1% /lib/init/rw
udev         tmpfs    285M   28K  285M   1% /dev
tmpfs        tmpfs    306M  4.0K  306M   1% /dev/shm
overflow     tmpfs    1.0M     0  1.0M   0% /tmp

There is overflow on filesystem which supposed to be not happen. So, what the solution? Is easy! just reboot your server and overflow will gone and “/tmp” will come back to your /dev/sda1.

OS

Ec2 fix ebs volume space

2 Mar , 2013  

When creating new EBS from Snapshot in EC2, this EBS will using the old space. For instance, snapshot was 1GB and we launch new EBS with 5GB capacity. After attaching into EC2 server, we’ll find out that the space still the old one (1GB). To fix this EBS volume space, just use “resize2fs”:

1
resize2fs /dev/xvda1