Django

Boost Django running unit testing faster with mysql database loaded into Ramdisk / Memory

27 Sep , 2012  

Running unit testing in Django using “python manage test ” is not enough. Usually unit-testing takes a lot of time to make it done. FYI, I use Fedora 17 and MySQL for my development and usually, it takes about 2-4 minutes to get testing on single application done. Using SQLite is not good option since I always want to running unit-testing under production database which have MySQL + INNODB. Yes, you can say that InnoDB will make it slower rather MyISAM. But, again, I want to run unit testing on the production database architecture.

After googling, I found that “Load MySQL into RamDisk” will boost performance since it will stored into memory. This is interesting, then i give it a shot! Here we go:

1. Mount ramdisk
We mounting tmpfs with size 1GB into /mnt/ramdisk and set permission to “mysql”.

1
2
3
sudo mkdir -p /mnt/ramdisk
sudo mount -t tmpfs -o size=1G tmpfs /mnt/ramdisk
sudo chown mysql:mysql /mnt/ramdisk

More…

,

Python

How to compare and get only the matched between two list in Python

26 Sep , 2012  

There is conditions when we need to matching two of big list of data which it’s called “intersection”.
Python is a great language which provide many ways to achieve solution of this problem. First, we have the list datasets:

1
2
first = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
second = [2, 4, 5, 8, 9]

Now the solutions:

1. List comprehension
List comprehension provide concise way to doing map(), filter() or reduce().

1
[x for x in first if x in second]

2. Joining sets
We can doing intersection by joining two sets together

1
sorted(list(set(first) & set(second)))

More…

,

Django

Solving Django fixtures unit testing load data IntegrityError: (1452, ‘Cannot add or update a child row: a foreign key constraint fails using INNODB engine

26 Sep , 2012  

When you’re using MySQL with InnoDB for unit-testing, you may stumbled into this problems when try loading fixtures in test case.

1
IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`DATABASE`.`TABLE_NAME`, CONSTRAINT `COLUMN_id_refs_id_ea62e552` FOREIGN KEY (`room_id`) REFERENCES `ANOTHER_TABLE` (`id`))')

You may solve this problem with this quickfix solution:

1. Change your database ENGINE into MyISAM when running unit-testing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
options = "INNODB"
import sys
if 'tests' in sys.argv:
    options = "MYISAM"

DATABASES = {
     'default': {
         'ENGINE': 'django.db.backends.mysql',
         'NAME': 'tripvillas_dev',
         'USER': 'root',
         'PASSWORD': 'trip',
         'HOST': '',
         'PORT': '',
         'OPTIONS': {'init_command': 'SET storage_engine=' + options},
     },
}

More…

,

Python

Solve python OSError: [Errno 24] Too many open files: ‘/dev/urandom’

25 Sep , 2012  

When you’re using random intensively, then at some conditions, you will get this errors:

1
OSError: [Errno 24] Too many open files: '/dev/urandom'

And this will makes another tasks that need to generate random getting affected.

To solve this issue, you need to raise OS-level limitation of /dev/random or /dev/urandom. I use Fedora 17 at this example, which this limitation is handle by:

1
/etc/security/limit.conf

And you can modify it:

1
2
3
4
#<domain>      <type>  <item>         <value>
#
*               soft     nofile          65535
*               hard     nofile          65535

Unfortunately, you need to reboot server to make OS obey this limitation and opening access for 65535 request to /dev/random or /dev/urandom.

,

Java

Understanding Getter and Setter in Java

24 Sep , 2012  

Getter and Setter is new things for me since I have experience in Python which all attribute in a Class can be invoked anytime. It’s different in Java which all data should be thread-safe and manipulation between data should be isolated and not interfere another process/task.

Btw, did i mean “encapsulation” here? Well, there are a lot of long debate about Getter/Setter whether is part of encapsulation or not. But from my own perspective, Getter/Setter is more likely integration and interfacing.

Imagine, we have a Class that provide calculation of Student behaviour. We have sub-routine/function inside this class for processing student name. This function called “firstname”. There is another class which want to use this firstname() to get the student name. Then, Getter/Setter is the player here. Getter/Setter will invoke method firstname() to be able use by another class.
More…

,

IDE

Solve how to compile and run Java in SublimeText 2

24 Sep , 2012  

I found a lot of people discussing about how to compile and run the results of Java (single file) in SublimeText 2. Well, after trying another alternative and failing, finally I found how to make it works. It’s easy actualy, all you need are:

1. Edit SublimeText2 Java Build

1
~/.config/sublime-text-2/Packages/Java/JavaC.sublime-build

More…

,

Server

Solve slow DNS in HpCloud due to double reload on the website

24 Sep , 2012  

I found something weird with my Bind in HPCloud which it’s working well in my EC2. After trying several experiment, I found the correct way to set bind in HPCloud. This is for example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
;
; BIND data file for local loopback interface
;
$TTL    3600
@       IN      SOA     obroll.com. admin.obroll.com. (
                             30        ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
obroll.com  IN  A       15.185.178.20
@       IN      NS      ns1.obroll.com.
@       IN      A       15.185.178.20
ns1     IN      A       15.185.178.20
ns2     IN      A       15.185.178.20
www     IN      A       15.185.178.20

Yes, I put my domain name as first 🙂

Fedora

Configure Phpmyadmin in NGINX to combine with NODEJS or another non-php applications

21 Sep , 2012  

Configuring Phpmyadmin to be able working with another non-PHP applications is a little bit tricky. For instance, you have NODEJS application running in NGINX with root (“/”). When you’re opening http://yourdomain.com, then it will running NODEJS applications. Then, you want to make this domain have “/phpmyadmin” which it will running phpmyadmin. At this steps, you should handle this path from NGINX and pass it into PHP-FASTCGI.

This is example NGINX Configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
server {
    listen 80;
    server_name yourdomain.com;

    location ~ ^/(js/|stylesheets/|images/|img) {
        root /mypath/public;
        access_log off;
        expires max;
    }

    location / {
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
       
        # proxy_http_version 1.1;
        proxy_pass http://127.0.0.1:3000;
    }
}

More…

Django

Export SELECT WHERE query MYSQL into .sql files using mysqldump

21 Sep , 2012  

You will need this scripts if you have two databases and want to synchronize each other.

First, we need to export SELECT QUERY into .sql files by:

1
mysqldump --databases DATABASE_NAME --tables TABLE_NAME --where='YOUR_COLUMN_NAME IN ("SOME-ID", "SOME-ID")' -u root -pYOUR_PASSWORD > data.sql

The last thing, you need to edit this “sql” files and change the DATABASE_NAME into your another DATABASE_NAME that will imported.
Then you can import this into your ANOTHER_DATABASE by:

1
mysql -u root -pYOUR_PASSWORD DATABASE_NAME --database=YOUR_ANOTHER_DATABASE < data.sql

Django

Django test form errors and validation results in Unit Testing

21 Sep , 2012  

Testing forms in Unit testing Django is necessary and important. Instead of using “self.client” for opening URL, we can post data into form in the webpage. We want to testing different input into forms and seeing how forms handle these input.

Here is an example how to testing form in Django unit-testing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from django.test import TestCase
from django.test.client import Client
from django.core.urlresolvers import reverse

class FormTestingExample(TestCase):
    """Testing Django Form"""
    def setUp(self):
        self.client = Client()

    def testing_some_form(self):
        data = {"key": "value"}
        response = self.client.post(reverse("app:namespace"), data, follow=True)
        self.assertEqual(response.status_code, 200)

        print response.context['your-form'].errors

Yes, all forms results can be accessed in context inside response.
Your form value and errors will be stored inside context.

Here is some example to test form errors and validation:

1
2
3
# Must show error because data invalid
msg = ("Please insert name")
self.assertEqual(response.context['my_form'].errors['name'][0], msg)

Now you can test all your forms 🙂

,

Django

Test Django session in unit testing for created and modified session by views

20 Sep , 2012  

Sometimes we need to test Django session in our unit testing to see that view already modified / created a session. Basically, we only need to use File Backend Storage to handle sessions that made in Unit testing. So, let starts with create a new test :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from importlib import import_module

from django.test import TestCase
from django.test.client import Client
from django.core.urlresolvers import reverse
from django.conf import settings

class ExampleTestCase(TestCase):
    """Example Unit Testing"""

    def setUp(self):
        # Set session using file backend
        # http://code.djangoproject.com/ticket/10899
        settings.SESSION_ENGINE = 'django.contrib.sessions.backends.file'
        engine = import_module(settings.SESSION_ENGINE)
        store = engine.SessionStore()
        store.save()

        self.session = store
        self.client.cookies[settings.SESSION_COOKIE_NAME] = store.session_key

    def tearDown(self):
        store = self.session
        os.unlink(store._key_to_file())

More…

,

Fedora

Solving Alacarte can’t open gi._glib.GError: Error opening file: No such file or director in Fedora

19 Sep , 2012  

I see a lot of people have problem with alacarte but seems nobody know how to fix it quickly.
When you run “alacarte” from your command line, you will see :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Traceback (most recent call last):
  File "/usr/bin/alacarte", line 36, in <module>
    main()
  File "/usr/bin/alacarte", line 33, in main
    app.run()
  File "/usr/lib/python2.7/site-packages/Alacarte/MainWindow.py", line 64, in run
    self.loadMenus()
  File "/usr/lib/python2.7/site-packages/Alacarte/MainWindow.py", line 203, in loadMenus
    self.on_menu_tree_cursor_changed(menu_tree)
  File "/usr/lib/python2.7/site-packages/Alacarte/MainWindow.py", line 354, in on_menu_tree_cursor_changed
    self.loadItems(self.menu_store[menu_path][2], menu_path)
  File "/usr/lib/python2.7/site-packages/Alacarte/MainWindow.py", line 218, in loadItems
    icon = util.getIcon(item)
  File "/usr/lib/python2.7/site-packages/Alacarte/util.py", line 151, in getIcon
    pixbuf = info.load_icon()
  File "/usr/lib64/python2.7/site-packages/gi/types.py", line 43, in function
    return info.invoke(*args, **kwargs)
gi._glib.GError: Error opening file: No such file or directory

When, this problem occured because “ONE of your desktop application can’t find icon that has been referenced before”. Why this happen? Seems you’re deleting your applications and Alacarte won’t start because it’s can’t find your applications.

How to fix it? It’s simple! all of your Alacarte app-made located in

1
~/.local/share/applications

You will see alacarte desktop like:

1
2
3
alacarte-made-1.desktop
alacarte-made-2.desktop
alacarte-made-3.desktop

Just open this files and see which one that have applications deleted / inaccessible / wrong path.
You can remove them or edit into correct path.

Soon after you fix it, then you can start your “alacarte” again 🙂

,

OS

Solve Fedora 17 boot problem repair filesystem fsck

19 Sep , 2012  

It’s weird when my Fedora suddenly crash and it’s give this error message when boot-up and asking for “repair filesystem”:

1
2
3
4
5
6
7
8
9
10
11
12
Loading initial ramdisk...
_Fedora-16-Beta-: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options)
dracut Warning: e2fsck returned with 4
dracut Warning:_Fedora-17-contains a file system with errors, check forced.
dracut Warning:_Fedora-17: Inodes that were part of a corrupted orphan linked list found.
dracut Warning: * An error occured during the file system check.
dracut Warning: * Dropping you to a shell; the system will try
dracut Warning: * to mount the filesystem(s), when you leave the shell.
dracut Warning:
Dropping to debug shell.
sh: can't access tty; job control turned off
(Repair filesystem):/#

The solution for this cases is easy:

1. Get your root partition :

1
blkid

For instance, I have fedora installed in “/dev/sda6”

1
2
3
4
5
6
/dev/sda1: SEC_TYPE="msdos" LABEL="DELLUTILITY" UUID="3030-3030" TYPE="vfat"
/dev/sda2: LABEL="Recovery" UUID="CC70378A703779F2" TYPE="ntfs"
/dev/sda3: LABEL="OS" UUID="AC7C4EC27C4E86D4" TYPE="ntfs"
/dev/sda5: LABEL="DATA" UUID="2E24594324590EE9" TYPE="ntfs"
/dev/sda6: LABEL="_Fedora-17-x86_6" UUID="861accd0-45a3-4129-a477-f931e5dc9816" TYPE="ext4"
/dev/sda7: UUID="87c4e897-c3b7-467e-b275-2838e50cc1a5" TYPE="swap"

2. Repair

1
fsck /dev/sda6 -vY

Do hard reboot and you will get your Fedora back! Remember to have backup of your HDD 🙂

,

Django

Solve local settings not being imported from settings.py in Django

18 Sep , 2012  

Usually we create local_settings.py for overwrite settings.py configuration due to environment or custom conditions.
We can put local_settings.py into django project that have same place with settings.py.
To load local_settings.py, we can use :

1
2
3
4
try:
    from local_settings import *
except ImportError:
    pass

But problem come when local_settings not being called by settings.py.
This problem usually because you have import python modules in local_settings but it’s doesn’t installed.

To test, open your django shell (python manage.py shell) and import local_settings.

1
2
python manage.py shell
import local_settings

Then you will see error which python module that not installed 🙂

,

Python

Create an independent Python logger and save reports into file

17 Sep , 2012  

This is just a trivial task which we want to run a python files and logging all actions.
We want to save all the reports into file. Then, you can just doing this way:

1
2
3
4
5
6
7
8
9
10
11
12
13
import os
import logging

logger = logging.getLogger('test')
log_path = os.path.abspath(os.path.join(os.path.dirname(__file__), 'test.log'))
log_file = logging.FileHandler(log_path)
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
log_file.setFormatter(formatter)

logger.addHandler(log_file)
logger.setLevel(logging.INFO)

logger.error("Ouch! it's hurt!")

Explanation:

1.getLogger(‘test’)
We need to create logger name. This can be whatever name.

2. log_path
This is where the log stored.

3. log_file
This is we tell logging FileHandler to get where the log file will be created.

4. formatter
This is contain format style of logs

5. logger.addHandler & logger.setLevel
This is we want to attach handler and level into logger that we want to use to log anything.

OS

Solve MYSQL: Cannot delete or update a parent row: a foreign key constraint fails InnoDB when DROP table

17 Sep , 2012  

There is conditions when we need to remove table / data in InnoDB and replaced with the new one. But the problems is, you will facing this error when DROP table that contain relational data :

1
MYSQL: Cannot delete or update a parent row: a foreign key constraint fails

How to solve this? It’s actually easy, open your mysql and run this command:

1
SET foreign_key_checks = 0;

This will DISABLE FOREIGN KEY CHECKS that running by INNODB when you’re try to DROP table / deleting data.
After you running that commmand, then you can start deleting data.

Make sure table / data deleted successfully by repeating your “DELETE” twice / three times.
Then, after finish, you should switch this feature back by:

1
SET foreign_key_checks = 1;

,

Python

ValueError: “unsupported format character ‘a’ (0x61) at index 9” in Python

14 Sep , 2012  

Do you ever got this error message when running code like:

1
print("Hello world this should be a long paragraph and you need %s, %s and some of % needed by %s" % (var1, var2, var3, var4))

Well, when this problem show-up, it’s not about Unicode and you should convert all variable value into ASCII.
This error will show if you forgot to put “s” after “%”.

1
2
print("Hello world %" % var1)
ValueError: "unsupported format character 'a' (0x61) at index 9" in Python

Solution:

1
print("Hello world %s" % var1)

Seems you’re tired and need a break for a while 🙂

,

Python

Solving retrieve objects from Amazon S3 using boto get_all_keys() and get limited results

14 Sep , 2012  

When you have more than 1000 objects in S3, you will realize that get_all_keys() doesn’t give you all objects you have.
This is because get_all_keys() have limitation. To fetch / retrieve all objects, please using list().

For example :

1
2
3
4
s3 = boto.connect_s3()
bucket = s3.get_bucket('your-bucket-name')
for obj in bucket.list():
    print obj.name

Now you can retrieve all objects in S3.

,

Server

Solving Bind DNS un-resolved when nslookup domain without WWW

13 Sep , 2012  

This is common mistake when we try to nslookup our domain that working with “WWW” but not without “WWW”. This is some bind9 configuration :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
;
; BIND data file for local loopback interface
;
$TTL    3600
@       IN      SOA     obroll.com. admin.obroll.com. (
                             30        ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      ns1.obroll.com.
ns1     IN      A       15.185.178.20
ns2     IN      A       15.185.178.20
www     IN      A       15.185.178.20

More…

,

Python

Convert string with tuple format into tuple objects in Python

10 Sep , 2012  

There will come condition when you have MySQL data that have structure like tuples in Python. For example, given data in categories.txt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
('Woodenware', 2594),
('Woodworking', 2595),
('Woodworking Equipment and Supplies', 2596),
('Word Processing Service', 2597),
('Work Clothing - Retail', 2598),
('Work Clothing - Whol and Mfrs', 2599),
('Wrecker and Towing Services', 2600),
('Wrecking Contractors', 2601),
('Writers', 2602),
('Writers Service', 2603),
('X-Ray Apparatus and Supplies', 2604),
('X-Ray Inspection Service', 2605),
('X-Ray Laboratories - Industrial', 2606),
('X-Ray Laboratories - Medical', 2607),
('X-Ray Protection', 2608),
('Yarn - Retail', 2609),
('Yoga Instruction', 2610),
('Yogurt - Retail Shops', 2611),
('Yogurt - Wholesale', 2612),
('Youth Organizations and Centers', 2613),
('Zoos', 2614)

More…

,

OS

Backup / migrating files into Amazon S3 bucket using RSYNC

10 Sep , 2012  

The idea is simple, we want to rsync our data into Amazon S3. FYI, I use Fedora 17 at this example.
We can use this powerfull tools called “s3fs” to mount S3 into our local. Download the latest version in http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz.

1. Install dependecies
To be able using this tools, you need to have :

1
sudo yum install fuse fuse-devel libcurl libxml2-devel libcurl-devel libstdc++-devel openssl-devel curl-devel mailcap

On Debian / Ubuntu :

1
apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support

2. Compile and install
Now you can extract by “tar -xvvf s3fs-1.61.tar.gz” and start compile :

1
2
3
./configure
make
sudo make install

More…

Python

Setup Amazon S3 backup system using Python with BOTO in Fedora

10 Sep , 2012  

Boto (https://github.com/boto/boto) is a must have tools when we need to communicate with Amazon Services using Python. Before we start, FYI, I use Fedora 17 in this articles. That’s mean several code may doesn’t works in your Linux OS (If we have different OS). But there are should be a minor issue since there no big difference between Linux distros. We can install this module by:

1
sudo pip-python install boto

You need AWS Access and Secret Key to be able accessing your Amazon account. You will find these values after logging in to your AWS account and clicking Account, then go to Security Credentials. Instead of defining AWS Access and Secret key each time we want to using Boto, we can put our configuration inside :

1. /etc/boto.cfg (For system-wide use)
2. ~/.boto (For spesific user use)
More…

,

Python

Why function overloading in Python doesn’t works meanwhile overriding is okay?

9 Sep , 2012  

This is basic question that frequently asked by a lot of people who learning Python. Basically, we know there are two method when calling function inside class :

1. Method Overriding
This function overiding allow Child or derived class to overwrite function in Parent class by define same function. This is common thing in inheritance method in Python. For example:

1
2
3
4
5
6
7
8
9
10
11
class Base(object):
    def show_status(self):
        print "Hello World"

class Child(Base):
    def show_status(self):
        print "Children replaced it!"

if __name__ == "__main__":
    data = Child()
    data.show_status()

More…

,

Python

Using Gevent to fetch urls website with python using non-blocking method

8 Sep , 2012  

I have a lot urls links and need to check which one that have 404 (Not Found) or Server Down. With python, everything looks possible and easy. We can use simple urllib2 module to fetch webpages based on given links. But, problem comes when we dealing with IO Bound. When we iterating links and doing fetchings, one process will waiting others process finished before it starting.

GIL, yes, i guarantee you have hear about Global Intrepeter Lock which make threading and multiprocessing became useless in multiple-core system. It caused by CPython memory management which not thread-safe ( http://wiki.python.org/moin/GlobalInterpreterLock ). More…

,

OS

How to replace primary storage with new EBS in Amazon EC2 instance

7 Sep , 2012  

This based on the need, we want create a big EBS, remove the old 8GB EBS and replace with the new one as primary storage. Using new EBS storage as primary is a little bit tricky and you should carefull when doing this. We should be careful when upgrading EBS storage in Amazon EC2. Make sure backup all data and databases before doing upgrading. Also, there no data added / changes in server in middle of migration because it may causes data loss.

First, you should make sure already backup all data in your instance. This is a list that you should notice :

1. /etc – in this folder will be contain bind, nginx, apache and a lot of configuration stuff
2. Backup your database
3. /var
4. Home folder – sometimes you put important stuff there
More…

Django

Make stand-alone python scripts to load Django environment

4 Sep , 2012  

Sometimes we need to make some executable scripts inside Django applications that standalone and independent.
Usually this for generate / processing data without accessing / open any Django views.

For instance, I have “Posts” application, I need to modified all Post and send them to some 3rdparty.
So, this will be:

1
2
3
4
project
   |___ posts
         |____ utils
                 |____ generate_post.py

Then, in generate_post.py, I just load Django environment by :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import os
import sys

CURRENT = os.path.dirname(__file__)
PROJECT_PATH = os.path.abspath(os.path.join(CURRENT, "..", ".."))

sys.path.append(PROJECT_PATH)

import settings
from django.core.management import setup_environ

setup_environ(settings)

# From here, you can start import any app models here
# Do whatever you want here

Yes, it’s simple. Basically, you need to load “settings” using “setup_environ”.
Before import settings, you should make sure to load django project path.