Python

Example multiprocessing in Python

27 Feb , 2013  

Here is example of using multiprocessing and queue in Python.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
from multiprocessing import Process, Queue
from multiprocessing.process import current_process


def arbitary_function(number):
    print "Executing number %s from %s" % (number, current_process().name)
    return True

def worker(worker_queue, result_queue):
    try:
        for number in iter(worker_queue.get, None):
            arbitary_function(number)
            result_queue.put("%s success with: %s" % (number,
                                                      current_process().name))
    except Exception, e:
        result_queue.put("%s failed with: %s" % (current_process().name,
                                                       e.message))


if __name__ == "__main__":
    numbers = [x for x in range(1000)]
    worker_queue = Queue()
    result_queue = Queue()

    for n in numbers:
        worker_queue.put(n)

    core_worker = 2
    workers = [Process(target=worker, args=(worker_queue, result_queue)) for i in range(core_worker)]

    for each in workers:
        each.start()

    for r in iter(results.get, None):
        print r

You can get the results from “result_queue.get()” and keep in mind that queue is generator.

It will give results:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Executing number 0 from Process-1
Executing number 1 from Process-1
Executing number 2 from Process-1
Executing number 4 from Process-2
Executing number 3 from Process-1
Executing number 5 from Process-1
Executing number 6 from Process-1
Executing number 7 from Process-1
Executing number 8 from Process-1
Executing number 9 from Process-2
Executing number 10 from Process-1
Executing number 11 from Process-2
Executing number 12 from Process-1
Executing number 13 from Process-2
Executing number 14 from Process-2
Executing number 15 from Process-1
Executing number 16 from Process-2
Executing number 17 from Process-1
Executing number 18 from Process-2
Executing number 19 from Process-1
Executing number 20 from Process-2

Fyi, If we want to run several multipleprocess in asynchronous way, better don’t use join(). Because it will block main process until all process finished.

,

Python

Example using Python Oauth2 to access Twitter

23 Feb , 2013  

Here is a quick example how to use Python Oauth2 to gain access into Twitter account. We need to install “oauth2″ python package first by :

1
pip install oauth2

Then here is a quick example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import oauth2 as oauth
import json

# Create your consumer with the proper key/secret.
consumer = oauth.Consumer(key="xxxxxxxxxxxxxxxx",
                          secret="xxxxxxxxxxxxxxxx")

# Request token URL for Twitter.
request_token_url = "https://api.twitter.com/oauth/request_token"

# Create our client.
client = oauth.Client(consumer)

# The OAuth Client request works just like httplib2 for the most part.
resp, content = client.request(request_token_url, "GET")
print json.dumps(resp, indent=4)
print json.dumps(content, indent=4)

It will showing results :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{
    "status": "200",
    "content-length": "144",
    "content-location": "https://api.twitter.com/oauth/request_token?oauth_body_hash=xxxxx&oauth_nonce=48355082&oauth_timestamp=1361601089&oauth_consumer_key=xxxx&oauth_signature_method=HMAC-SHA1&oauth_version=1.0&oauth_signature=xxxxLzwwvw1NtMvuD0PKA%3D",
    "x-transaction": "xxxx36f9200946491",
    "set-cookie": "k=xx.37xx0.136xxx509550; path=/; expires=Sat, 02-Mar-13 06:31:30 GMT; domain=.twitter.com, guest_id=v1%3A136xxx052057281; domain=.twitter.com; path=/; expires=Mon, 23-Feb-2015 18:31:30 GMT, _twitter_sess=BAh7CDoPY3JlxxxxxZjg3ZGYzMjhjYTdiNzM2MDQ1--1bxxxxx8ec0a; domain=.twitter.com; path=/; HttpOnly",
    "expires": "Tue, 31 Mar 1981 05:00:00 GMT",
    "x-mid": "b892643948470824023ffee4beed31c48388104a",
    "server": "tfe",
    "last-modified": "Sat, 23 Feb 2013 06:31:30 GMT",
    "x-runtime": "0.02175",
    "etag": ""d657e3e63axxxx83cea7d"",
    "pragma": "no-cache",
    "cache-control": "no-cache, no-store, must-revalidate, pre-check=0, post-check=0",
    "date": "Sat, 23 Feb 2013 06:31:30 GMT",
    "x-frame-options": "SAMEORIGIN",
    "content-type": "text/html; charset=utf-8",
    "-content-encoding": "gzip",
    "vary": "Accept-Encoding"
}
"oauth_token=xxxxxxxxxxxx&oauth_token_secret=xxxxx&oauth_callback_confirmed=true"

Web

Javascript URL window location works on Chrome but fail on Firefox

21 Feb , 2013  

This is commmon things that happen when we’re using “window.location.origin” to define the path for AJAX call or another purpose.
Actually, “window.location.origin” isn’t in browser standard. Firefox try the best to obey the standard which “window.location.origin” is not supported hence it give “undefined” result.

Substitution for this, you can use:

1
window.location.protocol + "//" + window.location.host

,

OS

Top showing wrong amount of memory usage in server

20 Feb , 2013  

“Top” (Go to console and type top) became a handy tools to track what applications that currently running in our server. But, if we’re using it for showing how much actual memory usage in server, that’s WRONG!
Why?

First, see this screenshot of my Fedora with my total RAM is 3.6 GB:

Top results (Free: 1.6 GB)

System Monitor: (Usage: 1.1GB)

See a different here? Well, if you google, actualy you can find a few articles already share about this. But for your information, Linux kernel use available memory for disk cache. Meaning, if you have 4GB RAM, your application ate 1GB, then for 3GB will be used for disk cache. Why disk cache? Because hardisk is slow here (SSD is pretty fast anyway). By cache them on memory, it will make the process became faster. But, you’re opening new application, then immediately Linux kernel will give this available memory to your new application.

Reference:
http://linuxtoolkit.blogspot.sg/2010/02/understanding-memory-usage-from-top-and.html
http://vivekrajendran.wordpress.com/tag/linux-incorrect-memory-usage/

Django

Django queryset distinct foreign key and count

19 Feb , 2013  

What we want to do here is:
1. We want distinct foreign key from the records
2. We want this unique foreign key contains number of records that belongs to.

Example:
Country -> has many -> Insurance

1
2
3
4
5
Country(models.Model):
    name = models.Charfields(..)

Insurance(model.Model):
    country = models.ForeignKey(Country)

We want to distinct Country in Insurance and show numbers of insurance in each unique Country.

1
Insurance.objects.order_by("country").values_list("country__name").distinct().annotate(Count("id"))

Django

Add export data into CSV files action in Django Admin

14 Feb , 2013  

Having “export data as CSV” action in Django Admin sometimes it useful to exchange data into another platform like Excel. To make this feature, we need to create action in Django admin.

1. Create admin export csv action
Create “action.py” in your app. Then, in this files, we create function to do queryset and export data into CSV files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
import csv
from django.http import HttpResponse
from datetime import datetime

def export_to_csv(modeladmin, request, queryset):
    """Exporting queryset results and filter into CSV"""
    # limit only staff and super_user accounts
    if request.user.is_staff or request.user.is_superuser:
        if queryset.count() > 0:
            # generate response and filename
            response = HttpResponse(mimetype="text/csv")
            today = datetime.now().strftime("%Y-%M-%d_%H:%M:%S")
            filename = "records-%s.csv" % today
            response["Content-Disposition"] = ('attachment; filename="%s"' %
                                               filename)
            writer = csv.writer(response)

            # Get column name
            columns = [field.name for field in queryset[0]._meta.fields]
            writer.writerow(columns)

            # Write data
            for obj in queryset:
                fields = map(lambda x: generate_value(obj, x), columns)
                writer.writerow(fields)

            return response

export_to_csv.short_description = "Export results to CSV"


def generate_value(obj, column):
    """Get fields value and convert to ASCIC for string type"""
    row = getattr(obj, column)
    if isinstance(row, basestring):
        row = row.encode('ascii', 'ignore')
    return row

This script will export “selected data” into CSV.
More…

,

Python

Example implementation of custom EventLoop in Pyglet

11 Feb , 2013  

Eventloop in Pyglet is a great feature when we want to control or integrate run() into another loop (eg: reactor.run() on twisted). In their documentation, http://www.pyglet.org/doc/programming_guide/customising_the_event_loop.html, the example of using custom eventloop in pyglet isn’t clear.

Here is example how to use EventLoop() in Pyglet to control the loop without pyglet.app.run():
More…

,

OS

How to draw cirle / ball objects in PyGame

3 Feb , 2013  

There is a lack documentation about how to draw circle or ball in Pygame. Here is an example taken from my game source code:

1
2
3
4
5
6
self.ball = pygame.Surface((25, 25))
self.ballrect = pygame.draw.circle(self.ball, (0, 255, 0), (15 / 2, 15 / 2), 15 / 2)
self.ball = self.ball.convert()
self.ball.set_colorkey((0, 0, 0))
self.ballrect.topleft = [0, 0]
self.screen.blit(self.ball, self.ballrect)

,

Python

Example bouncing ball using PyGame

2 Feb , 2013  

We will create a simple example of bouncing ball using Pygame. I use Fedora 18 and for installation and using Pygame “1.9.1release” from Fedora repository. Here is small code to make bouncing ball using Pygame:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
import sys
import os
import pygame

CURRENT = os.path.abspath(os.path.dirname(__file__))
sys.path.insert(0, CURRENT)

pygame.init()

size = width, height = 640, 400
speed = [1, 1]
black = 0, 0, 0

screen = pygame.display.set_mode(size)

ball = pygame.image.load(CURRENT + "/ball.png")
ballrect = ball.get_rect()
ballrect.topleft = [0, 0]

screen.blit(ball, ballrect)
   
while 1:
    for event in pygame.event.get():
        if event.type == pygame.QUIT: sys.exit()

    screen.fill(black)
    screen.blit(ball, ballrect)
   
    ballrect = ballrect.move(speed)
   
    if ballrect.left > width or ballrect.right > width:
        speed[0] = -speed[0]
    elif ballrect.left < 0:
        speed[0] = +abs(speed[0])
       
    if ballrect.top > height or ballrect.bottom > height:
        speed[1] = -speed[1]
    elif ballrect.top < 0:
        speed[1] = +abs(speed[1])

    pygame.display.flip()

It will give us results :

Remember to create your “ball.png” and put in the same folder with this scripts.

Update:
Here is the same scripts in decent format:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
import sys
import os
import pygame

class Game():
    def __init__(self):
        CURRENT = os.path.abspath(os.path.dirname(__file__))
        pygame.init()

        size = self.width, self.height = 640, 400
        self.speed = [1, 1]
        self.black = 0, 0, 0

        self.screen = pygame.display.set_mode(size)
       
        self.ball = pygame.image.load(CURRENT + "/ball.png")
        self.ballrect = self.ball.get_rect()
        self.ballrect.topleft = [0, 0]

        self.screen.blit(self.ball, self.ballrect)


    def start(self):
        while 1:
            for event in pygame.event.get():
                if event.type == pygame.QUIT: sys.exit()

            self.screen.fill(self.black)
            self.screen.blit(self.ball, self.ballrect)

            self.ballrect = self.ballrect.move(self.speed)

            if (self.ballrect.left > self.width or
                self.ballrect.right > self.width):
                self.speed[0] = -self.speed[0]
            elif self.ballrect.left < 0:
                self.speed[0] = +abs(self.speed[0])

            if (self.ballrect.top > self.height or
                self.ballrect.bottom > self.height):
                self.speed[1] = -self.speed[1]
            elif self.ballrect.top < 0:
                self.speed[1] = +abs(self.speed[1])

            pygame.display.flip()

if __name__ == "__main__":
    game = Game()
    game.start()

,