What's new in Django community blogs?

Django bugfix release issued: 1.10.4, 1.9.12, 1.8.17

Dec 01 2016 [Archived Version] □ Published at The Django weblog

Today we've issued the 1.10.4, 1.9.12, and 1.8.17 bugfix releases.

The release package and checksums are available from our downloads page, as well as from the Python Package Index. The PGP key ID used for this release is Tim Graham: 1E8ABDC773EDE252.


Django Under the Hood 2016 Recap

Nov 29 2016 [Archived Version] □ Published at Caktus Blog

Caktus was a proud sponsor of Django Under the Hood (DUTH) 2016 in Amsterdam this year. Organized by Django core developers and community members, DUTH is a highly technical conference that delves deep into Django. Django core developers and Caktus Technical Manager Karen Tracey and CEO/Co-founder Tobias McNulty both flew to Amsterdam to spend time...


Google Cloud Storage: Managing Files and Objects

Nov 29 2016 [Archived Version] □ Published at tuts+

In the first part of this two-part tutorial series, we had an overview of how buckets are used on Google Cloud Storage to organize files. We saw how to manage buckets on Google Cloud Storage from Google Cloud Console. This was followed by a Python script in which these operations were performed programmatically.

In this part, I will demonstrate how to manage objects, i.e. files and folders inside GCS buckets. The structure of this tutorial will be similar to that of the previous one. First I will demonstrate how to perform basic operations related to file management using Google Cloud Console. This will be followed by a Python script to do the same operations programmatically.

Just as bucket naming in GCS had some guidelines and constraints, object naming follows a set of guidelines as well. Object names should contain valid Unicode characters and should not contain Carriage Return or Line Feed characters. Some recommendations include not to have characters like "#", "[", "]", "*", "?" or illegal XML control characters because they can be interpreted wrongly and can lead to ambiguity.

Also, object names in GCS follow a flat namespace. This means physically there are no directories and subdirectories on GCS. For example, if you create a file with name /tutsplus/tutorials/gcs.pdf, it will appear as though gcs.pdf resides in a directory named tutorials which in turn is a subdirectory of tutsplus. But according to GCS, the object simply resides in a bucket with the name /tutsplus/tutorials/gcs.pdf.

Let's look at how to manage objects using Google Cloud Console and then jump onto the Python script to do the same thing programmatically.

Using Google Cloud Console

I will continue from where we left in the last tutorial. Let's start by creating a folder.

Create a folder or upload files directly to GCS

To create a new folder, click on the Create Folder button highlighted above. Create a folder by filling in the desired name as shown below. The name should follow the object naming conventions.

Creating a folder in GCS

Now let's upload a file in the newly created folder.

Uploading a file in GCS

After the creation, the GCS browser will list the newly created objects. Objects can be deleted by selecting them from the list and clicking on the delete button.

Delete an object from GCS

Clicking on the refresh button will populate the UI with any changes to the list of objects without refreshing the whole page.

Managing Objects Programmatically

In the first part, we saw how to create a Compute Engine instance. I will use the same here and build upon the Python script from the last part.

Writing the Python Script

There are no additional installation steps that need to be followed for this tutorial. Refer to the first part for any more details about installation or development environment.

gcs_objects.py

The above Python script demonstrates the major operations that can be performed on objects. These include:

  • creation of a new object in a bucket
  • listing of all objects in a bucket
  • deletion of a specific object

Let's see how each of the above operations looks when the script is run.

Conclusion

In this tutorial series, we saw how Google Cloud Storage works from a bird's eye view, which was followed by in-depth analysis of buckets and objects. We then saw how to perform major bucket and object related operations via Google Cloud Console. 

Then we performed the same using Python scripts. There is more that can be done with Google Cloud Storage, but that is left for you to explore.


How to Run Unix Commands in Your Python Program

Nov 29 2016 [Archived Version] □ Published at tuts+

Unix is an operating system which was developed in around 1969 at AT&T Bell Labs by Ken Thompson and Dennis Ritchie. There are many interesting Unix commands we can use to carry out different tasks. The question is, can we use such commands directly within a Python program? This is what I will show you in this tutorial.

The Unix command ls lists all files in the directory. If you put ls as is in a Python script, this is what you will get when you run the program:

This shows that the Python interpreter is treating ls as a variable and requires it to be defined (i.e. initialized), and did not treat it as a Unix command.

os.system()

One solution to this issue is to use os.system() from Python’s os module.

As mentioned in the documentation, os.system():

Executes the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations.

So, we can run the ls command in Python as follows:

This will return the list of files in your current directory, which is where your .py program is located.

Let’s take another example. If you want to return the current date and time, you can use the Unix command date as follows:

In my case, this was what I got as a result of the above script:

Tue May 24 17:29:20 CEST 2016

call()

Although os.system() works, it is not recommended as it is considered a bit old and deprecated. A more recommended solution is Python’s subprocess module call(args) function. As mentioned in the documentation about this function:

Run the command described by args. Wait for command to complete, then return the returncode attribute.

If we want to run the ls Unix command using this method, we can do the following:

Let’s see how we can return the date using the subprocess module, but let’s make the example more interesting.

The above example can be run more simply using check_output(), as follows:

The output of the above scripts is:

It is Tue May 24 19:14:22 CEST 2016

The above examples show the flexibility in using different subprocess functions, and how we can pass the results to variables in order to carry out further operations.

Conclusion

As we saw in this tutorial, Unix commands can be called and executed using the subprocess module, which provides much flexibility when working with Unix commands through its different functions. You can learn more about this module and its different functions from the Python documentation.


Google Cloud Storage: Managing Buckets

Nov 28 2016 [Archived Version] □ Published at tuts+

Google Cloud Storage (GCS) is a very simple and powerful object storage offering from Google as a part of its Google Cloud Platform (GCP). It provides a highly durable, scalable, consistent and available storage solution to developers and is the same technology that Google uses to power its own object storage. 

It is billable with a pay for what you use model, and GCP comes with a 60-day trial period, so it is free to try out to see if it fits the needs of your organization. GCS has different service levels (also called storage classes) which can be chosen as needed (detailed discussion on these is out of the scope of this tutorial). GCS can be used for a variety of purposes, such as serving static/dynamic website content, storing user-specific application files, disaster recovery, or making large data objects downloadable to users.

Those who have worked on GCP will know that everything in GCP revolves around projects. Each project can have many buckets around which the architecture of Google Cloud Storage is structured. Buckets are the basic containers on GCS that contain the data stored. These are used as basic blocks to organize your data and look like folders on an operating system, but they cannot be nested. 

Each bucket can contain any number of objects, which can be folders and/or files. A bucket is assigned a storage class and geographic location when being created. These settings can be specified while creating the bucket but cannot be changed later.

Buckets have specific naming conventions which need to be followed strictly, otherwise GCP will not allow you to create a bucket. Bucket names are globally unique, so they need to chosen in a way that prevents conflicts. However, a name used by a deleted bucket can be reused. 

Also, the name cannot be changed once it's been assigned to a bucket. The only solution if you want to change it is to create a new bucket with the desired name, move the contents from the previous bucket to the new one, and then delete the previous bucket.

In this tutorial, I will cover how to manage buckets from the Google Cloud Console. This is followed by a Python script where I will demonstrate performing the same operations programmatically.

Using Google Cloud Console

First, let's see how to manage buckets using the web user interface provided by GCP known as Google Cloud Console

Open Storage Browser in a web browser of your choice. If you are a first-time user, you will be prompted to create a project first. Also, an option will be shown to sign up for a free trial. Go ahead with the free trial signup, otherwise you will not be allowed to create a new bucket by yourself. By default, GCP only provides one free bucket per App Engine instance. 

Once done with all these formal processes, navigating to this page should open up the page shown below.

Google Cloud Storage Browser

To create a new bucket, click on the Create Bucket button highlighted above. Create a bucket by filling in a desired name as shown below. The name should follow the bucket naming conventions.

Creating a new bucket

After you've created a bucket, the GCS browser will list it. Buckets can be deleted by selecting them from the list and clicking on the delete button.

Delete a bucket

Clicking on the refresh button will populate the UI with any changes to the list of buckets without refreshing the whole page.

Managing Buckets Programmatically

First, let's create a Google Compute Engine instance as that will allow quick demonstration of the targeted concepts rather than dealing with extra authentication steps on local machines. To create a GCE instance, open the link and click on the Create Instance button as shown below.

Create a new Compute Engine instance

A form will come up asking for relevant details, which can be filled in at your convenience. Once the GCE instance is created, open up the SSH client as shown below, which by default opens in a new browser window.

SSH into Compute Engine instance

The SSH client screen will look something like as shown below. All the further operations in this tutorial will be done directly on the SSH client itself.

SSH Client for newly created Compute Engine instance

Writing a Python Script

Below are the commands you need to run to set up the newly created server for a Python development environment.

Below is the dependency that needs to be installed for writing this script.

On production systems, it is not advisable to install libraries using "sudo". Please follow Python virtualenv best practices for this.

gcs_bucket.py

The above Python script demonstrates the major operations that can be performed on a bucket. These include:

  • creation of a new bucket in a project
  • listing of all buckets in a project
  • getting details of a specific bucket
  • deleting a specific bucket

Let's see what these operations look like when the script is run.

Conclusion

In this tutorial, you saw how to manage buckets on Google Cloud Storage. This was also accompanied by a small introduction to creating a Google Compute Engine instance and using it via an SSH client.

In the next tutorial, I will cover how to manage objects, i.e. folders and files inside a bucket. 


How to break Python

Nov 28 2016 [Archived Version] □ Published at The B-List: Latest entries under tags  django python

Don’t worry, this isn’t another piece about Python 3. I’m fully in favor of Python 3, and on record as to why. And if you’re still not convinced, I suggest this thoroughly comprehensive article on the topic, which goes over not just the bits people get angry about but also the frankly massive amount of cool stuff that only works in Python 3, and that you’re missing out on if you still only use Python 2 ...

Read full entry


Watermarking images on Django sites

Nov 22 2016 [Archived Version] □ Published at Blog | Machinalis

Have you ever noticed how stock photography sites add watermarks to the images shown on their catalogs? They do that to make sure people don’t just take the free samples and use them without proper licensing. Google Maps does it for the satellite imagery as well. Turns out this is pretty easy to do and we’ll show you how to do it for Django sites sites using Pillow and django-imagekit.

All the code is available on GitHub and a Vagrantfile is provided if you want to try the live Django demo by yourself.

Watermarking images

First we’ll show you three simple watermarking techniques that can be used in an Python project that uses images, and then we’ll use what we’ve built to add watermarks to images on a sample Django site.

Text overlay

The first thing we’ll do is add a semi-transparent text legend on the center of an image. We’ll use Pillow’s Image, ImageDraw and ImageFont modules.

The following function adds a text overlay on the center of the supplied image:

# processors.py

from PIL import Image, ImageDraw, ImageFont


_default_font = ImageFont.truetype('/usr/share/fonts/dejavu/DejaVuSans-Bold.ttf', 24)


def add_text_overlay(image, text, font=_default_font):
    rgba_image = image.convert('RGBA')
    text_overlay = Image.new('RGBA', rgba_image.size, (255, 255, 255, 0))
    image_draw = ImageDraw.Draw(text_overlay)
    text_size_x, text_size_y = image_draw.textsize(text, font=font)
    text_xy = ((rgba_image.size[0] / 2) - (text_size_x / 2), (rgba_image.size[1] / 2) - (text_size_y / 2))
    image_draw.text(text_xy, text, font=font, fill=(255, 255, 255, 128))
    image_with_text_overlay = Image.alpha_composite(rgba_image, text_overlay)

    return image_with_text_overlay

We create a new Image with the same dimensions as the original image on which we’re going to draw the text. In order to do that we create an ImageDraw.Draw instance and call ImageDraw.Draw.text (we also use ImageDraw.Draw.textsize to get the size of the text which we’ll need to know where to place it.)

We also need an ImageFont instance that can be created from an existing TrueType font (using ImageFont.truetype).

Finally we use Image.alpha_composite to combine the original image (converted to RGBA mode) and the text overlay.

/static/media/uploads/watermarking-images-django/screenshot-text-overlay.jpg

Watermark

The next technique is superimposing a semi-transparent watermark on top of the image. The code is similar to the one used for text overlays:

# processors.py

from PIL import Image, ImageDraw


def add_watermark(image, watermark):
    rgba_image = image.convert('RGBA')
    rgba_watermark = watermark.convert('RGBA')

    image_x, image_y = rgba_image.size
    watermark_x, watermark_y = rgba_watermark.size

    watermark_scale = max(image_x / (2.0 * watermark_x), image_y / (2.0 * watermark_y))
    new_size = (int(watermark_x * watermark_scale), int(watermark_y * watermark_scale))
    rgba_watermark = rgba_watermark.resize(new_size, resample=Image.ANTIALIAS)

    rgba_watermark_mask = rgba_watermark.convert("L").point(lambda x: min(x, 25))
    rgba_watermark.putalpha(rgba_watermark_mask)

    watermark_x, watermark_y = rgba_watermark.size
    rgba_image.paste(rgba_watermark, ((image_x - watermark_x) // 2, (image_y - watermark_y) // 2), rgba_watermark_mask)

    return rgba_image

Before we can combine the original image with the watermark we have to scale the watermark to make sure it fits the image. We use Image.resize with the new size, which we want it to be around half of the original image surface area, and the watermark image.

We then create a mask using the resized watermark converting it to grayscale with Image.convert, filtering out pixels with a value higher than 25 using Image.point and finally replacing the alpha channel on the watermark image with the mask using Image.putalpha.

The last step is to combine the original image with the watermark with Image.paste.

/static/media/uploads/watermarking-images-django/screenshot-watermark.jpg

Invisible Watermark

The last technique we’ll show is encoding arbitrary data into the image without affecting its appearance significantly (Also known as Steganography). We’ll do this by storing the individual bits of our data package into the least significant bit (LSB) of the original image. It is fairly rudimentary and easily detectable and defeatable, but it’ll give you an idea of what we can do with Pillow.

# processors.py

import numpy as np

from io import BytesIO
from pickle import dump

from PIL import Image, ImageMath


def lsb_encode(data, image):
    bytes_io = BytesIO()
    dump(data, file=bytes_io)
    data_bytes = bytes_io.getvalue()
    data_bytes_array = np.fromiter(data_bytes, dtype=np.uint8)
    data_bits_list = np.unpackbits(data_bytes_array).tolist()
    data_bits_list += [0] * (image.size[0] * image.size[1] - len(data_bits_list))
    watermark = Image.frombytes(data=bytes(data_bits_list), size=image.size, mode='L')
    red, green, blue = image.split()
    watermarked_red = ImageMath.eval("convert(a&0xFE|b&0x1,'L')", a=red, b=watermark)
    watermarked_image = Image.merge("RGB", (watermarked_red, green, blue))
    return watermarked_image

Before we can do anything with the image, we need to convert the data package to bits. We do this pickling the data onto a BytesIO and then use NumPy‘s fromiter and unpackbits and finally padding the result so it fits the dimensions of the original image.

We could have use struct instead of pickle, but the latter is a lot easier to use.

We then create a grayscale image using Image.frombytes and a bytes instance from the bits array we got before.

Finally we split the original image into red, green and blue channels, replace the least significant bit of each byte of the red channel with the information from our watermark using PIL.ImageMath.eval and reconstitute the original image using the new red channel and the old blue and green ones using Image.merge.

The problem with this approach is that even a minor change to the image will mess with the information stored in it. There are more robust techniques we could have used, but the LSB is one of the easiest to understand and implement.

Assuming the image remains intact (which requires the usage of a lossless format), How do we get the data out of it?

# processors.py

import numpy as np

from io import BytesIO
from pickle import load, UnpicklingError

from PIL import Image, ImageMath


def lsb_decode(image):
    try:
        red, green, blue = image.split()
        watermark = ImageMath.eval("(a&0x1)*0x01", a=red)
        watermark = watermark.convert('L')
        watermark_bytes = bytes(watermark.getdata())
        watermark_bits_array = np.fromiter(watermark_bytes, dtype=np.uint8)
        watermark_bytes_array = np.packbits(watermark_bits_array)
        watermark_bytes = bytes(watermark_bytes_array)
        bytes_io = BytesIO(watermark_bytes)
        return load(bytes_io)
    except UnpicklingError:
        return ''

We split the watremarked image into its red, green and blue channels, we filter out the original image’s information from the red channel using PIL.ImageMath.eval and convert the result to grayscale.

We then use NumPy‘s fromiter and packbits on the modified red channel’s data to get the bytes of our original data package, and we unpickle the results.

Watermarks in Django

We can apply what we showed you into a Django site using django-imagekit. It provides template filters and model tools to apply arbitrary processing to Django images.

Everything in django-imagekit is done trough Specs. Specs apply processors to images to generate new ones. They can be used in templates using filters, or in models using ImageSpecField to generate images dynamically, or ProcessedImageField to save the generated images alongside the model.

We’ll show you how to use the specs with templates and later we’ll mention briefly the other methods.

# processors.py

from django.conf import settings
from imagekit import ImageSpec, register

class TextOverlayProcessor(object):
    font = ImageFont.truetype('/usr/share/fonts/dejavu/DejaVuSans-Bold.ttf', 36)

    def process(self, image):
        return add_text_overlay(image, 'django-watermark-images', font=self.font)


class WatermarkProcessor(object):
    watermark = Image.open(settings.WATERMARK_IMAGE)

    def process(self, image):
        return add_watermark(image, self.watermark)


class HiddenWatermarkProcessor(object):
    def process(self, image):
        return lsb_encode('django-watermark-images', image)


class TextOverlay(ImageSpec):
    processors = [TextOverlayProcessor()]
    format = 'JPEG'
    options = {'quality': 75}


class Watermark(ImageSpec):
    processors = [WatermarkProcessor()]
    format = 'JPEG'
    options = {'quality': 75}


class HiddenWatermark(ImageSpec):
    processors = [HiddenWatermarkProcessor()]
    format = 'PNG'


register.generator('items:text-overlay', TextOverlay)
register.generator('items:watermark', Watermark)
register.generator('items:hidden-watermark', HiddenWatermark)

We’ve integrated the watermarking techniques into processors which in term are used by custom ImageSpec implementations. We then register said specs as generators so they can be used with the generateimage filter:

{% load imagekit %}

...

<div class="row item-row">
    <div class="col-lg-2 item-label">Image</div>
    <div class="col-lg-10"><img class="img-responsive item-image" src="{{ object.image.url }}"></div>
</div>
<div class="row item-row">
    <div class="col-lg-2 item-label">Text Overlay</div>
    <div class="col-lg-10">{% generateimage 'items:text-overlay' source=object.image -- class="img-responsive item-image" %}</div>
</div>
<div class="row item-row">
    <div class="col-lg-2 item-label">Watermark</div>
    <div class="col-lg-10">{% generateimage 'items:watermark' source=object.image -- class="img-responsive item-image" %}</div>
</div>
<div class="row item-row">
    <div class="col-lg-2 item-label">Hidden Watermark</div>
    <div class="col-lg-10">{% generateimage 'items:hidden-watermark' source=object.image -- class="img-responsive item-image" %}</div>
</div>

The snippet above is part of a template to display the following model:

# models.py

import uuid

from django.db import models
from django.urls import reverse_lazy
from django.utils.translation import ugettext_lazy as _

from django_extensions.db.models import TitleDescriptionModel, TimeStampedModel


def image_upload_to(instance, filename):
    return 'original_image/{uuid}/{filename}'.format(uuid=uuid.uuid4().hex, filename=filename)


class Item(TitleDescriptionModel, TimeStampedModel):
    image = models.ImageField(_('original image'), upload_to=image_upload_to)

That’s pretty much all you need. If you wanted to use an ImageSpecField or a ProcessedImageField instead of filters, we need to change the model:

# models.py


from .processors import TextOverlay, Watermark, HiddenWatermark


class Item(TitleDescriptionModel, TimeStampedModel):
    image = models.ImageField(_('original image'))
    text_overlay_image = ImageSpecField(source='image', processors=[TextOverlay()], format='JPEG')
    watermark_image = ImageSpecField(source='image', processors=[Watermark()], format='JPEG')
    hidden_watermark_image = ImageSpecField(source='image', processors=[HiddenWatermark()], format='PNG')

and later reference the new fields in templates:

<div class="row item-row">
    <div class="col-lg-2 item-label">Image</div>
    <div class="col-lg-10"><img class="img-responsive item-image" src="{{ object.image.url }}"></div>
</div>
<div class="row item-row">
    <div class="col-lg-2 item-label">Text Overlay</div>
    <div class="col-lg-10">{{ object.text_overlay_image.url }}</div>
</div>
<div class="row item-row">
    <div class="col-lg-2 item-label">Watermark</div>
    <div class="col-lg-10">{{ object.watermark_image.url }}</div>
</div>
<div class="row item-row">
    <div class="col-lg-2 item-label">Hidden Watermark</div>
    <div class="col-lg-10">{{ object.hidden_watermark_image.url }}</div>
</div>

Conclusions

As you can see, all we needed was a little bit of knowledge of Pillow’s API and the rest of work was handled for us by django-imagekit. There are a lot of things that these two frameworks can do for us like generating thumbnails, changing formats, etc.

One thing you should notice is that all these methods work on images during a request and, depending on the complexity of the task, might introduce an unacceptable delay in giving the response back to the user. We usually offload this work (generally using Celery) and respond to the client as soon as possible.

We’ve only shown you a dead simple steganography method. If you want to see more advanced techniques, check out Stéganô.

Feedback

As usual, comments, suggestions and pull requests are more than welcomed.


Security advisory: Vulnerability in password reset (master branch only)

Nov 21 2016 [Archived Version] □ Published at The Django weblog

Today, Florian Apolloner, a member of the Django security team, discovered and fixed a critical security issue in the new PasswordResetConfirmView that was added to the Django master branch on July 16th, 2016. The view didn't validate the password reset token on POST requests and therefore allowed anyone to reset passwords for any user.

This issue doesn't affect any released versions of Django. Per our security policy, security issues in master, but not present in any released version, are disclosed and fixed in public without pre-notification.

The issue demonstrates the complexity of class-based generic views, and the Django team advises caution when using them for security-sensitive functionality. We'll consider removing the class-based authentication views that are in the master branch, planned for Django 1.11. The discussion for this will take place publicly on the django-developers mailing list.


eGenix PyRun - One file Python Runtime 2.2.3 GA

Nov 21 2016 [Archived Version] □ Published at eGenix.com News & Events

Introduction

eGenix PyRun is our open source, one file, no installation version of Python, making the distribution of a Python interpreter to run based scripts and applications to Unix based systems as simple as copying a single file.

eGenix PyRun's executable only needs 11MB for Python 2 and 13MB for Python 3, but still supports most Python application and scripts - and it can be compressed to just 3-4MB using upx, if needed.

Compared to a regular Python installation of typically 100MB on disk, eGenix PyRun is ideal for applications and scripts that need to be distributed to several target machines, client installations or customers.

It makes "installing" Python on a Unix based system as simple as copying a single file.

eGenix has been using eGenix PyRun internally in the mxODBC Connect Server product since 2008 with great success and decided to make it available as a stand-alone open-source product.

We provide both the source archive to build your own eGenix PyRun, as well as pre-compiled binaries for Linux, FreeBSD and Mac OS X, as 32- and 64-bit versions. The binaries can be downloaded manually, or you can let our automatic install script install-pyrun take care of the installation: ./install-pyrun dir and you're done.

Please see the product page for more details:

    >>> eGenix PyRun - One file Python Runtime

News

This minor level release of eGenix PyRun comes with the following enhancements:

Enhancements / Changes

  • Removed lzma module from PyRun for Python 3.x again, since this caused too many issues with incompatible/missing libzma.so references. The module is still being built as optional add-on and can be used if the necessary libs are available, but it will no longer prevent PyRun from working altogether.

install-pyrun Enhancements

  • Updated install-pyrun to default to eGenix PyRun 2.2.3 and its feature set.
For a complete list of changes, please see the eGenix PyRun Changelog.

Downloads

Please visit the eGenix PyRun product page for downloads, instructions on installation and documentation of the product.

Support

Commercial support for this product is available directly from eGenix.com.

Please see the support section of our website for details.

More Information

For more information on eGenix PyRun, licensing and download instructions, please write to [email protected].

Enjoy !

Marc-Andre Lemburg, eGenix.com


eGenix PyRun - One file Python Runtime 2.2.2 GA

Nov 18 2016 [Archived Version] □ Published at eGenix.com News & Events

Introduction

eGenix PyRun is our open source, one file, no installation version of Python, making the distribution of a Python interpreter to run based scripts and applications to Unix based systems as simple as copying a single file.

eGenix PyRun's executable only needs 11MB for Python 2 and 13MB for Python 3, but still supports most Python application and scripts - and it can be compressed to just 3-4MB using upx, if needed.

Compared to a regular Python installation of typically 100MB on disk, eGenix PyRun is ideal for applications and scripts that need to be distributed to several target machines, client installations or customers.

It makes "installing" Python on a Unix based system as simple as copying a single file.

eGenix has been using eGenix PyRun internally in the mxODBC Connect Server product since 2008 with great success and decided to make it available as a stand-alone open-source product.

We provide both the source archive to build your own eGenix PyRun, as well as pre-compiled binaries for Linux, FreeBSD and Mac OS X, as 32- and 64-bit versions. The binaries can be downloaded manually, or you can let our automatic install script install-pyrun take care of the installation: ./install-pyrun dir and you're done.

Please see the product page for more details:

    >>> eGenix PyRun - One file Python Runtime

News

This minor level release of eGenix PyRun comes with the following enhancements:

Enhancements / Changes

  • Upgraded PyRun to Python 2.7.12, Python 3.4.5 and Python 3.5.2.
  • Fixed rpath setting to properly include the $ORIGIN marker. Without this, the rpath setting doesn't work.
  • Added missing lzma module to default PyRun 3.x installation. Please note that this adds an additional dependency on libzma.so.5 for PyRun for Python 3.4 and 3.5.

install-pyrun Enhancements

  • Updated install-pyrun to default to eGenix PyRun 2.2.2 and its feature set.
For a complete list of changes, please see the eGenix PyRun Changelog.

Downloads

Please visit the eGenix PyRun product page for downloads, instructions on installation and documentation of the product.

Support

Commercial support for this product is available directly from eGenix.com.

Please see the support section of our website for details.

More Information

For more information on eGenix PyRun, licensing and download instructions, please write to [email protected].

Enjoy !

Marc-Andre Lemburg, eGenix.com


eGenix mxODBC 3.3.6 GA

Nov 16 2016 [Archived Version] □ Published at eGenix.com News & Events

Introduction

mxODBC provides an easy-to-use, high-performance, reliable and robust Python interface to ODBC compatible databases such as MS SQL Server, Oracle Database, IBM DB2, Informix and Netezza, SAP Sybase ASE and Sybase Anywhere, Teradata, MySQL, MariaDB, PostgreSQL, SAP MaxDB and many more:

>>>   mxODBC Product Page

The eGenix mxODBC - Python ODBC Database Interface product is a commercial extension to our open-source  eGenix mx Base Distribution:

>>>   mx Base Distribution Page

News

The 3.3.6 release of our mxODBC is a patch level release of our popular Python ODBC Interface for Windows, Linux, Mac OS X and FreeBSD. It includes these enhancements and fixes:

Features:

  • The mxODBC default *RowFactory helpers will no longer try to add column names which are not valid Python identifiers to the Row class code. Such columns are still available via index (e.g. row[0]) or named access (e.g. row['123']).

Bug Fixes:

  • IMPORTANT: Fixed a bug in context managers not properly detecting exceptions. This resulted exceptions getting silenced, transactions not getting committed and could lead to data corruption. Thanks to Jan Murre for the report.
The complete list of changes is available on the mxODBC changelog page.

Features

mxODBC 3.3 was released on 2014-04-08. These are the highlights of the new release:

mxODBC 3.3 Release Highlights

Stored Procedures

  • mxODBC now has full support for input, output and input/output parameters in stored procedures and stored functions, allowing easy integration with existing databases systems.

User Customizable Row Objects

  • Added support for user customizable row objects by adding cursor/connection .rowfactory and .row constructor attributes. When set, these are used to wrap the normal row tuples returned by the .fetch*() methods into dynamically created row objects.
  • Added new RowFactory classes to support cursor.rowfactory and cursor.row. These allow dynamically creating row classes that provide sequence as well as mapping and attribute access to row fields - similar to what namedtuples  implement, but specific to result sets.

Fast Cursor Types

  • Switched to forward-only cursor types for all database backends, since this provides a much better performance for MS SQL Server and IBM DB2 drivers.
  • Added a new .cursortype attribute to allow adjusting and inspecting the ODBC cursor type to be used for an mxODBC cursor object. Default is to use forward-only cursors, but mxODBC also support several other useful cursor types such as static cursors with full support for result set scrolling.

mxODBC 3.3 Driver Compatibility Enhancements

Oracle

  • Added work-around for Oracle Instant Client to be able to use integer output parameters.
  • Added a work-around for Oracle Instant Client to have it return output parameters based on the input placeholder Python parameter types. It would otherwise return all parameters as strings.

MS SQL Server

  • mxODBC now defaults to 100ns connection.timestampresolution for MS SQL Server 2008 and later, and 1ms resolution for MS SQL server 2005 and earlier. This simplifies interfacing to SQL Server timestamp columns by preventing occasional precision errors.
  • Tested mxODBC successfully with new MS SQL Server Native Client 11 for Linux. Unicode connection strings still don't work, but everything else does.
  • Added documentation on how to use Kerberos with mxODBC and SQL Server for authentication on both Windows and Linux.

Sybase ASE

  • Added work-around for the Sybase ASE ODBC driver, which doesn't always pass back NULL correctly to mxODBC on 64-bit Unix systems.
  • Changed the variable type binding mode default for the Sybase ASE ODBC driver from Python type binding to SQL type binding, which resolves issues with e.g. the Unicode support for that driver.

IBM DB2

  • Added work-around for the IBM DB2 ODBC driver, which doesn't always pass back NULL correctly to mxODBC on 64-bit Unix systems.

PostgreSQL

  • Added work-around to force Python type binding for the PostgreSQL ODBC drivers. More recent versions of the driver report supporting SQL type binding, but they don't implement it.
  • Added work-around to have PostgreSQL ODBC drivers properly work with binary data for BYTEA columns.

MySQL

  • mxODBC now supports native Unicode with the recent MySQL ODBC drivers - provided you use the Unicode variants of the drivers.
  • Changed the default binding mode for MySQL ODBC drivers to Python type binding. This works around a problem with date/time values when talking to MySQL 5.6 servers.
For the complete set of features, please have a look at the mxODBC product page.

Editions

mxODBC is available in these two editions:

  • The Professional Edition, which gives full access to all mxODBC features.
  • The Product Development Edition, which allows including mxODBC in applications you develop.
For a complete overview of the available editions, please see the product page.

Downloads

Please visit the eGenix mxODBC product page for downloads, instructions on installation and documentation of the packages.

Note that in order to use the eGenix mxODBC product, you first need to install our open-source eGenix mx Base Distribution.

You can also simply use:

pip install egenix-mxodbc

and then request 30-day evaluation licenses from our web-site.

Upgrading

Users are encouraged to upgrade to this latest mxODBC release to benefit from the new features and updated ODBC driver support.

We have taken special care not to introduce backwards incompatible changes, making the upgrade experience as smooth as possible.

Customers who have purchased mxODBC 3.3 licenses can continue to use their licenses with this patch level release.

For upgrade purchases, we will give out 20% discount coupons going from mxODBC 2.x to 3.3 and 50% coupons for upgrades from mxODBC 3.x to 3.3. Please contact the eGenix.com Sales Team with your existing license serials for details for an upgrade discount coupon.

If you want to try the new release before purchase, you can request 30-day evaluation licenses by visiting our web-site or writing to [email protected], stating your name (or the name of the company) and the number of eval licenses that you need.

More Information

For more information on the eGenix.com Python products, licensing and download instructions, please write to [email protected].

Enjoy !

Marc-Andre Lemburg, eGenix.com


RapidCon 2016: RapidPro Developer's Recap

Nov 16 2016 [Archived Version] □ Published at Caktus Blog

Developer Erin Mullaney was just in Amsterdam for RapidCon, a UNICEF-hosted event for developers using RapidPro, an SMS tool built on Django. The teams that have worked on RapidPro and its predecessor RapidSMS have gotten to know each other virtually over the years. This marks the second time they’ve all come from across the globe...


How does the Django Cross-site request forgery protection work?

Nov 16 2016 [Archived Version] □ Published at Practicing web development under tags  django security

Dan Poirier wrote an article on the Cactus Group blog about common web site security vulnerabilities. In it he talked about the CSRF protection in Django. Although he is right about a CSRF token having to be part of the POST request, this is not the entire story.

It is not my intention to claim that mister Poirier does not know how the CSRF protection in Django works. I only want to present a more complete version.

First things first, for those of you that have not read the Dan Poiriers article, here’s a short summary of the CSRF related parts.

Cross-site request forgery (CSRF or XSRF) is a type of attack where a malicious site is trying to make your browser send requests to another site in an attempt to leverage the permissions of the user—you. (For more information and examples, check the original article or the OWASP page on CSRF.)

Besides making sure that GET requests do not change data the article talks about the CSRF protection provided by Django. Specifically it states the following (emphasis mine):

Django’s protection is to always include a user-specific, unguessable string as part of such requests, and reject any such request that doesn’t include it. This string is called the CSRF token. Any form on a Django site that does a POST etc has to include it as one of the submitted parameters. Since the malicious site doesn’t know the token, it cannot generate a malicious POST request that the Django site will pay any attention to.

This is where the author is not incorrect (the POST request indeed has to include the CSRF token), but this is only one half of the mechanism. The other half is a cookie that is set by the original site where the user is logged in. Only when the server receives both values from the browser (and they match) will the POST request be valid. This is described as the “Double Submit Cookie” defence in the Cross-Site Request Forgery (CSRF) Prevention Cheat Sheet by OWASP.

Example

Here’s an example with a very simple form (source code on GitHub). Let’s first request the form:

$ curl -i http://localhost:8000/post_to_me/

The response will look something like this:

HTTP/1.0 200 OK
Date: Tue, 15 Nov 2016 20:25:58 GMT
Server: WSGIServer/0.2 CPython/3.4.3
Content-Type: text/html; charset=utf-8
Vary: Cookie
X-Frame-Options: SAMEORIGIN
Set-Cookie:  csrftoken=wVFdNQ1Hz487w7yk2mVjre2qlsclXi99w2jEcGyvXorojDLd7jH09NGhmbavG3tx; expires=Tue, 14-Nov-2017 20:55:58 GMT; Max-Age=31449600; Path=/

<!doctype html>
<html>
  <body>
    <form action="/post_to_me/" method="post">
      <input type='hidden' name='csrfmiddlewaretoken' value='JBWuPGvKU54xW9YIwEIknst1azSkBmg3JIAVew2yipnOJFbBBBu1517SbiQuk7Ar' />
      <tr><th><label for="id_name">Name:</label></th><td><input id="id_name" name="name" type="text" required /></td></tr>
      <input type="submit" value="Post" />
    </form>
  </body>
</html>

Note the csrftoken value in the Set-Cookie header. Also note the csrfmiddlewaretoken value in the form in the body of the response. We’ll use these values in our examples where we send POST requests.

First a demonstration that we can successfully post a value using both the cookie and the value in the form (in the --data parameter):

$ curl -s -D - -o /dev/null \
-H 'Cookie: csrftoken=wVFdNQ1Hz487w7yk2mVjre2qlsclXi99w2jEcGyvXorojDLd7jH09NGhmbavG3tx' \
--data 'csrfmiddlewaretoken=JBWuPGvKU54xW9YIwEIknst1azSkBmg3JIAVew2yipnOJFbBBBu1517SbiQuk7Ar&name=mark' \
http://localhost:8000/post_to_me/

The response is a nice 200 OK:

HTTP/1.0 200 OK
Date: Tue, 15 Nov 2016 20:30:35 GMT
Server: WSGIServer/0.2 CPython/3.4.3
X-Frame-Options: SAMEORIGIN
Content-Type: text/html; charset=utf-8

If we do not send the cookie along, we expect that the POST request will fail:

$ curl -s -D - -o /dev/null \
--data 'csrfmiddlewaretoken=JBWuPGvKU54xW9YIwEIknst1azSkBmg3JIAVew2yipnOJFbBBBu1517SbiQuk7Ar&name=mark' \
http://localhost:8000/post_to_me/

And indeed the server responds with a 403 Forbidden status:

HTTP/1.0 403 Forbidden
Date: Tue, 15 Nov 2016 20:32:47 GMT
Server: WSGIServer/0.2 CPython/3.4.3
X-Frame-Options: SAMEORIGIN
Content-Type: text/html

We could try to include the cookie but leave out the value in the form to check if the cookie alone is enough:

curl -s -D - -o /dev/null \
-H 'Cookie: csrftoken=wVFdNQ1Hz487w7yk2mVjre2qlsclXi99w2jEcGyvXorojDLd7jH09NGhmbavG3tx' \
--data 'name=mark' \
http://localhost:8000/post_to_me/

However, this has the same result:

HTTP/1.0 403 Forbidden
Date: Tue, 15 Nov 2016 20:33:35 GMT
Server: WSGIServer/0.2 CPython/3.4.3
X-Frame-Options: SAMEORIGIN
Content-Type: text/html

As you can see Django requires both components to be present. The actual value of the token is less relevant. Sure, it is “unguessable”, but that is (in my humble opinion) not the most relevant part. The CSRF token is also not stored. Django could not care that much about the actual value—as long as the value in the cookie matches the one in the POST data, the token is considered valid.

To demonstrate that I can make up my own values if I want to (as long as they are 32 or 64 characters in length):

$ curl -s -D - -o /dev/null \
-H 'Cookie: csrftoken= markmarkmarkmarkmarkmarkmarkmark' \
--data 'csrfmiddlewaretoken=markmarkmarkmarkmarkmarkmarkmark&name=mark' \
http://localhost:8000/post_to_me/

This succeeds:

HTTP/1.0 200 OK
Date: Tue, 15 Nov 2016 20:35:55 GMT
Server: WSGIServer/0.2 CPython/3.4.3
X-Frame-Options: SAMEORIGIN
Content-Type: text/html; charset=utf-8

This example uses an older version of the CSRF token. As of Django 1.10 the CSRF form token value is salted and changed every request. (For details, see this commit and the related issue.) This does not change the mechanism behind the defence though.

Why does this defence work?

Back to the attack. Why does it matter that the POST request has to have matching tokens both via the cookie and the POST data?

Cookies are set for a specific domain. Your browser protects this in two ways:

  • You cannot set a cookie for a different domain.
  • Cookies for one domain are not sent to another domain.

This defence against CSRF works because although the evil site can force the browser to make a request to the site it wants to abuse, the attacker can only manipulate the request and its data. The attacker cannot set, modify or even read a cookie for a different domain than its own. As a result the attacker cannot determine or even guess which CSRF token should be in the request and thus the request will fail.

Note that this is the protection chosen by Django. Other forms of defence are possible. See the aforementioned Cross-Site Request Forgery (CSRF) Prevention Cheat Sheet for more information.

As already stated at the top of the article, I do not want to imply that mister Poirier does not know how the CSRF protection in Django works; perhaps he decided to leave out some of these details to make his article more succinct. Either way, in my opinion his article only told half of the story with regards to CSRF protection. So I decided to talk about the other half.


Create an Excellent Python Dev Env

Nov 12 2016 [Archived Version] □ Published at Dougal Matthews under tags  pipsi pyenv virtualenv

There are a huge number of Python dev tools around, a number of them are essential for my day to day development. However, they tend to suffer from a lack of discoverability and it takes a while to find what works for you.

I'm going to quickly share what I use, some of these are well known, some are probably not. I'd expect most people to pick and choose from this post as you are unlikely to want everything I use, but there should be something useful for most people.

These are my primary goals:

  • Be able to install any Python version easily.
  • Don't ever touch the system Python.
  • An easy way to setup virtualenvs for specific projects.
  • Install and isolate a number of Python tools.

How do we get there?

pyenv

pyenv pitches itself as "simple python version management" and it does just that. Once setup, you can easily install and switch between Python versions, including specific point releases. pyenv install --list reveals it knows how to install a whopping 271 different Python versions at the moment from cpython 2.1.3 up to 3.7-dev and pypy and stackless.

The install process is a bit manual, but there is an install tool that makes it easier. After installing, I do something like this:

pyenv install -s 2.7.12;
pyenv install -s 3.5.2;
pyenv install -s 3.4.5;
pyenv install -s pypy-5.4.1;
pyenv global 2.7.12 3.5.2 3.4.5 pypy-5.4.1;

This installs the Python versions I typically need, and then sets them as the global default. The order is important, 2.7.12 becomes the default for python as it is first and 3.5.2 becomes the default for python3.

If you just want to use a specific Python version in a directory, and it's subdirectories, you can run the command pyenv local 3.5.2 and it will create a .python-version file. Warning, if you do this in your home directory by mistake it can be very confusing.

One feature I'd love pyenv to have, is a way to tell it to install a Python version (like 2.7 or 3.5) and have it automatically install the latest point release (and add a new command that removes and updates them when needed)

pyenv-virtualenv

For a long time I was a big user of virtualenvwrapper, however, my transition to pyenv and fish caused some issues. I stumbled on pyenv-virtualenv (not to be mistaken with pyenv-virtualenvwrapper which also doesn't support fish) which covers all my needs. I wrote a few fish functions to make it a little easier to use. It isn't hard, but maybe just a little verbose.

For example, here is a handy way to make a temporary virtualenv, I found this feature of virtualenvwrapper (the mktmpenv command) particularly useful.

function venv-tmp
  set venv_tmp_name "tmp-"(random)
  pyenv virtualenv (expr substr (python --version 2>&1) 8 20) $venv_tmp_name
  venv-activate $venv_tmp_name
end

function venv-tmp-cleanup
  for val in (pyenv versions | grep "/envs/tmp-")
    venv-rm (basename $val)
  end
end

Generally it doesn't give me much over what virtualenvwrapper did (other than fish support) but I do like that it is managed by pyenv and integrates well.

pipsi

pipsi is a more recent addition to my setup. It is a fairy simple tool which allows you to install Python CLI tools in their own virtualenv and then the command is added to your path. The main advantage here is that they are all isolated and don't need to have compatible requirements. Uninstalling is also much cleaner and easier - you just delete the virtualenv.

I install a bunch of Python projects this way, here are some of the most useful.

  • tox: My defacto way of running tests.
  • mkdocs: A beautifully simple documentation tool (I might be biased).
  • git-review: The git review command for gerrit integration.
  • flake8: Python linting, mostly installs like this for vim.

Putting it all together

So, overall I don't actually use that many projects, but I have very happy with how it works. I have the setup automated, and it looks like this.

# pyenv
if [ ! -d ~.pyenv ]; then
    curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
    git clone https://github.com/yyuu/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv
else
    pyenv update
fi;

pyenv install -s 2.7.12;
pyenv install -s 3.5.2;
pyenv install -s 3.4.5;
pyenv install -s pypy-5.4.1;
pyenv global 2.7.12 3.5.2 3.4.5 pypy-5.4.1;
~/.pyenv/shims/pip install -U pip pipsi
rm -rf ~/.local/venvs
~/.pyenv/shims/pipsi install tox
~/.pyenv/shims/pipsi install mkdocs
~/.pyenv/shims/pipsi install git-review
~/.pyenv/shims/pipsi install 1pass
~/.pyenv/shims/pipsi install flake8
~/.pyenv/shims/pipsi install yaql
~/.pyenv/shims/pipsi install livereload

The summary is, first install pyenv and setup the Python versions you need. Then install pipsi into the default pyenv environment and use that to install the other tools. The system Python should never be touched.

A couple of things are missing as you'll need to setup paths and so on, so please do look at the install guides for each.


x3 Python Software Developers + (Visa Sponsorship)

Nov 11 2016 [Archived Version] □ Published at Djangojobs.Net latest jobs

Scurri is a rapidly growing software company that wants to enable every online retailer to offer an Amazon-like experience to their customers, when it comes to shipping. We do this by providing a delivery management solution to our customers. We have ambitious goals and an exceptional team.

Some of the challenges you will face in this role are:

  • Integrations with lots of third-party APIs
  • Scaling to handle the volume from large retailers
  • High Availability
  • Designing software that can be easily adapted.
  • You will join a team that follows lean and agile practices, believes that untested code is broken by design, does continuous deployments (but decouples releases from deployments) and continuously improves.

You need to have:

  • Strong engineering principles.
  • 3+ years experience with Python. Our platform is based on Django.
  • Strong working knowledge of git.
  • Experience with unit and acceptance testing tools and practices. Tests are part of what done means.
  • Excellent written and spoken English and good communication skills overall.
  • Experience with JavaScript and front-end technologies is a bonus.
  • Experience with AWS is a bonus.

This position is for a permanent role in our organisation. We pay a competitive salary and you get a chance to make delivery better for hundreds of thousands of consumers.

Also for the right candidate, we will pay Visa/work permit fees.

You can enjoy discounted gym membership rates, bike to work scheme and Scurri social club.


django-planet aggregates posts from Django-related blogs. It is not affiliated with or endorsed by the Django Project.

Social Sharing

Feeds

Tag cloud

admin administration adsense advanced ajax amazon angular angularjs apache api app appengine app engine apple application security aprendiendo python architecture articles asides audrey authentication automation backup bash basics best practices binary bitbucket blog blog action day blogging book books buildout business c++ cache capoeira celery celerycam celerycrawler challenges chat cheatsheet cherokee choices class-based-views cliff cloud cms code codeship codeship news coding command community computer computers computing configuration consumernotebook continuous deployment continuous integration couchdb coverage css custom d data database databases db debian debugging deploy deployment deployment academy design developers development devops digitalocean django django1.7 django cms djangocon django framework django-nose django-readonly-site django-rest-framework django-tagging django templates django-twisted-chat django web framework tutorials documentation dojango dojo dotcloud dreamhost dughh easy_install eclipse education elasticsearch email encoding english error europe eventbrite events expressjs extensions fabric facebook family fashiolista fedora field file filter fix flash flask form forms frameworks friends fun functional reactive programming gae gallery games geek general gentoo gis git github gmail gnome google google app engine guides gunicorn hack hackathon hacking hamburg haskell heroku holidays hosting howto how-to how-tos html http i18n image imaging indifex install installation intermediate internet ios iphone java javascript jinja2 jobs journalism jquery json justmigrated kde la latex linear regression linkedin linode linux login mac machine learning mac os x markdown math memcached meme mercurial meta meteor migration mirror misc model models mod_wsgi mongodb months mozilla multi-language mvc mysql nelenschuurmans newforms news nginx nodejs nosql oauth ogólne openshift opensource open source open-source operations orm osx os x ottawa paas packages packaging patterns pedantics pelican penetration test performance personal philippines philosophy php pi pil pinax pip piston planet plone plugin pony postgis postgres postgresql ppoftw presentation private programmieren programming programming &amp; internet project projects pycharm pycon pycon-2013-guide pydiversity pygrunn pyladies pypi pypy pyramid python python3 queryset quick tips quora rabbitmq rails rant ratnadeep debnath reactjs recipe redis refactor release request resolutions rest reusable app review rhel rtnpro ruby ruby on rails scala scaling science screencast script scripting security server setup shell simple smiley snaking software software collections software development south sphinx sprint sql ssh ssl static storage supervisor support svn sysadmin tag tag cloud talk nerdy to me tastypie tdd techblog technical technology template templates template tags test testing tests tip tips tools tornado transifex travel travel tips for geeks tumbles tutorial tutorials twisted twitter twoscoops typo3 ubuntu uncategorized unicode unittest unix use user authentication usergroup uwsgi uxebu virtualenv virtualenvwrapper web web 2.0 web application web applications web design &amp; development webdev web development webfaction web framework websockets whoosh windows wordpress work workshop wsgi yada znc zope