What's new in Django community blogs?

django-partial-index

Aug 07 2017 [Archived Version] □ Published at Latest Django packages added

PostgreSQL partial index support for Django


TransactionManagementError during testing with Django 1.10

Aug 04 2017 [Archived Version] □ Published at atodorov.org under tags  django fedora.planet qa

During the past 3 weeks I've been debugging a weird error which started happening after I migrated KiwiTestPad to Django 1.10.7. Here is the reason why this happened.

Symptoms

After migrating to Django 1.10 all tests appeared to be working locally on SQLite however they failed on MySQL with

TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

The exact same test cases failed on PostgreSQL with:

InterfaceError: connection already closed

Since version 1.10 Django executes all tests inside transactions so my first thoughts were related to the auto-commit mode. However upon closer inspection we can see that the line which triggers the failure is

self.assertTrue(users.exists())

which is essentially a SELECT query aka User.objects.filter(username=username).exists()!

My tests were failing on a SELECT query!

Reading the numerous posts about TransactionManagementError I discovered it may be caused by a run-away cursor. The application did use raw SQL statements which I've converted promptly to ORM queries, that took me some time. Then I also fixed a couple of places where it used transaction.atomic() as well. No luck!

Then, after numerous experiments and tons of logging inside Django's own code I was able to figure out when the failure occurred and what events were in place. The test code looked like this:

response = self.client.get('/confirm/')

user = User.objects.get(username=self.new_user.username)
self.assertTrue(user.is_active)

The failure was happening after the view had been rendered upon the first time I do a SELECT against the database!

The problem was that the connection to the database had been closed midway during the transaction!

In particular (after more debugging of course) the sequence of events was:

  1. execute django/test/client.py::Client::get()
  2. execute django/test/client.py::ClientHandler::__call__(), which takes care to disconnect/connect signals.request_started and signals.request_finished which are responsible for tearing down the DB connection, so problem not here
  3. execute django/core/handlers/base.py::BaseHandler::get_response()
  4. execute django/core/handlers/base.py::BaseHandler::_get_response() which goes through the middleware (needless to say I did inspect all of it as well since there have been some changes in Django 1.10)
  5. execute response = wrapped_callback() while still inside BaseHandler._get_response()
  6. execute django/http/response.py::HttpResponseBase::close() which looks like

    # These methods partially implement the file-like object interface.
    # See https://docs.python.org/3/library/io.html#io.IOBase
     
    # The WSGI server must call this method upon completion of the request.
    # See http://blog.dscpl.com.au/2012/10/obligations-for-calling-close-on.html
    def close(self):
        for closable in self._closable_objects:
            try:
                closable.close()
            except Exception:
                pass
        self.closed = True
        signals.request_finished.send(sender=self._handler_class)
    
  7. signals.request_finished is fired

  8. django/db/__init__.py::close_old_connections() closes the connection!

IMPORTANT: On MySQL setting AUTO_COMMIT=False and CONN_MAX_AGE=None helps workaround this problem but is not the solution for me because it didn't help on PostgreSQL.

Going back to HttpResponseBase::close() I started wondering who calls this method. The answer was it was getting called by the @content.setter method at django/http/response.py::HttpResponse::content() which is even more weird because we assign to self.content inside HttpResponse::__init__()

Root cause

The root cause of my problem was precisely this HttpResponse::__init__() method or rather the way we arrive at it inside the application.

The offending view last line was

return HttpResponse(Prompt.render(
     request=request,
     info_type=Prompt.Info,
     info=msg,
     next=request.GET.get('next', reverse('core-views-index'))
))

and the Prompt class looks like this

from django.shortcuts import render

class Prompt(object):
    @classmethod
    def render(cls, request, info_type=None, info=None, next=None):
        return render(request, 'prompt.html', {
            'type': info_type,
            'info': info,
            'next': next
        })

Looking back at the internals of HttpResponse we see that

  • if content is a string we call self.make_bytes()
  • if the content is an iterator then we assign it and if the object has a close method then it is executed.

HttpResponse itself is an iterator, inherits from six.Iterator so when we initialize HttpResponse with another HttpResponse object (aka the content) we execute content.close() which unfortunately happens to close the database connection as well.

IMPORTANT: note that from the point of view of a person using the application the HTML content is exactly the same regardless of whether we have nested HttpResponse objects or not. Also during normal execution the code doesn't run inside a transaction so we never notice the problem in production.

The fix of course is very simple, just return Prompt.render()!

Thanks for reading and happy testing!


Fastest *local* cache backend possible for Django

Aug 04 2017 [Archived Version] □ Published at Peterbe.com

I did another couple of benchmarks of different cache backends in Django. This is an extension/update on Fastest cache backend possible for Django published a couple of months ago. This benchmarking isn't as elaborate as the last one. Fewer tests and fewer variables.

I have another app where I use a lot of caching. This web application will run its cache server on the same virtual machine. So no separation of cache server and web head(s). Just one Django server talking to localhost:11211 (memcached's default port) and localhost:6379 (Redis's default port).

Also in this benchmark, the keys were slightly smaller. To simulate my applications "realistic needs" I made the benchmark fall on roughly 80% cache hits and 20% cache misses. The cache keys were 1 to 3 characters long and the cache values lists of strings always 30 items long (e.g. len(['abc', 'def', 'cba', ... , 'cab']) == 30).

Also, in this benchmark I was too lazy to test all different parsers, serializers and compressors that django-redis supports. I only test python-memcached==1.58 versus django-redis==4.8.0 versus django-redis==4.8.0 && msgpack-python==0.4.8.

The results are quite "boring". There's basically not enough difference to matter.

Config Average Median Compared to fastest
memcache 4.51s 3.90s 100%
redis 5.41s 4.61s 84.7%
redis_msgpack 5.16s 4.40s 88.8%


drf-jwt-devices

Aug 04 2017 [Archived Version] □ Published at Latest Django packages added


django-croppie

Aug 04 2017 [Archived Version] □ Published at Latest Django packages added


Linting SCSS with sass-lint

Aug 01 2017 [Archived Version] □ Published at Lincoln Loop

Linting SCSS with sass-lint

Complex projects always require multiple developers and design heavy web sites are no exception. Sass/SCSS is still our language of choice when writing CSS, both for its wide support and flexibility. The latter can be a double-edged sword when it comes to maintainability with each developer bringing their own interpretation to the language's syntax. One way to solve this is with sass-lint, a Node-based linting tool, to keep everyone on the same page when writing Sass/SCSS code.

We've used sass-lint on a few projects now, most notably a large-scale CMS with dozens of components supported by 8-10 developers. Like other linters, rules are enforced via a .sass-lint.yml rules file and developers can determine what triggers either an error or warning. The syntax for a rule is pretty easy to grok:

rules:
  class-name-format:
    - 2
    - convention: hyphenatedlowercase

All rules are indented under rules:. In the example above we are enforcing class name format via class-name-format:. - 2 tells sass-lint to throw an error if there is a validation (0 disarms, 1 is a warning). - convention: hyphenatedlowercase tells the linter to enforce a specific style of class name. In this case we're going with hyphenated lowercase class names, or something like this: .tooltip-content. Based on the above rule, .TOOLTIPcontent would force sass-lint to throw an error that might look like this:

scss/base/_globals.scss
  452:2  error  Class '.TOOLTIPcontent' should be written in lowercase with hyphens  class-name-format

Luckily for those perfectionists with deadlines, all the available rules for sass-lint can be found in the repo via the rules documentation. If you need a little help getting started you can check out our base rules file.

What if you need to make an exception? What if some third-party code forces you to use .TOOLTIPContent? That's simple enough with Sass/SCSS comments:

// sass-lint:disable class-name-format
.TOOLTIPcontent {
  position: absolute;
}
// sass-lint:enable class-name-format

Since class-name-format impacts the whole block we need to disable and reenable it, but for most other rules you can handle everything on a single line:

// sass-lint:disable class-name-format
.TOOLTIPcontent {
  position: absolute;
  top: 10px; left: 20px; // sass-lint:disable one-declaration-per-line
}
// sass-lint:enable class-name-format

If you have the need to be especially heavy-handed, you can use sass-lint:disable-all and sass-lint:enable-all as needed. Need to go surgical? You can separate rules via commas: sass-lint:disable hex-notation, indentation.

Using sass-lint

Now that you're convinced to start linting your Sass/SCSS code, how do actually go about it? Installation couldn't be simpler with sass-lint:

npm install -g sass-lint

sass-lint comes with a handy CLI tool which might be enough for most situations:

sass-lint -c .sass-lint.yml '**/*.scss' -v -q

However, linting is the type of thing you want to do as you code. Several popular editors have plugins that allow you to do just that:

There's a good chance your favorite editor has a plugin that uses sass-lint. An alternative is to integrate sass-lint into your build tools or linting your Sass/SCSS code after each save via watch-like tools. This sort of approach is best left to future articles.

While just one tool in our ever-growing box, sass-lint has helped us increase developer productivity and deliver higher quality code to client development teams. Passive linting has eliminated some of the tedium of code reviews and feels like the only way to work. Happy linting!


Django bugfix release: 1.11.4

Aug 01 2017 [Archived Version] □ Published at The Django weblog

Today we've issued the 1.11.4 bugfix release.

The release package and checksums are available from our downloads page, as well as from the Python Package Index. The PGP key ID used for this release is Tim Graham: 1E8ABDC773EDE252.


How to Write, Package and Distribute a Library in Python

Aug 01 2017 [Archived Version] □ Published at tuts+

Python is a great programming language, but packaging is one of its weakest points. It is a well-known fact in the community. Installing, importing, using and creating packages has improved a lot over the years, but it's still not on par with newer languages like Go and Rust that learned a lot from the struggles of Python and other mature languages. 

In this tutorial, you'll learn everything you need to know about writing, packaging and distributing your own packages. 

How to Write a Python Library

A Python library is a coherent collection of Python modules that is organized as a Python package. In general, that means that all modules live under the same directory and that this directory is on the Python search path. 

Let's quickly write a little Python 3 package and illustrate all these concepts.

The Pathology Package

Python 3 has an excellent Path object, which is a huge improvement over Python 2's awkward os.path module. But it's missing one crucial capability—finding the path of the current script. This is very important when you want to locate access files relative to the current script. 

In many cases, the script can be installed in any location, so you can't use absolute paths, and the working directory can be set to any value, so you can't use a relative path. If you want to access a file in a sub-directory or parent directory, you must be able to figure out the current script directory. 

Here is how you do it in Python:

To access a file called 'file.txt' in a 'data' sub-directory of the current script's directory, you can use the following code: print(open(str(script_dir/'data/file.txt').read())

With the pathology package, you have a built-in script_dir method, and you use it like this:

Yep, it's a mouthful. The pathology package is very simple. It derives its own Path class from pathlib's Path and adds a static script_dir() that always returns the path of the calling script. 

Here is the implementation:

Due to the cross-platform implementation of pathlib.Path, you can derive directly from it and must derive from a specific sub-class (PosixPath or WindowsPath). The script dir resolution uses the inspect module to find the caller and then its filename attribute.

Testing the Pathology Package

Whenever you write something that is more than a throwaway script, you should test it. The pathology module is no exception. Here are the tests using the standard unit test framework: 

The Python Path

Python packages must be installed somewhere on the Python search path to be imported by Python modules. The Python search path is a list of directories and is always available in sys.path. Here is my current sys.path:

Note that the first empty line of the output represents the current directory, so you can import modules from the current working directory, whatever it is. You can directly add or remove directories to/from sys.path. 

You can also define a PYTHONPATH environment variable, and there a few other ways to control it. The standard site-packages is included by default, and this is where packages you install using via pip go. 

How to Package a Python Library

Now that we have our code and tests, let's package it all into a proper library. Python provides an easy way via the setup module. You create a file called setup.py in your package's root directory. Then, to create a source distribution, you run: python setup.py sdist

To create a binary distribution called a wheel, you run: python setup.py bdist_wheel

Here is the setup.py file of the pathology package:

It includes a lot of metadata in addition to the 'packages' item that uses the find_packages() function imported from setuptools to find sub-packages.

Let's build a source distribution:

The warning is because I used a non-standard README.md file. It's safe to ignore. The result is a tar-gzipped file under the dist directory:

And here is a binary distribution:

The pathology package contains only pure Python modules, so a universal package can be built. If your package includes C extensions, you'll have to build a separate wheel for each platform:

For a deeper dive into the topic of packaging Python libraries, check out How to Write Your Own Python Packages.

How to Distribute a Python Package

Python has a central package repository called PyPI (Python Packages Index). When you install a Python package using pip, it will download the package from PyPI (unless you specify a different repository). To distribute our pathology package, we need to upload it to PyPI and provide some extra metadata PyPI requires. The steps are:

  • Create an account on PyPI (just once).
  • Register your package.
  • Upload your package.

Create an Account

You can create an account on the PyPI website. Then create a .pypirc file in your home directory:

For testing purposes, you can add a "pypitest" index server to your .pypirc file:

Register Your Package

If this is the first release of your package, you need to register it with PyPI. Use the register command of setup.py. It will ask you for your password. Note that I point it to the test repository here:

Upload Your Package

Now that the package is registered, we can upload it. I recommend using twine, which is more secure. Install it as usual using pip install twine. Then upload your package using twine and provide your password (redacted below):

For a deeper dive into the topic of distributing your packages, check out How to Share Your Python Packages.

Conclusion

In this tutorial, we went through the fully fledged process of writing a Python library, packaging it, and distributing it through PyPI. At this point, you should have all the tools to write and share your libraries with the world.

Additionally, don’t hesitate to see what we have available for sale and for study in the marketplace, and please ask any questions and provide your valuable feedback using the feed below.


Senior Django Developer

Jul 31 2017 [Archived Version] □ Published at Djangojobs.Net latest jobs

A Web Technology Startup seeks a full-time Senior Python Engineer for a role in NYC.

About LeafLink:

LeafLink is a SaaS platform that provides a marketplace for licensed marijuana dispensaries to order from their cannabis vendors, as well as a suite of software tools for those vendors to manage their orders and operations to help grow their brands. With over 1,400 dispensaries and more than 250 leading brands on board in Colorado, Washington, California, Oregon, and Nevada, LeafLink is setting the industry standard for how cannabis brands and retailers do business with each other.

LeafLink is funded by leading VC firms with offices in NYC, Denver, Los Angeles, and San Francisco.

Qualifications:

  • 4+ years working in a fast-paced software environment as an engineer
  • 2+ years working directly with Django and Python
  • 2+ years working with AWS and similar cloud computing services
  • 2+ years working with PostgreSQL
  • 1+ years working with Celery or similar queue systems
  • 1+ years working with Elasticsearch or similar technology
  • 1+ years working with Vue.js or similar client side web application development
  • Proven track record for quality code, which has proper test coverage and documentation
  • You should be comfortable and embrace working in an agile / sprint-based environment
  • Comfortable in a Linux (Ubuntu) environment
  • Familiar with basic security and infrastructure design

Experience With:

  • Nginx and wsgi environments
  • RESTful API development
  • Automated deployment and configuration tools

Bonus Points:

  • You are not a designer but have an eye for user experience and can do slight modifications to CSS
  • Strong production infrastructure experience: backups, load balancing, etc.
  • You have personal interest in the cannabis industry and are familiar with elements of compliance
  • You have worked in a startup environment
  • Excited by interacting directly with clients to understand their needs and gather requirements
  • Lead or contribute to open source software projects

Benefits:

  • Healthcare matching
  • 3 weeks paid vacation a year
  • Fun and growing office which may or may not contain a disco ball


django-mako

Jul 31 2017 [Archived Version] □ Published at Latest Django packages added

The first simple, elegant Django Mako library


Django Coleman

Jul 30 2017 [Archived Version] □ Published at Latest Django packages added


django-allauth-cas

Jul 28 2017 [Archived Version] □ Published at Latest Django packages added

CAS support for django-allauth.


Django Hardcopy

Jul 27 2017 [Archived Version] □ Published at Latest Django packages added

Render PDFs from HTML in Python/Django using Headless Chrome


Building Your Own Website on a Budget in 2017

Jul 26 2017 [Archived Version] □ Published at Nerdy Dork under tags  web sites wordpress

I find that a lot of people don’t know where to go to start their own website. It is difficult to know because their are so many options. So here is my personal opinion on how to get up and running with your own blog and/or small business website. Domain name There are lots of […]

The post Building Your Own Website on a Budget in 2017 appeared first on Dustin Davis.


IoT_pki

Jul 25 2017 [Archived Version] □ Published at Latest Django packages added


django-planet aggregates posts from Django-related blogs. It is not affiliated with or endorsed by the Django Project.

Social Sharing

Feeds

Tag cloud

admin administration adsense advanced ajax amazon angular angularjs apache api app appengine app engine apple application security aprendiendo python architecture argentina articles asides audrey aurelia australia authentication automation backup bash basics best practices big data binary bitbucket blog blog action day blogging book books buildout business c++ cache capoeira celery celerycam celerycrawler challenges chat cheatsheet cherokee choices christianity class-based-views cliff clojure cloud cms code codeship codeship news coding command community computer computers computing configuration consumernotebook consumer-notebook continuous deployment continuous integration cookiecutter couchdb coverage css custom d data database databases db debian debugging deploy deployment deployment academy design developers development devops digitalocean django django1.7 django admin django cms djangocon django framework django-nose django-readonly-site django-rest-framework django-tagging django templates django-twisted-chat django web framework tutorials documentation dojango dojo dotcloud dreamhost dughh easy_install eclipse education elasticsearch email encoding english error europe eventbrite events expressjs extensions fabric facebook family fashiolista fedora field file filter fix flash flask foreman form forms frameworks friends fun functional reactive programming gae gallery games geek general gentoo gis git github gmail gnome goals google google app engine guides gunicorn hack hackathon hacking hamburg haskell heroku holidays hosting howto how-to howtos how-tos html http i18n igalia image imaging indifex install installation intermediate internet ios iphone java javascript jinja2 jobs journalism jquery json justmigrated kde la latex linear regression linkedin linode linux login mac machine learning mac os x markdown math memcached meme mercurial meta meteor migration mirror misc model models mod_wsgi mongodb months mozilla multi-language mvc mysql nasa nelenschuurmans newforms news nginx nodejs nosql oauth ogólne openshift opensource open source open-source openstack operations orm osx os x ottawa paas packages packaging patterns pedantics pelican penetration test performance personal personal and misc philippines philosophy php pi pil pinax pip piston planet plone plugin pony postgis postgres postgresql ppoftw presentation private programmieren programming programming & internet project projects pycharm pycon pycon-2013-guide pydiversity pygrunn pyladies pypi pypy pyramid python python3 queryset quick tips quora rabbitmq rails rant ratnadeep debnath reactjs recipe redis refactor release request resolutions rest reusable app review rhel rtnpro ruby ruby on rails scala scaling science screencast script scripting security server setup shell simple smiley snaking software software collections software development south sphinx sprint sql ssh ssl static storage supervisor support svn sysadmin tag tag cloud talk nerdy to me tastypie tdd techblog technical technology template templates template tags test testing tests tip tips tools tornado training transifex travel travel tips for geeks tumbles tutorial tutorials twisted twitter twoscoops typo3 ubuntu uncategorized unicode unittest unix use user authentication usergroup uwsgi uxebu vagrant validation virtualenv virtualenvwrapper web web 2.0 web application web applications web design & development webdev web development webfaction web framework websockets whoosh windows wordpress work workshop wsgi yada year-end-review znc zope