What's new in Django community blogs?

django_18_fast_migrations

Apr 06 2017 [Archived Version] □ Published at Latest Django packages added


django-filem

Apr 06 2017 [Archived Version] □ Published at Latest Django packages added

A simple media file manager for Django


django-filem

Apr 06 2017 [Archived Version] □ Published at Latest Django packages added

A simple media file manager for Django


django-background-tasks

Apr 05 2017 [Archived Version] □ Published at Latest Django packages added

A database-backed work queue for Django


django-background-tasks

Apr 05 2017 [Archived Version] □ Published at Latest Django packages added

A database-backed work queue for Django


Django 1.11 Release Notes a Reading

Apr 05 2017 [Archived Version] □ Published at GoDjango Screencasts and Tutorials

Django 1.11 is released for the world to use. It comes with a lot of changes, which can take some time to read.

In my last video I mentioned I would read these release notes if there was interest so here you go. I have read all of the release notes so you can just put on some headphones and hit play.

Django 1.11 Release Notes a Reading


Django Souvenirs

Apr 05 2017 [Archived Version] □ Published at Latest Django packages added


Django Souvenirs

Apr 05 2017 [Archived Version] □ Published at Latest Django packages added


Scraping Webpages in Python With Beautiful Soup: Search and DOM Modification

Apr 05 2017 [Archived Version] □ Published at tuts+

In the last tutorial, you learned the basics of the Beautiful Soup library. Besides navigating the DOM tree, you can also search for elements with a given class or id. You can also modify the DOM tree using this library. 

In this tutorial, you will learn about different methods that will help you with the search and modifications. We will be scraping the same Wikipedia page about Python from our last tutorial.

Filters for Searching the Tree

Beautiful Soup has a lot of methods for searching the DOM tree. These methods are very similar and take the same kinds of filters as arguments. Therefore, it makes sense to properly understand the different filters before reading about the methods. I will be using the same find_all() method to explain the difference between different filters.

The simplest filter that you can pass to any search method is a string. Beautiful Soup will then search through the document to find a tag that exactly matches the string.

You can also pass a regular expression object to the find_all() method. This time, Beautiful Soup will filter the tree by matching all the tags against a given regular expression.

The code will look for all the tags that begin with "h" and are followed by a digit from 1 to 6. In other words, it will be looking for all the heading tags in the document.

Instead of using regex, you could achieve the same result by passing a list of all the tags that you want Beautiful Soup to match against the document.

You can also pass True as a parameter to the find_all() method. The code will then return all the tags in the document. The output below means that there are currently 4,339 tags in the Wikipedia page that we are parsing.

If you are still not able to find what you are looking for with any of the above filters, you can define your own function that takes an element as its only argument. The function also needs to return True if there is a match and False otherwise. Depending on what you need, you can make the function as complicated as it needs to be to do the job. Here is a very simple example:

The above function is going through the same Wikipedia Python page and looking for unordered lists that have more than 20 children.

Searching the DOM Tree Using Built-In Functions

One of the most popular methods for searching through the DOM is find_all(). It will go through all the tag's descendants and return a list of all the descendants that match your search criteria. This method has the following signature:

The name argument is the name of the tag that you want this function to search for while going through the tree. You are free to provide a string, a list, a regular expression, a function, or the value True as a name.

You can also filter the elements in the DOM tree on the basis of different attributes like id, href, etc. You can also get all the elements with a specific attribute regardless of its value using attribute=True. Searching for elements with a specific class is different from searching for regular attributes. Since class is a reserved keyword in Python, you will have to use the class_ keyword argument when looking for elements with a specific class.

You can see that the document has 1,734 tags with a class attribute and 425 tags with an id attribute. If you only need the first few of these results, you can pass a number to the method as the value of limit. Passing this value will instruct Beautiful Soup to stop looking for more elements once it has reached a certain number. Here is an example:

When you use the find_all() method, you are telling Beautiful Soup to go through all the descendants of a given tag to find what you are looking for. Sometimes, you want to look for an element only in the direct children on a tag. This can be achieved by passing recursive=False to the find_all() method.

If you are interested in finding only one result for a particular search query, you can use the find() method to find it instead of passing limit=1 to find_all(). The only difference between the results returned by these two methods is that find_all() returns a list with only one element and find() just returns the result.

The find() and find_all() methods search through all the descendants of a given tag to search for an element. There are ten other very similar methods that you can use to iterate through the DOM tree in different directions.

The find_parent() and find_parents() methods traverse up the DOM tree to find the given element. The find_next_sibling() and find_next_siblings() methods will iterate over all the siblings of the element that come after the current one. Similarly, the find_previous_sibling() and find_previous_siblings() methods will iterate over all the siblings of the element that come before the current one.

The find_next() and find_all_next() methods will iterate over all the tags and strings that come after the current element. Similarly, the find_previous() and find_all_previous() methods will iterate over all the tags and strings that come before the current element.

You can also search for elements using CSS selectors with the help of the select() method. Here are a few examples:

Modifying the Tree

You can not only search through the DOM tree to find an element but also modify it. It is very easy to rename a tag and modify its attributes.

Continuing from our last example, you can replace a tag's contents with a given string using the .string attribute. If you don't want to replace the contents but add something extra at the end of the tag, you can use the append() method. 

Similarly, if you want to insert something inside a tag at a specific location, you can use the insert() method. The first parameter for this method is the position or index at which you want to insert the content, and the second parameter is the content itself. You can remove all the content inside a tag using the clear() method. This will just leave you with the tag itself and its attributes.

At the beginning of this section, you selected a level two heading from the document and changed it to a level three heading. Using the same selector again will now show you the next level two heading that came after the original. This makes sense because the original heading is no longer a level two heading. 

The original heading can now be selected using h3:nth-of-type(2). If you completely want to remove an element or tag and all the content inside it from the tree, you can use the decompose() method.

Once you've decomposed or removed the original heading, the heading in the third spot takes its place.

If you want to remove a tag and its contents from the tree but don't want to completely destroy the tag, you can use the extract() method. This method will return the tag that it extracted. You will now have two different trees that you can parse. The root of the new tree will be the tag that you just extracted.

You can also replace a tag inside the tree with something else of your choice using the replace_with() method. This method will return the tag or string that it replaced. It can be helpful if you want to put the replaced content somewhere else in the document.

In the above code, the main heading of the document has been replaced with a b tag. The document no longer has an h1 tag, and that is why print(soup.h1) now prints None.

Final Thoughts

After reading the two tutorials in this series, you should now be able to parse different webpages and extract important data from the document. You should also be able to retrieve the original webpage, modify it to suit your own needs, and save the modified version locally.

If you have any questions regarding this tutorial, please let me know in the comments.


Digging Into Django QuerySets

Apr 05 2017 [Archived Version] □ Published at Caktus Blog

Digging Into Django QuerySets Object-relational mappers (or ORMs for short), such as the one that comes built-in with Django, make it easy for even new developers to become productive without needing to have a large body of knowledge about how to make use of relational databases. They abstract away the details of database access, replacing...


django-slowdown

Apr 04 2017 [Archived Version] □ Published at Latest Django packages added


django-slowdown

Apr 04 2017 [Archived Version] □ Published at Latest Django packages added


Django 1.11 released

Apr 04 2017 [Archived Version] □ Published at The Django weblog

The Django team is happy to announce the release of Django 1.11.

This version has been designated as a long-term support (LTS) release, which means that security and data loss fixes will be applied for at least the next three years. It will also receive fixes for crashing bugs, major functionality bugs in newly-introduced features, and regressions from older versions of Django for the next eight months until December 2017.

As always, the release notes cover the medley of new features in detail, but a few highlights are:

You can get Django 1.11 from our downloads page or from the Python Package Index. The PGP key ID used for this release is Tim Graham: 1E8ABDC773EDE252.

With the release of Django 1.11, Django 1.10 has reached the end of mainstream support. The final minor bugfix release (1.10.7) was issued today. Django 1.10 will receive security and data loss fixes for another eight months until December 2017.

Django 1.9 has reached the end of extended support. The final security release (1.9.13) was issued today. All Django 1.9 users are encouraged to upgrade to Django 1.10 or later.

See the downloads page for a table of supported versions and the future release schedule.


mypy-django

Apr 04 2017 [Archived Version] □ Published at Latest Django packages added

PEP-484 type hints bindings for the Django web framework


mypy-django

Apr 04 2017 [Archived Version] □ Published at Latest Django packages added

PEP-484 type hints bindings for the Django web framework


django-planet aggregates posts from Django-related blogs. It is not affiliated with or endorsed by the Django Project.

Social Sharing

Feeds

Tag cloud

admin administration adsense advanced ajax amazon angular angularjs apache api app appengine app engine apple application security aprendiendo python architecture articles asides audrey authentication automation backup bash basics best practices binary bitbucket blog blog action day blogging book books buildout business c++ cache capoeira celery celerycam celerycrawler challenges chat cheatsheet cherokee choices christianity class-based-views cliff cloud cms code codeship codeship news coding command community computer computers computing configuration consumernotebook continuous deployment continuous integration couchdb coverage css custom d data database databases db debian debugging deploy deployment deployment academy design developers development devops digitalocean django django1.7 django admin django cms djangocon django framework django-nose django-readonly-site django-rest-framework django-tagging django templates django-twisted-chat django web framework tutorials documentation dojango dojo dotcloud dreamhost dughh easy_install eclipse education elasticsearch email encoding english error europe eventbrite events expressjs extensions fabric facebook family fashiolista fedora field file filter fix flash flask foreman form forms frameworks friends fun functional reactive programming gae gallery games geek general gentoo gis git github gmail gnome google google app engine guides gunicorn hack hackathon hacking hamburg haskell heroku holidays hosting howto how-to howtos how-tos html http i18n image imaging indifex install installation intermediate internet ios iphone java javascript jinja2 jobs journalism jquery json justmigrated kde la latex linear regression linkedin linode linux login mac machine learning mac os x markdown math memcached meme mercurial meta meteor migration mirror misc model models mod_wsgi mongodb months mozilla multi-language mvc mysql nelenschuurmans newforms news nginx nodejs nosql oauth ogólne openshift opensource open source open-source openstack operations orm osx os x ottawa paas packages packaging patterns pedantics pelican penetration test performance personal personal and misc philippines philosophy php pi pil pinax pip piston planet plone plugin pony postgis postgres postgresql ppoftw presentation private programmieren programming programming & internet project projects pycharm pycon pycon-2013-guide pydiversity pygrunn pyladies pypi pypy pyramid python python3 queryset quick tips quora rabbitmq rails rant ratnadeep debnath reactjs recipe redis refactor release request resolutions rest reusable app review rhel rtnpro ruby ruby on rails scala scaling science screencast script scripting security server setup shell simple smiley snaking software software collections software development south sphinx sprint sql ssh ssl static storage supervisor support svn sysadmin tag tag cloud talk nerdy to me tastypie tdd techblog technical technology template templates template tags test testing tests tip tips tools tornado transifex travel travel tips for geeks tumbles tutorial tutorials twisted twitter twoscoops typo3 ubuntu uncategorized unicode unittest unix use user authentication usergroup uwsgi uxebu virtualenv virtualenvwrapper web web 2.0 web application web applications web design & development webdev web development webfaction web framework websockets whoosh windows wordpress work workshop wsgi yada znc zope