merging or adding of two dictionaries

• merging or adding of two dictionaries this feature available from python 3.5 + versions on words.

Example:

x={'a':1,'b':2}

y={'c':3,'d':4}

z={'a':5,'b':6}

p={'a':5,'b':7}

xy={**x,**y}

print(xy)

xz={**x,**z}

print(xz)

zp={**z,**p}

print(zp)

Output:

{'a': 1, 'b': 2, 'c': 3, 'd': 4}

{'a': 5, 'b': 6}

{'a': 5, 'b': 7}

args and kwargs in python

Normal Arguments and arbitrary arguments and keyword arguments in python:

• Normal argument type depends on value/data

• Arbitrary argument by default type is tuple, Represents *

• keyword argument  by default type is dictionary,Represents **

Example:

def f1(req,*args,**kwargs):
    if req:
        print(req,type(req))
    if args:
        print(args,type(args))
    if kwargs:
        print(kwargs,type(kwargs))

f1(10)

f1(10,2.3,5,"siva")

f1(10,2.3,5,"siva",key1=20,key2="siva",key3=True)

Output:

10  <class 'int'>

10  <class 'int'>
(2.3, 5, 'siva')  <class 'tuple'>

10  <class 'int'>
(2.3, 5, 'siva')  <class 'tuple'>

{'key1': 20, 'key2': 'siva', 'key3': True}  <class 'dict'>

constructors in python

_ _new_ _ 

• __new__ is for creating objects.

• Use __new__ when you need to control the creation of a new instance.

• __new__ is the first step of instance creation. It's called first, and is responsible for returning a new instance of your class.

• __new__ is static, class method .

• __new__ takes cls as parameter.

• __new__ is good for immutable object as they cannot be changed once they are assigned. So we can return new instance which has new state.

class A:

    

    def __new__(cls):

        print "A.__new__ is called"  # -> this is never called

A( )

_ _init_ _

• __init__ is for initializing objects.

•  Use __init__ when you need to control initialization of a new instance.

• In contrast, __init__ doesn't return anything,it's only responsible for initializing the instance after it's been created.

• __init__ is instance method. 

• __init__ takes self as parameter.

• __init__ called on it automatically.

class A:

    def __init__(self):
        print "A.__init__ called"


A( )



class A:

    def __init__(self):
        return 29


A( )


class A(object):  # -> don't forget the object specified as base

    def __new__(cls):
        print "A.__new__ called"
        return super(A, cls).__new__(cls)

    def __init__(self):
        print "A.__init__ called"


A( )

Example:

class ExampleClass(object):

     def __new__(cls, value):

         print"Creating new instance..."

         #Call the superclass constructor to create the instance.

         instance = super(ExampleClass, cls).__new__(cls)

         return instance

     def __init__(self, value):

        print"Initialising instance..."

        self.x= value

exampleInstance = ExampleClass(42)

print(exampleInstance.x)

why method takes self parameter in python

when objects are instantiated,the object itself is passed in to the self parameter of the method.

class test:

     def m1(self):

           print("hai")

t1=test()

t1.m1()

note:

• t1 passed to the self parameter of the m1( ) method

methods in python

Methods:

• It is possible to define two kinds of methods with-in a class that can be called without an instance.

Class Method:

• A class method takes cls as first parameter.
• A class method can access or modify class state.
• We use @classmethod decorator in python to create a class method.
• We generally use class method to create factory methods.

Syntax:

class C(object):

    @classmethod

    def fun(cls, arg1, arg2, ...):

       ........................

       ........................

       ........................

Static Method:

• A static method needs no specific parameters.
• A static method can’t access or modify it.
• We use @staticmethod decorator to create a static method in python.
• We generally use static methods to create utility functions.

Syntax:

class C(object):

    @staticmethod

    def fun( arg1, arg2, ...):

       ....................

       ....................

       ....................

Example:

class Test(object):

    x=20

    def m1(self):

        print"hai,i am in instance method"

        print(Test.x)

 

    @classmethod

    def m2(cls):

        print "hai,i am in class method"

        print(Test.x)

 

    @staticmethod

    def m3():

        print "hai,i am in static method"

        print(Test.x)

 

Test.m2()

Test.m3()

Test.m1()

web scraping

• When performing data science tasks, it’s common to want to use data found on the internet. 

• You’ll usually be able to access this data in csv format, or via an Application Programming Interface(API). 

• However, there are some times when the data you want can only be accessed as part of a web page. 

• In cases like this, you’ll want to use a technique called web scraping. 

Web-Scraping:

• to get the data from the web page into a format you can work with in your analysis.

• We are going to use Python as our scraping language, together with a simple and powerful library, BeautifulSoup.

pip install BeautifulSoup4 or beautifulsoup4

Before we start jumping into the code, let’s understand the basics of HTML and some rules of scraping.

<!DOCTYPE html>  ----> type deceleration
<html>  
    <head>
    </head>
    <body>
        <h1> First Scraping </h1>
        <p> Hello World </p>
    <body>

</html>


This is the basic syntax of an HTML webpage. Every <tag> serves a block inside the webpage:

1. <!DOCTYPE html>: HTML documents must start with a type declaration.

2. The HTML document is contained between <html> and </html>.

3. The meta and script declaration of the HTML document is between <head> and </head>.

4. The visible part of the HTML document is between <body> and </body> tags.

5. Title headings are defined with the <h1> through <h6> tags.

6. Paragraphs are defined with the <p> tag.

• Other useful tags include <a> for hyperlinks, <table> for tables, <tr> for table rows, and <td> for table columns.

• Also, HTML tags sometimes come with id or class attributes. 

• The id attribute specifies a unique id for an HTML tag and the value must be unique within the HTML document. 

• The class attribute is used to define equal styles for HTML tags with the same class. We can make use of these ids and classes to help us locate the data we want.

Scraping Rules:

• You should check a website’s Terms and Conditions before you scrape it. 

• Be careful to read the statements about legal use of data. 

• Usually, the data you scrape should not be used for commercial purposes.

• Do not request data from the website too aggressively with your program (also known as spamming), as this may break the website. 

• Make sure your program behaves in a reasonable manner (i.e. acts like a human). 

• One request for one webpage per second is good practice.

• The layout of a website may change from time to time, so make sure to revisit the site and rewrite your code as needed.

Examle:1

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

print(soup)

Example:2

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

print(source)

Example:3

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

print(soup.title)

Example:4

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

print(soup.title.string)

Example:5

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

print(soup.title.text)

Example:6

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( ) 

soup=bs.BeautifulSoup(source,'lxml')

print(soup.p)

Example:7

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

print(soup.find_all('p'))

Example:8

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

for paragraph in soup.find_all('p'):

    print(paragraph)

Example:9

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

for paragraph in soup.find_all('p'):

    print(paragraph.string)

Example:10

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

for paragraph in soup.find_all('p'):

    print(paragraph.text)

Example:11

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

print(soup.get_text( ))

Example:12

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

for url in soup.find_all('a'):

    print(url.text)

Example:13

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

for url in soup.find_all('a'):

    print(url.get('href'))

Example:14

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

nav=soup.nav

print(nav)

Example:15

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

nav=soup.nav

for url in nav.find_all('a'):

    print(url.get('href'))

Example:16

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

body=soup.body

for paragraph in body.find_all('p'):

    print(paragraph.text)

Example:17

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

for div in soup.find_all('div'):

    print(div.text)

Example:18

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

for div in soup.find_all('div',class_='body'):

    print(div.text)

Example:19

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

table=soup.table

print(table)

Example:20

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

table=soup.find('table')

print(table)

Example:21

import bs4 as bs

import urllib

source=urllib.urlopen('https://pythonprogramming.net/parsememcparseface/').read( )

soup=bs.BeautifulSoup(source,'lxml')

table=soup.find('table')

table_rows=table.find_all('tr')

for tr in table_rows:

    td=tr.find_all('td')

    row=[i.text for i in td]

    print(row)

Example:22

import bs4 as bs

import urllib

import pandas as pd

dfs=pd.read_html('https://pythonprogramming.net/parsememcparseface/')

for df in dfs:

    print(df)

Example:23

import bs4 as bs

import urllib

import pandas as pd

dfs=pd.read_html('https://pythonprogramming.net/parsememcparseface/',header=0)

for df in dfs:

    print(df)

Example:24

import bs4 as bs

import urllib

import pandas as pd

source=urllib.urlopen('https://pythonprogramming.net/sitemap.xml').read()

soup=bs.BeautifulSoup(source,'xml')

print(soup)

Example:25

import bs4 as bs

import urllib

import pandas as pd

source=urllib.urlopen('https://pythonprogramming.net/sitemap.xml').read()

soup=bs.BeautifulSoup(source,'xml')

for url in soup.find_all('loc'):

    print(url.text)

Advanced Scraping Techniques:

• BeautifulSoup is simple and great for small-scale web scraping, But if you are interested in scraping data at a larger scale, you should consider using these other alternatives: Scrapy

• Try to integrate your code with some public APIs. The efficiency of data retrieval is much higher than scraping webpages. 

• For example, take a look at Facebook Graph API, which can help you get hidden data which is not shown on Facebook webpages.

• Consider using a database backend like MySQL to store your data when it gets too large.

Test-cases

Testing : 

• It involves identifying bug/error/defect in a software without correcting it.

• Testing is the process of evaluating a system or its component(s) with the intent to find whether it satisfies the specified requirements or not.

Debugging : 

• It involves identifying and fixing the problems/bugs. 

Who does Testing?

• In the IT industry, large companies have a team with responsibilities to evaluate the developed software in context of the given requirements. 

different types of testing:

• Manual Testing

• Automation Testing

Manual Testing:

• Manual testing includes testing a software manually, i.e., without using any automated tool or any script.

Automation Testing:

• Automation testing, which is also known as Test Automation, is when the tester writes scripts and uses another software to test the product.

• Selenium

• QTP

• Jmeter

• Loadrunner

• TestLink

• Quality Center(ALM)

Selenium

It is a software testing tool used for regression testing. It is an open source testing tool that provides playback and recording facility for regression testing. The Selenium IDE only supports Mozilla Firebox web browser.

• It provides the provision to export recorded script in other languages like Python,Java, Ruby, C#.

• It can execute multiple tests at a time

• Autocomplete for Selenium commands that are common Walk through tests

• Identifies the element using id, name , X-path, etc.

• Store tests as Python Script,Ruby Script, HTML, and any other format.

• It provides an option to assert the title for every page

• It supports selenium user-extensions file

• It allows to insert comments in the middle of the script for better understanding and debugging

In-order to work with selenium with python program first we install selenium.

pip install selenium

we are testing web-applications through web browsers, we need some web-browser related drivers.download that drivers and install that drivers.

download and install geckodriver for Chrome or Firefox or IE.

What is Gecko and GeckoDriver? 

• Gecko is a web browser engine used in many applications developed by Mozilla Foundation and the Mozilla Corporation.

• Gecko Driver is the link between your tests in Selenium and the Firefox browser.

•  GeckoDriver is a proxy for using W3C WebDriver-compatible clients to interact with Gecko-based browsers i.e. Mozilla Firefox

example1:

from selenium import webdriver

browser=webdriver.Chrome()

browser.get('https://www.facebook.com')

example2:

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

import time

browser=webdriver.Chrome()

browser.get('https://www.facebook.com')

element=browser.find_element_by_name('email')

element.send_keys('divyakola43@gmail.com'+Keys.RETURN)

element1=browser.find_element_by_name('pass')

element1.send_keys('sivakrishna143'+Keys.RETURN)

example3:

import time

from selenium import webdriver

from selenium.webdriver.common.action_chains import ActionChains

from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()

driver.get('http://google.com')

element = driver.find_element_by_link_text('About')

ActionChains(driver) \

    .key_down(Keys.CONTROL) \

    .click(element) \

    .key_up(Keys.CONTROL) \

    .perform()

time.sleep(30) # Pause to allow you to inspect the browser.

driver.quit()

example4:

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

browser = webdriver.Chrome()

browser.get('http://www.yahoo.com')

elem = browser.find_element_by_name('p')  # Find the search box

elem.send_keys('seleniumhq' + Keys.RETURN)

browser.quit()

example5:

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()

driver.get("http://www.python.org")

assert "Python" in driver.title

elem = driver.find_element_by_name("q")

elem.clear()

elem.send_keys("pycon")

elem.send_keys(Keys.RETURN)

assert "No results found." not in driver.page_source

driver.close()

example6: unittest

import unittest2 as unittest

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

class PythonOrgSearch(unittest.TestCase):

    def setUp(self):

        self.driver = webdriver.Chrome()

    def test_search_in_python_org(self):

        driver = self.driver

        driver.get("http://www.python.org")

        self.assertIn("Python", driver.title)

        elem = driver.find_element_by_name("q")

        elem.send_keys("pycon")

        elem.send_keys(Keys.RETURN)

        assert "No results found." not in driver.page_source

    def tearDown(self):

        self.driver.close()