<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 練習
觀察 https
Step1:  使用 requests 做靜態爬蟲
Step2:  使用 Selenium 做動態爬蟲
<ASSISTANT_TASK:>
Python Code:
import os
import requests
import re
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from fake_useragent import UserAgent
from pprint import pprint
url = 'https://afuntw.github.io/Test-Crawling-Website/pages/gallery/index.html'
fu = UserAgent()
Explanation: 練習
觀察 https://afuntw.github.io/Test-Crawling-Website/pages/gallery/index.html 並撰寫爬蟲程式
判斷是否為 JavaScript rendered website
下載網頁影片
設定 Implicit Wait
透過 XPath 定位圖片
透過靜態網站爬蟲會看到的圖片是
<img src="https://i.imgur.com/0s6Iiu3.png" height="184" width="337" style="display: inline-block;">
透過動態網站爬蟲會看到的圖片是
<img src="https://i.imgur.com/db3tGBG.png" height="320" width="320" style="display: inline-block;">
End of explanation
resp = requests.get(url)
soup = BeautifulSoup(resp.text, 'lxml')
imgs = soup.find_all('img', class_=re.compile('.*img-change'))
imgs = [i['src'] for i in imgs]
imgs = list(set(imgs))
pprint(imgs)
Explanation: 使用 requests 做靜態爬蟲
End of explanation
driver = webdriver.Chrome()
results = os.path.abspath('../results')
if not os.path.exists(results):
    os.makedirs(results)
try:
    # webdriver setting
    driver.get(url)
    driver.maximize_window()
    driver.implicitly_wait(10)
    
    # xpath
    imgs = driver.find_elements(By.XPATH, '/html/body/div/div/div/a/img')
    imgs = [i.get_attribute('src') for i in imgs]
    imgs = list(set(imgs))
    print(imgs)
    
    # download
    for img in imgs:
        headers = {'User-Agent': fu.random}
        img_resp = requests.get(img, stream=True, headers=headers)
        
        filename = os.path.basename(img)
        print('catch - {}'.format(filename))
        filename = os.path.join(results, filename)
        
        with open(filename, 'wb') as f:
            for chunk in img_resp.iter_content(2048):
                f.write(chunk)
            print('save - {}'.format(filename))
    
except Exception as e:
    print(e)
finally:
    driver.quit()
Explanation: 使用 Selenium 做動態爬蟲
End of explanation
<END_TASK>