Pyppeteer - Chromium 다운로드 시 SSLCertVerificationError 발생할 경우

 

맥북에서 Pyppeteer 패키지 (Headless chrome/chromium automation library)로 웹브라우저 크롤링 작업을 좀 하려했는데 오류가 발생했다

 

[예시 코드]

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('http://www.google.com')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

[오류 발생]

[W:pyppeteer.chromium_downloader] Starting Chromium download. Download may take a few minutes.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests_html_test.py", line 5, in <module>
    response.html.render()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests_html.py", line 586, in render
    self.browser = self.session.browser  # Automatically create a event loop and browser
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests_html.py", line 730, in browser
    self._browser = self.loop.run_until_complete(super().browser)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests_html.py", line 714, in browser
    self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, args=self.__browser_args)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyppeteer/launcher.py", line 307, in launch
    return await Launcher(options, **kwargs).launch()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyppeteer/launcher.py", line 120, in __init__
    download_chromium()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyppeteer/chromium_downloader.py", line 139, in download_chromium
    extract_zip(download_zip(get_url()), DOWNLOADS_FOLDER / REVISION)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyppeteer/chromium_downloader.py", line 81, in download_zip
    r = http.request('GET', url, preload_content=False)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /chromium-browser-snapshots/Mac/588429/chrome-mac.zip (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

urllib3 패키지에서 SSLCertVerificationError 예외가 발생했다

권한 문제인가 싶어서 sudo로 실행해봐도 동일하길래 구글링을 좀 해봤다

---

[ 해결 ] urllib3 버전 문제

urllib3 패키지 버전이 1.25 미만이어야 제대로 돌아간다고 해서 새로 설치했다

(기존에 1.26.2 버전이 설치되어 있었다)

pip install -U "urllib3<1.25"

스크립트를 다시 실행해보니

[W:pyppeteer.chromium_downloader] Starting Chromium download. Download may take a few minutes.
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py:842: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
100%|██████████| 86.8M/86.8M [00:02<00:00, 42.7Mb/s]
[W:pyppeteer.chromium_downloader] Chromium download done.
[W:pyppeteer.chromium_downloader] chromium extracted to: /Library/Application Support/pyppeteer/local-chromium/588429

다운로드가 제대로 완료되었다 (구현한 기능도 제대로 돌아간다 ㅎㅎ)

끝~!

 

[참고]

https://github.com/miyakogi/pyppeteer/issues/219