情報収集自動化

RSSを使って情報収集自動化

情報のソースとなるRSSフィードを選び出します。
各自いつも見ているNewsサイトなどのRSSフィードを取得してください。
今回はGoogleNewsです。

Pythonライブラリの追加

以下のライブラリをインストールしてください。
※pipが無ければapt-getしてください。

$ pip install feedparser
$ pip install extractcontent3
$ pip install google-api-python-client
$ pip install google-auth-httplib2
$ pip install google-auth-oauthlib 
$ pip install google-cloud-translate 

Pythonプログラム

test.pyを作成して以下のコードを記述します。
ここではGoogleNewsのRSSフィードからタイトルとソース記事を引っ張ってきます。

test.py

# coding: utf-8
import feedparser

def getRss():
    rssUrl = 'https://news.google.com/news/rss/headlines/section/topic/TECHNOLOGY'
    rssLang = '?hl=en-US&gl=US&ceid=US:en'
    feed = feedparser.parse(rssUrl + rssLang)
    for entry in feed.entries:
            title = entry.get('title')
            print('title:' + title)
            link = entry.get('link')
            print('link:' + link)

if __name__ == '__main__':
    getRss() 

結果はこちら

$ python3 test.py
title:PS5 And Xbox Shortages To Continue Into 2022
link:https://news.google.com/__i/rss/rd/articles/CBMiVWh0dHBzOi8va290YWt1LmNvbS94Ym94LWhlYWQtZXhwbGFpbnMtY29uc29sZS1zaG9ydGFnZXMtd2lsbC1jb250aW51ZS1pbnRvLTE4NDc3ODc0NzjSAVlodHRwczovL2tvdGFrdS5jb20veGJveC1oZWFkLWV4cGxhaW5zLWNvbnNvbGUtc2hvcnRhZ2VzLXdpbGwtY29udGludWUtaW50by0xODQ3Nzg3NDc4L2FtcA?oc=5
        　：
        以下略
        　：

URLから本文を取得

RSSフィードから取得したURLから記事本文を読み込みます。

Pythonライブラリの追加

$ pip install extractcontent3

Pythonプログラム

URLからextractcontent3を使用して記事本文を抜き出します。

test.py

# coding: utf-8
import feedparser
import requests
from extractcontent3 import ExtractContent

extractor = ExtractContent()
# オプション値を指定する
opt = {"threshold":50}
extractor.set_option(opt)

def getBody(link):
    try :
        res = requests.get(link)
        extractor.analyse(res.text)
        text, title = extractor.as_text()
        print('===================================================')
        print('title:' + title)
        print('===================================================')
        print('body :' + text.replace('. ', '.\n'))
        print('===================================================')
    except Exception as e :
        print(e)

def getRss():
    rssUrl = 'https://news.google.com/news/rss/headlines/section/topic/TECHNOLOGY'
    rssLang = '?hl=en-US&gl=US&ceid=US:en'
    feed = feedparser.parse(rssUrl + rssLang)
    for entry in feed.entries:
            link = entry.get('link')
            getBody(link)

if __name__ == '__main__':
    getRss()

結果はこちら

$ python3 test.py
===================================================
title:PS5 And Xbox Shortages To Continue Into 2022
===================================================
body :Its likely that if you are reading these words on this website, you already know how hard it can be to get your hands on the new consoles.
Well, Ive got some bad news straight from Xbox VP of gaming Phil Spencer: Finding a console aint getting easier anytime soon.

As spotted and transcribed by VGC, Phil Spencer spoke to The Wrap recently and explained that while chip shortages were partly to blame for the ongoing console shortages, there was more going on with the situation.
I think its probably too isolated to talk about it as just a chip problem, Spencer said.
When I think about, what does it mean to get the parts necessary to build a console today, and then get it to the markets where the demand is, there are multiple kind of pinch points in that process.
And I think regretfully its going to be with us for months and months, definitely through the end of this calendar year and into the next calendar year.
Spencer claims he and everyone at Mircosoft is working hard to get consoles out the door and into peoples homes, but he expects it will continue to be a challenge for quite a while.
The thing thats most disappointing is just the fan disappointment, Spencer admitted.
People really want this new generation of consoles—theyre good consoles, both from us and the other platform holders—and they want the new functionality.

===================================================

次はひとまず要約や翻訳は置いておいてBlogger自動投稿を行います。