mirror: 初始快照

.gitignore

@@ -0,0 +1,3 @@
+/.vscode
+__pycache__
+error.csv

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 PWXCOO
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,102 @@
+# chinese-xinhua
+
+中华新华字典数据库和 API 。收录包括 14032 条歇后语，16142 个汉字，264434 个词语，31648 个成语。
+
+## Project Structure
+
+```
+chinese-xinhua/
+|
++- data/ <-- 数据文件夹
+|  |
+|  +- idiom.json <-- 成语
+|  |
+|  +- word.json <-- 汉字
+|  |
+|  +- xiehouyu.json <-- 歇后语
+|  |
+|  +- ci.json <-- 词语
+```
+
+## Database Introduction
+
+### 成语 (idiom.json)
+
+```json
+[
+    {
+        "derivation": "语出《法华经·法师功德品》下至阿鼻地狱。”",
+        "example": "但也有少数意志薄弱的……逐步上当，终至堕入～。★《上饶集中营·炼狱杂记》",
+        "explanation": "阿鼻梵语的译音，意译为无间”，即痛苦无有间断之意。常用来比喻黑暗的社会和严酷的牢狱。又比喻无法摆脱的极其痛苦的境地。",
+        "pinyin": "ā bí dì yù",
+        "word": "阿鼻地狱",
+        "abbreviation": "abdy"
+    },
+    ...
+]
+```
+
+### 词语 (ci.json)
+
+```json
+[
+    { 
+        "ci": "宸纶", 
+        "explanation": "1.帝王的诏书﹑制令。" 
+    },
+    ...
+]
+```
+
+### 汉字 (word.json)
+
+```json
+[
+    {
+        "word": "嗄",
+        "oldword": "嗄",
+        "strokes": "13",
+        "pinyin": "á",
+        "radicals": "口",
+        "explanation": "嗄〈叹〉\n\n 同啊”。表示省悟或惊奇\n\n 嗄!难道这里是没有地方官的么?--宋·佚名《新编五代史平话》\n\n 嗄á叹词。在句首，〈表〉疑问或反问～，这是什么？～，你想干什么？\"嗄\"另见shà㈠。\n\n 嗄shà\n\n ⒈声音嘶哑～声。\n\n 嗄a 1.助词。表示强调﹑肯定或辩解。 2.助词。方言。表示疑问或反诘。\n\n 嗄xià 1.见\"嗄饭\"。 2.见\"嗄程\"。",
+        "more": "嗄 ga、a 部首 口 部首笔画 03 总笔画 13  嗄2\nshà\n〈形〉\n(1)\n声音嘶哑的 [hoarse]\n终日嚎而嗌不嗄。--《老子》\n(2)\n又如嗄哑,嗄嘶(嗓音嘶哑)\n嗄\nshà\n〈叹〉\n(1)\n什么 [what]--表示否定\n我要丢个干干净,看你嗄法把我治。--清·蒲松龄《聊斋俚曲集》\n(2)\n旧时仆役对主人、下级对上级的应诺声 [yes]\n带进来”。两边军士应一声嗄”,即将牛皋推至面前。--《说岳全传》\n另见á\n嗄1\ná\n〈叹〉\n同啊”(á)。表示省悟或惊奇 [ah]\n嗄!难道这里是没有地方官的么?--宋·佚名《新编五代史平话》\n另见shà\n嗄1\nshà　ㄕㄚ╝\n嗓音嘶哑。\n郑码janr，u55c4，gbke0c4\n笔画数13，部首口，笔顺编号2511325111354\n嗄2\ná　ㄚˊ\n同啊2”。\n郑码janr，u55c4，gbke0c4\n笔画数13，部首口，笔顺编号2511325111354"
+    },
+    ... 
+]
+```
+
+### 歇后语 (xiehouyu.json)
+
+```json
+[
+    {
+        "riddle": "飞机上聊天",
+        "answer": "高谈阔论"
+    },
+    ...
+]
+```
+
+## Changelog
+
+<details><summary>查看更新日志  </summary> 
+
+- 20181216: 成语数据集去重
+- 20181216: API 功能下线
+- 20180803: 添加词语数据集
+- 20180206: 添加成语，歇后语，汉字数据集
+
+</details>
+
+
+## Copyright
+
+本仓库的所有的数据都是我从网上收集整理的。仓库本来的目的是因为我以前想做一个成语接龙的东西，但是苦于没有现成可用的数据库，自己就从各个网站抓取整理了一份。放在 Github 是为了方便自己的使用，同时也能方便有类似需求的人不用去做这些 trival 的工作。所有抓取数据的[脚本](./scripts/README.md)都在仓库里。
+
+**本仓库无任何商业目的！如果有侵权行为将及时删除！**
+
+---
+
+## 关于本镜像
+
+本仓库系 pwxcoo/chinese-xinhua 的忠实镜像。原项目以 MIT 许可发布（Copyright 2018 PWXCOO），著作权归原作者所有。上游版本为 commit fe6d6c2e8baa82187f4c96bbe042e43f96c05666（2019-01-17），抓取于 2026-10-02；来源与完整许可见本仓 LICENSE。

data/ci.json

@@ -0,0 +1 @@
+[{"ci": "宸纶", "explanation": "1.帝王的诏书﹑制令。"}, {"ci": "宸明", "explanation": "1.圣明。借指皇帝。"}, {"ci": "宸命", "explanation": "1.圣旨;皇帝的委命。"}, {"ci": "宸谟", "explanation": "1.帝王的谋略。"}, {"ci": "宸谋", "explanation": "1.帝王的谋划。"}, {"ci": "宸念", "explanation": "1.皇帝的思虑。"}, {"ci": "宸阙", "explanation": "1.指皇宫。"}, {"ci": "宸赏", "explanation": "1.谓帝王的游赏。 \n2.帝王的赏识﹑赞赏。"}, {"ci": "宸声", "explanation": "1.皇帝的命令﹑旨意。"}, {"ci": "宸枢", "explanation": "1.指帝位。"}, {"ci": "宸算", "explanation": "1.帝王的谋略。"}, {"ci": "宸题", "explanation": "1.帝王的亲笔题字。"}, {"ci": "宸听", "explanation": "1.谓帝王的听闻。"}, {"ci": "宸网", "explanation": "1.帝王车上悬挂的珠饰网帘。"}, {"ci": "宸威", "explanation": "1.帝王的威严。"}, {"ci": "宸卫", "explanation": "1.帝王的仪仗﹑侍卫。"}, {"ci": "宸文", "explanation": "1.帝王所作之诗文。"}, {"ci": "宸象", "explanation": "1.指紫微垣。 \n2.借喻君王。"}, {"ci": "宸心", "explanation": "1.帝王的心意。"}, {"ci": "宸轩", "explanation": "1.帝王所居宫室。"}, {"ci": "宸严", "explanation": "1.帝王的威严。亦喻指君王。"}, {"ci": "宸筵", "explanation": "1.帝王举行的宴会。"}, {"ci": "宸曜", "explanation": "1.喻指帝王的恩泽。"}, {"ci": "宸掖", "explanation": "1.帝王内宫。"}, {"ci": "宸仪", "explanation": "1.帝王的仪仗。 \n2.帝王的仪容。"}, {"ci": "宸扆", "explanation": "1.借指帝廷﹑君位。扆，帝王座后的屏风。"}, {"ci": "宸音", "explanation": "1.皇帝的旨意或话语。"}, {"ci": "宸游", "explanation": "1.亦作\"宸游\"。 \n2.帝王之巡游。"}, {"ci": "宸舆", "explanation": "1.帝王所乘车。借指皇帝。"}, {"ci": "宸宇", "explanation": "1.屋檐。比喻庇荫。 \n2.指帝居。"}, {"ci": "宸垣", "explanation": "1.指帝京。清吴长元有《宸垣识略》一书，所记皆当时京师之事。"}, {"ci": "宸藻", "explanation": "1.指帝王的诗文。"}, {"ci": "宸造", "explanation": "1.对皇帝的敬称。"}, {"ci": "宸瞔", "explanation": "1.帝王的眷注﹑恩宠。瞔，同\"眷\"。"}, {"ci": "宸赠", "explanation": "1.皇帝的赏赐。"}, {"ci": "宸札", "explanation": "1.帝王的手札。"}, {"ci": "宸章", "explanation": "1.皇帝所作的诗文。"}, {"ci": "宸正", "explanation": "1.帝后之正位。"}, {"ci": "宸旨", "explanation": "1.帝王的旨意。"}, {"ci": "宸衷", "explanation": "1.帝王的心意。 \n2.偶沿用于称国家最高领导。"}, {"ci": "宸州", "explanation": "1.帝京，京城。"}, {"ci": "宸瞩", "explanation": "1.谓帝王的注视﹑观赏。"}, {"ci": "宸注", "explanation": "1.帝王的关注。"}, {"ci": "莐藩", "explanation": "1.即知母。根状茎可入药。"}, {"ci": "莐藸", "explanation": "1.草名。即五味子。"}, {"ci": "晨安", "explanation": "1.早安。清晨的问候语。"}, {"ci": "晨步", "explanation": "1.清晨散步。"}, {"ci": "晨参暮礼", "explanation": "1.早晩参拜。"}, {"ci": "晨紁", "explanation": "1.见\"晨妆\"。"}, {"ci": "晨唱", "explanation": "1.指鸡清晨啼鸣。"}, {"ci": "晨朝", "explanation": "1.清晨。"}, {"ci": "晨炊", "explanation": "1.清晨做饭;早饭。"}, {"ci": "晨旦", "explanation": "1.天亮。"}, {"ci": "晨发", "explanation": "1.早晨发布。 \n2.清晨出发。"}, {"ci": "晨风", "explanation": "1.清晨的风。 \n2.鸟名。"}, {"ci": "晨妇", "explanation": "1.谓妇人干政。"}, {"ci": "晨鹄", "explanation": "1.鸟名。"}, {"ci": "晨祼", "explanation": "1.古代宗庙祭祀仪节名。"}, {"ci": "晨光", "explanation": "1.曙光;阳光。"}, {"ci": "晨晖", "explanation": "1.清晨的阳光。"}, {"ci": "晨昏", "explanation": "早上和晚上常做夜班令人感觉晨昏颠倒，早晚不分｜晨昏随侍在侧。"}, {"ci": "晨昏定省", "explanation": "1.同\"昏定晨省\"。 \n2.旧时指朝夕服侍慰问双亲。"}, {"ci": "晨昏蒙影", "explanation": "即曙暮光”(851页)。"}, {"ci": "晨景", "explanation": "1.清晨的阳光。"}, {"ci": "晨暝", "explanation": "1.旦与夕。"}, {"ci": "晨暮", "explanation": "1.早晨和傍晚。"}, {"ci": "晨秦暮楚", "explanation": "1.犹言朝秦暮楚。比喻反复无常。"}, {"ci": "晨曲", "explanation": "1.清晨的歌唱;清晨的乐曲。"}, {"ci": "晨膳", "explanation": "1.早餐。"}, {"ci": "晨省", "explanation": "1.早晨向父母问安。亦指昏定晨省之礼。"}, {"ci": "晨事", "explanation": "1.指早晨祭祀。"}, {"ci": "晨暾", "explanation": "1.朝阳。"}, {"ci": "晨乌", "explanation": "1.初升的太阳。古代神话谓日中有乌，故以乌为日之代称。"}, {"ci": "晨夕", "explanation": "1.犹朝夕。旦与暮。天天;时时。"}, {"ci": "晨霞", "explanation": Line truncated

data/idiom.json

@@ -0,0 +1 @@
+[{"derivation": "语出《法华经·法师功德品》下至阿鼻地狱。”", "example": "但也有少数意志薄弱的……逐步上当，终至堕入～。★《上饶集中营·炼狱杂记》", "explanation": "阿鼻梵语的译音，意译为无间”，即痛苦无有间断之意。常用来比喻黑暗的社会和严酷的牢狱。又比喻无法摆脱的极其痛苦的境地。", "pinyin": "ā bí dì yù", "word": "阿鼻地狱", "abbreviation": "abdy"}, {"derivation": "三国·魏·曹操《整齐风俗令》阿党比周，先圣所疾也。”", "example": "《论语·卫灵公》众恶之，必察焉；众好之，必察焉”何晏集解引三国魏王肃曰或众～，或其人特立不群，故好恶不可不察也。”", "explanation": "指相互勾结，相互偏袒，结党营私。", "pinyin": "ē dǎng bǐ zhōu", "word": "阿党比周", "abbreviation": "edbz"}, {"derivation": "《汉书·诸葛丰传》今以四海之大，曾无伏节死谊之臣，率尽苟合取容，阿党相为，念私门之利，忘国家之政。”", "example": "无", "explanation": "阿党偏袒、偏私一方。为了谋求私利相互偏袒、包庇。", "pinyin": "ē dǎng xiāng wéi", "word": "阿党相为", "abbreviation": "edxw"}, {"derivation": "鲁迅《我们要批评家》然而新的批评家不开口，类似批评家之流便趁势一笔抹杀‘阿狗阿猫’。”", "example": "无", "explanation": "旧时人们常用的小名。引申为任何轻贱的，不值得重视的人或著作。", "pinyin": "ā gǒu ā māo", "word": "阿狗阿猫", "abbreviation": "agam"}, {"derivation": "见阿家阿翁”。", "example": "既然如此，你我两个，便学个不痴不聋的～。★《儿女英雄传》二三回", "explanation": "阿名词的前缀。姑丈夫的母亲。翁丈夫的父亲。指公公婆婆。", "pinyin": "ā gū ā wēng", "word": "阿姑阿翁", "abbreviation": "agaw"}, {"derivation": "唐·赵璾《因话录》卷一郭暖尝与升平公主琴瑟不调。尚父拘暖，自诣朝童结罪。上召而慰之曰‘谚云不痴不聋，不作阿家阿翁。’”", "example": "无", "explanation": "阿名词的前缀。家通姑”，丈夫的母亲。翁丈夫的父亲。指公公婆婆。", "pinyin": "ā gū ā wēng", "word": "阿家阿翁", "abbreviation": "agaw"}, {"derivation": "语出旧题汉·班固《汉武故事》汉武帝幼时曾对姑母长公主说若得阿娇(姑母之女)作妇，当以金屋贮之。”", "example": "～原相配，太其玉镜宁无时？★明·周履靖《锦笺记·诒婚》", "explanation": "阿娇指汉武帝刘彻的姑母（长公主刘娇）的女儿。原指汉武帝刘彻要用金屋接纳阿娇为妇。这里泛指美丽高贵的女子。", "pinyin": "ā jiāo jīn wū", "word": "阿娇金屋", "abbreviation": "ajjw"}, {"derivation": "鲁迅《我的第一个师父》这和名孩子为阿猫阿狗，完全是一样的意思容易养大。”", "example": "在德国无论～见面，一律互相举手高喊希特勒万岁”。★邹韬奋《法西斯作风的罪恶》", "explanation": "旧时人们常用的小名。引申为任何轻贱的，不值得重视的人或著作。", "pinyin": "ā māo ā gǒu", "word": "阿猫阿狗", "abbreviation": "amag"}, {"derivation": "元·张国宾《合汗衫》第四折张寿友曰‘阿弥陀佛，这个是谁？’卜儿云‘这便是媳妇儿。’”", "example": "和尚念经，～。", "explanation": "佛教语，信佛的人用作口头诵颂的佛号，表示祈祷祝福或感谢神灵的意思。", "pinyin": "ē mí tuó fó", "word": "阿弥陀佛", "abbreviation": "emtf"}, {"derivation": "见阿鼻地狱”。", "example": "生前不曾修福，死堕～。★《敦煌变文集·妙法莲花经讲经文》", "explanation": "阿毗梵语的译音，意译为无间”，即痛苦无有间断之意。常用来比喻黑暗的社会和严酷的牢狱。又比喻无法摆脱的极其痛苦的境地。", "pinyin": "ā pí dì yù", "word": "阿毗地狱", "abbreviation": "apdy"}, {"derivation": "据《晋书·卫玠传》及《王澄传》载玠好言玄理。琅玡王澄字平子，兄昵称之曰阿平”。有高名，少所推许，每闻玠言，辄叹息绝倒。故时为之语曰卫玠谈道，平子绝倒。”", "example": "无", "explanation": "以之比喻对对方的言论极为佩服。亦用为讥讽言论极为乖谬，常贻笑大方。", "pinyin": "ā píng jué dǎo", "word": "阿平绝倒", "abbreviation": "apjd"}, {"derivation": "《孟子·公孙丑上》宰我、子贡、有若，智足以知圣人，污不至阿其所好。”", "example": "吾何能～为？★鲁迅《坟·摩罗诗力说》", "explanation": "阿曲从；其他的；好爱好。指为取得某人的好感而迎合他的爱好。", "pinyin": "ē qí suǒ hào", "word": "阿其所好", "abbreviation": "eqsh"}, {"derivation": "恽代英《再复毕几庵书》故办杂志者，当研究社会心Line truncated

scripts/README.md

@@ -0,0 +1,11 @@
+## Scripts
+
+这里是抓取整理数据的脚本们。
+
+
+- [addAbbreviation.py](./addAbbreviation.py):  给成语添加缩写
+- [clean.ipynb](./clean.ipynb): 去重重复成语
+- [chengyu.py](./chengyu.py): 抓取成语
+- [ci.py](./ci.py): 抓取词语
+- [word.py](./word.py): 抓取汉字
+- [xiehouyu.py](./xiehouyu.py): 抓取歇后语

scripts/addAbbreviation.py

@@ -0,0 +1,14 @@
+import pandas as pd
+
+chengyu = pd.read_json('chengyu.json')  
+
+replace = {'ā':'a', 'á':'a', 'ǎ':'a', 'à':'a', 'ō':'o', 'ó':'o', 'ǒ':'o', 'ò':'o',\
+           'ē':'e', 'é':'e', 'ě':'e', 'è':'e'}
+def abbreviation(pinyin):
+    each = pinyin.split(' ')
+    return ''.join(list(map(lambda x: replace[x[:1]] if x[:1] in replace else x[:1], each)))
+
+chengyu['abbreviation'] = chengyu['pinyin'].apply(abbreviation)
+
+chengyu.to_json('idiom.json', force_ascii=False, orient='records')
+    

scripts/chengyu.py

@@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+
+"""
+
+author: pwxcoo
+date: 2018-02-05 
+description: 抓取下载成语并保存
+
+"""
+
+import requests, json
+from bs4 import BeautifulSoup
+
+def downloader(url):
+    """
+    下载成语并保存
+    """
+    response = requests.get(url)
+
+    if response.status_code != 200:
+        print(f'{url} is failed!')
+        return
+    
+    print(f'{url} is parsing')
+    html = BeautifulSoup(response.content.decode('gbk', errors='ignore'), "lxml")
+    table = html.find_all('table')[-2]
+
+    prefix = 'http://www.zd9999.com'
+    words = [prefix + a.get('href') for a in table.find_all('a')]
+
+    res = []
+    for i in range(0, len(words)):
+        response = requests.get(words[i])
+        print(f'{[words[i]]} is parsing')
+        if response.status_code != 200:
+            print(f'{words[i]} is failed!')
+            continue
+
+        wordhtml = BeautifulSoup(response.content.decode('gbk', errors='ignore'), "lxml")
+        explanation = wordhtml.find_all('table')[-3].find_all('tr')
+        res.append({'word':explanation[0].text.strip(),\
+                    'pinyin': explanation[1].find_all('tr')[0].find_all('td')[1].text.strip(),\
+                    'explanation': explanation[1].find_all('tr')[1].find_all('td')[1].text.strip(),\
+                    'derivation': explanation[1].find_all('tr')[2].find_all('td')[1].text.strip(),\
+                    'example': explanation[1].find_all('tr')[3].find_all('td')[1].text.strip()})
+    return res
+
+if __name__ == '__main__':
+    res = downloader('http://www.zd9999.com/cy/')
+    for i in range(2, 199):
+        res += downloader(f'http://www.zd9999.com/cy/index_{i}.htm')
+    print(len(res))
+    with open('chengyu.json', mode='w+', encoding='utf-8') as json_file:
+        json.dump(res, json_file, ensure_ascii=False)

scripts/ci.py

@@ -0,0 +1,72 @@
+"""
+
+author: pwxcoo
+date: 2018-08-02 
+description: 多线程抓取下载词语并保存
+
+"""
+
+import requests, csv
+from bs4 import BeautifulSoup
+import time
+from multiprocessing.dummy import Pool as ThreadPool
+
+def downloader(url):
+    """
+    下载词语并保存
+    """
+    res = []
+    try:
+        response = requests.get(url)
+        
+        if response.status_code != 200:
+            print(f'{url} is failed!')
+            return len(res)
+
+        print(f'{url} is parsing')
+        html = BeautifulSoup(response.content.decode('gbk', errors='ignore'), "lxml")
+        a = html.find_all('a', target="_blank")
+        
+        prefix = 'http://www.zd9999.com'
+        words = [prefix + w.get('href') for w in a]
+    
+        for i in range(0, len(words)):
+            print(f'{[words[i]]} is parsing')
+
+            try:
+                response  = requests.get(words[i])
+                wordhtml = BeautifulSoup(response.content.decode('gbk', errors='ignore').replace('<br/>', '\n').replace('<br>', '\n')\
+                            , "lxml")
+                td = wordhtml.find_all('table')[5].find_all('td')
+                res.append([td[0].text.strip(), td[1].text.strip()])
+     
+            except Exception as e:
+                with open('../data/error.csv', mode='a+', encoding='utf-8', newline='') as error_file:
+                    csv.writer(error_file).writerows([0, e, words[i]])
+                print(f'{words[i]} is failed! {e}')
+                continue
+
+    except Exception as e:
+        with open('../data/error.csv', mode='a+', encoding='utf-8', newline='') as error_file:
+            csv.writer(error_file).writerows([1, e, url])
+        print(f'{url} is failed! {e}')
+
+    with open('../data/ci.csv', mode='a+', encoding='utf-8', newline='') as csv_file:
+        csv.writer(csv_file).writerows(res)
+    
+    return len(res)
+
+if __name__ == '__main__':
+    start_time = time.time()
+
+    pool = ThreadPool(100)
+    urls = ['http://www.zd9999.com/ci/index.htm']
+    for i in range(2, 1959):
+        urls.append(f'http://www.zd9999.com/ci/index_{i}.htm')
+    responses = pool.map(downloader, urls)
+    pool.close()
+    pool.join()
+     
+    end_time = time.time()
+    
+    print(f'总共耗时 {end_time - start_time}, 抓取了 {sum(responses)} 条数据')

scripts/clean.ipynb

@@ -0,0 +1,114 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Clean Data (Duplicate Removal)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"/home/pwxcoo/projects/chinese-xinhua/archived/idiom.json\") as fp:\n",
+    "    idioms = json.load(fp)\n",
+    "\n",
+    "len(idioms)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "idioms[:5]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "unique_idioms = set(idiom[\"word\"] for idiom in idioms)\n",
+    "unique_idioms"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "len(unique_idioms)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "check = dict()\n",
+    "for idiom in idioms:\n",
+    "    if idiom[\"word\"] in check and str(idiom) > str(check[idiom[\"word\"]]):\n",
+    "#         print(idiom)\n",
+    "#         print(check[idiom[\"word\"]])\n",
+    "        check[idiom[\"word\"]] = idiom\n",
+    "    else:\n",
+    "        check[idiom[\"word\"]] = idiom\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"/home/pwxcoo/projects/chinese-xinhua/data/idiom-clean.json\", \"w+\", encoding=\"utf-8\") as fp:\n",
+    "    json.dump(list(check.values()), fp, ensure_ascii=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

scripts/word.py

@@ -0,0 +1,58 @@
+# -*- coding: utf-8 -*-
+
+"""
+
+author: pwxcoo
+date: 2018-02-05 
+description: 抓取下载汉字并保存
+
+"""
+
+import requests,json
+from bs4 import BeautifulSoup
+
+def downloader(url):
+    """
+    下载汉字并保存
+    """
+    response = requests.get(url)
+
+    if response.status_code != 200:
+        print(f'{url} is failed!')
+        return
+    
+    print(f'{url} is parsing')
+    html = BeautifulSoup(response.content.decode('gbk', errors='ignore'), "lxml")
+    a = html.find_all('a', target="_blank")
+
+    prefix = 'http://www.zd9999.com'
+    words = [prefix + w.get('href') for w in a]
+
+    res = []
+    for i in range(0, len(words)):
+        response = requests.get(words[i])
+        print(f'{[words[i]]} is parsing')
+        if response.status_code != 200:
+            print(f'{words[i]} is failed!')
+            continue
+
+        wordhtml = BeautifulSoup(response.content.decode('gbk', errors='ignore').replace('<br/>', '\n').replace('<br>', '\n')\
+                     , "lxml")
+        td = wordhtml.find_all('table')[4].find_all('td')
+        explanation = td[12].text.strip()
+        res.append({'word': td[1].text.strip(),\
+                    'oldword': td[4].text.strip(),\
+                    'strokes': td[6].text.strip(),\
+                    'pinyin': td[8].text.strip(),\
+                    'radicals': td[10].text.strip(),\
+                    'explanation': explanation[explanation.find('\r\n'):].strip(),\
+                    'more': td[14].text.strip()})
+    return res
+
+if __name__ == '__main__':
+    res = downloader('http://www.zd9999.com/zi/index.htm')
+    for i in range(2, 102):
+        res += downloader(f'http://www.zd9999.com/zi/index_{i}.htm')
+    print(len(res))
+    with open('word.json', mode='w+', encoding='utf-8') as json_file:
+        json.dump(res, json_file, ensure_ascii=False)

scripts/xiehouyu.py

@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+
+"""
+
+author: pwxcoo
+date: 2018-02-04 
+description: 抓取下载歇后语并保存
+
+"""
+
+import requests, json
+from bs4 import BeautifulSoup
+
+destination_site = ['http://xhy.5156edu.com/html2/xhy.html', 'http://xhy.5156edu.com/html2/xhy_2.html']
+
+
+def downloader(url):
+    """
+    下载歇后语并保存
+    """
+    response = requests.get(url)
+    if response.status_code != 200:
+        print(f'{url} is failed!')
+        return
+    print(f'{url} is parsing')
+    html = BeautifulSoup(response.content.decode('gbk'), "lxml")
+    tr = html.find('table', style='word-break:break-all').find_all('tr')
+    return [{'riddle':t.find_all('td')[0].text, 'answer':t.find_all('td')[1].text} for t in tr][1:]
+
+if __name__ == '__main__':
+    res = downloader('http://xhy.5156edu.com/html2/xhy.html')
+    for i in range(2, 282):
+        res += downloader(f'http://xhy.5156edu.com/html2/xhy_{i}.html')
+    print(len(res))
+    with open('xiehouyu.json', mode='w+', encoding='utf-8') as json_file:
+        json.dump(res, json_file, ensure_ascii=False)