• Data science is only useful when the data are used to
  answer a question / tell a story.
• 多型態資料（esp. 數值資料與文本資料）互動是當紅的關注焦點。
  • 如何 (從 ptt) 的語言與網路分析/預測鄉民對於某特定政策的 想法/立場/情緒 ?
  • 如何從使用者的觀影/購物/閱讀/偏好等行為事件進行 推薦/行銷 ?

• DASH: A new gradute certificate: data management, statistics, text analysis, geospatial analysis, digital prosopography, and data visualization and information design.

• Digital Humanities Projects at Stanford

Natural language is primarily hard because it is messy. There are few rules. And yet we can easily understand each other most of the time.

We are awash with text, from books, papers, blogs, tweets, news, and increasingly text fromspoken utterances.

• 多元
• 異質
• 個人與群體差異
• 時空變異

Human language is highly ambiguous … It is also ever changing and evolving. Peopleare great at producing language and understanding language, and are capable of expressing, perceiving, and interpreting very elaborate and nuanced meanings. Atthe same time, while we humans are great users of language, we are also very poorat formally understanding and describing the rules that govern language. (Neural Network Methods in Natural Language Processing, 2017. http://amzn.to/2u0JtPl)

• sound segments
• character and word
• n-gram (n-word sequences) > lexical bundles / chunks
• sentence, paragraph
• theme
• natural units (a speech, poem, lyrics, manifesto, etc)

• 字與詞
• 為何需要斷詞
• 斷詞系統

library(jiebaR)
# 用 worker() 初始化斷詞引擎
seg <- worker()
# 各種參數設定情參考 ?worker
# 斷詞簡單例子
seg["據台大語言所小編謝舒凱表示，宅宅也是非常用功 der"]

##  [1] "據"     "台大"   "語言所" "小編"   "謝舒凱" "表示"   "宅宅"  
##  [8] "也"     "是"     "非常"   "用功"   "der"

seg["你這種人還是死死去最好了"]

## [1] "你"   "這種" "人"   "還是" "死"   "死去" "最好" "了"

• 德文複合詞: Rindfleischetikettierungsberwachungsaufgabenbertragungsgesetz (“the law concerning the delegation of duties for the supervision of cattle marking and the labelling of beef”)

• 土耳其文: OSMANLILŞTIRAMAYABIĪLECEKLERIĪMIĪZDENMIĪŞSIĪNIĪZCESIĪNE (“as if you were of those whom we might consider not converting into an Ottoman”)

#txt_jp <- "政治とは社会に対して全体的な影響を及ぼし、
#社会で生きるひとりひとりの人の人生にも様々な影響を及ぼす複雑な領域である。"
#tokens(txt_jp)
txt_khmer <- "តៃវ៉ាន់បោះជំហានឆ្ពោះទៅរកការធ្វើពាណិជ្ជកម្មនៅអាស៊ីដើម្បីកាត់បន្ថយភាពអាស្រ័យលើប្រទេសចិន
"
#Taiwan Steps up Asia Business to Reduce Dependence on China
#taivean baohchomhan chhpaohtow rokkarothveu peanechchokamm now asai daembi 
#katbanthoy pheap asry leu bratesa chen
tokens(txt_khmer)

## tokens from 1 document.
## text1 :
##  [1] "តៃវ៉ាន់"    "បោះជំហាន"  "ឆ្ពោះទៅ"  "រកការធ្វើ" "ពាណិជ្ជកម្ម" "នៅ"      "អាស៊ី"     
##  [8] "ដើម្បី"    "កាត់បន្ថយ"  "ភាព"      "អាស្រ័យ"    "លើ"      "ប្រទេស"   "ចិន"

• 語彙的多義行為 Polysemy: A word can have multiple senses in varied contexts
• 語彙使用的歧異現象 Ambiguity: lexical, structural, scope (每個人都喜歡他的孩子), anaphora/deictic expressions (他上禮拜開始就應該待在這裡)

• A fine-grained sense inventory and lexical relations database

Penn treebank pos tagset

(Mandarin Chinese Words and Parts of Speech: A Corpus-based Study. 2017. Routledge)

• 文本語流 sense and sentiment flow
• conversational maxim

• Emotion: A relatively brief episode of coordinated brain, automatic and behavioural changes that facilitate a response to an external or internal event of significance for the organism.
• Mood: More long-lasting affective state (days to months); more difuse
• Feeling : Subjective representation of emotion, private to individual experiencing them

• Attitude: relatively enduring, affectively coloured beliefs, preferences and predispositions toward objects or persons.
• Temperament: affective styles that are apparent early in life, and
• Personality trait : stable individual difference across different situations.
• Cognition: do not have facial expressions, not always physiological changes, though cognitive arousal can get rise to emotions.

• Mining opinions and sentiments from natural language / texts (e.g., product review, social media discussion, survery responses)
• It requires a deep understanding of the explicit/implicit, regular/irregular patterns of language use.
• Employing machine learning techniques / dictionary look-up methods to detect the
  sentiment polarity, emotion types and valence shifting.

• Text mining/analytics
• Literature sentiment
• Political speech
• Historical text

• Multimodality affect state (paralinguistics)
• Lexical items vs lexical chunks/bundles
• Sense-sentiment concurrent processing
• Tracing varieties and changes

• 情緒表達語言的單位是什麼
• 情緒表達和語意表達是否皆具備組合性

• From Cues to Rules (lee, 2010)
• Local Grammar of Affect (Bednarek. 2008)
• Use pattern grammar (Hunston, 1999) in approaching evaluative (semantic) prosody (Lee and Hsieh, 2016).

- 不同視野下的詞彙行為/知識表徵化

Operationalized lexical knowledge representation

E.g., 我們不(只)關心「打」的詞義有哪些，更關心

• 在什麼標準|脈落下「打」的詞義分成幾個；

• 這樣的標記訊息(Categorical and/or Numerical) 跟該單位在其他脈絡下被觀察到的行為 (習得，情緒，發展，語言教學，神經表徵，心理反應等) 之間的關聯為何？

• 以這樣的第二層知識出發提供「語言學養分」給 NLP/Machine Learning (rather than shallow linguistic feature engineering)

Reused, Reproduced, Reshaped and Reinforced.

• It takes the functional position (usage-based view) in determining units and patterns (in Chinese), as well as the ontological grounding on the relation between linguistic objects and situations (bits of reality). (Langacker 1987, 1988, 1999; Croft 2002; Tomasello 2003; Bybee 2006, 2010)

• Lexical data at different levels are modularized (only for practical reasons), such as syntax-semantics module, emotion module, discourse and pragmatic module, diachronic module, etc. Researchers from different fields can initiate a new cooperation based upon.

Concurrent processing of Word Sense and Sentiment

• 他曾打了勾勾說要娶她當新娘，她可沒忘喔！
• 明天班上有烤肉，不知道要不要去，因為我車票打好了，不太想去改。
• 少爺打了一個寒顫，冷靜的想了很久。

source

• (WHAT): Text Analytics is the process of converting unstructured text data into meaningful data for analysis
• (WHY) : to measure customer opinions, product reviews, feedback, to provide search facility, sentimental analysis and entity modeling to support fact based decision making, etc.
• (HOW): combining many linguistic, statistical, and machine learning techniques.

It involves lexical analysis, categorization, clustering, pattern recognition, tagging, annotation, information extraction, link and association analysis, visualization, and predictive analytics. Text Analytics determines key words, topics, category, semantics, tags from the millions of text data available in an organization in different files and formats. The term Text Analytics is roughly synonymous with text mining.

• Preparing / Preprocessing text and data.
  • Text is unstructured or partially structured data that must be prepared for analysis. We extract features from text. We define measures.
  • Quantitative data are messy or missing, too They may require transformation prior to analysis. Data preparation consumes much of a data scientist’s time.

• Exploratory data analysis and Infographics (data visualization for the purpose of discovery. We look for groups in data, find outliers, identify common dimensions, patterns, and trends.)

• Prediction models (Regression; Classification and Clustering;) and Evaluations (Recommender systems, collaborative filtering, association rules, optimization methods based on linguistic heuristics, as well as a myriad of methods for regression, classification, and clustering fall under the rubric of machine learning).

• 前處理 Pre-processing (編碼處理，中文斷詞)
• 字串處理 String manipulation, esp. (linguistic) pattern extraction
• 語料庫 corpus 與詞庫 lexicon

• 蒐集與前處理 collecting raw text and preprocessing,
• 表徵 representing text,
  • using Term Frequency-Inverse Document Frequency (TFIDF) to compute the usefulness of each word in the texts,
  • word vectors
• 分類、分群、主題與關鍵詞偵測 categorizing documents by topics using topic modeling,
• 語意與情緒（情感）分析 semantic/sentiment analysis, and
• 應用導向的洞見與預測 gaining greater insights.

如何能？！語言學、語言資源 與 自然語言處理 ！！

• 語料庫 corpus
• 詞庫 lexicon / lexical (knowledge) resources
• 知識本體 ontologies

• From text to texts ？
• 語料庫 (Corpus) 是自然語言處理與文本解析的基礎建設。 a large collection of texts used for various purposes in Natural Language Processing (NLP).
• 標記 (annotation) 是核心。It’s linguistic in nature.

Good annotations support good applications

(Benoit, 2016)

• (corpus) sampling strategies
• Descriptive, distribution, correlation, regression, information theory, ….

• bag-of-words
• document-feature matrix
• ngram, lexical chunks/bundles
• word vectors

• [word-level] tokenization (and word segmentation), stemmer, POS tagger >
• [phrase-level] chunker, NER (Named Entity Recognizer) >
• [sentence-level] syntactic parser >
• [discourse-level] dialogue act

• 自然語言處理和機器學習是好盆友
• training and test data
• accuracy, precision, recall.

• tmcn.word2vec
• text2vec

(Hamilton et al, 2016)

• Pathetic became more negative. It used to be similar in meaning to passionate but gained connotations of “weakness” over time.

• somehow changed the game
• using R
  • wordVectors
  • R interface to Keras
    
  • R wrapper for fastText

inter-coder agreement, reliability 信度, validation 效度,

Pros and Cons

優點

• flexible; theoretically-motivated annotation/code frame effrots
• can apply to texts, speech, video, etc.
• 可以用來解決一般機器學習系統 high precision low recall 的問題。把潛在的語意與情緒發掘出來。

缺點

• manually intensive
• thus can be expensive

• 最簡單可以用 Excel 來做：
  • One (or more) column(s) for text data； One column for topic label (as gold standard)
  • 通常至少有多於 3000 份標好的文件。
• 大型的專案要考慮到永續、相容、交換等問題，建議使用標記系統。
  • 語料庫和語言處理社群 GATE
    
  • 質性研究社群 CAT (Coding Analysis Toolkit)
  • lopetator
• labeling 和 annotation 的差異。

• Text analytics / mining
  • (Semantic and/with) Sentiment analysis
• Text learning (classification, regression, clustering,..)
• Text summarization and generation
• Text and multimodality

• 文本數據取得與前處理 data collection and preprocessing
• 文本探索分析 exploratory textual data analysis
• 文本語意表徵 converting text into a quantitative matrix;
• 文本分析機器學習 modeling the data: using quantitative or statistical methods to analyse this matrix in order to generate inferences about the texts or their authors.
• 文本分析結果表達、呈現與敘事

數據取得與前處理 Data collection and preprocessing

• 決定語料文本的來源與樣本
• 決定要聚焦/移除的單位 identifying texts and units of texts for analysis;
• 定義要抽取/標記的特徵
• 自然語言處理協助 (automatic wordhood/pos/NER/sense tagging, dependency/lexical relation parsing,…)

文本探索分析 Exploratory textual data analysis

• 抽取單位在文本中的量化特徵及其分佈 extracting from the texts quantitatively measured features—such as coded content categories, word counts, word types, dictionary counts, or parts of speech
• 形成假說
• 視覺化

文本語意表徵 text semantic representation

文本分析機器學習 text analytics and machine learning

文本分析結果表達、呈現與敘事 Infovis, presentation and story-telling

• 步驟之間是 back and forth

• tm, tmcn
• quanteda, preText, text2vec
• syuzhet, sentimentr, tidytext,

• 川普的用字是否“低端”?

• hybrid mash-ups that pull from data/models
• 整合: 說的比較容易：需要從教育與學習破壞開始
• what are the celebrating skills does AHSSer develop?