Preparing Your Data

Preparing Your Data

This page will help you determine what text is valid and what could potentially skew your results. Our technology performs best when samples come from written or spoken language, including conversational language, formal or informal language from a variety of sources including blog posts, survey responses, social media posts, transcribed calls, short text samples, or text messages.

Raw text works best, meaning that it's unnecessary to tokenize, lemmatize, stem, remove stop words, or remove punctuation.

During data preparation, raw text may need to be aggregated or parsed depending on the goal of your analysis. Raw text should be prepared in a way that corresponds with the level of insight you aim to produce. For example, parse raw text into discrete sentences before calling the API if you aim to produce sentence-level insights; aggregate raw text into paragraphs before calling the API if you aim to produce paragraph-level insights. The API will analyze input text submissions as a whole unit.

Refer to the table below for details surrounding what to include and what to exclude from your text before using the Receptiviti API.

Understanding Word Count Discrepancies

You might notice that the API response item under summary/word_count displays a word count that differs from the count you see in the text editor document that contains your samples. This is because the Receptiviti API separates words on hyphens and apostrophes, so for example, he's our go-to guy is six 'words' counted as he s our go to guy.

From a counting perspective, the Receptiviti API regards words primarily as tokens, and in some cases emphasizes their individual units rather than recognizing them as complete, contextual words. This means that it dissects the text into discrete components, such as it's becoming it and s and subsequently assesses and processes these fragments separately.

Optimizing Outcomes Using Aggregated Text Methods

Depending on the level of insight you aim to produce, it can be useful to take different approaches to structuring your text samples for analysis. For example, sentence-level insights work best when you want to identify emotions related to key topics of interest. Or, you can split conversational language into chronological segments to evaluate linguistic progression over time. Below are several examples of different useful text aggregation methods:

Analysis of the Individual

• Use case: Understanding the character traits of a speaker.
• Method: Aggregate all language data from a single speaker.

Analysis of Answers to Key Questions

• Use case: Exploring and comparing trends in responses across individual questions.
• Method: Aggregate language data based on individual question (combine the responses that were given in answer to each question).

Analysis of Answers by Question Theme

• Use case: Exploring and comparing trends in responses across multiple related questions.
• Method: Aggregate language data based on question theme (combine the responses that were given in answer to each set of related questions).

Analysis of Conversation Over Time (longitudinal)

• Use case: Evaluating trends in conversation progression or long-term progression.
• Method: Split the conversation’s language into chronological segments; segments can be split based on word count, time, a conversation outline, days, weeks, etc.

Analysis of Topics by Emotion

• Use case: Identifying emotions related to key topics of interest.
• Method: Parse language data into sentences and score using pre-built or custom taxonomies and our emotions framework.

Analysis of Rapport (Language Style Matching)

• Use case: Understanding the extent to which two or more speakers are attentively engaged.
• Method: Analyze and compare each speaker pair’s language.

Code Snippets to Remove Unwanted Elements

@Mentions

# python
import re
def remove_at_mentions(line):
    return re.sub("@\w+", "", line)
def scrub_text(text):
    scrubbed_text = remove_at_mentions(text).strip()
    return scrubbed_text

scrubbed_text = scrub_text("@dogwalker I hope you had a nice long walk!")

URLs

# python
# Regex from https://gist.github.com/dperini/729294
import re
def remove_urls(line):
  URL_REGEX = "(?:(?:(?:https?|ftp):)?\/\/)(?:\S+(?::\S*)?@)?(?:(?!(?:10|127)(?:\.\d{1,3}){3})(?!(?:169\.254|192\.168)(?:\.\d{1,3}){2})(?!172\.(?:1[6-9]|2\d|3[0-1])(?:\.\d{1,3}){2})(?:[1-9]\d?|1\d\d|2[01]\d|22[0-3])(?:\.(?:1?\d{1,2}|2[0-4]\d|25[0-5])){2}(?:\.(?:[1-9]\d?|1\d\d|2[0-4]\d|25[0-4]))|(?:(?:[a-z0-9\u00a1-\uffff][a-z0-9\u00a1-\uffff_-]{0,62})?[a-z0-9\u00a1-\uffff]\.)+(?:[a-z\u00a1-\uffff]{2,}\.?))(?::\d{2,5})?(?:[/?#]\S*)?"
  return re.sub(URL_REGEX, "", line)

remove_urls("have you visited https://www.receptiviti.com/company to find out more")

Email headers and Email Metadata

# python
from email import message_from_string
def extract_body_from_email(text):
    msg = message_from_string(text)
    if msg.is_multipart():
        for part in msg.walk():
            content_type = part.get_content_type()
            content_disposition = str(part.get('Content-Disposition'))
            found_body = False
            if content_type == 'text/plain' and 'attachment' not in content_disposition and not found_body:
                body = part.get_payload(decode=True)
    else:
        body = msg.get_payload(decode=True)
    return body

    text =  """Delivered-To: [email protected]
    Received: by 2002:a05:6000:1188:0:0:0:0 with SMTP id g8csp2511771wrx;
            Mon, 14 Feb 2021 05:11:19 -0700 (PDT)
    MIME-Version: 1.0
    In-Reply-To: <[email protected]>
    From: Test User <[email protected]>
    Date: on, 14 Feb 2021 08:11:06 -0400
    Subject: Fwd: Spam email from you
    To: All full-time employees <[email protected]>
    Content-Type: multipart/related; boundary="000000000000754a2e05af44eeac"
    --000000000000754a2e05af44eeac
    Content-Type: multipart/alternative; boundary="000000000000754a2c05af44eeab"

    --000000000000754a2c05af44eeab
    Content-Type: text/plain; charset="UTF-8"

    As part of our ongoing security awareness, if you see emails like this,
    please mark them as phishing.
    Note the warning signs - the actual email address doesn't match sender, the
    signature isn't right, and there's a giant red warning banner :)

    [image: Screen Shot 2020-09-14 at 9.24.41 am.png]

    Marking something as phishing is slightly different than using the spam
    button in gmail.
    The email goes through different processes / to a different team and helps
    Google prevent these from landing in our inboxes.

    Thanks all!
    *Test User* Desig, Nation | Head, Software @ Receptiviti |
    [email protected]

    This message is intended only for the use of the intended recipients, and
    it may be privileged and confidential. If you are not the intended
    recipient, you are hereby notified that any review, re-transmission,
    conversion to hard copy, copying, circulation or other use of this message
    is strictly prohibited and may be illegal. If you are not the intended
    recipient, please notify me immediately by return email and delete this
    message from your system. Thank you.

    --000000000000754a2c05af44eeab
    Content-Type: text/html; charset="UTF-8"
    Content-Transfer-Encoding: quoted-printable

    <div dir=3D"ltr"><div>As part of our ongoing security awareness, if you see
     emails like this, please mark them as phishing. <br></div><div>Note the wa
    rning signs - the actual email address doesn&#39;t match sender, the signat
    ure isn&#39;t right, and there&#39;s a giant red warning banner :)</div><div>
    <br></div><div><img alt=3D"Screen Shot 2020-09-14 at 9.24.41 am.png" src
    =3D"cid:1748c828c6b7ef93e481" width=3D"542" height=3D"309"></div><div><font face=3D"
    Times"><span style=3D"font-size:12px">This message is intended only for the use of the
    intended recipients, and it may be privileged and confidential. If you are not the intended
    recipient, you are hereby notified that any review, re-transmission,
    conversion to hard copy, copying, circulation or other use of this message
    is strictly prohibited and may be illegal. If you are not the intended
    recipient, please notify me immediately by return email and delete this
    message from your system. Thank you.</span></font></div></div>

    --000000000000754a2c05af44eeab--
    --000000000000754a2e05af44eeac
    Content-Type: image/png; name="Screen Shot 2020-09-14 at 9.24.41 am.png"
    Content-Disposition: inline; filename="Screen Shot 2020-09-14 at 9.24.41 am.png"
    Content-Transfer-Encoding: base64
    Content-ID: <1748c828c6b7ef93e481>
    X-Attachment-Id: 1748c828c6b7ef93e481

    --000000000000754a2e05af44eeac--"""

extract_body_from_email(text)

HTML

# python
# pip3 install bs4
from bs4 import BeautifulSoup
def strip_html_basic(message_string, parser="lxml-xml"):
    soup = BeautifulSoup(message_string, parser)
    for tag in soup("style"):
        tag.decompose()
    plain = soup.get_text("\n", strip=True)
    return plain

html_doc = """
<html><head><title>The Receptiviti Story</title></head>
<body>
<p class="story">Every word counts...
<a href="http://receptiviti.com/i" class="link">I</a>,
<a href="http://receptiviti.com/fountain" class="link">Fountain</a> and
<a href="http://receptiviti.com/puppy" class="link">Puppy</a>;
When language first emerged, they were not made equally.</p>
<p class="story">...</p>
"""
strip_html_basic(html_doc)