Category Archives: Heyley’s blog

Week 30

My task: To find a small dataset that we can use in a short demonstration at Stetson’ Research Showcase.

The goal is to choose articles that can best demonstrate how our system works.

I’m looking for news articles are fake/untrustworthy and that have already been proven to be fake/untrustworthy in hopes that when they are put through our system, we can conclude them as such as well. For example,

1.)

British police find Putin’s passport at scene of Salisbury poison attack – Friday, 16 March 2018

http://www.the-postillon.com/2018/03/salisbury-putin.html

2.)

Photo of Neil Armstrongs’s Spacesuit Doesn’t Match Footprints on the Moon

https://www.elitereaders.com/photo-neil-armstrongs-spacesuit-dont-match-footprints-moon/

3.)

Coca-Cola Recalls Dasani Water After Clear Parasite Worm Was Found In Bottles Across U.S.

http://news4ktla.com/coca-cola-recalls-dasani-water-clear-parasite-worm-found-bottles-across-u-s/

 

Week 29

Here is the final code used to extract all the certain sentences from all of the articles.

It resulted in nearly 96,000 .csv documents of certain sentences ready for testing.

article-parsing.py

# coding: utf-8

import re
import pandas as pd
from nltk.tokenize import sent_tokenize
from sklearn.feature_extraction.text import CountVectorizer
import numpy as np
from sklearn.ensemble import RandomForestClassifier
import os
import csv

name = ['sent','hedge']

#training data for hedge-detection
train = pd.read_csv("../../HedgeDetection/bag-of-words/train/TRAIN_biomedical_fullarticles.csv",
        names=name, delimiter = "\t", quoting=3)
vectorizer = CountVectorizer(analyzer='word',
                             stop_words = "english",
                             ngram_range = (1,2),
                             lowercase = True)
train_data_feature = vectorizer.fit_transform(train['sent'])
forest = RandomForestClassifier(n_estimators = 100)
forest = forest.fit(train_data_feature, train["hedge"])
title = pd.read_csv('title.csv', names = ['article'])

i = 0
while i < title['article'].shape[0]:
    with open('fulltext/'+title['article'][i], 'r')as f:
        if os.stat('fulltext/'+title['article'][i]).st_size != 0:
            text = f.read()
            sentences = re.sub(r'\s+',' ',text)
            article_tokenize_list = sent_tokenize(sentences)
            #print(article_tokenize_list)
            article_data_feature = vectorizer.transform(article_tokenize_list)
            #print(article_data_feature)
            article_data_feature = article_data_feature.toarray()

Week 26

My task: To correct last week’s code.

This week, I decided to try to tackle the main problem of testing all of the articles in our dataset in sections rather than all at once. I began with using last week’s code to correctly write a code that will separate the sentences from one of the articles and test them to decide whether each sentence is certain (hedge = 1) or uncertain (hedge = 0).

Here are the results of that test:

               article

0  ainews:00012214.txt

1  ainews:0004CF01.txt

2  ainews:000580DE.txt

3  ainews:00059AEB.txt

4  ainews:0006365E.txt

ainews:00012214.txt

100003

['Published: 04:06 EST, 3 September 2015 | Updated: 03:49 EST, 4 September 2015 In the home of the future, lights will turn themselves off and cookers and washing machines will switch on automatically.', "In the meantime, Samsung has launched a kit that will let you make your current house 'smart' for just 200.", 'The SmartThings kit will let users control, monitor and secure their home from anywhere in the world, using an app.', 'Smart home in a box: The SmartThings kit will let users control, monitor and secure their home from anywhere in the world, using an app.', 'It comprises a hub and four sensors (pictured) The system comprises a hub and four sensors: a motion sensor, multi-sensor, presence sensor and a smart outlet to connect one device, such as a light, for example.', "'It's for people who aren't sure but want to turn their home into a smart home,' Mark Younger, managing director of SmartThings in the EU, told MailOnline.", 'He said setting up the system is as easy to set-up, taking just 15 minutes.', "'Routines' can perform a series of actions (pictured left).", "For example an 'I'm leaving the house' routine could comprise the app switching all connected lights off, turning down a smart thermostat and checking a curling iron attached to a smart outlet is turned off.", "Sensors can also detect break-ins (pictured right) 'We're trying to break the popular misconception that you need a professional to come into our home to set things up and make it smart.'", 'A SmartThings Hub to connect with smart sensors, lights, locks, cameras and more A Multi Sensor to monitor whether doors, windows, drawers or your garage are open or closed A Presence Sensor to know when people, pets and cars arrive or leave home A Power Outlet to control lights, electronics and small appliances Users plug the hub into a router and then place the four sensors around their house.', "'The hub acts as the heart of the home and connects wireless devices in your home,' Younger explained.", 'Users can then control the four sensors - as well as hundreds of compatible devices from the likes of Honeywell, Bose and Yale - using a free app, which will be available across all mobile platforms from September 10.', 'For example, they may wish to place a motion sensor by their bed, so if they get up in the middle of the night, a connected light could automatically be switched on.', "By placing it by the front door, a motion sensor could also be used as part of a burglary alarm system, which could trigger an alarm or flash lights if movement was detected in the middle of the night, or send a push notification the homeowner's phone.", 'Samsung says the kit takes around 15 minutes to set up.', "'We're trying to break the popular misconception that you need a professional to come into our home to set things up and make it smart,' a spokesperson explained 'Multi-sensors' are designed to be fixed on doors and windows and detect vibrations.", 'So they could tell a user if they have left a window open, if someone is knocking on the front door, or if a medicine cabinet has been opened, for example.', "The presence sensor is designed to show a person or pet's whereabouts and could be affixed to a school backpack, for example.", "If one was attached to a backpack, it could send a push notification to a user's phone when their child arrived home from school.", 'Moisture sensors, which are not included in the starter kit, could be also placed in basements or below sinks for example to detect any leaks of even flooding.', "'The biggest insurance claim is water damage in the UK and [these sensors] could prevent major spend on damage,' Younger said.", 'All the sensors also measure temperature, meaning they could help users heat their home in a more targeted and efficient way, if they have a smart thermostat.', 'Users can then control the four sensors as well as hundreds of compatible devices - from the likes of Honeywell, Bose and Yale - using a free app, which will be available across all mobile platforms from September 10 (examples of the app in action pictured) The kit will include one smart outlet, which can be plugged into an appliance to make it operable remotely or automatically.', 'For example, it could be attached to a coffee maker and programmed to start brewing for a certain time in the morning, or to a light which would automatically switch off at a chosen time.', 'Younger described the hub as the heart of the home, which connects various wireless devices including the four sensors in the kit.', "Once set up, users need never touch it again and it can be plugged in to a house's router and stored out of site, he said.", "The second-generation device includes a back-up battery, so if there's a power cut, it will keep working and if the internet drops out, it will also carry on monitoring the home.", 'The device is designed to be future-proof so users need only buy it once.', "It's compatible with all communication standards and future ones too so it will be able to 'talk' to most smart gadgets.", "Younger said: 'We're making one hub to rule them all.", "We don't want consumers having to worry about how devices communicate, we want them to be able to pick the ones they want.", "Younger said the free app is 'where all the action happens from' and users can control any connected device from it as well as choreograph routines.", "For example an 'I'm leaving the house' routine could comprise the app switching all connected lights off, turning down a smart thermostat and checking a curling iron attached to a smart outlet is turned off for peace of mind, he explained.", "The app has a 'market place' where users can download routines built by the SmartThings community.", "These could be something like a simple routine of rule, which says 'if my door is open between midnight and 6am, flash the lights and sound an alarm'.", 'In time, Samsung will also launch its first service which will allow users to see video footage from around their home and to allow friends of neighbours receive notifications about their home, which could be useful if an owner is on holiday.', 'Younger says the community will build on gradually.', "So, for example, users may one day be able to see a live stream of their pets while they're at work, or use their phone as a baby monitor.", 'The system could also one day incorporate data from wearables such as fitness monitors and smart watches.', 'Samsung says the app will develop further over time.', "So, for example, users may one day be able to see a live stream of their pets while they're at work, or use their phone as a baby monitor.", "The system could also one day incorporate data from wearables such as fitness monitors and smart watches (current app uses pictured) The starter kit will be available from September 10 via Currys PC World and Samsung's website.", 'In the US, it will available on Amazon too.', 'It will cost 199 for the kit, including the hub, three sensors and one smart outlet.', 'The hub will be available to buy separately for 99 and all additional sensors will cost 30 each, with smart outlets at 45 each.', 'It is not yet known when the kit will be rolled out in the US.', "Younger believes the number sensors in the starter kit are sufficient to make a home noticeably smarter and certain actions much more convenient, but said the sky's the limit as to how many can be added to the system.", 'One Samsung employee is rumoured to have between 400 and 500 sensors in his home to make it as automated as possible.', 'The views expressed in the contents above are those of our users and do not necessarily reflect the views of MailOnline.', 'By posting your comment you agree to our house rules.']

  (0, 35)       1

  (0, 44)       1

  (0, 913)      1

  (0, 3176)     1

  (0, 9124)     2

  (0, 10881)    1

  (0, 20147)    1

  (0, 22922)    2

  (0, 25122)    1

  (0, 26250)    1

  (1, 458)      1

  (1, 6475)     1

  (1, 12204)    1

  (1, 13944)    1

  (1, 14471)    1

  (1, 15129)    1

  (2, 6059)     1

  (2, 14471)    1

  (2, 16204)    1

  (2, 26738)    1

  (2, 26748)    1

  (2, 27328)    1

  (3, 3872)     1

  (3, 6059)     1

  (3, 14471)    1

   :     :

  (45, 1336)    1

  (45, 1924)    1

  (45, 3193)    1

  (45, 6286)    1

  (45, 22916)   1

  (46, 14030)   1

  (47, 1880)    1

  (47, 4485)    1

  (47, 14672)   1

  (47, 15129)   1

  (47, 17065)   1

  (47, 17148)   1

  (47, 24896)   1

  (47, 27455)   1

  (48, 829)     1

  (48, 970)     1

  (48, 15129)   1

  (48, 18778)   1

  (49, 9650)    1

  (49, 16715)   1

  (49, 21041)   1

  (49, 26738)   1

  (49, 27089)   2

  (50, 12204)   1

  (50, 22274)   1

[[0 0 0 ... 0 0 0]

 [0 0 0 ... 0 0 0]

 [0 0 0 ... 0 0 0]

 ...

 [0 0 0 ... 0 0 0]

 [0 0 0 ... 0 0 0]

 [0 0 0 ... 0 0 0]]

[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

 1 1 1 1 1 1 1 1 1 1 1 1 1 1]

      hedge                      sent

1       1  In the meantime, Samsung has launched a kit th...

2       1  The SmartThings kit will let users control, mo...

3       1  Smart home in a box: The SmartThings kit will ...

4       1  It comprises a hub and four sensors (pictured)...

5       1  'It's for people who aren't sure but want to t...

6       1  He said setting up the system is as easy to se...

7       1  'Routines' can perform a series of actions (pi...

8       1  For example an 'I'm leaving the house' routine...

9       1  Sensors can also detect break-ins (pictured ri...

10      1  A SmartThings Hub to connect with smart sensor...

11      1  'The hub acts as the heart of the home and con...

12      1  Users can then control the four sensors - as w...

13      1  For example, they may wish to place a motion s...

14      1  By placing it by the front door, a motion sens...

15      1  Samsung says the kit takes around 15 minutes t...

16      1  'We're trying to break the popular misconcepti...

17      1  So they could tell a user if they have left a ...

18      1  The presence sensor is designed to show a pers...

19      1  If one was attached to a backpack, it could se...

20      1  Moisture sensors, which are not included in th...

21      1  'The biggest insurance claim is water damage i...

22      1  All the sensors also measure temperature, mean...

23      1  Users can then control the four sensors as wel...

24      1  For example, it could be attached to a coffee ...

25      1  Younger described the hub as the heart of the ...

26      1  Once set up, users need never touch it again a...

27      1  The second-generation device includes a back-u...

28      1  The device is designed to be future-proof so u...

29      1  It's compatible with all communication standar...

30      1  Younger said: 'We're making one hub to rule th...

31      1  We don't want consumers having to worry about ...

32      1  Younger said the free app is 'where all the ac...

33      1  For example an 'I'm leaving the house' routine...

34      1  The app has a 'market place' where users can d...

35      1  These could be something like a simple routine...

36      1  In time, Samsung will also launch its first se...

37      1  Younger says the community will build on gradu...

38      1  So, for example, users may one day be able to ...

39      1  The system could also one day incorporate data...

40      1  Samsung says the app will develop further over...

41      1  So, for example, users may one day be able to ...

42      1  The system could also one day incorporate data...

43      1        In the US, it will available on Amazon too.

44      1  It will cost 199 for the kit, including the hu...

45      1  The hub will be available to buy separately fo...

46      1  It is not yet known when the kit will be rolle...

47      1  Younger believes the number sensors in the sta...

48      1  One Samsung employee is rumoured to have betwe...

49      1  The views expressed in the contents above are ...

50      1  By posting your comment you agree to our house...

Through testing, it was determined that all of the sentences in the first article in the dataset are certain.