恶意网址实时检测系统 Malicious URL Detection System

Please wait for a moment...

The system is collecting features and predicting in real time.
This may take seconds to look up feature information such as WHOIS, Server and pages number indexed by search engines.

反钓鱼：恶意网址实时检测系统
Anti-Phishing: A System for Malicious URLs Detection

URL:

Classifier:

*请输入一个完整的网址，包括协议，主机名。如：http://www.news.uestc.edu.cn/?n=UestcNews.Front.Document.ArticlePage&Id=61086
*Please enter a complete URL, including protocol, server name, such as: http://www.bbc.com/sport/horse-racing/41884841

Report on Detection

本系统用于恶意网址检测。系统仍在持续地收集网址以及相关的特征信息。系统已内置了多个分类器，并进行实时训练。请在上框中输入一个完整的网址，点击“Check”后，检测报告将在此处显示。
The system is used for malicious URLs detection with classifiers listed above. The classifiers have been trained appropriately with collected URLs as well as prepared features by the system. Please input a URL in the box above and click 'Check'. Then the prediction result and related information will be shown here.

This project now mainly uses four types of features:

(1) Domain-based features. Such as WHOIS information, domain and its sections, days after the domain was registered and remaining days before the domain expires.

(2) Host-based features. Such as where the server is located, sponsor of the host, and technologies used by the site.

(3) Reputation-based features. Such as Alexa rank, Alexa linkin, pages number indexed by Baidu, and so on.

(4) Lexical Features. Such as the number of dots in URL, number of parameters, and sensitive keywords contained in the URL.

All 62 features used in this system are list below:

TABLE 1

The description of all 62 features used in this system

Type	feature	description
Domain	domain_length	The length of the domain.
	server_name_length	The length of the "server name" section.
	pure_domain_length	The length of the pure domain.
	if_com	If the top level domain name is “.com”.
	if_org_gov	If the top level domain name is “.gov” or “.org”.
	expire_days	Remaining days before the domain expires.
	register_days	Days after the domain is registered.
Reputation based	baidu_index	The number of pages indexed by Baidu.
	baidu_inverse_index	The number of the pages which link to the objective URL and indexed by Baidu.
	beian	If it is registered by Chinese "BEIAN System".
	google_pr	Google page rank.
	google_index	The number of pages indexed by Google.
	alexa_rank	The rank of the website in Alexa.
Host based	if_godaddy	If the website is hosted by "Godaddy.com".
	if_usa	If the hosting server is located in USA.
	if_cn	If the hosting server is located in China.
	if_latin_american	If the hosting server is located in latin America.
	if_europe	If the hosting server is located in Europe.
	if_japan	If the hosting server is located in Japan.
	if_aus	If the hosting server is located in Austrilia.
	if_north_ame	If the hosting server is located in North America (except USA).
	if_india	If the hosting server is located in India.
	if_africa	If the hosting server is located in Africa.
	if_middle_east	If the hosting server is located in Middle East.
	if_se_asia	If the hosting server is located in South East Asia.
	if_korea	If the hosting server is located in Korea.
	if_jsp	If the website uses "JSP" technology.
	if_cgi	If the website uses "CGI" technology.
	if_html	If the website uses static HTML technology.
	if_asp	If the website uses "ASP" technology.
	if_php	If the website uses "PHP" technology.
	if_http	If the web protocol is "HTTP".
	if_https	If the web protocol is "HTTPS".
	if_ftp	If the web protocol is "FTP".
Lexical	dot_num	The number of the dots in the URL.
	param_num	The number of the parameters in the URL.
	if_paypal_u	If "paypal" is contained in the PATH section.
	if_taobao_u	If "taobao" is contained in the PATH section.
	if_ali_u	If "ali" is contained in the PATH section.
	if_jd_u	If "jd" is contained in the PATH section.
	if_safety_u	If "safety" is contained in the PATH section.
	if_verify_u	If "verify" is contained in the PATH section.
	if_google_u	If "google" is contained in the PATH section.
	if_apple_u	If "apple" is contained in the PATH section.
	if_facebook_u	If "facebook" is contained in the PATH section.
	if_amazon_u	If "amazon" is contained in the PATH section.
	if_porn_u	If "porn" related words are contained in the PATH section.
	if_gamble_u	If "gamble" related words are contained in the PATH section.
	if_awarding_u	If "awarding" related words are contained in the PATH section.
	if_paypal_d	If "paypal" is contained in the DOMAIN section.
	if_taobao_d	If "taobao" is contained in the DOMAIN section.
	if_ali_d	If "ali" is contained in the DOMAIN section.
	if_jd_d	If "jd" is contained in the DOMAIN section.
	if_safety_d	If "safety" is contained in the DOMAIN section.
	if_verify_d	If "verify" is contained in the DOMAIN section.
	if_google_d	If "google" is contained in the DOMAIN section.
	if_apple_d	If "apple" is contained in the DOMAIN section.
	if_facebook_d	If "facebook" is contained in the DOMAIN section.
	if_amazon_d	If "amazon" is contained in the DOMAIN section.
	if_porn_d	If "porn" related words are contained in the DOMAIN section.
	if_gamble_d	If "gamble" related words are contained in the DOMAIN section.
	if_awarding_d	If "awarding" related words are contained in the DOMAIN section.

All features are numeric.

The dataset used by this system was mainly from four sources:

Malicious URLs were collected from “PhishTank” and “360 Safety Center”. PhishTank is a collaborative clearing house for data and information about phishing on the Internet [38]. It also provides an open API for developers and researchers to integrate anti-phishing data into applications. “360.com” is a famous Chinese network security company [21]. They periodically publish Chinese malicious websites reports, including fishing sites, fraud sites, hacking sites, and so on.

Benign URLs were collected from “Hao123” and “DMOZ”. Hao123 is a Chinese yellow page website. All sites indexed by Hao123 are reliable because they have been manually verified and registered by Chinese BEIAN system, which is a censorship mechanism developed by the Chinese government. DMOZ is an acronym for Directory Mozilla [40]. It is an open Web directory whose entries are constantly examined by volunteer editors, who go through a vetting process.

The system has collected about 350000 URLs and their features (WHOIS, indexing information by search engines, server information). A sample of about 50000 is available for download:

url_sample_50000.csv

The full dataset is not provided now for the protection of the original research work by our team on this issue. It will be available after the publication of the related work in the future.

The invocation to this system from other machines is quite simple. We encapsulate the malicious detection as a remote service with java Servlet and it can be invoked using URL like this:

http://url.jspfans.com/servlet/Remote?url=http%3a%2f%2fwww.google.com&classifier=LR

The first parameter is 'url'. Please provide the url you want to check. Remember that the URL should be encoded. For example, "http://www.google.com" should be encoded as: "http%3a%2f%2fwww.google.com".

The second parameter is 'classifier'. Please provide the classifier name you want to use. Available values are: 3-NN, L-SVM, LR, and LDA.

The service is temporarily disabled due to the pressure of the system load on cloud host.

The basic view point of this research is that "Feature Engineering" plays an crucial role in malicious URLs detection.

“Feature engineering” is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in improved model accuracy on unseen data”. So basically speaking feature engineering is a process of creating features that makes machine learning algorithms work better.

We reduce the dimension of the dataset to 2 using t-Distributed Stochastic Neighbor Embedding (t-SNE) techniques, and the distribution of the dataset is illustrated as follows:

Fig.1 Visualization of high dimensional URLs dataset by t-SNE

It can be seen from the figure above that data points are not linearly separable. Such distribution challenges many models, such as linear SVM and Logistic Regression. Most classifiers cannot work well on such URLs dataset.

We propose several "feature engineering" methods with regards to space transformation in this research to manipulate the features to make basic classifiers work better as a result that the detection of malicious URLs is improved dramatically. The details of these methods, which consist of Singular Value Decomposition, newly proposed Distance Metrics Learning, Nystrom Method, and integrations of them, will be disclosed in the near future.

The classifiers used in this system are the fastest and widely verified ones such linear Support Vector Machine (L-SVM), Logistic Regression (LR), Linear Discriminant Analysis (LDA) and k-Nearest Neighbor (KNN), as it is shown in page "URL Detection". The performance of them can be promoted by the proposed "feature engineering" methods.

Introduction

Internet is frequently used by criminals for illegal activities, such as financial fraud, global voice phishing, online gambling, fake TV shopping, fraudulent prize winning and spam SMS in social networks. The dark side of Internet has emerged and bedeviled the world.

In recent years, mainland Chinese citizens suffered more than 20 billion Yuan loss per year from global voice phishing, most of which were with the aid of phishing or fake websites located outside China. Besides, widespread usage of smart phones also stimulates the rapid increase of mobile and OR code phishing activities, especially towards old people without much knowledge about phishing. More than 10000 phishing websites were reported to Anti-phishing Alliance of China (APAC) per month on average from Aug 2011 to May 2017.

This project is launched by us to collect malicious URLs dataset, extract features, and provide sustainable malicious detection support for anti-phishing researches and industrial applications. We here call for the global cooperation and effort to fight against the dark side of the Internet, and make the online world a better place for all people.

System Development

The system is developed using Java EE, with many cutting-edge technologies such as distributed cache (Hazelcast), Map-reduce computing (Hazelcast), and NoSQL database (Cassandra). The technical architecture of the system is shown in Fig.2

Fig.2 Technical architecture of the learning system

The system consists of four layers.

(1) Distributed database layer. Cassandra was deployed on 2 machines. Thanks to the Gossip protocol, Cassandra does not need central node.

(2) Distributed cache layer. Almost all of the functional modules involving data reading and writing. In order to reduce the access response time, it is very important to build caches for “hot data”. We built three type of caches for the system: shared cache, replicated cache and local cache. a) Shared cache is used as a distributed big data grid. All of the modules share the same data in the shared cache; b) Replicated cache is built on every server. Each server have the same copy of the cached data. Replicated cache is used to store the most frequently read data but with rare update; c) Local cache is used to store the data which are used only for local frequent computation purpose.

(3) Service layer. Most of the aforementioned core functions were encapsulated in service layer. The distance metric learning, classifier training were implemented according to the models, using JAVA, and developed as standard services. They are generally invoked by crawling service while new instance incomes. URL and feature crawling services involve HTTP client, XML/HTML parsing and web services invocation. Business logic here mainly refers to how to organize the data used in the GUI and handle the events invoked by users.

(4) Access interface layer. The system may be used by human or invoked by other systems. So we provide two type of interface: web pages and web services API. The web part of the system was developed using Java Server Faces. The pages is built via XHTML, namely Facelet, and they exchange data with managed beans. Managed beans are java classes and they retrieve data from service layer to pages or propagate the actions/events from the pages to the service layer. As for web services, they provide common APIs to interact with other systems.

The technical architecture was deployed into a distributed environment as a cluster. The modules connected by dual arrows was deployed in the same Tomcat application. With shared distributed caches and databases, each of the routine connected by the arrows in Fig.2 can be replicated, and then deployed as a member in the cluster. Different Tomcats were integrated by a common Ngnix providing inverse proxy service.

In terms of distributed computing, Hazelcast supports “Map-Reduce” and “ExecutorService”. Nyström method and DMZ benefit from these mechanisms. In model training tasks, such as the search of the closest points in DML, clustering in Nyström method, were encapsulated in independent threads, which were implemented by “ExecutorService”, and then ran in the distributed cluster. In space transformation tasks, the kernel and DMZ matrix information was stored in a shared memory, and then allocated the computation of the transformation of each single instance into the map tasks.

反钓鱼：恶意网址实时检测系统Anti-Phishing: A System for Malicious URLs Detection

反钓鱼：恶意网址实时检测系统
Anti-Phishing: A System for Malicious URLs Detection