2015年8月27日 星期四

Maltelligence

Why Maltelligence is created?
After hiding myself in a corner at my home every weekend for almost half year, I finally completed the new tool – Maltelligence (which is named MalShell in the past). It shall be made available at Github by end of this week.

As described in my last blog of seven months ago, I want to develop a tool for threat/malware analysts to collect OSINT of APT or watering hole attacks’ artifacts from various sources. It extends the functionality of previous MalProfile Tool, which keeps only history (based of Passive DNS) of selected DNS-IP pair and whois information of c2 used in malware samples. The original idea is to develop an OS like framework so that Maltelligence users can write plugins, for craving out relevant data from Cuckoo sandbox, analyzed results from Viper and network infrastructure artifacts from MalProfile ported to Maltelligence. The information is stored inside the Maltelligence database for further analysis and for using in attribution.

As I am not a ‘real coder’, I invited my research buddies: Leng, Michael, Frank & Dan to work on this project with me.  They are now working as per planned and have achieved various milestones this month. I am looking forward to see their final product. 

Although I am only responsible to write the network infrastructure plugin, I found why not I first release this part separately so that users can start collecting the network infrastructure artifacts as soon as possible?  Therefore, I trimmed down my part and release a self-workable Maltelligence as pre-release beta.   I hope the community can provide further feedbacks and comments, which will definitely enrich the development of the future Maltelligence.


Design concepts
At first, I want to build a Malware/Threat Analyst Desktop that allows analyst interactively analysis and investigate various malicious “artifacts”, including sample files, url, dns(domain), IP address, whois data, email, pcap, memory images, etc. All results and collected artifacts will be properly kept in a highly organised database or on file system folders for further analysis, profiling and attribution.

Maltelligence also allows users to write their own plugin to analysis each kind of objects by making queries from various OSINT (or paid data sources) and has capability to monitor “temporal” or “contextual” changes on selected malicious artifact. The information can be presented in a tabular table or even nicely formatted 2D graphs (like: Maltego and Elasticsearch/Kibana).

Maltelligence should provide easy online helps and fairly controlled command line environment. Analyst can selectively execute commands interactively in response to the queried results. All results will display all output to the stdout first, but users can selectively save all or partial results into the database. All typed commands can be saved into a command log, which can be run like a tailed-made shell script for repeated operations with some key parameters provided by the users.

I want to implement some basic analytic functions into Maltelligence in a way that normal IT Sec professionals can get the best and easy results by accessing the collected data.  Finally, it will be great if the queried results can be visualised on either 2D or 3D graphs or diagrams, interactively and replay the states along the time span. 


What is Maltelligence?
Maltelligence is a tool developed by Maltelligence Research Group to automatically collect malicious network infrastructure information and malware samples RECURSIVELY from various open source intelligence (OSINT) sources including VirusTotal, whois, passive DNS, IP subnets, AS number and Geolocation information.

You may run Maltelligence PERIODICALLY to capture and profile the behaviour of malicious group of domain, IP, whois and html content along different stage/time of APT attacks. Maltelligence is a project from the insight of MalProfile


What artifacts collected?
If a domain or an IP address is submitted, Maltelligence will update the domain and its second level domain to its domain table, then it will grasp the latest key data from the respective whois server.  If IP address is submitted, Maltelligence will update the Geo-location, the Country of residing, the AS number, AS registrant, Subnet information if the ASN parameter is turned on in MalProfile.ini.  Please note that I used paid service from Maxmind’s geoip database and tcpiputils.com to data mine this set of information. 

Using the web module, Maltelligence collects JS script, iframe, images and links of the first web page of a domain.  Downloaded JS scripts will be stored at the repository directory under sub-directory name of respective domain by date.  When making a recursive Passive DNS query to VirusTotal, Maltelligence will store everything, including associated domains and IP addresses provided by VirusTotal. 

If download hash is available, using your VirusTotal key, the samples will be automatically downloaded and store at the repository directory.  If any sample is downloaded, Maltelligence will also check if this sample was once uploaded to VirusTotal. If record is found, Maltelligence will collect the analysis details from VirusTotal and saved inside the av_classification table. 

Other than using Maltelligence’s report.py module, users can structure their own queries to dump the collected information from Maltelligence database.  I would like to see contributors upload their queries scripts in near future.


How to use Maltelligence?
Maltelligence consists of three parts: (a) the engine to collect of the OSINT artifacts, (b) the analytic logic to display the collected data and (c) the presentation of output by visualizing the information in 2D or 3D graphs or diagrams.

In this trimmed version, I included only the domain, ip & whois part.  First, you need to submit identified malicious data into Maltelligence.  By using update [-u] or [-c] option, you provide a single record to Maltelligence, it will try to find the relevant data from the Internet. 

You can make a submission in batch mode [-b] by creating a text file and put all related network infrastructure (domain and/or IP address) plus sample hash (md5, sha1 or sha256) in the same line, then submit the  [--path] of the file with a [--tag] to Maltelligence.

Maltelligence will automatically to grasp the whois, geoip, subnet and AS number related information from the Internet and save the information to Maltelligence database. If hash is available, it will go the VirusTotal to grasp the sample and put it under a separate folder, determined by the tag, for further analysis.  To use Maltelligence, you have apply a API key from VirusTotal. 

By setting up appropriate cron jobs plus using the [-m] option, Maltelligence can be used as an automated engine to collect malware associated network infrastructure data running in the background. I use it to collect recessive Passive DNS from identified malware (as specified under the c2 table).

For example, I have set up a cron job to monitoring the IP Address and whois changes of selected domains and set up another cron job to monitor the updates of parked domains of selected subnets under a few AS number.

The way to learn how to use Maltelligence, is download it from Github and run the batch [-b] mode  using with some sample files provided under ./reamde directory. Then check the output by calling the pre-defined queries contained in the report.py module.


Where is the Maltelligence user manual?
Maltelligence is created as my part time project without funding.  I made it open source and assume users will provide feedbacks to us so that a more sophistical version can be released. Only readme and FAQ are the only user documentation.  I am going to write some blogs to illustrate how I make use of Maltellignece to analyze malicious network infrastructure. You can find some introduction video on youtubeBy modifying the reporting module, structured queries can be presented like the attached.  Enjoy it …!



Visualisation of future Maltelligence