DAPT 2020
Network Data Source NetFlows, DNS
Network Data Labeled Yes, Netflows are labeled
Host Data Source Syslog, auditd, apache, auth, various services
Host Data Labeled No
   
Overall Setting Enterprise IT
OS Types n/a
Number of Machines 4
Total Runtime 5 days
Year of Collection 2020
Attack Categories Reconnaissance
Persistence
Lateral Movement
Exfiltration
Benign Activity Real, benign traffic generated by students (?)
   
Packed Size 460 MB
Unpacked Size n/a
Download Link goto

Overview

The Dataset for Advanced Persistent Threats (DAPT) was created with the goal of providing a cybersecurity dataset representative of an enterprise being the victim of an APT campaign. The authors define such a campaign as a chain of attacks that covers certain categories - in this case Reconnaissance, Foothold Establishment, Lateral Movement, and Data Exfiltration. “Cover Up” is later mentioned as a fifth category, but declared out of scope for this work.

Environment

The environment is simulated leveraging four distinct machines (as VMs):

  • Private VM, hosting services like Samba, WordPress, FTP or MySQL
  • Public VM, hosting (vulnerable) services like DVWA, BadStore and Metasploitable
  • Gateway Router VM, connecting to the outside, entry point for the attacker
  • Log Server, hosting an ELK cluster for log collection and management

Private and public VM as well as the log server belong to the internal network, with access to the outside provided by the gateway router.

Activity

This setup ran for a duration of five days, with the first one containing only benign activities, and each of the remaining four days consisting of both benign and malicious activities. Benign behavior does not seem to be simulated, but rather performed by students: “normal users (students with basic knowledge of website maintenance and access), used shopping interface to check out items, browse different options, create posts on the website, add comments on particular items, etc.”.

Contained Data

Logs are collected centrally by the log server, and include the following sources:

  • pcaps
  • Logs of system events (syslog)
  • MySQL access logs
  • Auditd host IDS logs
  • Apache Access Logs
  • Authentication Logs
  • Logs from services (WordPress, docker, samba, ftp)
  • DNS logs

Pcaps are not provided as-is, but in the form of network flows using CICFlowMeter, and are grouped per day - an explanation for these features used is available on the linked GitLab homepage. The remaining logs are sadly not sorted in an obvious manner - additionally, while the network flows are dated from 15/07 to 19/07 (the mentioned 5 days of simulation), a substantial number of logs are dated well outside this range, which is not mentioned or explained by the authors. Labels are only available for network flows, consisting of “activity” (the specific attack) and “stage” (the current APT stage) associated with that flow.

Papers

Data Examples

Network flows taken from csv/enp0s3-public-wednesday.pcap_Flow.csv

Flow ID,Src IP,Src Port,Dst IP,Dst Port,Protocol,Timestamp,Flow Duration,Total Fwd Packet,Total Bwd packets,Total Length of Fwd Packet,Total Length of Bwd Packet,Fwd Packet Length Max,Fwd Packet Length Min,Fwd Packet Length Mean,Fwd Packet Length Std,Bwd Packet Length Max,Bwd Packet Length Min,Bwd Packet Length Mean,Bwd Packet Length Std,Flow Bytes/s,Flow Packets/s,Flow IAT Mean,Flow IAT Std,Flow IAT Max,Flow IAT Min,Fwd IAT Total,Fwd IAT Mean,Fwd IAT Std,Fwd IAT Max,Fwd IAT Min,Bwd IAT Total,Bwd IAT Mean,Bwd IAT Std,Bwd IAT Max,Bwd IAT Min,Fwd PSH Flags,Bwd PSH Flags,Fwd URG Flags,Bwd URG Flags,Fwd Header Length,Bwd Header Length,Fwd Packets/s,Bwd Packets/s,Packet Length Min,Packet Length Max,Packet Length Mean,Packet Length Std,Packet Length Variance,FIN Flag Count,SYN Flag Count,RST Flag Count,PSH Flag Count,ACK Flag Count,URG Flag Count,CWR Flag Count,ECE Flag Count,Down/Up Ratio,Average Packet Size,Fwd Segment Size Avg,Bwd Segment Size Avg,Fwd Bytes/Bulk Avg,Fwd Packet/Bulk Avg,Fwd Bulk Rate Avg,Bwd Bytes/Bulk Avg,Bwd Packet/Bulk Avg,Bwd Bulk Rate Avg,Subflow Fwd Packets,Subflow Fwd Bytes,Subflow Bwd Packets,Subflow Bwd Bytes,FWD Init Win Bytes,Bwd Init Win Bytes,Fwd Act Data Pkts,Fwd Seg Size Min,Active Mean,Active Std,Active Max,Active Min,Idle Mean,Idle Std,Idle Max,Idle Min,Activity,Stage
192.168.3.29-23.219.38.49-55438-80-6,192.168.3.29,55438,23.219.38.49,80,6,17/07/2019 02:35:09 PM,13621707,2,3,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.367061191376,3405426.75,4812396.59671,10224310.0,15786.0,3381611.0,3381611.0,0.0,3381611.0,3381611.0,10255969.0,5127984.5,7229740.45224,10240183.0,15786.0,0,0,0,0,64,96,0.146824476551,0.220236714826,0.0,0.0,0.0,0.0,0.0,0,0,0,0,1,0,0,0,1.0,0.0,0.0,0.0,0,0,0,0,0,0,2,0,3,0,-1,294,0,0,15786.0,0.0,15786.0,15786.0,10224310.0,0.0,10224310.0,10224310.0,Normal,Benign
192.168.3.29-52.32.232.251-43846-443-6,192.168.3.29,43846,52.32.232.251,443,6,17/07/2019 02:33:27 PM,42843,1,2,0.0,74.0,0.0,0.0,0.0,0.0,39.0,35.0,37.0,2.82842712475,1727.23665476,70.0231076255,21421.5,30236.5930703,42802.0,41.0,0.0,0.0,0.0,0.0,0.0,42802.0,42802.0,0.0,42802.0,42802.0,0,1,0,0,32,64,23.3410358752,46.6820717503,0.0,39.0,28.25,18.927493230699998,358.25,0,0,0,1,1,0,0,0,2.0,37.6666666667,0.0,37.0,0,0,0,0,0,0,1,0,2,74,-1,118,0,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Normal,Benign
192.168.3.29-23.52.177.21-48386-443-6,192.168.3.29,48386,23.52.177.21,443,6,17/07/2019 02:35:41 PM,40215051,34,38,1319.0,154463.0,604.0,0.0,38.7941176471,117.385864466,13032.0,0.0,4064.81578947,3923.31162416,3873.7237956,1.79037445458,566409.169014,4739207.87646,39937006.0,9.0,40201818.0,1218236.90909,6950749.31165,39937058.0,35.0,40215051.0,1086893.27027,6572656.5157900015,39986014.0,19.0,0,0,0,0,1088,1232,0.8454546035510001,0.944919851028,0.0,13032.0,2134.0,3467.00436256,12020119.25,0,1,0,0,0,0,0,0,1.0,2163.63888889,38.7941176471,4064.81578947,0,0,0,0,0,0,34,1319,38,154463,-1,264,6,0,277958.0,0.0,277958.0,277958.0,39937006.0,0.0,39937006.0,39937006.0,Normal,Benign
192.168.3.29-23.52.177.21-48386-443-6,192.168.3.29,48386,23.52.177.21,443,6,17/07/2019 02:36:21 PM,308,2,1,31.0,46.0,31.0,0.0,15.5,21.920310216799997,46.0,46.0,46.0,0.0,250000.0,9740.25974026,154.0,127.279220614,244.0,64.0,64.0,64.0,0.0,64.0,64.0,0.0,0.0,0.0,0.0,0.0,0,1,0,0,64,32,6493.50649351,3246.75324675,0.0,46.0,30.75,21.6852484422,470.25,0,0,0,1,1,0,0,0,0.0,41.0,15.5,46.0,0,0,0,0,0,0,2,31,1,46,-1,1444,1,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,Normal,Benign
192.168.3.29-35.166.166.56-36318-443-6,192.168.3.29,36318,35.166.166.56,443,6,17/07/2019 02:35:41 PM,60356767,14,13,1400.0,3351.0,726.0,0.0,100.0,227.224389265,1566.0,0.0,257.769230769,559.304799706,78.7152830767,0.447340063791,2321414.11538,4292337.46351,10198459.0,11.0,60314624.0,4639586.4615400005,5204739.05298,10240112.0,152.0,60356174.0,5029681.166669998,5234408.022419998,10240302.0,412.0,0,0,0,0,448,432,0.231954107151,0.21538595664,0.0,1566.0,169.678571429,413.776917314,171211.337302,0,1,0,0,0,0,0,0,0.0,175.962962963,100.0,257.769230769,0,0,0,0,0,0,14,1400,13,3351,-1,124,4,0,58797.1666667,41584.0185332,143680.0,41653.0,10000565.1667,356527.4967570001,10198459.0,9290250.0,Normal,Benign
192.168.3.29-35.166.166.56-36316-443-6,192.168.3.29,36316,35.166.166.56,443,6,17/07/2019 02:35:41 PM,60357348,13,12,1399.0,3351.0,725.0,0.0,107.615384615,234.37809143299998,3014.0,0.0,279.25,864.292472068,78.6979573721,0.414199775643,2514889.5,4417205.88404,10197659.0,12.0,60314634.0,5026219.5,5237027.23113,10240181.0,120.0,60356339.0,5486939.90909,5231849.26686,10240419.0,1658.0,0,0,0,0,416,400,0.215383883334,0.198815892309,0.0,3014.0,182.692307692,603.149916305,363789.821538,0,1,0,0,0,0,0,0,0.0,190.0,107.615384615,279.25,0,0,0,0,0,0,13,1399,12,3351,-1,124,4,0,60124.6666667,42696.81756130001,147279.0,42522.0,9999265.16667,356332.788651,10197659.0,9289242.0,Normal,Benign
0.87.248.248-3.0.0.0-0-0-0,0.87.248.248,0,3.0.0.0,0,0,17/07/2019 02:33:58 PM,119999806,2,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0250000404167,59999903.0,19.7989898732,59999917.0,59999889.0,59999917.0,59999917.0,0.0,59999917.0,59999917.0,0.0,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0.0166666936112,0.00833334680558,0.0,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0,0.0,0.0,0.0,0.0,0,0,0,0,0,0,2,0,1,0,-1,-1,0,0,0.0,0.0,0.0,0.0,59999903.0,19.7989898732,59999917.0,59999889.0,Normal,Benign
[...]