ProcessingofaFlexibleNetworkTrafﬁcFlowInformation }w !"#$%&'()+,-./012345<yA|

(1)

MASARYKOVA UNIVERZITA

FAKULTA INFORMATIKY

}w !"#$%&'()+,-./012345<yA|

Processing of a Flexible Network Traffic Flow

Information

MASTER’STHESIS

Petr Velan

Brno, Spring 2012

(2)

(3)

Declaration

Hereby I declare, that this paper is my original authorial work, which I have worked out on my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source.

Petr Velan

Advisor: RNDr. Radek Krejˇcí

(4)

(5)

Acknowledgement

I would like to thank my advisor Radek Krejˇcí for his patience, consulta- tions and guidance throughout the whole process, my family for their support, and finally Zuzana Balážová for the proofreading.

(6)

(7)

Abstract

There are several widely used methods of storing and processing network traffic flow data. Majority of existing tools implement only NetFlow v9 protocol [1], which is not flexible enough to encompass growing demand for additional flow information. The goal of this work is to assess different approaches of flexible flow data storage and manipulation in IPFIX [2] format.

It includes creation of a tool that implements the most appropriate data storage method and would allow to query the flow data efficiently. The comparison of performance and efficiency with some of the other existing tools is also part of the work.

(8)

(9)

Keywords

NetFlow, IPFIX, FastBit, nfdump, fbitdump, database

(10)

(11)

1 Introducion

Network monitoring has become an important source of information for security incident handling and accounting. Simple statatistics, such as an amount of transferred data can be usually obtained from network interfaces. Such information can be computed and accessed without any significant overhead. This might be enough to monitor the amount of network traffic in time, but it provides no information about the data above link layer of ISO/OSI [3] model.

Every possible information about transferred data can be obtained using a method called deep packet inspection (DPI). Only the L3 layer of each packet needs to be examinded by network routers in order to determine the proper direction for the packet. Deep packet inspection examines even the data in packet payload, which allows the monitoring device to access information about the packet up to application layer of ISO/OSI model.

This can be used for accounting and security purposes, because the origin and target as well as content of the communication is known. The method cannot be used directly with encrypted traffic. Another disadvatage of this approach is its computational complexity. It is highly impractical to use DPI on high-speed links, because a common device is unable to process all data at such speed.

Flow monitoring provides a compromise between the two previously mentioned approaches. A flow is generally defined as a sequence of packets which share a common feature and pass through an observation point. Only packet headers are inspected in flow monitoring, which can be quite fast because the header has defined structure. This allows information about the packet up to transport layer to be accessed. Obtained information is aggregated into flow records, so that one connection with multiple packets is stored as one flow record. The key elements by which the packets are aggreagated into flows cany vary, but in IP networks they usually include source and destination ports and addresses, together with transport protocol.

1.1 Flow Monitoring Architecture

Flow monitoring can run on routers as an additional service. When the device is fully occupied, routing takes precedence and flows are sampled, or not processed at all. One way to avoid this problem is to setup a monitoring device dedicated to creating flows. The network line is duplicated by

(14)

1. INTRODUCION

hub or a splitter so that the network performance remains intact and the monitoring device can process all data independently (see Figure 1.1). Cre- ated flows are usually exported to another device called a collector. This allows to set up several metering points in a network and have the flow data exported to single point, where they can be processed. The flow data are exported using flow information protocol.

Figure 1.1: Standalone Probe Setup

1.2 Flow Information Protocols

Cisco NetFlow [4] technology became a de-facto standard in the flow monitoring area. NetFlow v5, using static data format, stays the most widely deployed flow information protocol. Cisco also introduced NetFlow v9 protocol with possible (and limited) flexibility of the information sent from an observation device to a collector. In 2002, Internet Engineering Task Force (IETF) chartered working group with the purpose of standardising vendor- neutral flow information protocol. In 2008, the working group published IPFIX protocol definition as RFC 5101.

IPFIX protocol provides even more flexibility than NetFlow v9. Both protocols are based on template mechanism specifying format of the data records. In contrast to NetFlow, IPFIX allows vendors to define required non-standard information elements on their own. These elements are con- nected to a vendor’s enterprise number, which is assigned by Internet As- signed Number Authority (IANA)¹. Such flexibility brings additional de-

1. http://www.iana.org/assignments/ipfix/ipfix.xml

(15)

1. INTRODUCION

mands for collectors’ capabilities of processing unknown enterprise information elements.

1.3 Flow Collectors

As IPFIX protocol allows to export more complex and flexible information, collectors must be designed to handle such data. Most of the current IPFIX support in collectors is limited to processing only elements that are present in NetFlow v9. This means that they can process IPFIX format provided it contains only standard subset of IPFIX elements.

As far as we have been able to determine, there are no tools that allow- ing to work with full IPFIX data format including enterprise elements so far, except for SiLK[5]. The SiLK is a collection of several tools that can be used to store and process flow records. It does so in a UNIX way, i.e. each tool has its specific purpose and the whole data processing consists of putting those tools together. The SiLK is more widespread in America, while in Eu- rope there is stronger tendency to use nfdump [6], which provides similar functionality in fewer and more general tools. For this reason, the SiLK is not further explored in this work.

The flow collector for nfdump framework isnfcapd(1). It can receive flow information over UDP on arbitrary port and supports IPv4 and IPv6 protocol, allows user to set size of internal buffer to optimize performance and provides system of extensions that can enable storing of specific flow elements.

Another IPFIX capable collector is IPFIXcol [7], which is developed by CESNET. The IPFIXcol is a modular collector which is capable of using different input and storage plugins at the same time. It supports UDP, TCP and SCTP protocols for receiving flows from exporter. A few plugins for storing the flow data are available and support IPFIX file format, FastBit [8]

database and PostgreSQL database as a backend.

1.4 Goals

In this work, we will discuss the suitability of different data storages for flexible flow data. Then we will select the one that is the most appropriate and implement a tool that will work with flows using this data storage.

The performance of the created application will be compared to existing solutions.

Except for the creation of the tools to store data in different formats, and

(16)

1. INTRODUCION

unless other sources are cited, the work is genuine.

1.5 Related Work

A comparison of file based storage and database approach was done in [9]. The autors used nfdump [6] and MySQL database [10] to store approximately 30GB of flow data. Then they compared running time of several types of queries using both storages. In this work, we will use similar approach to compare selected data formats, but we will account for the desired flexibility.

(17)

2 NetFlow Data Storage Formats

Requirements for flow data storage are quite specific. Each flow template defines different set of elements to be stored, but the stored data must be processed as a whole. There might be different template for each transport protocol, but we still might want to know, how much data was transfetred in total, or which connections were iniciated from specific node, regardless of the protocol.

Due to these requirements, several data storages that use flat file format were developed in the past, specifically for storing data in NetFlow format.

These storages are often combined with traditional SQL databases for storage of derived and computed data.

A brief overview of how common data storage methods can be used to store flow data follows.

2.1 Flat File Format

The most widely used format for storing flow data in Europe is probably nfdump file format. The following section relates mostly to this format.

Using flat file format, data are writen to regular file that has a firm structure. As NetFlow v9 supports different templates, these are usually man- aged by extensions. Usage of an extension is indicated by setting a control bit for this extension so that the reader knows, which data to expect.

This storage format does not support any kind of indexed search, all data must be accessed on each query, unless only limited output is expected.

The extensibility of this format is limited. It is not able to store new unknown elements, because extensions must be hardcoded into the format.

Adding new extensions on the fly would result in incompatibility with other tools. Making the format more extensible by specifying used format in header of each file would make the resulting format more row-oriented database like.

Although flat files do not seem to be a good storage format for IPFIX data, we will compare the performance of this format with others, since it is now widely used in nfdump. Showing that IPFIX can be used with storage format, that has comparable or better qualities as current flat file formats used for NetFlow v9, would encourage more users towards using the new standard.

(18)

2. NETFLOWDATASTORAGEFORMATS

2.2 Row-oriented SQL Database

SQL databases mostly use row-oriented approach and have enough flexibility to store data in NetFlow v9 and IPFIX format. We mention them because they are most widely used solution for data storages. Each template can be stored as separate table, so that the flows will correspond to table rows. Indexes can be built as necessary on appropriate columns to acceler- ate data acquisition. When query result is expected to have elements from more tables, SQLUNION operator can be used to merge individual tables.

Although SQL databases are very powerful and flexible, there might be some unnecesary overhead when dealing with large data sets. Flow data usually do not need some of the complex SQL features such as scheme al- teration, JOIN or DELETE command. Filtering based on element values and basic aggregate functions often suffice. Since the data are written only once and read multiple times after, common properties of SQL databases like atomicity and isolation are not required.

While working with flow data, we often want to access only subset of elements of a template. Using row oriented database would mean reading whole rows with redundant data.

2.3 NoSQL Column-oriented Database

Since flow data analysis does not necessarily need whole power of SQL query language, and the data are usually not modified, noSQL databases can be used. This section shows how FastBit can be used to store flow data.

The flow data are stored in directories, each directory for one template.

Every column has its own file. This allows to access only the required elements to answer a query, but could result in slower access time when reading one row at a time. Indexes can be built easily for each column, so that the select queries can be accelerated.

Storing each column in separate file has another advantage. The file contains uniform data type and the values often have small range. This can be used to compress the files more efficiently than it would be possible with row-based or flat file storage approach.

Storing the columns in different files might rapidly increase the total number of files that must be accessed to answer a query. There is no reason to expect the files to be stored physically nearby, so that the seek time of the storage drive is minimised. On the other hand, using fewer larger files should, when there is enough space and little fragmentation, lead to sequential read on queries that access all data.

(19)

3 Comparison of Common Data Storages

In high speed networks monitoring, performance is always very important.

The collector can be flooded with flow records as a result of various attacks and it is important that the information about this incident is not lost. Us- ing one collector to store data from multiple exporters can also have high demands on the performance of the collector.

This chapter describes the data collection and storage process performance comparison. Four data storage methods were selected to be compared from different categories presented in previous chapter. Flat files are represented by nfdump storage format and IPFIX file format.

• nfcapd(1) is a collector for nfdump framework. It captures flows and stores them in nfdump format.

• IPFIX file format is defined by RFC 5655 [11] and since it is standard format, we include it here for completeness.

• PostgreSQL [12] is an open source object-relational database system.

It offers data types to store IPv4, IPv6 and mac addresses, which makes it easy to use as a flow storage engine. It represents Row- oriented SQL databases.

• FastBit database was created at Lawrence Berkeley National Labo- ratory. It is designed to handle large amount of data effectively and uses. Use of various types of indexes is fully supported in FastBit.

The database is provided as library which can be incorporated into other applications.

3.1 Data Collection Process

The performance of data storage can be measured in number of flows that it is capable of storing in time. We will use twenty minutes of flow traffic¹ exported from one of the CESNET network metering points. This traffic will be captured bytcpdump(1) utility and replayed at different speeds to each storage plugin. Number of stored flows for different speeds (i.e. different time durations) will be compared.

Since the captured data cannot be sent by tcpreplay[13] to localhost, we need to send the data over the network once again from different machine

1. The traffic contains templates for TCP, UDP and ICMP traffic both for IPv4 and IPv6 as well as some other templates. The most of the traffic consists of TCP and UDP IPv4 records.

(20)

3. COMPARISON OFCOMMONDATASTORAGES

to the target. Both machines used in the measurement have at least two gigabit ethernet ports. The data are sent directly from one machine to the other without any intermediate devices, so that the latency and jitter of the network is minimized. Since the data was captured on another interface, we need to change the destination mac address. Without this modification, the data would be ignored by target machine. Thetcpreplay-edit(1) tool can be used for this task. The sending machine is fully capable of saturating the gigabit line.

The IPFIXcol will be used to store data in different formats. It is capable of processing IPFIX data transmitted as UDP packets, which is easy to record and replay, since UDP is not connection oriented. Currently IPFIXcol has plugins for PostgreSQL, IPFIX file format and FastBit storage. We will also test the IPFIXcol throughput so that we can be sure that the storage process is really the performance limiting factor.

The machine used for all tests has following configuration: 1x Intel(R) Xeon(R) E5410 @ 2.33 GHz CPU, 12 GB RAM, 1x Seagate Barracuda ES.2 ST3500320NS hard drive (500 GB of space, 7200 RPM, SATA 3 Gb/s interface) with etx4 file system. The operation system used is Scientific Linux 6.

3.1.1 Nfdump File Format

There is currently no storage plugin capable of storing data in nfdump file format for IPFIXcol. However, we can use nfcapd, which is a native application for storing flows in this format. Since nfdump version 1.6.6, which was published on13^thof March 2012, there is an experimental basic support for receiving data in the IPFIX format. Not all exported elements match the elements that the nfdump expects, thus information about number of packets or bytes is missing. However, for the performance testing, this should not make much of a difference.

nfcapd Buffer Settings

The nfcapd tool that serves as the collector for nfdump framework allows user to specify a buffer size for incoming packets. Documentation recom- mend using buffer of size at least 100 000 bytes for high speeds. We compared default buffer size set by nfcapd and size of 200 000 bytes. The results are shown in Figures 3.1 and 3.2. Several measurements were made and the error bars represent the error of the standard deviation.

The default settings gives better results for our testing environment, therefore that is the one that will be used in further tests.

(21)

0 1000 2000 3000 4000 5000 6000

2000 4000 6000 8000 10000 12000

UDP errors

Packets per second nfcapd default buﬀer

nfcapd 200k buﬀer

Figure 3.1: nfcapd Buffer - UDP Errors

31600000 31620000 31640000 31660000 31680000 31700000 31720000 31740000

2000 4000 6000 8000 10000 12000

Captured Flows

Packets per second nfcapd default buﬀer

nfcapd 200k buﬀer

Figure 3.2: nfcapd Buffer - Captured Flows

(22)

3.1.2 PostgreSQL

The version 8.4.9 of PostgreSQL was used for testing as it is present in Sci- entific Linux 6 distribution. IPFIXcol storage plugin uses C API provided by libpq library [14]. Each IPFIX template is represented by SQL table.

Originally, each flow caused one library call with SQLINSERT state- ment to the database, while each call was considered a separate transaction. We made a few changes to improve the efficiency of the plugin. Firstly, flows from same IPFIX packet that have same template are stored using one INSERT call. Secondly, it is possible to set number ofINSERT calls that be- long to one SQL transaction. On average, this saved about 30 % of the time needed to process one IPFIX packet.

The connection to the database is using UNIX socket, which should have slightly better performance than networking layer, because there are no routing decisions involved. Some of the database default settings in postgresql.conf were changed in order to raise the storage performance. The changes are described in table 3.1. The meaning of the variables can be found in PostgreSQL documentation [15]. Since no indexes are used when inserting data, the most important is the fsync option. Others are used mainly to improve the querying performance in section 5.

Table 3.1: PostgreSQL Settings

Variable Default Value New Value

shared_buffers 32 MB 2 048 MB

work_mem 1MB 512 MB

maintenance_work_mem 16 MB 4 096 MB

fsync on off

synchronous_commit on off

full_page_writes on off

effective_cache_size 128 MB 2 048 MB

3.1.3 FastBit Plugin

The FastBit plugin version 1.2.1 uses FastBit library to create tables and store the data. In new version 1.3.0, the plugin stores the data directly, which should increase performance significantly. We will compare performance of both versions and then use the better to compare with other plugins.

(23)

The plugin version 1.3.0 have new option, which allows to specify size of internal buffers for data, that are used to store the values before they are written to disk. The value might be different per hardware configuration, therefore we will try to find an optimum buffer size before doing the actual comparisons. The size of the buffer is the number of elements that it can store i.e. for short integer the buffer of size 1 000 takes 2 000 bytes.

FastBit Plugin 1.3.0 Buffer Size

In the first phase of this test, we started with buffer size of 20 000 and increased it by 30 000 steps as long, as the performance increased. The number of stored flows and number of UDP packet errors were measured for each buffer size and for four different transmission speeds. The number of UDP errors shows how many packets were dropped before they could be processed.

The test showed that the settings with buffers of size 50 000 and 80 000 have the best performance. Hence we conducted more experiments with those sizes and added a size of 70 000. We took five measurements for each of the three data rates and each buffer size. The results were computed as averages of all measurements. Error bars were added to represent the standard error value to show how much the individual measurements var- ied. The Figure 3.3 shows numbers of UDP errors and the Figure 3.4 shows numbers of captured flows.

The problem with the measurements is that the deviation is too big compared to the number of errors. But in average, the 70 000 size buffer has better performance than the other two, hence we will use it in further tests.

The reason why large buffers are not as effective as might be expected is that the write process takes too long to complete. Meanwhile other buffers are filled and packets are dropped.

FastBit Plugin 1.3.0 and 1.2.1 Versions Comparison

We will measure the performance of earlier version of the FastBit plugin the same way as we compared the buffer sizes. We will take the data we have for plugin 1.3.0 with 70000 buffer and do the tests with version 1.2.1 just as we did earlier. The results are shown in Figures 3.5 and 3.6.

The performance difference is quite significant, we can only deduce that the FastBit library uses ineffective algorithm to buffer the data and store them in files. The results have been reported to library maintainer.

(24)

0 200 400 600 800 1000 1200 1400 1600

8000 9000 10000 11000 12000

UDP errors

Packets per second 50k buﬀer

70k buﬀer 80k buﬀer

Figure 3.3: FastBit Buffer - UDP Errors

31700000 31705000 31710000 31715000 31720000 31725000 31730000 31735000 31740000 31745000

8000 9000 10000 11000 12000

Captured Flows

Packets per second 50k buﬀer

70k buﬀer 80k buﬀer

Figure 3.4: FastBit Buffer - Captured Flows

(25)

0 200000 400000 600000 800000 1000000

8000 9000 10000 11000 12000

UDP errors

Packets per second Version 1.2.1

Version 1.3.0

Figure 3.5: FastBit Versions comparison - UDP Errors

5000000 10000000 15000000 20000000 25000000 30000000

8000 9000 10000 11000 12000

Captured Flows

Packets per second Version 1.2.1

Version 1.3.0

Figure 3.6: FastBit Versions comparison - Captured Flows 3.2 Methodology

We will use similar method for comparing individual storages as used for previous tests. Following steps will be applied:

(26)

• Save current number of UDP errors.

• Start the collector.

• Wait 1 second for collector to initialize.

• Start iostat utility to write CPU and disk utilization to file every 5 seconds.

• Start tcpreplay on second machine to simulate and exporting process.

• After all data are sent, wait 5 seconds and then send terminating signal to collector.

• Wait for collector to stop.

• Save current number of UDP errors.

• Save number of stored flow records.

The test data contain 31 740 636 flow records in 1 270 138 packets, which is about 25 flows per packet. Total size of the data is 1 710.8 MB including packet headers. The IPFIX data size is 1 660 MB.

The data will be sent at following speeds: 2 000 pps, 4 000 pps, 6000 pps, 8000 pps, 10000 pps and 12000 pps. The test will be repeated five times for each storage and speed. To eliminate possible deviations caused by other factors, the results that are off will be removed and corresponding test repeated. The parameters of different speeds are summarized in Table 3.2.

Table 3.2: Exporter Data Rate Parameters Rate (pps) Bitrate (Mbps) Duration (seconds)

2 000 22.6 635

4 000 45.2 318

6 000 67.8 204

8 000 90.0 154

10 000 113.0 122

12 000 135.5 103

The IPFIX file storage will not have number of flows measured, since it is more difficult to process and determine how many flow records are

(27)

stored in the file. The number of UDP errors corresponds with the number of flows, so it is not that important.

Each value in the graphs has an error bar to show the standard deviation error.

3.3 Results

The Figures 3.7 and 3.8 shows the UDP errors and number of captured flow records for each data speed and storage except PostgreSQL, which is dis- cussed in next subsection.

0 500 1000 1500 2000 2500

2000 4000 6000 8000 10000 12000

UDP errors

Packets per second FastBit plugin

nfcapd default buﬀer IPFIX ﬁle plugin

Figure 3.7: Storage Comparison - UDP Errors

As we can see from the Figure 3.7, the IPFIX file format has very good performance, since there is no additional data processing necessary in the output plugin. This format might provide enough performance for high speed networks and could be used to store the data before further processing, possibly in distributed environment.

The second best result was achieved by FastBit storage plugin. Even though it stores data in multiple files to achieve high level of flexibility, the performance is better than that of nfcapd.

(28)

31690000 31700000 31710000 31720000 31730000 31740000 31750000

2000 4000 6000 8000 10000 12000

Captured Flows

Packets per second FastBit plugin

nfcapd default buﬀer

Figure 3.8: Storage Comparison - Captured Flows

3.3.1 PostgreSQL Performance

The performance of the PostgreSQL database is very low, even though that no indexes are being built on the data and despite the optimizations performed in database and storage plugin setup. The database would have to be distributed to several machines to be able to process decent data rate.

Since it might be interesting to know where the limits of such a database are, we performed further tests for lower data rates. The results are displayed in Figures 3.9 and 3.10. The reason that the results are not directly compared to other storages in a graph is that the scale of such a graph would be too small to show anything relevant for other storages.

The PostgreSQL database in our setup is able to losslessly capture data rate of 250 pps (2.82 Mb/s). That is approximately 4.25 x slower, than the original exporting speed. Table 3.3 shows information about the speed used in further measurements. The database would have to be much better optimized before it could be used for purpose of storing flow data in high speed networks.

During the tests we noticed that the PostgreSQL plugin stored 3 450 less flow records than the exporter sent. Although this is negligible for the performance testing, it is a bug that should be fixed in production environment.

(29)

0 200000 400000 600000 800000 1000000 1200000 1400000

2000 4000 6000 8000 10000 12000

UDP errors

Packets per second

PostgreSQL Plugin

Figure 3.9: PostgreSQL - UDP Errors

0 5000000 10000000 15000000 20000000 25000000 30000000 35000000

2000 4000 6000 8000 10000 12000

Captured Flows

Packets per second

PostgreSQL Plugin

Figure 3.10: PostgreSQL - Captured Flows

(30)

Table 3.3: Exporter Data Rate Parameters for PostgreSQL Rate (pps) Bitrate (Mbps) Duration (seconds)

250 2.82 5 081

500 5.65 2 540

1 000 11.3 1 270

3.3.2 CPU and HDD Utilization

Besides performance the resource usage of individual storages is also important. Since flows from more exporters might be captured and processed on single machine, the CPU and hard disk utilization of storages might be the limiting factor. Figures 3.11 and 3.12 shows CPU idle time and hard disk utilization when running individual plugins respectively.

0.00 20.00 40.00 60.00 80.00 100.00

0 50 100 150 200

CPU idle %Time (seconds)

Time (seconds) FastBit plugin

nfcapd IPFIX ﬁle plugin IPFIX PostgreSQL plugin

Figure 3.11: Storages CPU Utilization

The nfcapd needs the least CPU time to process incomming data. We can assume that since it is a single purpose tool, the performance optimization can be quite effective.

On the other hand, the IPFIXcol supports multiple formats of input and storage plugins, thus its interfaces are designed to be quite general and

(31)

0.00 10.00 20.00 30.00 40.00 50.00 60.00

0 50 100 150 200

HDD utilization %

nfcapd default buﬀer IPFIX ﬁle plugin IPFIX PostgreSQL plugin

Figure 3.12: Storages Disk Utilization

there might be some performance overhead. Some more tests of the IPFIX- col core are provided in next section.

The hard disk utilization is very similar across all storages, except for the PostgreSQL plugin. The PostgreSQL plugin does not store all data at 6 000 pps and after sending a shutdown signal to the collector, it takes the plugin another minute to process buffered data. It shows that the performance problems with the plugin are not the throughput of the drive, but amount overhead when processing the data. The CPU utilization confirms this, since the plugin makes full use of one core and uses other cores as well.

It seems that the nfcapd can run easily in multiple instances. The IPFIX- col is designed to process multiple input streams in one instance running new thread for each storage plugin. Testing the performance for multiple inputs is not part of this work, but the throughput and CPU utilization will be measured for IPFIXcol without storage plugin, since it is important to differentiate between core and storage plugin CPU consumption.

3.3.3 IPFIXcol Performance

The Figure 3.13 show the CPU utilization for IPFIXcol with FastBit storage plugin and with dummy storage plugin, which does not perform any ac- tion.

(32)

0.00 20.00 40.00 60.00 80.00 100.00

0 50 100 150 200

CPU idle %Time (seconds)

dummy plugin

Figure 3.13: IPFIXcol CPU Utilization

The graph shows that the dummy plugin uses less than 5 % of total CPU power. Therefore computational overhead of IPFIXcol core should not be a bottleneck of the application.

We will try to determine the maximum throughput of the IPFIXcol core in our setup and compute an average time it takes to pass one packet to the storage plugin. The Figure 3.14 shows number of UDP errors depending on the data rate of exporter.

The IPFIXcol core can process the data at 18 000 pps, the average time needed for one packet is less than 0.06 ms. The IPFIXcol with FastBit plugin can process 6 000 pps, which is less than 0.17 ms. The IPFIXcol core takes about 30 % of total time needed to process one packet, which roughly corresponds with its CPU usage. Since IPFIXcol uses multiple threads and ring buffer to pass data between them, there might be some space for concur- rency optimization.

3.3.4 Disk Space Usage

Another criterion to consider when comparing data storage methods is the amount of data needed to store the information. We can expect the nfdump format to be very space effective and the FastBit format as well, because there is only small amount of overhead that defines the data format.

(33)

0 20000 40000 60000 80000 100000 120000 140000

15000 16000 17000 18000 19000 20000

UDP errors

Packets per second IPFIXcol dummy Plugin

Figure 3.14: IPFIXcol UDP Errors

The IPFIX file stores the row flow data as it comes from the network, thus it may contain unnecessary template definitions and other IPFIX transport specific information.

The PostgreSQL database uses complex internal structures to store the data, along with other information like statistics table information.

The Table 3.4 shows the space needed by each storage to store the test data.

Table 3.4: Disk Space Usage Storage Space Occupied (MB)

nfdump 1 575

FastBit 1 626

IPFIX file 1 660

PostgreSQL 4 064

We suspect that the reason that nfcapd saves about 50 MB of disc space is that the extensions for some of the elements contained in the exported flow records were not used. As a result, such a data were not stored at all.

(34)

Nfcapd supports runtime data compression using LZO algorithm². This can be particularly important for long term data storage, where the disk space usage is very important factor. After discussion with FastBit main- tainers we came to conclusion, that storage layer like Zettabyte File System (ZFS) might provide the compression technique transparently to the application. It should be possible to store the data on regular file system to provide necessary performance while moving old data to file system with compression.

3.4 Conclusion

Table 3.5 sums up the advantages and disadvantages of tested storages.

Table 3.5: Storage Comparion

Storage Space Usage Performance Storage Flexibility

nfdump good good average

FastBit good good good

IPFIX file good good poor

PostgreSQL poor poor good

The results show that the FastBit storage might be very good solution for flexible flow data. We will implement a querying tool for the FastBit storage and compare its performance with PostgreSQL and nfdump.

2. http://www.oberhumer.com/opensource/lzo/

(35)

4 Tool for IPFIX Data Manipulation

Large portion of this work was dedicated to creation of data manipulation tool for flow data in FastBit storage format. The main goal was to create nfdump-like tool which would be flexible enough to handle IPFIX elements.

The name of the tool is fbitdump.

4.1 Requirements

We have imposed several requirements upon the fbitdump tool, based on the features provided by nfdump:

• It must be possible to display any IPFIX element, including enterprise elements.

• The output format will be customizable. Some basic output formats must be provided for compatiblity with nfdump.

• It will be possible to limit the number of printed flows.

• Input data specification must be easy and similar to nfdump.

• The tool will convert output values to human readable form based on elements semantics.

• Big numbers must be suffixed with units. It will be possible to disable this feature.

• The tool will manage indexes on data. It will be possible to create and delete indexes for chosen elements.

• A filter for data must exist. It will be possible to select only flows specified by the filter. The input must be user friendly, it must be possible to input values for elements with known semantics as human readable values (e.g. IP addresses, timestamps).

• It will be possible to aggregate flows based on specified elements.

The tool will provide an option for default flow aggregation in order to achieve compatibility with nfdump.

• Output rows will be sortable by specified element. The default order will be ascending, the default sorting element the flows start time, when it is present.

(36)

4. TOOL FORIPFIX DATAMANIPULATION

• Statistics for selected element will be generated on demand. This is similar to aggregation based on the element and limiting default number of printed rows. It must be possible to specify a column to sort the results by. The default sorting order for statistics will be de- scending.

• The tool must provide an option for translating IP addresses to do- main names.

4.2 Implementation

This section describes some of the implementation decisions that have been made throughout the creation process.

Linux was chosen as the target platform for the tool. This is a logical choice as nfdump supports the same. Additionally, the tool is developed and tested on systems with Red Hat Packet Manager (RPM), therefore the RPM packaging support is provided.

4.2.1 Programming Language

The tool is expected to run as fast as possible, which is why chose C++ as a programming language. Any interpreted language would present unnecessary slowdown. The C++ was chosen over C because the FastBit library provides its native interface for C++. Furthemore, there is a wider choice of XML libraries than for C language.

The currently used XML library is pugixml [16], which has very mini- malistic source code and is provided as part of the fbitdump project sources.

The tool heavily uses C++ STL library, namely strings, vectors and sets.

It simplifies the work with flexible configuration described in following section.

4.2.2 Columns Configuration

The most substantial difference between fbitdump and nfdump is the required flexibility. Nfdump works with strictly defined set of flow elements, which allows it to have their semantics at source code level. This is not possible for fbitdump, which should be able to process even unknown, new elements.

There must be a place to specify the semantics for displaying unknown elements, unless the data are displayed as raw numbers or byte strings.

(37)

Passing such configuration on commandline is clearly impractical, therefore a configuration file must exist. Working with such a flexible configuration is quite a challenging task and it is crutial to understand the requirements for such configuration and implications for the application design.

Plain columns

Each element is printed in different column in the output. The configuration file must allow to define the used columns and their attributes. The appropriate and easy to parse format for structured configuration data is XML. Example of configuration with one column is in listing 4.1.

The column name printed in header isDate flow start, its width is 23 characters. Alias refering to the columns is %ts. The column takes plain value from column namede0id152, allows it to be aggregated usingmin function and the printed value is considered a 64 bittimestamp containing number of miliseconds since 1.1.1970 and the value is formatted accord- ingly.

Listing 4.1: fbitdump XML Configuration

<name>Date flow start</name>

</value>

</column>

</columns>

</configuration>

The FastBit columns can have arbitrary names, but the IPFIXcol Fast- Bit ouput plugin have following naming convention. The column name is derived from enterprise number, element number and optionally part number, when the column is bigger than 64 bits. For example, IPv6 source IP is stored in two columns, e0id27p0 ande0id27p1. To support such scheme, child elements of <value type="plain" /> can have attributeparts, which specifies the number of database columns in which the element is stored.

The example ofvalue element configuration for IPv6 column is shown in listing 4.2.

The currently supported semantics attributes for plain values are:

(38)

Listing 4.2: fbitdump IPv6 Column Configuration

</value>

• ipv4: Element is expected to be uint32 column and is printed as IPv4 address.

• ipv6: Element is expected to be uint64 column and is printed as IPv6 address. Theparts attribute must be set to2.

• tmstmp64: Element is expected to be uint64 column and is printed as timestamp.

• tmstmp32: Element is expected to be uint32 column and is printed as timestamp.

• tcpflags: Element is expected to be uint8 column and is printed as TCP flags. The format is same as in nfdump.

• flows: This denotes a special column with no relation to the database.

It is computed as number of flows, i.e. number of database rows.

The aggregation functions that can be used aresum,min,max,avg, and allow to sum up values of aggregated columns, find minimum, maximum value, or compute an average.

Multiple aliases can be specified for one column. The aliases can be used to specify output format, aggregation or statistics columns. They are also used in filter and any other place that requires column specification.

Operation columns

Having only plain columns is not enough. The nfdump supports computed columns such asDuration orPackets per second. Since this feature can be useful, we will implement it in a general way. We introduce value typeop- eration. The configuration of Duration element using operation is shown in listing 4.3.

The operation element allows to specify atributename, which defines type of operation. The name can be one of’+’,’-’,’/’, ’*’ and has implicit meaning. The element can have semantics, just as with type plain. In the

(39)

Listing 4.3: fbitdump Duration Column Configuration

<name>Duration</name>

<alias>%duration</alias>

</operation>

</value>

</column>

example,%te−%tsis computed, which means end timestamp minus start timestamp. The result is formated as duration.

The only currently supported semantics attribute for operation plain values is:

• duration: Formats miliseconds as seconds with three decimal places.

One last column configuration element not yet mentioned is<default- value/>. When element is not available in specified table, This value is printed instead of implicit“NULL” string.

4.2.3 Summary Configuration

Nfdump shows a few summary and statistics rows at the bottom of the output. The numbers of processed and printed flows, bytes and packets can be useful, as it gives the user a graps on how many flows, bytes and packets are present in the selected nfdump files.

Since the fbitdump is required to be flexible, it cannot presume the ex- istence of such elements within data files. Therefore columns that should be summed up are defined in configuration file as show in listing 4.4. The columns are reference by their aliases. Only plain value columns can be used.

The use of statistics can extend the execution time, because the all required data must be accessed and summed up. Therefore the statistics are optional. However, when fbitdump is invoked to compute statistics for specific column, the bottom statistics are enabled automatically, since much of the the required data is probably processed already.

(40)

Listing 4.4: fbitdump Summary Columns Configuration

</summary>

4.2.4 IPFIX Template Differences Output format

Users accustomed to nfdump expect that all data are stored in same file regardless of its type. The output is adjusted to accomodate printing of different protocols in one output format. This means, that IPv4 and IPv6 addresses are prined in one column, because it can be expected that one flow will not contain both IPv4 and IPv6 address. Similarly ICMP message type is printed in columns with ports, because ICMP does not have source and destination ports.

Such assumptions cannot be imposed on fbitdump output. IPFIX is sup- posed to be flexible format, thus any combination of elements must be supported in one template. Trying to support column reuse for multiple elements causes more problems that is potentially solves. Thus fbitdump output has strictly one column per element, with obvious exception of computed columns (operations). Rows that have no value for required column make use of<default-value/>element described above.

The default output formats supported by nfdump are longer in fbitdump, since more columns are required for printing the same information.

To alleviate the problem of long output lines, fbitdump supports formats specifically for IPv4 or IPv6, as this is most commonly used scenarion. Thus options likelineorlong have their equivalents for IPv4 only (line4,long4) and for IPv6 only (line6,long6).

FastBit interface

FastBit library allows to work with individual data tables throughibis::part interface. Class ibis::table provides a level of abstraction above the parts.

The table have methods for querying the data and iterating over the results usingibis::table::cursor.

When the table query contains a column in a SELECT clause, which does not exist in the partition (an element not present in a template), the query

(41)

returns no result. Therefore it is vital to check that only the existing elements are requested while our code is required to handle the default values for non-existing elements.

Since FastBit is noSQL database, its data manipulation language does not contain support for complex operations on columns like adding, sub- stracting or dividing. But this feature is needed for operation columns defined in fbitdump XML configuration. Hence the computation of such operations must be done in the fbitdump itself. This causes a limitation for filtering, because the operation columns cannot be used in filter. It would be possible to implement such a feature in fbitdump code, but that would be quite inefficient. The code for filtering is quite universal so that this feature can be added just by implementing one function, if the need ever arises.

When using fastbit table for aggregation, it is necessary to aggregate only parts with the same elements, because FastBit does not support aggregation of parts with different schema. More precisely, the elements in the SELECT clause must exist in all parts.

The aggregation process must respect the limitations mentioned above.

On the other hand we often want to aggregate more different parts together while ignoring their differences. It is important to know how the process works so that the user gets the expected results.

In the following, summary columns are the columns that use some aggregation function (e.g. sum, min, man) and aggregation columns are the columns whose combination is expected to be unique, i.e. the GROUP BY columns in SQL notation. The aggregation is done as follows:

• Parts that do not have required columns for summary functions are excluded.

• Intersection of aggregation columns and partition elements is computed for each partition.

• A table is created from all parts with same intersecion from previous step. This ensures that we can aggregate flows by ports, protocol, IPv4 and IPv6 addresses so that the flows with IPv4 adresses will end up in one table, and the flows with IPv6 addresses in another, even if the ports and protocol are the same.

• The query is executed on the resulting tables, a combination of the flows from all tables is outputed in a specified order.

(42)

4.3 Conclusion

The tool was developed to fulfil all of the initial requirements. During the process, need for some new functionality arose and the application was modified acccordingly. The complete features of the fbitdump are described in its manual pages, that is included in the fbitdump package.

(43)

5 Data Querying Performance Comparison

5.1 Data Collection Process

The data for the querying tools comparison will be collected from one of the CESNET metering points, just as we did in Chapter 3. We will capture 24 hours of flow traffic withtcpdump(1). Then, we will replay the captured data to nfcapd(1) andipfixcol(1) with Fastbit and PostgreSQL plugins. It will take four days for PostgreSQL to store the data, even without any indexes.

5.2 Methodology

This section describes how the performance will be measured. The query response time is the important figure which we define as the time invoking a query and retrieving complete result set. The large result sets will not be printed to screen since this would only add constant time to each query.

All tests will be performed on the same configuration as the tests in Chapter 3.

The performance of each of the queries will be observed on different data sets. We will increase the data set by hour to see how each tool handles larger queries. We will repeat each query execution three times to eliminate the effect of other factors and add error bars to resulting graphs to show standard error value of the standard deviation.

The psql(1) command line utility is used to issue SQL commands to PostgreSQL database.

5.2.1 Data Access Optimalizations

Since FastBit and PostgreSQL are designed to make use of indexes, we will build indexes on all appropriate columns beforehand. We will let Post- greSQL decide whether to use indexes or not. For long term storage, it is possible to remove the indexes and save disk space. The indexes will be build on columns, that are used for filtering.

We will compare following three setups:

1. nfdump with its default behavior.

2. PostgreSQL deciding whether to use indexes or not.

3. fbitdump using available indexes.

(44)

5. DATAQUERYINGPERFORMANCECOMPARISON

5.2.2 Use Cases

This section describes the queries that will be used to test the querying tools.

We will test sequential scan, random access and data aggregation with following queries:

Listing: This use case tests the sequential access to the data. It prints all data specified by time window. Note that since nfdump and fbitdump have the flows stored in multiple files or folders, there is no need for them to access all data. Only PostgreSQL will have to have limitation on flow timestamps.

Listing + Filtering: This query will simply print all flow records that sat- isfy given condition. We will select all queries to DNS servers making the tools to print only flows with destination port 53.

Grouping: In this query, we will determine number of flows, packets and bytes for each protocol.

Grouping + Filtering: This query will select only flows with destination port 53 and group them by destination IPv4. The result should show all used DNS servers with number and size of their queries.

All queries will print only following elements when available:source port,destination port,source IPv4,source IPv6,destination IPv4andsource IPv4 address,protocol,bytes,packets,flow start,number of flows.

Table 5.1 shows skeletons of queries for PostgreSQL database invoked to get desired results. For the listing and listing + filtering cases, select state- ments can be executed one by one for each table. When grouping, subquery is needed in order to put all tables together usingUNION command. The queries in the table are simplified using pseudo-syntax for clarity.

Table 5.2 shows corresponding queries for nfdump, Table 5.3 for fbitdump.

5.3 Data Statistics

This section provides basic information on the actual data sample used for the tests. The Table 5.4 shows how much disk space is used by each storage.

An index on destination port column was created in FastBit and the last column shows how much space the index takes.

The PostgreSQL database has an index on flow start time column, to speed up theBETWEEN search, and an index on destination port to speed

(45)

Table 5.1: Queries for PostgreSQL Use Case PostgreSQL Query

Listing SELECT [usable columns]

FROM "Template[258,261,264,265,267,268,270,271]"

WHERE "flowStartMilliseconds" BETWEEN from AND to;

Listing + Filtering

SELECT [usable columns]

FROM "Template[258,261,264,267]"

WHERE "flowStartMilliseconds" (BETWEEN from AND to) AND ("destinationTransportPort" = 53);

Grouping SELECT [usable columns]

FROM (SELECT UNION

"Template[258,261,264,265,267,268,270,271]"

WHERE "flowStartMilliseconds" BETWEEN from AND to) as subquery

GROUP BY "protocolIdentifier";

Grouping + Filtering

SELECT [usable columns]

FROM (SELECT UNION "Template[258,264]"

WHERE ("flowStartMilliseconds" BETWEEN from AND to) AND ("destinationTransportPort" = 53)) as subquery GROUP BY "destinationIPv4address";

up the filtering. The creation of flow start time index took 184 minutes and it occupies 42 853 MB and the creation of index on destination port column took 189 minutes and the index takes 42 001 MB.

The creation of the FastBit index for whole sample took 21 minutes and the size of the data increased by 12 737 MB. The index management must be done carefully, since the size of the data can increase significantly. The time used for index creation should not be an issue since it can be done while capturing the data.

The data contains 2 000.5 million of flows in total.

5.4 Results

This section presents the results of the measurements performed using the methodology described above.

Some PostgreSQL queries have significantly higher standard deviation

(46)

Table 5.2: Queries for nfdump Use Case nfdump Query

Listing nfdump -R

/datadir/nfcapd.from_date:nfcapd.to_date -o "fmt:%ts %pr %sap -> %dap %pkt %byt"

Listing + Filtering

nfdump -R

/datadir/nfcapd.from_date:nfcapd.to_date

-o "fmt:%ts %pr %sap -> %dap %pkt %byt" "dst port 53"

Grouping nfdump -R

/datadir/nfcapd.from_date:nfcapd.to_date -o "fmt:%pr %pkt %byt %fl" -A proto Grouping +

Filtering

nfdump -R

/datadir/nfcapd.from_date:nfcapd.to_date -o "fmt:%pr %da %pkt %byt %fl"

-A dstip "ipv4 and dst port 53"

than nfdump and fbitdump. This might be caused by part of the result set remaining in the memory to be used by subsequent queries. It is particularly notable in queries that access smaller amount of data, since it is more likely that the cache will contain desired information.

5.4.1 Listing Case

The Figure 5.1 shows comparison of performance of the querying tools in the listing case.

The intention was to list the whole data sample, but that would be too time consuming, so we cut out all queries that took longer than 40 minutes to complete. We can see that the nfdump was able to process more than twice the data than fbitdump. PostgreSQL was able to output 40 minutes of flows in 45 minutes, which is three times less than fbitdump.

The results shows that nfdump is highly optimized for flows printing, fbitdump and PostgreSQL needs to create the columns dynamically and therefore there is more computational overhead.

(47)

Table 5.3: Queries for fbitdump Use Case fbitdump Query

Listing fbitdump -R

-o "fmt:%ts %pr %sa4 -> %da4 %sa6 -> %da6 %sp %dp

%pkt %byt"

Listing + Filtering

fbitdump -R

-o "fmt:%ts %pr %sa4 -> %da4 %sa6 -> %da6 %sp %dp

%pkt %byt" "dst port 53"

Grouping fbitdump -R

/datadir/nfcapd.from_date:nfcapd.to_date -o "fmt:%pr %pkt %byt %fl" -A %proto Grouping +

Filtering

fbitdump -R

-o "fmt:%pr %da4 %pkt %byt %fl" -A %da4 "%dp = 53"

00:00 05:00 10:00 15:00 20:00 25:00 30:00 35:00

00:00 00:40 01:20 02:00 02:40 03:20 04:00 04:40

Query response time (minutes)

Network data (hours)

fbitdump nfdump PostgreSQL

Figure 5.1: Listing Query

(48)

Table 5.4: Disk Space Usage

Storage Space Occupied (MB) Space Occupied - Indexes (MB)

raw data 109 672 -

nfdump 99 247 -

FastBit 102 480 12 737

PostgreSQL 256 020 84 854

5.4.2 Listing + Filtering Case

Comparison of performance of the querying tools in listing + filtering use case is shown in Figure 5.2. The fbitdump performs similarly to nfdump for shorter flow data time window, but the nfdump processing speed is better when printing more flows. The nfdump is twice as fast as the fbitdump when processing the full data sample, but nfdump has a trend to be even more effective. The fbitdump might perform better when a less frequent destination port is used.

The PostgreSQL database queries were run only for first few hours.

00:00 05:00 10:00 15:00 20:00 25:00 30:00 35:00 40:00 45:00

0 2 4 6 8 10 12 14 16 18 20 22 24

Network data (hours) fbitdump

nfdump nfdump

Figure 5.2: Listing + Filtering Query

(49)

5.4.3 Grouping Case

The Figure 5.3 shows the performance of the querying tools in grouping case. The PostgreSQL performance is quite low since a subquery needed to be used to make union of all tables before the grouping could take place.

The result set consisted of only a few rows in this case, hence the nfdump does not benefit from its superior printing speed. The fbitdump gives the result for full data twice as fast as nfdump and the tendency of the graph hints that it might be better for larger data sets. The response time for the first eleven hours of the flow data was quite low, though it is likely to be due to the index being cached in memory, so that the fbitdump does less disk reads in subsequent queries. This can be considered a features since the user is likely to refine her query in a few iterations.

00:00 05:00 10:00 15:00 20:00 25:00 30:00 35:00 40:00 45:00 50:00 55:00

0 2 4 6 8 10 12 14 16 18 20 22 24

Network data (hours) fbitdump

nfdump PostgreSQL

Figure 5.3: Grouping Query

5.4.4 Grouping + Filtering Case

This use case has significantly larger result set than the grouping case. It tests the performance of the tools when aggregating using column with high entropy. The Figure 5.4 shows the results the for grouping + filtering case. The graph is similar to that of the grouping case, the fbitdump being almost twice as fast as nfdump for the whole data set.

ProcessingofaFlexibleNetworkTrafﬁcFlowInformation }w  !"#$%&'()+,-./012345<yA|

}w !"#$%&'()+,-./012345<yA|

Processing of a Flexible Network Traffic Flow

Information

Petr Velan

Declaration

Acknowledgement

Abstract

Keywords

Contents

1 Introducion

2 NetFlow Data Storage Formats

3 Comparison of Common Data Storages

4 Tool for IPFIX Data Manipulation

5 Data Querying Performance Comparison

ProcessingofaFlexibleNetworkTrafﬁcFlowInformation }w !"#$%&'()+,-./012345<yA|