All You Need To Know About Spectre And Meltdown A pair of bugs has silently infested CPUs from Intel, AMD, and ARM for years.

 

After two days of whirlwind developments, we finally have more of a complete picture of the new vulnerabilities that impact processors from the leading vendors. Reports initially surfaced two days ago that Intel’s processors are susceptible to a new hardware-based bug that cannot be patched with a mere microcode update. A report from The Register, based in part on a blog post, said that incoming Windows and Linux patches would correct the vulnerability but come with a 5-30% performance loss depending on the workload.

The industry remained silent due to NDAs that were scheduled to expire on January 9, the same date as a round of patches were scheduled to appear. After a day of silence while its stock slumped, Intel issued a statement and claimed the issue is not a hardware bug. Intel also announced that it’s working with other titans of the industry, such as AMD and ARM Holdings, to “develop an industry-wide approach to resolve this issue promptly and constructively.” AMD has since released a statement and claimed that it has minimal exposure to the primary vulnerability.

The root issues behind the vulnerabilities weren’t clearly defined at the time, but a slew of releases from several of the parties involved, along with Google’s Project Zero team, have shed light on two new exploits that have served as the catalyst for the recent developments. We’ll cover the new exploits below; then we’ll get to the updates from Intel, ARM, AMD, and Nvidia.

Performance First

Before we dive into the nuts and bolts, recent tests indicate the patch does not impart a cataclysmic performance loss in most workloads. Phoronix tested the Linux patch, and Computerbase.de tested a patched Windows Insider build.

The good news? Most desktop applications appear to be safe in both Windows 10 and Linux. That includes most workloads that are largely confined to the user space, such as gaming and normal productivity applications. There does appear to be a slowdown to storage I/O operations (2-7%), but for now it’s hard to ascertain if that is due to the patch or other kernel updates. The Windows 10 patch was rolled out to the Windows Insider builds in November, and there haven’t been reports of performance issues.

The bad news? The patch does incur a performance overhead to some enterprise applications. Phoronix recorded significant performance regressions in the object-relational PostgreSQL database. Redis also suffers a performance loss. Many industry analysts feel the real impacts will come in virtualized environments, but we have yet to see benchmarks. Google has already updated all its cloud infrastructure, which includes its cloud computing services, and we haven’t yet heard of significant user backlash due to reduced performance.

Meet Meltdown & Spectre

Google’s Project Zero touched off the vulnerability scare when it discovered that it could access data held in the protected kernel memory through two exploits that are now known as Meltdown and Spectre. Google does not believe these exploits have ever been used in the wild, but it’s impossible to tell if they have or not.

 

Meltdown is both easy to execute and easy to fix. This exploit allows applications to read from the protected kernel memory. That ability can allow hackers to read passwords, encryption keys, or other data from the memory. Intel’s statement specifically noted that the exploits cannot corrupt, modify, or delete data, but those points are moot if the attacker can access passwords and encryption keys. The biggest concern for data centers and cloud service providers is that the exploit also allows an application resident in one virtual machine to access the memory of another remote virtual machine. This means an attacker could rent an instance on a public cloud and collect information from other virtual machines on the same server.

Researchers have been able to execute a Meltdown exploit only on Intel processors, although ARM has submitted patches to protect itself from the same method of attack. In fact, the attack exploits Intel’s out-of-order execution implementation that is present on every Intel processor made since 1995. Researchers discovered Meltdown last year. The exploit is reportedly simple enough that a script kiddie could execute the attack, so a fix is of utmost importance.

Apple already patched this exploit in the MacOS December OSX patch (10.13.2). Windows is also pushing emergency patches out immediately. The Linux kernel has also been patched. These patches do have performance impacts, as we noted above, that largely revolve around how frequently the application issues kernel calls.

The Spectre exploit is much more nefarious and impacts Intel, AMD, and ARM. This exploit can access kernel memory or data from other applications. Researchers contend that fixing this exploit would require a fundamental re-tooling of all processor architectures, so we’ll live with the threat of this vulnerability for the foreseeable future. Fortunately, this exploit is extremely hard to execute and requires an elevated level of knowledge of the interior workings of the target processor.

These two exploits are categorized into three variants. Variants 1 and 2 are Spectre, whereas Variant 3 is Meltdown. Intel is vulnerable to all three.

Variant 1: bounds check bypass (CVE-2017-5753)
Variant 2: branch target injection (CVE-2017-5715)
Variant 3: rogue data cache load (CVE-2017-5754)

Levels Of Exposure

We reached out to AMD, and the company responded with the following information, which has since been publicly released.

Most notably, AMD claims that is has zero vulnerability to Variant 3 (Meltdown), stating that the patches that are currently being issued for Meltdown do not apply to its processors due to “architectural differences.” This is excellent news for AMD, as it therefore has no exposure to the current round of potentially performance-sapping patches. That bodes very well for the company as it reenters the data center with a competitive line of EPYC processors.

The Ryzen desktop processors are also not susceptible to Meltdown. Linus Torvalds has also granted AMD an exemption to the performance penalties incurred by the Linux patch for Meltdown.

AMD is vulnerable to Variant 1, which is a Spectre exploit. As noted above, many contend that Spectre is not likely to see an effective patch any time soon, and some researchers claim the vulnerability exists in every modern processor architecture in existence. They also claim that fixing the issues could require a redesign of fundamental processor architectures. AMD said it has a patch that can mitigate Variant 1 with minimal performance impact and further stated that it has a “near zero risk of exploitation” from Variant 2, which is also a Spectre exploit.

Nvidia also issued a statement regarding the vulnerabilities:

Nvidia’s core business is GPU computing. We believe our GPU hardware is immune to the reported security issue and are updating our GPU drivers to help mitigate the CPU security issue. As for our SoCs with ARM CPUs, we have analyzed them to determine which are affected and are preparing appropriate mitigations.

ARM Holdings has added a security update to its website that outlines its exposure to the vulnerabilities, and like Intel, it is susceptible to all three variants.

The legal ramifications of these developments could be troublesome. The Law Offices of Howard G. Smith has already announced an investigation on behalf of Intel Corporation investors, and there will likely be more similar developments in the coming weeks. Intel has a history of establishing a reserve to cover pending large-scale hardware replacements, but the company has not disclosed a new fund to deal with the vulnerabilities. The company has also stated that it does not expect any impact to its business.

Intel’s statement on the matter specifically says that the exploits are not caused by a “bug” or a “flaw” that is unique to Intel products. Intel also noted that the exploits can “gather sensitive data from computing devices that are operating as designed.” These statements likely indicate Intel will defend any potential claims because “the hardware is working correctly.” Depending on when these vulnerabilities became known (some claim that Meltdown-type attacks have been a known entity since 2010), these points may be challenged in court. ARM and other vendors may also face similar challenges.

Intel’s CEO, Brian Krzanich, also sold $39 million in stocks in November 2017 (this doesn’t include the amount he paid for the stock options). These transaction initially appeared innocuous (and they may be) because Krzanich sold the stock under a 10b5-1(c) plan, which is a pre-planned sale of stocks intended to prevent claims of insider trading. The sale left Krzanich with the Intel-mandated minimum of 250,000 stocks. The sale was pre-planned on October 30. Now, though, MarketWatch claims Intel was made aware of the vulnerability on June 1, which may draw attention to the matter from regulatory officials. Business Insider said a representative for the Securities and Exchange Commission declined to comment on the matter.

Considering the lengthy preparation period, we imagine there will not be any major service disruptions to the cloud service providers. However, we expect more details to come to light concerning performance impacts of the new patches on various workloads. Stay tuned.

Related Articles: Understanding The Meltdown And Spectre Exploits: Intel, AMD, ARM, And Nvidia

Database Monitoring with PRTG Monitoring your databases enables you to ensure that queries are processed in time.

Monitoring your databases enables you to ensure that, on the one hand, database queries are processed in time, and, on the other hand, that the database itself performs within the defined parameters. Furthermore, database monitoring with PRTG makes it possible to be alerted via a corresponding sensor status if database queries return an unexpected result value.

PRTG comes with built-in native sensors for the most common databases:

  • Microsoft SQL servers
  • MySQL servers
  • PostgreSQL servers
  • Oracle SQL servers

It is also possible to monitor many other database servers. For this concern, PRTG uses the ActiveX Data Objects (ADO) interface.

There are two types of database sensors:

Sensors monitoring databases directly: Monitor databases from the user perspective. These sensors send a request to the database server and receive corresponding values. You can optionally process data tables and show values in individual channels or monitor transactions.
Sensors monitoring database performance: Monitor databases with a more abstract view on the servers. Usually, these sensors monitor performance counters via Windows Management Instrumentation (WMI).
SENSORS MONITORING DATABASES DIRECTLY

PRTG provides several sensors that can “look into” the content of databases. Sensors of this type connect to the database server, execute a defined query, and show the execution time of the whole request and the query. You can use these sensors to process the data table and show requested values in individual channels.

The following sensor types are available for this kind of monitoring:

  • Microsoft SQL v2 Sensor: Monitor your Microsoft SQL server 2005 or later.
  • MySQL v2 Sensor: Monitor your MySQL server version 5.0 or later.
  • Oracle SQL v2 Sensor: Monitor your Oracle database server version 10.2 or later.
  • PostgreSQL Sensor: Monitor your PostgreSQL database version 7.x or later.

For these sensor types, you can define valid SQL statements that the sensors send to the database server. Define the queries in an SQL script file and store it into the respective \Custom Sensors\sql\ subfolder of your PRTG installation (see section Data Storage for details).

You can select this SQL script when you add the sensor to PRTG. With every scanning interval, the sensor executes this script with the defined query against the database and the database returns corresponding values in individual channels (see the example below for sample channel value selections). Use the Sensor Channels Settings to define limits for specific values.

icon-i-round-redThese sensor types need .NET 4.5 or later installed on the computer running the PRTG probe.

Alternatively, you can monitor almost all available database servers with the ADO SQL v2 Sensor via an ActiveX Data Objects (ADO) connection.

 

Related Article: PRTG MANUAL: MONITORING DATABASES

How To Extend Datastore Capacity in the vSphere Client When your Datastore require more space, you can dynamically increase the capacity

Use one of the following methods to increase a VMFS datastore:

  • Add a new extent. An extent is a partition on a storage device. You can add up to 32 extents of the same storage type to an existing VMFS datastore. The spanned VMFS datastore can use any or all of its extents at any time. It does not need to fill up a particular extent before using the next one.
  • Grow an extent in an existing VMFS datastore, so that it fills the available adjacent capacity. Only extents with free space immediately after them are expandable.

Note
If a shared datastore has powered on virtual machines and becomes 100% full, you can increase the datastore’s capacity only from the host with which the powered on virtual machines are registered.

Prerequisites

Required privilege: Host.Configuration. Storage Partition Configuration

Procedure

Log in to the vSphere Client and select a host from the Inventory panel.

Click the Configuration tab and click Storage.

From the Datastores view, select the datastore to increase and click Properties.

Extend Datastore ESXi
Extend Datastore ESXi
Click Increase.

Select a device from the list of storage devices and click Next.

Extend Datastore ESXi
Extend Datastore ESXi
Extend Datastore ESXi
Extend Datastore ESXi

Option

To add a new extent – Select the device for which the Expandable column reads NO.

To expand an existing extent – Select the device for which the Expandable column reads YES

Review the Current Disk Layout to see the available configurations and click Next.

Select a configuration option from the bottom panel.

Depending on the current layout of the disk and on your previous selections, the options you see might vary.

Option

Use free space to add new extent – Adds the free space on this disk as a new extent.

Use free space to expand existing extent – Expands an existing extent to a required capacity.

Use free space – Deploys an extent in the remaining free space of the disk. This option is available only when you are adding an extent.

Use all available partitions – Dedicates the entire disk to a single extent. This option is available only when you are adding an extent and when the disk you are formatting is not blank. The disk is reformatted, and the datastores and any data that it contains are erased.

Extend Datastore ESXi
Extend Datastore ESXi

Set the capacity for the extent.

The minimum extent size is 1.3GB. By default, the entire free space on the storage device is available.

Click Next.

Extend Datastore ESXi
Extend Datastore ESXi

Review the proposed layout and the new configuration of your datastore, and click Finish.

Extend Datastore ESXi
Extend Datastore ESXi

What to do next

After you grow an extent in a shared VMFS datastore, refresh the datastore on each host that can access this datastore, so that the vSphere Client can display the correct datastore capacity for all hosts.

Related Article: Increase VMFS Datastore Capacity in the vSphere Client

List for troubleshooting Process and threads on Linux

To the get the sum of all threads running in the system:

ps -eo nlwp | tail -n +2 | awk '{ num_threads += $1 } END { print num_threads }'

To get the number of threads for a given pid:

ps -o nlwp <pid>

Locate a Process

top

or

ps

To display all process names, use the following command –

$ ps -e

List the process associated with  a service/program

ps aux | grep my_service /  PID

To Kill a Process

kill PID

To see the thread count of process, use the following command-

$ cat /proc/<pid>/status

Configure IPTABLES to Allow Access to Common Services This article gives the steps to open firewall ports on CentOS in Iptables IPv4

Basics

  • Iptables rules can be changed on the fly by using the iptables binary.
  • The rules that are set using iptables command are in memory only and will vanish when the daemon is restarted.
  • The firewall rules added on the fly can be saved to the configuration file easily in CentOS/RHEL with the command service iptables save
    • This is no need to edit the configuration file unless you really want to.
  • The following examples are aimed at hardening the inbound traffic, but allowing all outbound traffic.
    • You can completely lock down all inbound, outbound and forwarded traffic if needed. It generally just causes a lot more administration and usually isn’t necessary.

Basic Commands

iptables –flush delete all firewall rules from memory.
iptables –list List current firewall policies
service iptables save (CentOS/RHEL) save current rules in memory to configuration file (/etc/sysconfig/iptables)
service iptables restart restart iptables daemon and load firewall rules from configuration file.
iptables-save > /root/firwallrules.fw save firewall rules in memory to a specific configuration file.
iptables-restore > /root/firwallrules.fw restore firewall rules from a specific configuration file to memory.

Basic iptables Command Parameters

  • -A append to policy chain
  • INPUT | OUTPUT | FORWARD policy chain identifiers
  • -p protocol
  • -m match
  • -s source
  • –dport destination port
  • –state connection state
  • -j jump target ACCEPT | DROP

Backup Current Iptables Configuration to File

Before you begin, it is recommended to backup your current firewall rules.

iptables-save > /path/to/somewhere/filename

Example:

iptables-save > /home/user1/iptable-rules-20130308.fw

Remove All Current Rules

iptables --flush

Set Policy Chains Default Rule

iptables -P INPUT DROP
 iptables -P OUTPUT ACCEPT
 iptables -P FORWARD ACCEPT

Allow Loopback

iptables -A INPUT -i lo -j ACCEPT

Allow All Established and Related Connections

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

Allow ICMP “ping” from LAN (TCP Port 22)

iptables -A INPUT -p icmp -m icmp -s 192.168.0.0/24 --icmp-type echo-request -j ACCEPT

Allow SSH from LAN (TCP Port 22)

iptables -A INPUT -p tcp -m tcp -s 192.168.0.0/24 --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT

Allow RSYNC from LAN (TCP Port 873)

iptables -A INPUT -p tcp -m tcp -s 192.168.0.0/24 --dport 873 -m state --state NEW,ESTABLISHED -j ACCEPT

Allow HTTP (TCP Port 80)

iptables -A INPUT -p tcp -m tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT

Allow HTTPS (TCP Port 443)

iptables -A INPUT -p tcp -m tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT

Allow MySQL Server Access from LAN (TCP Port 3306)

iptables -A INPUT -p tcp -m tcp -s 192.168.0.0/24 --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT

Allow Nagios NRPE Client Access from Nagios Server (TCP Port 5666)

iptables -A INPUT -s 192.168.0.100 -p tcp -m tcp --dport 5666 -m state --state NEW,ESTABLISHED -j ACCEPT

Save Current Rules in Memory to Configuration File

service iptables save

Restart Service

service iptables restart

iptables: insert a rule at a specific line number

# list the rules with line numbers

iptables -nL --line-numbers

# insert a rule at line 5

iptables -I INPUT 5 -p tcp -m state --state NEW -m tcp --dport 4000 -j ACCEPT

Related Articles: Configure iptablesiptables: insert a rule at a specific line number