ASPM on Linux#

This section is a review of ASPM and the Linux tweaks/debugging utilities available for testing ASPM. I can’t find another wiki to place this in yet on https://wiki.kernel.org/, it should be moved eventually.

ASPM review#

ASPM is a PCI-E enhancement. It allows for a device to go completely into electrically idle state, meaning it will not send or receive electrical signals for a while. To achieve this the PCI-E specification has come up with instructions a PCI-E endpoint (device) should follow for signaling to a root complex (the bus) that it is going idle, or waking up. Communication at the PCI-E bus can be tricky to align with an endpoint and because of this there are patterns a PCI-E device will use to train the link to come out of electrical idle states. There are several states a device will enter when using ASPM, namely L1, L0s.

PCIE cards should always support ASPM, what the ASPM requirements says today is that L1 is mandatory and L0s is optional unless the formfactor specifications explicitly requies it. Not sure which form factors explicitly require L0s (anyone?). Additionally software must not enable L0s in either direction on a given Link unless components on both sides of the Link each support L0s.

The way it typically works internally on endpoints (devices) is that there are idle timers (counters) in the chipset. There is a set point at which the PCIe link is idle enough to enter L0s, and a second point at which we’re idle enough to enter L1. A device could potentially ‘support’ L0s but internally the timers could be set such that L0s and L1 happen at the same time or L0s happens after L1, so the link will essentially never enter L0s. ASPM compliance may vary by device, ASPM specification has varied as new releases have been made.

Determining if you have ASPM enabled#

You can verify by using lspci -vvv as root.

When ASPM is enabled#

This is an example that has ASPM enabled:

05:00.0 Network controller: Atheros Communications Inc. AR928X Wireless Network Adapter (PCI-Express) (rev 01)
        Subsystem: Atheros Communications Inc. Device 3099
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 19
        Region 0: Memory at dbdf0000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [60] Express (v1) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM L1 Enabled; RCB 128 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [90] MSI-X: Enable- Count=1 Masked-
                Vector table: BAR=0 offset=00000000
                PBA: BAR=0 offset=00000000
        Capabilities: [100] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [140] Virtual Channel <?>
        Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
        Kernel driver in use: ath9k
        Kernel modules: ath9k

When ASPM is disabled#

This is an example output when ASPM is disabled:

localhost ~ # lspci -vvvv -s 03:00
03:00.0 Network controller: Atheros Communications Inc. AR928X Wireless Network Adapter (PCI-Express) (rev 01)
        Subsystem: Atheros Communications Inc. Device 309a
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 17
        Region 0: Memory at f0100000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [60] Express (v1) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [90] MSI-X: Enable- Count=1 Masked-
                Vector table: BAR=0 offset=00000000
                PBA: BAR=0 offset=00000000
        Capabilities: [100] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [140] Virtual Channel <?>
        Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
        Kernel driver in use: ath9k
        Kernel modules: ath9k

Why is ASPM disabled for my device?#

ASPM should automatically be negotiated by the BIOS based on all the endpoints connected on a root complex. IF your device has ASPM disabled it is likely because:

  • the BIOS determined that needed to happen

  • PCIE requires ASPM but L0s is optional so you might have L0s disabled and only L1 enabled

  • you have a buggy BIOS

  • you have no BIOS and your systems programmers didn’t address ASPM yet

ASPM for 802.11#

Testing on battery lifetime shows that the difference between having L1 and L1/L0s both could at max save up to five minutes of battery life in the best possible scenario where the 802.11 link is the most idle. This comes right down to the frequency the driver is accessing registers on the chip.

Linux kernel ASPM support#

An Operating System should not need to muck with ASPM, the BIOS would have dealt with the capability exchanges between the root complex and the different endpoints. Of course, BIOSes are buggy though – so the Linux kernel does have the capability to oversee and review the capabilities by itself and overrule the BIOS. ASPM support in the Linux kernel is also used to expose ASPM capabilities for PCIE devices to userspace (need confirmation, I see this being done in the code, but makes no sense).

You can also muck with ASPM settings to debug root complex/endpoints. This is a feature which should not be used by the average user, this is designed more for developers, choosing the wrong parameters can damage your device.

The option to debug ASPM is available through the CONFIG_PCIEASPM kernel configuration:

config PCIEASPM
        bool "PCI Express ASPM support(Experimental)"
        depends on PCI && EXPERIMENTAL && PCIEPORTBUS
        default n
        help
          This enables PCI Express ASPM (Active State Power Management) and
          Clock Power Management. ASPM supports state L0/L0s/L1.

          When in doubt, say N.

Enabling this will compile drivers/pci/pcie/aspm.c.

Force enable or disable ASPM#

This boot parameter is available:

pcie_aspm=           [PCIE] Forcibly enable or disable PCIe Active State Power
                     Management.
             off     Disable ASPM.
             force   Enable ASPM even on devices that claim not to support it.
                     WARNING: Forcing ASPM on may cause system lockups.</code>

Note that this will try to enable ASPM on all devices, check below of a way to do it for devices individually.

Enabling ASPM with enable_aspm#

enable_aspm is a script which you can use to enable ASPM for your device, once you know its address and the root complex on which its using. You can read below how to find out what root complex a device uses. The script can be downloaded from:

http://drvbp1.linux-foundation.org/~mcgrof/scripts/enable-aspm

There are only 3 parameters you will need to change:

ROOT_COMPLEX="00:1c.1"
ENDPOINT="03:00.0"

# We'll only enable the last 2 bits by using a mask
# of :3 to setpci, this will ensure we keep the existing
# values on the byte.
#
# Hex  Binary  Meaning
# -------------------------
# 0    0b00    L0 only
# 1    0b01    L0s only
# 2    0b10    L1 only
# 3    0b11    L1 and L0s
ASPM_SETTING=3

Also if you have new setpci, you should add .B after address in all setpci calls.

Enabling ASPM with setpci#

The PCIE Link Control Register is properly parsed with lspci -vvv but you might want to know exactly how to determine if your device has ASPM support manually. This is required to know exactly where to poke a PCIE device to force enable ASPM manually for a specific root complex or endpoint device. This section covers how to do this.

802.11 ASPM device tweaks#

Devices are typically not expected to have to do anything for ASPM but some tuning can be done on the PCI-E devices to reprogram L0s/L1 entrance and exit latency timers, the number of FTS packets we send to exit L0s, amongst other things.

On this page
Edit on GitHub
Show Source