PPPoE

Broadcast Control
User Authentication
Speed Control
Accounting
What is PPPoE?
 Point to Point Protocol over Ethernet
 Specification is RFC-2516
 From the specification: “It (PPPoE) is intended to be used with broadband
remote access technologies that provide a bridged Ethernet topology, when
access providers wish to maintain the session abstraction associated with
PPP.
 PPP (Point to Point Protocol) is traditionally used on serial links – things like
modems, T1’s, DS3’s, etc. PPPoE adapts the PPP protocol to a broadcast
medium (Ethernet).
 Makes available all the tools used for managing dialup users to the
broadband provider.
Advantages to using PPPoE
 Reduces broadcast traffic on the network.
 Adds privacy – customers can not talk directly to each other – only
through gateway.
 Access control – customers must authenticate before being allowed
online.
 Accounting – easy to track customer use, many options for billing.
 Speed control – set upload/download rates
 With some creative networking can be used to send late or non-
paying customers to a ‘pay up’ page – more on this later.
 Scales well - can handle thousands of users.
Who uses PPPoE?
 Nearly all DSL service is provisioned as PPPoE (or it’s close cousin
PPPoA)
 Why?
 PPPoE facilitates multiple providers sharing a single physical network.
 PPPoE works closely with the RADIUS protocol (an Authentication
Protocol we will cover shortly).
 PPPoE allows a user to specify a username, a password, and a service
name (for selecting the service provider you wish to use).
 The service name is used to route the authentication request to the
correct providers RADIUS server for authentication.
Disadvantages to PPPoE
 Requires a broadcast domain between the client and server.
 Not supported directly on Windows prior to Windows XP. Requires
(buggy) 3rd party software for 95, 98, NT, 2000, ME
 Some customers dislike the ‘dialup’ feel of a PPPoE connection.
 There are a few poor PPPoE client implementations in consumer
routers – they either fail to connect or retry extremely rapidly.
 PPPoE uses overhead in the packet of 8 bytes – the maximum
packet content is reduced from 1500 to 1492 octets. This causes
problems for sites with broken PMTUD (discussed shortly)
 PPPoE is not good on marginal links – it takes the server and client
some time to figure out a link has failed and to establish a new
connection (read this as marginal 900Mhz links that renegotiate).
Broadcast Traffic
 Broadcast traffic can be the bane of the wireless ISP. It wastes
bandwidth and broadcast storms can bring the network to a halt.
PPPoE helps to eliminate the broadcast storms.
 PPPoE uses a (very large) broadcast packet to locate the PPPoE
server.
 The server and client must be on a broadcast domain – it won’t work
through a router.
 Possible ways to do this:
 Small PPPoE server at each tower.
 Central PPPoE server, entire network one broadcast domain.
 Central PPPoE server, VLANS to each tower or AP.
 Canopy SM’s can be set to only allow PPPoE traffic.
Privacy
 PPPoE can lightly encrypt the traffic between the client and the
server providing some (very weak) protection against
eavesdropping.
 Blocking everything other than PPPoE at the SM prevents Windows
customers from seeing each other in “Network Neighborhood”.
 Yes – you really can and should click “All IPv4”, “ARP”, and “All
others” in the Packet Filter Configuration. This prevents anything
other than PPPoE from getting through.
Access Control and Accounting
 Authentication can either be done at the PPPoE server or using an
external RADIUS server.
 Local authentication checks the user against a database at the
PPPoE server.
 RADIUS (Remote Authentication Dial In User Service) was
developed to authenticate dialup users. RADIUS is in widespread
usage and can easily be modified to support additional capabilities.
 RADIUS can be used to set nearly every aspect of a PPPoE
session ie. IP Address, DNS servers, Filters, Rate Limits, Time
Limits, Transfer Limits, etc.
 RADIUS servers can proxy requests to other RADIUS servers – this
makes roaming possible.
Access Control and Accounting
 PPPoE and RADIUS are your hook into many of the tools that have
been developed by and for ISP’s to allow them to control and bill
users.
 The PPPoE server reports to the RADIUS server the details of a
connection at termination (or periodically if configured) including
time online, number of packets, etc.
 Parsing the RADIUS accounting data is a basic function of any of
the dedicated ISP billing software packages – Billmax, Rodopi,
Platypus, Freeside, etc. This allows you to bill based on usage if
so desired. Some billing packages can enforce usage limits as well
if needed (shut off the user, restrict speeds or take other action).
 Fri Jan 12 20:10:37 2007
 Service-Type = Framed-User
 Framed-Protocol = PPP
 NAS-Port-Id = 5702
 NAS-Port-Type = 15
 User-Name = "toledohealth-sm"
 Calling-Station-Id = "00:0F:66:39:D9:3D"
 Called-Station-Id = "pppoe"
 Attr-87 = "ether2"
 Acct-Session-Id = "81700f93"
 Framed-IP-Address = 64.246.97.246
 Acct-Authentic = RADIUS
 Acct-Session-Time = 86288
 Acct-Input-Octets = 5492944
 Attr-52 = "\000\000\000"
 Acct-Input-Packets = 55159
 Acct-Output-Octets = 56887685
 Attr-53 = "\000\000\000"
 Acct-Output-Packets = 75482
 Acct-Status-Type = Stop
 Acct-Terminate-Cause = User-Request
 NAS-Identifier = "MikroTik"
 NAS-IP-Address = 64.246.96.225
 Acct-Delay-Time = 0
 Client-IP-Address = 64.246.96.225
 Realm = "NULL"
 Timestamp = 1168650637
 Request-Authenticator = Verified
Sample Radius User Entries
mark Auth-Type = System
Service-Type = Framed-User,
Framed-Protocol = PPP,
Framed-IP-Address = 1.2.3.4, <- assigns a specific IP to this user
Framed-Compression = Van-Jacobson-TCP-IP,
Framed-MTU = 1500

luckfarm Auth-Type = System, Simultaneous-Use=1

Service-Type = Framed-User,
Framed-Protocol = PPP,
Framed-IP-Address = 255.255.255.254, <- special meaning – assign a IP address from the pool
Framed-Routing = None,
Framed-MTU = 1500,
Framed-Compression = Van-Jacobson-TCP-IP
Complex RADIUS user entry:
hia Auth-Type = Local, Password = "none", Simultaneous-Use = 1
Session-Timeout = 43200,
Service-Type = Framed-User,
Framed-Protocol = PPP,
Framed-IP-Address = 65.165.122.145,
Framed-IP-Netmask = 255.255.255.240,
Framed-Route = "65.165.122.144/28 65.165.122.145 1",
Filter-Id = helm,
Framed-MTU = 1500,
Framed-Compression = Van-Jacobson-TCP-IP
Speed Control
 Microtik PPPoE servers (probably all PPPoE servers) can be set to apply
specific upload/download speeds to any PPPoE session.
 Example rate limit set from RADIUS:
mark Auth-Type = System
<removed>,
Mikrotik-Rate-Limit = "128k/512k 384k/2048k 256k/1024k 60/60 8“

This will give the user to 512k down, 128k up with a burst rate of 2M/384k. The burst
threshold is 1M/256k. The burst is limited to a 60 second window. The queue
has a priority of 8
PPPoE Servers
 There are several PPPoE server implementations:
 Mikrotik – inexpensive, smaller deployments (hundreds to a few
thousand users).
 ServPoET (midprice – up to 6000 simultaneous PPPoE sessions)
 Redback (expensive – thousands of sessions)
 Cisco (making Redback look inexpensive – as big as you want)
 Free/OpenBSD – runs in userland, pretty slow. not recommended.
 Others?
 Amplex uses the Mikrotik Implementation
 Low cost
 Reliable
 Plenty of people in the WISP community are familiar with it.
 Mikrotik includes many other useful features (other than PPPoE).
PPPoE Startup and Redundancy
 Multiple PPPoE Servers can exist on one broadcast network.
 The client (customer) side initiates a connection with a PADI
(PPPoE Active Discovery Initiation) packet.
 The servers respond with a PADO (PPPoE Active Discovery Offer)
– the client chooses one server from the (possibly multiple)
responses it sees. Usually the first responding server is chosen –
making for a crude load balancing mechanism.
Mikrotik PPPoE Configuration
Setting up a Mikrotik PPPoE Server
 Were going to assume you have a working Mikrotik installation with
at least 2 Ethernet interfaces.
 The Mikrotik can be set up completely from the command line or
using the GUI interface.
 We are going to put the gateway (connection to the Internet) on
Ether1 and the PPPoE server on Ether2.
 Mikrotik recommends NOT putting an IP address on the PPPoE
interface for security reasons – we routinely ignore this
recommendation.
 Amplex assigns public IP addresses to all of our customers – you
can also use private IP space and NAT if desired with PPPoE.
Create an IP Pool to assign to PPPoE customers

[admin@MikroTik] ip pool> print

# NAME RANGES
0 pppoe-pool 64.246.97.112-64.246.97.255

[admin@MikroTik] ip pool> ip pool add creates new item with specified property values.

name -- Name of the IP pool

ranges -- IP pool's ranges
next-pool -- Next pool range
copy-from -- item number

[admin@MikroTik] ip pool> ip pool add poolname 1.2.3.4-4.5.6.7

Create a PPPoE profile on the PPP tab
[admin@MikroTik] ppp> profile print
Flags: * - default
0 * name="default" use-compression=default use-vj-compression=default
use-encryption=default only-one=default change-tcp-mss=yes

1 * name="default-encryption" local-address=64.246.96.225
remote-address=pppoe-pool use-compression=no use-vj-compression=no
use-encryption=yes only-one=default change-tcp-mss=yes
dns-server=64.246.100.1,64.246.115.1,64.246.109.1

2 name="512-1024" use-compression=yes use-vj-compression=yes

use-encryption=yes only-one=yes change-tcp-mss=yes
dns-server=64.246.100.1,64.246.115.1,64.246.109.1
rate-limit=1200000/536000
Create the PPPoE Server:
[admin@MikroTik] interface> pppoe-server server print
Flags: X - disabled
0 service-name="pppoe" interface=ether2 max-mtu=1492 max-mru=1492
authentication=pap keepalive-timeout=10
one-session-per-host=yes max-sessions=0 default-profile=default-encryption
[admin@MikroTik] interface>
Create local users if not using RADIUS:
[admin@CanopyNorth] ppp secret> add name=test1 password=test service=pppoe profile=default

Use the print command to see a summary of users or print d for a detailed view.
[admin@CanopyNorth] ppp secret> print
Flags: X - disabled
# NAME SERVICE CALLER-ID PASSWORD PROFILE REMOTE-ADDRESS
0 mark pppoe xxxxxxxx default 172.23.0.10
1 dobbs pppoe xxxxxxxx default 172.23.0.11

[admin@CanopyNorth] ppp secret> print d

Flags: X - disabled
0 name="mark" service=pppoe caller-id="" password="xxxxxxxx" profile=default
local-address=72.23.0.10 remote-address=172.23.0.10 routes="" limit-bytes-in=0
limit-bytes-out=0
1 name="dobbs" service=pppoe caller-id="" password="xxxxxxxx" profile=default
local-address=72.23.0.11 remote-address=172.23.0.11 routes="" limit-bytes-in=0
limit-bytes-out=0
Or if using RADIUS:
[admin@MikroTik] ppp> aaa print
use-radius: yes
accounting: yes
interim-update: 30m

[admin@MikroTik] ppp> aaa set ?

use-radius -- Use or not radius

accounting -- Enable/Disable accounting
interim-update -- Defines time interval between communications with the router

[admin@MikroTik] ppp>
Tell Mikrotik where to find the RADIUS servers:

[admin@MikroTik] radius> add

creates new item with specified property values.

service -- Name of the service

called-id -- Called identity
domain -- The domain of the radius
address -- The address of radius
secret -- PPP secret name
authentication-port -- Default port 1645 to RFC
accounting-port -- Radius accounting port
timeout -- Time limit how long the radius client will try to connect to the radius server
accounting-backup -- Radius accounting backup
realm -- Explicitly stated realm (user domain)
copy-from -- item number
place-before -- item number
comment -- short description of the item
disabled --

[admin@MikroTik] radius> print

Flags: X - disabled
# SERVICE CALLED-ID DOMAIN ADDRESS SECRET
0 ppp 64.246.100.1 someothersecret
1 ppp 64.246.109.1 supersecretpassword
[admin@MikroTik] radius>
That’s it – you should have a working
PPPoE server
 Test it with a Windows XP machine or a router.
 In Windows create a connection just like you would create a dialup
connection except pick “Connect using a broadbrand connection
that requires a user name and password”.
 The PPPoE server can be plugged into an existing broadcast
network without disturbing your existing non-PPPoE setup.
 New customers can be provisioned with PPPoE, existing customers
can be moved to PPPoE as needed.
 Make sure you have a method to route the IP addresses you are
assigning from your PPPoE pool to the Mikrotik – either using static
routes in your router or automatically using OSPF.
PMTUD
How to break your network in odd ways…

 An ugly acronym… Path Maximum Transmission Unit Discovery

 PMTUD is a method that network devices use to discover the maximum
packet size for an end to end connection – nominally a MTU of 1500 bytes
on Ethernet.
 How it works:
 The transmitting device sends a full size packet (whatever it’s maximum is) with
the ‘don’t fragment (DF)’ TCP flag set.
 This tells routers along the way that they can not break the packet into two
smaller (fragmented) packets.
 If a router gets a DF packet and is set for a smaller MTU (like your PPPoE server)
then it returns an ICMP packet Type 3 Code 4 – “Fragmentation Needed & DF
Set” to the original router.
 The original router, on receipt of this ICMP packet, then reduces it’s packet size
and tries again. This continues until the sender and recipient agree on a MTU
size.
PMTUD
Why your network breaks in odd ways…
 Why do we need PMTUD to work when using PPPoE?
 Remember the part about PPPoE overhead? Your using reduced
packet sizes – probably 1492 bits long with PPPoE
 How do you break PMTUD?
 Block ICMP Unreachable (or all ICMP) packets.
 Use private IP addresses for routers along the path – ICMP unreachable
packets coming from private (RFC1918) IP space are generally blocked
on the Internet.
 What happens when PMTUD is broken?
 Web pages load slowly and/or refuse to show large images.
 Secure (https) web pages will not load.
 Really confuses some VPN implementations.
 Customers complain your blocking PayPal (it’s the end of the world as
we know it…)
 DON’T break PMTUD if you are using PPPoE. Better yet – don’t
break it in any case.
Canopy Surge
Suppressors

GPS Junction
Box – 1 and 6
Jumped on RJ-11’s
Typical
Mikrotik RB532 Wireless Site
With 564 Enclosure
Daughterboard
Main Power Supply
Homebrew Canopy
Power Injector – Battery Charger
Ethernet lines to Relay – batteries
Mikrotik, Power lines vs. line power
to 24VDC
SyncPipe Power

Relay – Ethernet
loopback,
2 24VDC Sealed detects mains
Lead Acid Batteries power failure
WISP Sitebox
Schematic
Dear User – pay your bill
How to suspend users without turning them off completely

 Here is a good trick for non-paying users… can be used anytime you can
control the IP assigned to the customer:
 Have your billing system assign a private IP address to suspended users – our
billing system creates the RADIUS file for us automatically with private IP’s for
suspended users.
 Have the router policy-route traffic with a private source address to a Unix box.
 Use the kernel firewall (ipfw) to rewrite packets arriving at the Unix box from the
private source IP addresses to the destination address of the Unix host.
 Have Apache serve up a web page of your choice.
 Email and other local services continue to work – but any web page they try to go
to returns your ‘pay up’ webpage.
 This method cuts down on customers that think something is broken with your
network when you have intentionally shut them off.
Cisco and Unix Configurations:
Create a filter list to match the traffic you want to policy route:

access-list 192 remark Suspended User HTTP Redirect

access-list 192 permit tcp 192.168.0.0 0.0.0.255 any eq www

(matches any http traffic with a source address of 192.168.0.0 to 192.168.0.254)

Create a route map to catch the traffic:

route-map AmplexServerPolicy permit 10

description Send Suspended users to Suspended page
match ip address 192
set interface FastEthernet0/0/0.1
set ip next-hop 64.246.100.1

(sets the next hop of any http packets coming from 192.168.0.xxx addresses to 64.246.100.1)

Apply the route-map to the customer facing interface:

interface FastEthernet5/0/0.1
description Wireless LAN
ip policy route-map AmplexServerPolicy

(applies the policy to the interface)

Cisco and Unix Configurations:

 On a FreeBSD box:
junior# ipfw list
00100 fwd 64.246.100.1 tcp from 192.168.0.0/24 to any dst-port 80
65535 allow ip from any to any
junior#

 In your Apache config:

<VirtualHost *>
ServerAdmin support@amplex.net
DocumentRoot /usr/local/www/suspended/
ErrorDocument 404 /index.html
# TransferLog /usr/local/www/suspended/logs/suspended-access_log
# ErrorLog /usr/local/www/suspended/logs/suspended-error_log
</VirtualHost>

 So what happens? HTTP requests for any page made from clients with private IP addresses are shown the web
page you place in /usr/local/www/suspended/index.html
 Our page just tells them the billing system has suspended them – they have to call or send money to get back
online. This could be integrated with a payment method but we have not done this yet.