Deep Packet Search and Screening - Civil Liberties Issue?

[INRM]

Currently the ISPs use deep-packet screening (not sure if the terminology's correct exactly)... basically scanning every single piece of data heading towards a given individual, tracking it, and if necessary automatically re-routing the data away from the user. And this is done with every single individual. Total tracking of every single message and action carried out by every single user. While I understand it has useful applications in screening out malware and such things, the broad scope and nature of this is kind of disturbing. This strikes me as completely Orwellian.

Additionally the US Government has actually ordered ISP companies to essentially (I'm not sure the exact wording) but basically structure their computer and network set ups to essentially make them easier for the government to wiretap as well.

This technology (and the US Government ordering networks to essentially be "wiretap" friendly, etc) I wonder if it really is in line with the constitution. Such technology can be used to allow online censorship as well -- China uses the technology to keep people from getting any information on religion, human rights violations and such, for example.


INRM

[TobiasTheViking]

I think you are overestimating how far and wide deep packet filtering goes.

No, the ISP's aren't checking each and every packet from every user on their network.

They aren't tracking everything everyone on their network does.

Yes, the government might have asked them to make it easy to wiretap connections, but again, this is not something being done automatically.

And the comparison to the great firewall of china doesn't really fit either..

Let me explain.

Yes, the ISP's have the ability to find out quite a bit about your internet habits by examining the packets that go from you, through them, to other servers on the internet... But this is a humongous task, that requires quite a bit of processing power.

I do not believe it is possible for any relatively big ISP to actually say "monitor for all packets on the bittorent network, and log their ip and access time" or "monitor for all packets having to do with subject X and log their ip and access time".. The processing power involved is far too big.

What the packet filtering is used for is, and this is paramount... QoS.. Quality of Service... They use packet filtering to say "P2P/Torrent/VoIP" will always be 2nd priority compared to ports 25 and 80(email and normal browsing).. And comcast has used packet filtering to make torrents, for instance, run really really slow.

Any real inspection can be done, on a per line basis, but not as a general blanket on all connections.

Usually, an ISP will simply monitor your bandwith usage, if it is too high they might(but usually won't) check what kind of traffic it is, and warn you, or disconnect you, for using too much traffic.

Now, the wiretap the government have asked for(i have not heard of this, but it wouldn't surprise me if it is true, so i will just assume it is).
The wiretap the government have asked for would only be akin to them asking the cell/land-line providers to "make it easy and fast for us to wiretap a connection", not "monitor all phone calls for the words X and Y".. Again, the processing power involved in this is far to great for it to be a feasible implementation.

This can all be done on a local network, of course, but on such a big scale... Sorry, i do not believe it.

And last, the comparison to the great firewall of china..

The internal network of china(as far as i understand it, feel free to correct me Wolfman) is closely monitored, and not really anonymous for when you post anything online.. BUT, again, no filtering is done for traffic as it happens.

What usually happens(and this is conjecture, i admit), is that the people who have to make sure nothing "offensive" is online, will look online, browse, monitor, etc, and when they find something offensive they will make sure it is taken offline, and then use the IP information to get to the person who posted this(this is rather easily doable, and could be done by the US government in most cases. I am quiet sure that this forum tracks ip's and stores them, and that the hosting server tracks them as well in a different log)..

But this is all reactive, not proactive.

This is not "Censor as it happens, by monitoring" but rather "censer afterwards and catch the culprit"

Now, for connections going from china through the firewall to the outside, that is a different matter..

Most of the traffic on the chinese internet is internal, that is, almost all of the data packets being send are within their own network, in a way, separated from the rest of the internet.

What little goes out(and by comparison it isn't a lot) is monitored(again, as i understand it). But because so much of it is internal, and handled in a quite different way from the way it is in america. The connections going through the firewall are quite miniscule in size..

It is still a lot, and i doubt they do Deep Packet Searches as such.. i assume(and this is all) that they log which ip connects to what address outside the server, and then check what is on said address, block it if need be, else leave it open(probably add it to a whitelist so the same pages aren't checked over and over).

Also, since some people from the inside would want to view uncensored material through the firewall they will use a proxy of some sort.. In this case i think it is again simply a matter of logging source and destination ip, seeing how much traffic is transfered, if it crosses a specific tresshold start deep packet searching just the connections going to that ip(if they even bother with this step), and then block it if it is not "in the best interest of the people".

I am not saying that everything i have written is the truth. It is simply the most logical way to implement what they want to implement. And it is extrapolated from certain limitations in hardware, etc...

I am sure i have missed something. The specifics are not what i think is important from my post, more, the general method and applications.

I don't think you have anything to worry about as such.

The packet filtering done by ISP's is done to remove the people that use the most traffic from the network, to save money. (it is usually the case that 5% use about 90% of the bandwidth, or so). Nothing worse than that.

The wiretap is not a general thing, more of a "make it easy and fast for us to wiretap". Which i think is a reasonable requirement, as long as the wiretap isn't started without a proper court order(which has nothing to do with the ISP, and everything to do with politics and the police).

Implementing censorship is extremely hard, especially if it must be done proactively, realtime. And honestly, unless you want to image NWO with about the same computer power as the rest of the world combined, it isn't really feasible(imo).

Feel free to correct me, this is just how i see it, and my opinion on the subject.

Sincerely
Tobias The Commie (this message brought to you from the ProPaganda Network)

[PixyMisa]

Some ISPs really are doing this, Tobias. There's been a huge fuss recently about ISPs working with NebuAd to insert targeted ads in web pages. http://news.google.com/news?q=nebuad

It's an enormous breach of privacy, of course, and the ISPs involved are now backing away from NebuAd as fast as they can.

[INRM]

I think people need to make a stink to the EFF and ACLU and such organizations regarding such things, and need to contact their representatives (keep it civil of course) about this issue.

This serious

[shadron]

Come on on now. It may be possible to tell in real time what all of an ISP's customers is doing at that depth, and to record every packet going to or from one or a few of their customers, but there is no way that recording could be done for everyone, all the time. You would need the processing power of the LHC many times over to do that, and then, what would you do with it? Write it all to DVDs? And what then? Now that you have it on a physical media, at the rate of, perhaps, tens of thousands of DVDs per hour, every hour, 24/7/365, you have to index it, handle it, store it and retrieve it, if it's to be useful. You can't write that kind of bandwidth to a single DVD - you'd have to split it into bits and write bits in parallel to a lot of devices. Then, to reconstruct a conversation, you'd have to find the right 100 or so DVDs and read them all back in and rebuild the conversation. What you are asking to be done simply beggars the imagination - take the entire volume of the internet and write it to a medium, adding time stamps and indexing overhead.

Dream on.

[PixyMisa]

I think people need to make a stink to the EFF and ACLU and such organizations regarding such things, and need to contact their representatives (keep it civil of course) about this issue. This serious Congress is already investigating NebuAd and the ISPs involved.

[Nick Bogaerts]

What the packet filtering is used for is, and this is paramount... QoS.. Quality of Service... They use packet filtering to say "P2P/Torrent/VoIP" will always be 2nd priority compared to ports 25 and 80(email and normal browsing).. And comcast has used packet filtering to make torrents, for instance, run really really slow.

Comcast went a bit futher than that...

They used deep-packet inspectiion to analyse the origin of P2P traffic, then forging TCP RST packets to block (not "slow down") the connections.
They did not do this to "prioritize" connections, since this behaviour was present at all times of day, even when the network was not congested.
They then lied to their customers about what they were doing.
Then then lied to the FCC about what they were doing.
Then then rented a crowd to fill up an FCC hearing on the matter to prevent critics from getting seats and voicing their concerns.

If you are in the least concerned about net neutrality and are a Comcast customer, I suggest you change your ISP.

Here in the UK, I suggest you boycott BT (for their illegal Phorm trials) and Virgin Media (for their attacks on net neutrality).

[TobiasTheViking]

Some ISPs really are doing this, Tobias. There's been a huge fuss recently about ISPs working with NebuAd to insert targeted ads in web pages. http://news.google.com/news?q=nebuad

It's an enormous breach of privacy, of course, and the ISPs involved are now backing away from NebuAd as fast as they can.
There is a difference between deep packet searching, and searching on port 80(http).. http transfer isn't really that much of the data being transfered.

It is quite doable to do some simple regex on normal http streams with current hardware.

Comcast went a bit futher than that...

They used deep-packet inspectiion to analyse the origin of P2P traffic, then forging TCP RST packets to block (not "slow down") the connections.
They did not do this to "prioritize" connections, since this behaviour was present at all times of day, even when the network was not congested.
They then lied to their customers about what they were doing.
Then then lied to the FCC about what they were doing.
Then then rented a crowd to fill up an FCC hearing on the matter to prevent critics from getting seats and voicing their concerns.

If you are in the least concerned about net neutrality and are a Comcast customer, I suggest you change your ISP.

Here in the UK, I suggest you boycott BT (for their illegal Phorm trials) and Virgin Media (for their attacks on net neutrality).

They were sending disconnects to the torrent clients, not inspecting the package... again, there is a difference between looking at the header, and the data.. and comcast simply looked at the ip in the header, then forced a disconnect..

The rest i agree with though, and comcast did blow it.

[INRM]

Nick Bogaerts,

Comcast went a bit futher than that...

They used deep-packet inspectiion to analyse the origin of P2P traffic, then forging TCP RST packets to block (not "slow down") the connections.
They did not do this to "prioritize" connections, since this behaviour was present at all times of day, even when the network was not congested.
They then lied to their customers about what they were doing.
Then then lied to the FCC about what they were doing.
Then then rented a crowd to fill up an FCC hearing on the matter to prevent critics from getting seats and voicing their concerns.

Now you all see why I have objections to such things?


INRM

[shadron]

Now you all see why I have objections to such things?

INRM

So, INRM, what is your proposal? Pass legislation preventing Comcast from looking at IP headers? Remember, they have to do that get traffic properly routed; that is why the header data is there. They have to do statistics gathering to support their ability to make a profit, to know where additional hardware is needed and where it is not. What you want to prevent is to keep them from copying packets to another user (wiretapping) and blocking channels when they deem someone is taking up exceptional bandwidth resources (mostly video downloading, and mostly illegal, not that the ISP especially cares until the law says it must).

So, by all means, carry on. However,

Currently the ISPs use deep-packet screening (not sure if the terminology's correct exactly)... basically scanning every single piece of data heading towards a given individual, tracking it, and if necessary automatically re-routing the data away from the user. And this is done with every single individual. Total tracking of every single message and action carried out by every single user. While I understand it has useful applications in screening out malware and such things, the broad scope and nature of this is kind of disturbing. This strikes me as completely Orwellian.

seems just a little overwrought, and a lot inaccurate. If you're going to convince anyone, you'd better get the story, and the emphasis, straight.

[Nick Bogaerts]

So, INRM, what is your proposal? Pass legislation preventing Comcast from looking at IP headers? Remember, they have to do that get traffic properly routed; that is why the header data is there.

An IP router needs to look at the IP header, obviously. Deep Packet Inspection refers to looking at the levels below that, in this instance the TCP protocol used by Bittorrent.
Comcast even went one step further, and started forging TCP datagrams.

[ddt]

Additionally the US Government has actually ordered ISP companies to essentially (I'm not sure the exact wording) but basically structure their computer and network set ups to essentially make them easier for the government to wiretap as well.

This technology (and the US Government ordering networks to essentially be "wiretap" friendly, etc) I wonder if it really is in line with the constitution. Such technology can be used to allow online censorship as well -- China uses the technology to keep people from getting any information on religion, human rights violations and such, for example.

I don't know about the US constitution, but this happens the same in Europe - even worse. The EU has mandated that ISP's tap and store "traffic data" - not so much the contents of the packets, but who initiates what traffic to whom. So from these data, the government can see that I sent an email to John Doe last Friday - not what was in the email. The EU has mandated that ISP's retain these data for 6 months; the Dutch government was so eager they have upped it for Dutch ISP's to 18 months.

As to the US, I wouldn't have any illusions that the tap possibilities are not only legally used by the FBI but also illegally (extrajudicial? :p) by the NSA.

If you care for your privacy: use encryption. Use Tor or similar networks.

[shadron]

An IP router needs to look at the IP header, obviously. Deep Packet Inspection refers to looking at the levels below that, in this instance the TCP protocol used by Bittorrent.
Comcast even went one step further, and started forging TCP datagrams.

OK, I can see what you're saying; they're doing a partial message rebuild so they can look at the higher level protocol. As someone pointed out above, that all takes processing power, and can't possibly be happening for everyone.

Thanks for the deeper education, Nick.

Additionally the US Government has actually ordered ISP companies to essentially (I'm not sure the exact wording) but basically structure their computer and network set ups to essentially make them easier for the government to wiretap as well.

No, the gov didn't order it, though some implicit threat may have been offered through regulatory control. Making them do it would have gotten the ISPs off he hook for liability, and would also have made the act non-covert; in the Pres's words, they offered to help in patriotism, but now they are left holding the bag despite whatever the administration may have promised. You can bet that, if legislation to cover future wiretapping isn't passed, it will be a cold day in hell that they will ever agree again on the same basis.

[TobiasTheViking]

OK, I can see what you're saying; they're doing a partial message rebuild so they can look at the higher level protocol. As someone pointed out above, that all takes processing power, and can't possibly be happening for everyone.

That's not what i said..

Comcast can do that, and did that.. though it wasn't really deep packet filtering, they just looked at the packet headers (ip, port, protocol), if protocol = bittorent, then send disconnect packet... Ain't even close to being a deep packet filter. And it is quite doable without too much processing power.

[Nick Bogaerts]

Comcast can do that, and did that.. though it wasn't really deep packet filtering, they just looked at the packet headers (ip, port, protocol), if protocol = bittorent, then send disconnect packet... Ain't even close to being a deep packet filter. And it is quite doable without too much processing power.

It's a little bit more complicated than that. The bitTorrent protocol runs over a TCP socket-- two (IP address, port number) pairs (one for each side). The TCP datagram itself is encapsuated within an IP datagram. Legitimately, any router other than the two at either end of the TCP connection should only be interested in the IP header. Not the TCP header. Now even then, there's no way, from a single TCP packet, to know that that packet forms part of a BitTorrent stream.

P2P in general however, follow extremely distinctive patterns of TCP connections -- typically a large number of TCP connections to and from high port numbers to different IP addresses.

What Comcast have is a set of heuristics designed to detect these kinds of connections, and whenever a bitTorrent client tried to connect to a Comcast customer's machine, it sent back a forged TCP RST packet, claiming to come from the customer's IP address and TCP port for that connection, with sequence numbers (used to keep track of the order in which packets are sent) the other machine woud have been expecting.

[PixyMisa]

There is a difference between deep packet searching, and searching on port 80(http) Not in this case. They were doing deep packet inspection. That's what NebuAd does.

[TobiasTheViking]

Wait, nebuad was doing deep packet inspection on all packages? and not just on http packages??

Because if they did it on all(which i seriously doubt) that would be really scary

[PixyMisa]

Wait, nebuad was doing deep packet inspection on all packages? and not just on http packages??

Because if they did it on all(which i seriously doubt) that would be really scary
We only know for sure they were scanning HTTP, but my point is that they were scanning the contents of the packets, not just the headers.

[TobiasTheViking]

We only know for sure they were scanning HTTP, but my point is that they were scanning the contents of the packets, not just the headers.

Yes, only on http traffic, which is rather a minuscule amount of traffic

[Nick Bogaerts]

Yes, only on http traffic, which is rather a minuscule amount of traffic

Only, like, 45%.

[shadron]

I suppose, like anything else, it depends on what you're measuring:

- the number of packets dedicated to the protocol
- the number of data bytes passed
- the number of TCP/UDP connections
- whatever other measurements might be possible.

My understanding is that the majority of the packets on the net were due, one way or anther, to video - streaming, or bittorrent, or other file copying protocol. Of course, most, if not all, of those are initiated from an http page. So, y'all are going to need to define your terms before you get into this one.