One server and multiple ips

ddzz2013 asked 2 years ago

Hi,

I have only one server which is used to crawl a few large sites. I have been given a number of static IPs from my ISP. I would like to use all of the given IPs while I have only one server. Can I assign randomly or sequentially one of the IPs to my server at any given time?

My purpose, firstly, is to avoid being blocked, if any, by the external websites. Secondly, I may have different virtual machines installed in the server such that one virtual machine can use one IP.

However, any solution should not comprise the speed of crawling.

Please help!

Thanks.

1 Answers
Shnerdly Staff answered 2 years ago

Thanks for your question ddzz2013.

I am unable to answer your question about using multiple external IP's on a single server though I am interested in the answer. I'm sure it can be done, it's just beyond my abilities.

I would point out that when an ISP assigns multiple blocks of IP Addresses, they are usually in a CIDR Block which makes them sequential. I'll use internal IP's as an example.

If you were assigned 172.20.133.0/24, that would get you all addresses on the 172.20.133. subnet with the exclusion of a couple for overhead. Usually 172.20.133.0 would be the IP of the ISP's server that you connect through and I think 172.20.133.255 is usually used for broadcast so it is also unavailable. There may also be some other exclusions.

My point it that when I find an undesirable crawler working my servers, I usually block their entire CIDR Block.

Know the answer? Login or sign up for an account to answer this question.
Sign Up