ITinfo: Search Engines Also Record Private Data

Site Directory
Membership
Train-the-Trainer
Trainer Certification
Certified Training Materials
ITinfo E-zine
Responsible Training
White Papers
Trainer Resources
What's New
Speaking Engagements
Onsite Training
ITrain Gear

Search Engines Pick Up Everything

Hmm, that's looks like an interesting search hit!

ITinfo Sponsor

ERROR: Random File Unopenable

The random file, as specified in the $random_file perl variable was unopenable.

The file was not found on your file system. This means that it has either not been created or the path you have specified in $trrandom_file is incorrect.

Search Engines Also Record Private Data

by Dave Murphy
ISSN 1535-3613

Search engine spiders, the software programs that crawl through millions of webpages each day indexing new and modified pages are finding more than webmasters would like. The spiders will record just about everything they find, including passwords, credit card numbers, classified documents, and other private information that the sites' webmasters never intended to be indexed.

Most popular search engines, such as Google, AltaVista, HotBot, Lycos, and Northern Light, will pick up webpages created in HTML (HyperText Markup Language), ASCII text, and, increasingly, PDF (Adobe's Portable Document Format). Unless documents are secured in protected directories or are included in a "robots.txt" instruction file on the website, the search engine's crawling bots will read the documents and include them in their master index that can then be searched by anyone with access to the Internet.

Recently, webmasters have found that other document formats are showing up in the major search engines: word processor files, spreadsheets, graphics, and other binary files that were posted to websites for easy access by authorized employees.

In most instances when sensitive data turns up the search engine databases it's the fault of an untrained web designer. Webmasters frequently use CGI (Common Gateway Interface) scripts to execute commands behind the scenes of a website. Unless the CGI programmer is aware of potential security vulnerabilities in his script, he may be leaving a gaping hole in the site's security. For example, a CGI script that collects and stores credit card data in an unprotected ASCII (American Standard Code for Information Interchange) file may leave the data open to a search engine's crawler. Using an MySQL database on a separate server and a web-interface such as PHP, both of which are available for free, would add a layer of security to the credit card data that would prevent search engines from locating and indexing the data.

Dave's Opinion

I'm careful to check out online retailers before I enter any private information on their websites. Often, I'll call the retailer and get a feel for how they do business. I often ask to talk to their webmaster and ask about his security practices. A few rules I follow: 1) try to buy only from large retailers, 2) check references for making my first purchase, 3) add my office address as a second shipping address to my credit card, and 4) have all shipments delivered to the office.

And, if you're thinking that the robots.txt fill will solve all your problems, consider this: the robots.txt file will only turn away crawling bots that comply to standards; not all are compliant. Also, the robots.txt file can be a clue to crackers as to which directories may hold the more interesting files.

Creating a secure website takes a bit of knowledge and a bit of skill.

Call for Comments

What do you think? Leave your comments on the message center.

References

Google
AltaVista
HotBot
Lycos
Northern Light
HTML
CGI
MySQL
PHP
Message Center

Previous issues are on our website at http://itrain.org/itinfo/.

International Association of Information Technology Trainers
PMB 616
6030-M Marshalee Dr
Elkridge, MD 21075-5987

410.567.5366
1.888.290.6200
fax: 801.650.0423
Membership Director: member@itrain.org

Return to ITrain Homepage

http://itrain.org/itinfo/2001/it011126.html
updated November 26, 2001