Ultimate Guide to Understanding HTTP Status Codes – 400 Series

Issues with a user-agent client can range from requesting resources that are no longer available (by mis-typing a URI) or even the server reacting to requests fitting a security-attack signature.

The most common is the 404 – document not found. 404 Document Not Found is returned when a resource is requested the web-server cannot find:

404 Document not Found when mistyping a URL

Let’s go over the most common and interesting 404 series response codes in a little more detail.

400 – Bad Request

400 Bad Request is (usually) the result of a user-agent sending something in the request a web-server could not process. Let’s generate a 400 Bad Request to understand what can cause this error.

We will go old-school on Linux and use a tool called netcat. Netcat can be used for sending raw data over sockets. After all, HTTP requests are just that – raw text.

First, let’s send a legitimate HEAD Request to google:

The web-server responded with 200.

Let’s force a 400 – Bad Request:

We purposfully sent a request the server could not understand – let’s look at the request method used:

mint ~ # nc www.google.com 80
BAD / HTTP/1,1
Host: www.google.com

The web-server could not process the method: BAD

It’s unlikely a modern web browser would send malformed requests like depicted above. However, if you’re using a custom developed HTTP application like a web-scraper, it’s time to debug.

Following the procedures below will resolve most all 400 Bad Request Errors.

1. Check The URL

If clicking on a link it is quite possible the web-author made a typo. Examine the entire URL paying particular attention to the path, document title, and query-string portions.

2. Clear all Cookies and Cache

Clearing the browser cache will solve 80% of 400 Bad Request Codes. Following these steps is a good universal fix for a lot of 400 Status Codes, too.

1. clear all cookies and browsing history
2. re-try the request

Let’s see how this is done with Chrome:

1. Go into Settings 

   

2. Go to Advanced

   

3. Click “Clear Browsing Data”

   

4. Select the following options

   

Clearing all cookies and browsing history will assure anything cached and/or corrupted (like cookies) are gone. More than likely this will fix your issues with most 400 Bad Request Codes

3. File Upload Size

The problem could very well be a back-end script itself, uploading too large of a file or a file with an incorrect media type.

1. If the 400 bad request presents itself while uploading a file, try a smaller file.
2. Make sure the file-type is acceptable. 2.
3. If uploading an image try both jpeg and png formats.
4. If an uploading a document usually text files .txt and Microsoft Word files are permitted.

4. Malformed Headers

Malformed headers would be a pretty low-level issue and could very well leave the web-application itself at fault.

Using the HTTPFox plugin with Firefox we can watch the entire HTTP transaction for anything looking out of the ordinary. 

Looks as if I need to clear the cache on my web-browser. Pretty simple to diagnose with HTTPFox.

5. Update Your Web Browser and Disable Plugins

Like anything HTTP goes through versions and revisions. If your browser is out of date it could very well be sending antiquated headers and requests.

Also, try removing or disabling all plug-ins and extensions. Trying the same request in another web browser can assist in isolating issues as well.

6. Check Your Internet Connection

A bad Internet Connection can cause data to be corrupted at lower layers, thus sending malformed data to the web server.

Using a good Internet Speed Test utility can help troubleshoot any issues with an Internet connection. Try running the utility 5 or 6 times for comparison.

7. Try Logging Out

If using a web application with user-based authentication and sessions try logging out and then clearing the browser cache.

401 – Unauthorized

401 Unauthorized is returned when a web server requests credentials for a secured document.

The HTTP Response will be 200 upon valid credentials being supplied

403 – Forbidden

403 Forbidden is usually one of four things:

• directory listings are not permitted
• currently authenticated user does not have sufficient permissions
• the web-server process has inadequate permissions
• requesting address has been banned

Directory Listing Not Permitted

Shows 403 Forbidden when trying to access a directory listing

IIS Showing Directory Permissions are not granted to unauthenticated users. The simplest method is to put an index file in the directory. Otherwise, directory listings to unauthenticated users will need to be explicitly set.

Authenticated User Does Not Have Permissions

The best method in this situation is making a group for all users who need access to a resource. Give this group the necessary permissions at the filesystem level. Then add permitted users to the group.

Web Server Process Does Not Have Sufficient Permissions

We are unable to access the test.php page in the webroot in /var/www/html. So first we need to:

• check permissions to the webroot:

• assign the www-data group to the webroot and give read + execute permissions:

   mint# ls -ld /var/www/html/
   drwx------ 3 root root 4096 Nov 24 03:47 /var/www/html/
  

  the /usr/var/html/ directory is owned by the root user and root group with no access to any other user.

  To fix a permissions issue like this, we need to modify permissions to the directory:

   mint# chgrp -R www-data /var/www/html
   mint# chmod 750 /var/www/html
   mint# chmod g+rx /var/www/html/test.php 
   mint html # chmod g-w /var/www/html/test.php 
   mint# ls -ld /var/www/html/
     drwxrw---- 3 root www-data 4096 Nov 24 03:47 /var/www/html/
   mint# ls -l /var/www/html/test.php 
      -rwxr-x--- 1 root www-data 91 Nov 16 06:52 /var/www/html/test.php
  

A little theory on how the problem was fixed:

1. change the owning group of the webroot to www-data
2. give proper permissions to the web-root for the www-data group. directory should be read and execute
3. give read and execute permissions to test.php for the www-data group

Permissions, as mentioned above, are for a production environment. But can be somewhat restrictive for development stages.

Since we took away write permissions to the directory and to test.php only root can add and edit files. This is good in production – we are 100% sure the web-process cannot write to and create files if a script is compromised.

However, this is very impractical for development. In a staging environment, it would be best to put developers in the www-data group. Then give them write permissions to the web-root and all its files. This is a good example of why modern DevOps keeps staging and production segmented.

Another common scenario for a 403 Forbidden Status may be an IP Address or IP Range being banned in a .htaccess file by the server Administrator.

404 – Not Found

404 Not Found means the web-server could not find a requested resource. When encountering a 404 error a few things can be done:

• check spelling of host, domain, path, and document

• check case of file and path. Linux and Unix web-servers (unless configured differently) usually distinquish case, with tradition of using all lowercase for each.

• use Google to find an alternative URL to the resource

• try getting a cached page from google

Using Advances Google Operators

Many times I have been able to locate a document with expired links (resulting in a 404) using a Google query similar to the one above. Listing of Advanced Google Operators

Try Getting the Same Resource From Google’s Cache

Using advanced Google Operators and Google’s Cache you may be able to find the resource even if it has been taken down.

405 – Method Not allowed

A 405 Method Not Allowed is presented when attempting a request method that is unpermitted by the web-server.

OPTIONS Request Not Allowed

406 – Not Acceptable

406 Not Acceptable will be given when the user-agent and client cannot establish acceptable content/encoding types or character-sets.

When resolving a 406 Status Code using a tool like HTTPFox is necessary.

Header values sent by the client to look over include:

• Accept:
• Accept-Charset
• Accept-Encoding
• Accept-Languages

A modern web-browser will never experience this issue. If it does, try upgrading your web-browser to later version.

407 – Proxy Authentication Required

407 Proxy Authentication Required will usually mean the user-agent needs to provide credentials to access a proxy server.

408 – Request Timeout

This status code will usually result from latency between your computer and the web-server.

To assist in resolving 408 Request Timeout Errors:

• try another website to make sure it is not the server infrastructure
• make sure no other downloads are running saturating your bandwidth
• reconnect to your WiFi
• reboot your router
• try an Internet Speed Test

If able to connect to another website fine, it is probably a remote network. If an Internet Speed Test is showing latency the problem could be your ISP or LAN.

Speedtest.net offers a great and free online app for testing Internet connection speeds. The app will check upload speed, download speed, and latency. Speedtest.net Speed Testing App Online

429 – Too Many Requests

429 will usually mean rate limiting has been enabled on the server. Rate limiting will only allow a certain amount of requests in an allotted amount of time.

When recieving 429 – Too Many Requests the user-agent will be blocked for a time period, usually by IP address.

To bypass 429 and 403 restricted errors try using Tor Browser

TOR Browser can be used obfuscate your IP address

451 – Unavailable for Legal Reasons

451 could be seen when a web-page violates copyrights, trademarks, intellectual property laws, or has been deemed a threat to national security.

Sample of a website seized by US Law Enforcement