Making HTTP Requests

Introduction

Now that you understand the fundamentals of APIs and HTTP, it's time to get your hands dirty with actual code. Python's requests library is the most popular and user-friendly way to make HTTP requests. Think of it as your personal HTTP client - it handles all the complex details of HTTP communication while giving you a clean, intuitive interface.

In this lesson, we'll explore how to make various types of HTTP requests, handle responses, work with different data formats, and manage common scenarios like authentication and error handling.

Getting Started with Requests

The requests library is not part of Python's standard library, so you'll need to install it:

pip install requests

Then import it in your code:

import requests

Basic GET Request

The simplest request is a GET request to retrieve data:

import requests

# Make a simple GET request
response = requests.get('https://httpbin.org/get')

# Check if the request was successful
if response.status_code == 200:
    print("Success!")
    print("Response:", response.text)
else:
    print(f"Error: {response.status_code}")

Understanding the Response Object

Every requests call returns a Response object with useful attributes:

response = requests.get('https://httpbin.org/get')

print("Status Code:", response.status_code)
print("Headers:", response.headers)
print("Content Type:", response.headers.get('content-type'))
print("URL:", response.url)
print("Is Redirect:", response.is_redirect)
print("Elapsed Time:", response.elapsed)

HTTP Methods

GET Requests with Parameters

import requests

# Basic GET request
response = requests.get('https://httpbin.org/get')
print("Basic GET:", response.json())

# GET with URL parameters
params = {
    'key1': 'value1',
    'key2': 'value2'
}
response = requests.get('https://httpbin.org/get', params=params)
print("With params:", response.json()['args'])

POST Requests

# POST with form data
data = {
    'username': 'john_doe',
    'password': 'secret123'
}
response = requests.post('https://httpbin.org/post', data=data)
print("POST data:", response.json()['form'])

# POST with JSON data
json_data = {
    'name': 'John Doe',
    'age': 30,
    'city': 'New York'
}
response = requests.post('https://httpbin.org/post', json=json_data)
print("POST JSON:", response.json()['json'])

PUT and DELETE Requests

# PUT request (update resource)
update_data = {'name': 'Jane Doe', 'age': 31}
response = requests.put('https://httpbin.org/put', json=update_data)
print("PUT response:", response.json()['json'])

# DELETE request (remove resource)
response = requests.delete('https://httpbin.org/delete')
print("DELETE status:", response.status_code)

Working with Headers

Headers provide additional information about your request:

import requests

# Custom headers
headers = {
    'User-Agent': 'MyApp/1.0',
    'Accept': 'application/json',
    'Authorization': 'Bearer your_token_here'
}

response = requests.get('https://httpbin.org/headers', headers=headers)
print("Sent headers:", response.json()['headers'])

Common Headers

Authentication

Basic Authentication

from requests.auth import HTTPBasicAuth

response = requests.get(
    'https://httpbin.org/basic-auth/user/pass',
    auth=HTTPBasicAuth('user', 'pass')
)

print("Auth successful:", response.status_code == 200)

Bearer Token Authentication

headers = {
    'Authorization': 'Bearer your_jwt_token_here'
}

response = requests.get('https://api.example.com/protected', headers=headers)

API Key Authentication

# In headers
headers = {
    'X-API-Key': 'your_api_key_here'
}

# Or as a parameter
params = {
    'api_key': 'your_api_key_here'
}

response = requests.get('https://api.example.com/data', 
                       headers=headers, params=params)

Handling Different Response Types

JSON Responses

response = requests.get('https://jsonplaceholder.typicode.com/users/1')

if response.status_code == 200:
    user_data = response.json()  # Automatically parse JSON
    print("User:", user_data['name'])
    print("Email:", user_data['email'])

Text Responses

response = requests.get('https://httpbin.org/html')

if response.status_code == 200:
    html_content = response.text
    print("HTML length:", len(html_content))
    print("First 200 chars:", html_content[:200])

Binary Responses

response = requests.get('https://httpbin.org/image/png')

if response.status_code == 200:
    image_data = response.content  # Binary content
    
    # Save to file
    with open('image.png', 'wb') as f:
        f.write(image_data)
    
    print("Image saved, size:", len(image_data), "bytes")

Error Handling

Checking Status Codes

response = requests.get('https://httpbin.org/status/404')

if response.status_code == 200:
    print("Success!")
elif response.status_code == 404:
    print("Resource not found")
elif response.status_code >= 500:
    print("Server error")
else:
    print(f"Other error: {response.status_code}")

Using raise_for_status()

try:
    response = requests.get('https://httpbin.org/status/500')
    response.raise_for_status()  # Raises exception for 4xx/5xx status codes
    print("Success!")
except requests.exceptions.HTTPError as e:
    print(f"HTTP Error: {e}")
except requests.exceptions.RequestException as e:
    print(f"Request Error: {e}")

Handling Timeouts

try:
    # Timeout after 5 seconds
    response = requests.get('https://httpbin.org/delay/10', timeout=5)
except requests.exceptions.Timeout:
    print("Request timed out")
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")

Comprehensive Error Handling

def safe_request(url, **kwargs):
    """Make a request with comprehensive error handling."""
    try:
        response = requests.get(url, **kwargs)
        response.raise_for_status()
        return response
    except requests.exceptions.Timeout:
        print("Request timed out")
    except requests.exceptions.ConnectionError:
        print("Connection error")
    except requests.exceptions.HTTPError as e:
        print(f"HTTP error: {e}")
    except requests.exceptions.RequestException as e:
        print(f"Request error: {e}")
    return None

# Usage
response = safe_request('https://httpbin.org/get', timeout=10)
if response:
    print("Data received:", len(response.text), "characters")

Advanced Request Features

Sessions for Connection Reuse

# Create a session to reuse connections
with requests.Session() as session:
    # Set default headers for all requests
    session.headers.update({
        'User-Agent': 'MyApp/1.0',
        'Accept': 'application/json'
    })
    
    # Make multiple requests with the same session
    response1 = session.get('https://httpbin.org/get')
    response2 = session.post('https://httpbin.org/post', json={'key': 'value'})
    
    print("First request:", response1.status_code)
    print("Second request:", response2.status_code)

Customizing Timeouts

# Different timeout values
timeouts = {
    'connect': 5,    # Time to establish connection
    'read': 10       # Time to read response
}

response = requests.get('https://httpbin.org/delay/3', 
                       timeout=(timeouts['connect'], timeouts['read']))

Handling Redirects

# Follow redirects (default behavior)
response = requests.get('https://httpbin.org/redirect/3')
print("Final URL:", response.url)
print("Redirect history:", [r.url for r in response.history])

# Disable redirects
response = requests.get('https://httpbin.org/redirect/1', allow_redirects=False)
print("Status code:", response.status_code)  # 302

Fetching Weather Data Examples

GitHub API

import requests

def get_github_user(username):
    """Get GitHub user information."""
    url = f'https://api.github.com/users/{username}'
    headers = {
        'Accept': 'application/vnd.github.v3+json',
        'User-Agent': 'Python-Requests-Demo'
    }
    
    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        
        user_data = response.json()
        return {
            'name': user_data.get('name'),
            'followers': user_data.get('followers'),
            'repos': user_data.get('public_repos'),
            'location': user_data.get('location')
        }
    except requests.exceptions.RequestException as e:
        print(f"Error fetching user {username}: {e}")
        return None

# Usage
user = get_github_user('octocat')
if user:
    print(f"Name: {user['name']}")
    print(f"Followers: {user['followers']}")
    print(f"Public repos: {user['repos']}")

Weather API

def get_weather(city, api_key):
    """Get current weather for a city."""
    base_url = 'http://api.weatherapi.com/v1/current.json'
    
    params = {
        'key': api_key,
        'q': city,
        'aqi': 'no'  # Disable air quality data
    }
    
    try:
        response = requests.get(base_url, params=params, timeout=10)
        response.raise_for_status()
        
        data = response.json()
        current = data['current']
        location = data['location']
        
        return {
            'city': location['name'],
            'country': location['country'],
            'temperature': current['temp_c'],
            'condition': current['condition']['text'],
            'humidity': current['humidity']
        }
    except requests.exceptions.RequestException as e:
        print(f"Error fetching weather for {city}: {e}")
        return None

# Usage (requires API key)
# weather = get_weather('London', 'your_api_key_here')
# if weather:
#     print(f"Weather in {weather['city']}: {weather['temperature']}°C, {weather['condition']}")

REST API CRUD Operations

class TodoAPI:
    """Simple Todo API client."""
    
    def __init__(self, base_url='https://jsonplaceholder.typicode.com'):
        self.base_url = base_url
        self.session = requests.Session()
        self.session.headers.update({
            'Content-Type': 'application/json',
            'Accept': 'application/json'
        })
    
    def get_todos(self, user_id=None):
        """Get all todos or todos for a specific user."""
        url = f'{self.base_url}/todos'
        params = {'userId': user_id} if user_id else {}
        
        response = self.session.get(url, params=params)
        response.raise_for_status()
        return response.json()
    
    def create_todo(self, title, user_id, completed=False):
        """Create a new todo."""
        url = f'{self.base_url}/todos'
        data = {
            'title': title,
            'userId': user_id,
            'completed': completed
        }
        
        response = self.session.post(url, json=data)
        response.raise_for_status()
        return response.json()
    
    def update_todo(self, todo_id, **updates):
        """Update an existing todo."""
        url = f'{self.base_url}/todos/{todo_id}'
        
        response = self.session.patch(url, json=updates)
        response.raise_for_status()
        return response.json()
    
    def delete_todo(self, todo_id):
        """Delete a todo."""
        url = f'{self.base_url}/todos/{todo_id}'
        
        response = self.session.delete(url)
        response.raise_for_status()
        return response.status_code == 200

# Usage
api = TodoAPI()

# Get todos for user 1
todos = api.get_todos(user_id=1)
print(f"User 1 has {len(todos)} todos")

# Create a new todo
new_todo = api.create_todo("Learn Python requests", user_id=1)
print(f"Created todo: {new_todo['title']}")

# Update the todo
updated = api.update_todo(new_todo['id'], completed=True)
print(f"Updated todo completed: {updated['completed']}")

# Clean up
deleted = api.delete_todo(new_todo['id'])
print(f"Todo deleted: {deleted}")

Best Practices

1. Always Handle Exceptions

# Good
try:
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    data = response.json()
except requests.exceptions.Timeout:
    print("Request timed out")
except requests.exceptions.HTTPError as e:
    print(f"HTTP error: {e}")
except ValueError:
    print("Invalid JSON response")

2. Use Sessions for Multiple Requests

# Good for multiple requests to same host
with requests.Session() as session:
    session.headers.update({'Authorization': 'Bearer token'})
    # Make multiple requests

3. Set Appropriate Timeouts

# Always set timeouts
response = requests.get(url, timeout=30)  # 30 seconds total
# or
response = requests.get(url, timeout=(10, 30))  # 10s connect, 30s read

4. Check Content Type

response = requests.get(url)
if 'application/json' in response.headers.get('content-type', ''):
    data = response.json()
else:
    data = response.text

5. Use Environment Variables for Secrets

import os

api_key = os.getenv('API_KEY')
if not api_key:
    raise ValueError("API_KEY environment variable not set")

response = requests.get(url, headers={'X-API-Key': api_key})

Key Points to Remember

• requests is Python's most popular HTTP library with an intuitive API
• Response objects contain status codes, headers, content, and metadata
• Different HTTP methods (GET, POST, PUT, DELETE) serve different purposes
• Headers provide additional request information and authentication
• Error handling is crucial with raise_for_status() and try/except blocks
• Sessions improve performance for multiple requests to the same host
• Timeouts prevent hanging requests and improve reliability

Now that you can make HTTP requests, we'll dive into working with JSON data and REST APIs. You'll learn how to parse JSON responses, send JSON requests, and work with the most common API patterns you'll encounter in real-world applications.