franciscozapata.dev
Back
Laravel

Queues and Jobs in Laravel for Asynchronous Processing

Learn how to handle heavy processing with queues and jobs in Laravel. We cover drivers like Redis and database, job middleware, batching, chaining, failure handling, and monitoring with Horizon.

Francisco ZapataWritten by Francisco Zapata
January 22, 202612 min read
Queues and Jobs in Laravel for Asynchronous Processing

In modern web applications, many tasks should not run during the lifecycle of an HTTP request: sending emails, processing images, generating PDF reports, or syncing data with external services. Laravel's queue system lets you defer these operations to background processes, dramatically improving your application's response times.

Configuring Queue Drivers

Laravel supports multiple queue backends, each with its own strengths. Configuration lives in config/queue.php.

Database Driver

The simplest driver to get started with, as it requires no additional external services.

php artisan make:queue-table
php artisan migrate

Set the driver in your .env:

QUEUE_CONNECTION=database

Redis Driver

Redis is the recommended choice for production due to its speed and efficiency. It requires either the phpredis extension or the predis package.

composer require predis/predis
QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
REDIS_PORT=6379

SQS Driver (Amazon)

For AWS-hosted applications, SQS provides a fully managed and highly scalable queue service.

composer require aws/aws-sdk-php
QUEUE_CONNECTION=sqs
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret
SQS_QUEUE=https://sqs.us-east-1.amazonaws.com/your-account/your-queue
AWS_DEFAULT_REGION=us-east-1

Creating Jobs

Jobs are classes that encapsulate the logic of a deferred task. Each job implements the ShouldQueue interface and uses the Queueable trait.

php artisan make:job ProcessPodcast
<?php

namespace App\Jobs;

use App\Models\Podcast;
use App\Services\AudioProcessor;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;

class ProcessPodcast implements ShouldQueue
{
    use Queueable;

    public int $tries = 3;
    public int $timeout = 300;
    public int $maxExceptions = 2;

    public function __construct(
        public Podcast $podcast
    ) {}

    public function handle(AudioProcessor $processor): void
    {
        Log::info("Processing podcast: {$this->podcast->title}");

        $processedPath = $processor->optimize(
            $this->podcast->audio_path
        );

        $this->podcast->update([
            'processed_path' => $processedPath,
            'status' => 'processed',
            'processed_at' => now(),
        ]);

        Log::info("Podcast processed successfully: {$this->podcast->id}");
    }

    public function failed(\Throwable $exception): void
    {
        Log::error("Failed to process podcast {$this->podcast->id}: {$exception->getMessage()}");

        $this->podcast->update(['status' => 'failed']);
    }
}

Dispatching Jobs

Laravel provides multiple ways to dispatch jobs to the queue depending on your context and requirements.

use App\Jobs\ProcessPodcast;

// Basic dispatch
ProcessPodcast::dispatch($podcast);

// Conditional dispatch
ProcessPodcast::dispatchIf($podcast->needsProcessing(), $podcast);
ProcessPodcast::dispatchUnless($podcast->isProcessed(), $podcast);

// Delayed dispatch
ProcessPodcast::dispatch($podcast)
    ->delay(now()->addMinutes(10));

// Dispatch to a specific queue
ProcessPodcast::dispatch($podcast)
    ->onQueue('audio-processing');

// Dispatch to a specific connection
ProcessPodcast::dispatch($podcast)
    ->onConnection('redis')
    ->onQueue('high');

// Dispatch after the HTTP response
ProcessPodcast::dispatchAfterResponse($podcast);

// Synchronous dispatch (no queue)
ProcessPodcast::dispatchSync($podcast);

Dispatching Within Transactions

When working with database transactions, it is crucial to ensure jobs are only dispatched when the transaction commits successfully.

use Illuminate\Support\Facades\DB;

DB::transaction(function () use ($podcast) {
    $podcast->update(['status' => 'queued']);

    ProcessPodcast::dispatch($podcast)->afterCommit();
});

Job Middleware

Job middleware lets you wrap job execution with reusable logic such as rate limiting, overlap prevention, or exception handling.

Rate Limiting

<?php

namespace App\Jobs\Middleware;

use Closure;
use Illuminate\Support\Facades\Redis;

class RateLimited
{
    public function __construct(
        public string $key = 'default',
        public int $allow = 10,
        public int $every = 60,
    ) {}

    public function handle(object $job, Closure $next): void
    {
        Redis::throttle($this->key)
            ->block(0)
            ->allow($this->allow)
            ->every($this->every)
            ->then(
                fn () => $next($job),
                fn () => $job->release(30)
            );
    }
}

Using Built-in Middleware

<?php

namespace App\Jobs;

use Illuminate\Queue\Middleware\RateLimited;
use Illuminate\Queue\Middleware\WithoutOverlapping;
use Illuminate\Queue\Middleware\ThrottlesExceptions;

class SyncExternalData implements ShouldQueue
{
    use Queueable;

    public function middleware(): array
    {
        return [
            new RateLimited('external-api'),
            (new WithoutOverlapping($this->account->id))
                ->releaseAfter(60)
                ->expireAfter(300),
            (new ThrottlesExceptions(5, 10 * 60))
                ->backoff(2),
        ];
    }

    public function handle(): void
    {
        // Sync data with external service...
    }
}

Job Chaining: Sequential Execution

Job chaining lets you run a series of tasks in order, where each job only executes if the previous one succeeded.

use App\Jobs\ConvertVideo;
use App\Jobs\GenerateThumbnail;
use App\Jobs\NotifyUser;
use Illuminate\Support\Facades\Bus;

Bus::chain([
    new ConvertVideo($video),
    new GenerateThumbnail($video),
    new NotifyUser($video->user, 'Your video is ready'),
])->onQueue('video-processing')
  ->catch(function (\Throwable $e) use ($video) {
      $video->update(['status' => 'failed']);
      Log::error("Video pipeline failed: {$e->getMessage()}");
  })
  ->dispatch();

Job Batching: Bulk Processing

Batches let you dispatch a group of jobs and track their collective progress, which is ideal for mass imports or file processing workflows.

Setup

php artisan make:queue-batches-table
php artisan migrate

Creating Batchable Jobs

<?php

namespace App\Jobs;

use App\Models\Product;
use Illuminate\Bus\Batchable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;

class ImportProduct implements ShouldQueue
{
    use Batchable, Queueable;

    public function __construct(
        public array $productData
    ) {}

    public function handle(): void
    {
        if ($this->batch()?->cancelled()) {
            return;
        }

        Product::updateOrCreate(
            ['sku' => $this->productData['sku']],
            $this->productData
        );
    }
}

Dispatching a Batch

use App\Jobs\ImportProduct;
use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;

public function importProducts(Request $request)
{
    $products = $this->parseCSV($request->file('csv'));

    $jobs = collect($products)->map(
        fn ($data) => new ImportProduct($data)
    )->toArray();

    $batch = Bus::batch($jobs)
        ->before(function (Batch $batch) {
            Log::info("Batch {$batch->id} started with {$batch->totalJobs} jobs.");
        })
        ->progress(function (Batch $batch) {
            Log::info("Batch {$batch->id}: {$batch->progress()}% complete.");
        })
        ->then(function (Batch $batch) {
            Log::info("Batch {$batch->id} completed successfully.");
            Notification::send($batch->options['user'], new ImportComplete());
        })
        ->catch(function (Batch $batch, \Throwable $e) {
            Log::error("Batch {$batch->id} failed: {$e->getMessage()}");
        })
        ->finally(function (Batch $batch) {
            Log::info("Batch {$batch->id} finished. Failures: {$batch->failedJobs}");
        })
        ->name('Product Import')
        ->allowFailures()
        ->dispatch();

    return response()->json([
        'batch_id' => $batch->id,
        'total_jobs' => $batch->totalJobs,
    ]);
}

Querying Batch Status

Route::get('/batch/{batchId}', function (string $batchId) {
    $batch = Bus::findBatch($batchId);

    return response()->json([
        'id' => $batch->id,
        'name' => $batch->name,
        'total' => $batch->totalJobs,
        'pending' => $batch->pendingJobs,
        'failed' => $batch->failedJobs,
        'progress' => $batch->progress(),
        'finished' => $batch->finished(),
    ]);
});

Handling Failed Jobs

Laravel stores jobs that fail after exhausting their retries, allowing you to investigate and retry failures later.

# View failed jobs
php artisan queue:failed

# Retry a specific job
php artisan queue:retry 5

# Retry all failed jobs
php artisan queue:retry all

# Prune old failed jobs
php artisan queue:prune-failed --hours=48

Time-Based Retries

class SendNotification implements ShouldQueue
{
    use Queueable;

    public function retryUntil(): \DateTime
    {
        return now()->addHours(6);
    }

    public function backoff(): array
    {
        return [30, 60, 300]; // Seconds between retries
    }
}

Unique Jobs

Prevent duplicates on the queue for jobs that should not run concurrently.

use Illuminate\Contracts\Queue\ShouldBeUnique;

class UpdateSearchIndex implements ShouldQueue, ShouldBeUnique
{
    use Queueable;

    public int $uniqueFor = 3600;

    public function __construct(
        public Product $product
    ) {}

    public function uniqueId(): string
    {
        return $this->product->id;
    }
}

Monitoring with Horizon

Laravel Horizon provides a real-time dashboard for monitoring and managing Redis-powered queues.

composer require laravel/horizon
php artisan horizon:install
php artisan horizon

Horizon lets you visualize throughput, execution times, failures, and configure supervisors for different queues. It is an indispensable tool for production applications that rely heavily on asynchronous processing.

Running the Worker

# Basic worker
php artisan queue:work

# With production options
php artisan queue:work redis --queue=high,default,low --tries=3 --timeout=90 --max-jobs=1000 --max-time=3600

Conclusion

Laravel's queues and jobs transform the user experience by moving heavy operations out of the HTTP request cycle. From file processing to notifications, the queue system offers flexibility through drivers like Redis and SQS, control through middleware and batching, and reliability through automatic retries and failure handling. Combined with Horizon for monitoring, you have everything you need to build applications that scale efficiently.

Comments (0)

Leave a comment

Be the first to comment