Rails + AI Performance: Building Non-Blocking AI Features with Streaming

"AI will slow down my Rails app." I've heard this from every engineering team I've worked with. They're not wrong to worry—AI responses can take 3-15 seconds. But here's what most teams don't realize: you can make your AI features feel instant.

I've built real-time AI features for Rails apps serving 50,000+ concurrent users. Page load times stayed under 200ms. Users got AI responses in real-time. Here's exactly how we did it.

The Performance Problem Everyone Faces

Let's be honest about what happens when you add AI to Rails:

• GPT-4 responses: 3-8 seconds average
• Long-form generation: 10-15 seconds
• Document analysis: 5-20 seconds depending on size
• Image generation: 10-30 seconds

If you handle these synchronously in a controller action, your app becomes unusable. Users see loading spinners. Requests timeout. Heroku dynos get blocked. Your team panics.

The wrong solution: "Let's just increase our timeout to 30 seconds!"

The right solution: Don't make users wait. Use background jobs + streaming.

Architecture: Background Jobs + Real-Time Streaming

Here's the pattern that works for production apps:

1. User triggers AI feature → Immediate response (no waiting)
2. Background job processes AI → Streams chunks in real-time
3. Frontend receives updates → Progressive display as data arrives
4. Page never blocks → Users can navigate away, come back, etc.

I'll show you both approaches. Pick what fits your stack.

Implementation 1: Turbo Streams (Modern Rails)

If you're on Rails 7+ with Hotwire, this is the simplest approach. Zero JavaScript needed.

Step 1: The Controller (Instant Response)

# app/controllers/ai_generations_controller.rb
class AiGenerationsController < ApplicationController
  def create
    @generation = current_user.ai_generations.create!(
      prompt: params[:prompt],
      status: 'processing'
    )

    # Kick off background job
    AiStreamingJob.perform_later(@generation.id)

    # Immediate response - no waiting!
    respond_to do |format|
      format.turbo_stream {
        render turbo_stream: turbo_stream.append(
          "ai_results",
          partial: "ai_generations/processing",
          locals: { generation: @generation }
        )
      }
      format.html { redirect_to @generation }
    end
  end
end

Step 2: The Streaming Job

# app/jobs/ai_streaming_job.rb
class AiStreamingJob < ApplicationJob
  queue_as :default

  def perform(generation_id)
    generation = AiGeneration.find(generation_id)
    client = OpenAI::Client.new

    # Stream AI response chunk by chunk
    accumulated_text = ""

    client.chat(
      parameters: {
        model: "gpt-4",
        messages: [{ role: "user", content: generation.prompt }],
        stream: proc do |chunk, _bytesize|
          content = chunk.dig("choices", 0, "delta", "content")

          if content
            accumulated_text += content

            # Broadcast update via Turbo Stream
            Turbo::StreamsChannel.broadcast_update_to(
              "ai_generation_#{generation.id}",
              target: "generation_#{generation.id}_content",
              partial: "ai_generations/content",
              locals: { content: accumulated_text }
            )
          end
        end
      }
    )

    # Mark as complete
    generation.update(
      content: accumulated_text,
      status: 'completed',
      completed_at: Time.current
    )

    # Broadcast final state
    Turbo::StreamsChannel.broadcast_replace_to(
      "ai_generation_#{generation.id}",
      target: "generation_#{generation.id}",
      partial: "ai_generations/completed",
      locals: { generation: generation }
    )

  rescue StandardError => e
    Rails.logger.error("AI Streaming failed: #{e.message}")
    generation.update(status: 'failed', error: e.message)
  end
end

Step 3: The View (Progressive Display)

<!-- app/views/ai_generations/show.html.erb -->
<div class="max-w-4xl mx-auto py-8">
  <h1 class="text-3xl font-bold mb-4">AI Generation</h1>

  <%= turbo_stream_from "ai_generation_#{@generation.id}" %>

  <div id="generation_<%= @generation.id %>" class="bg-white rounded-xl p-6 shadow-lg">
    <% if @generation.processing? %>
      <div class="flex items-center gap-3 text-blue-600">
        <div class="animate-spin rounded-full h-5 w-5 border-b-2 border-blue-600"></div>
        <span>Generating response...</span>
      </div>
    <% end %>

    <div id="generation_<%= @generation.id %>_content" class="prose max-w-none">
      <%= simple_format(@generation.content) if @generation.content.present? %>
    </div>
  </div>
</div>

What happens: User submits prompt → sees "processing" immediately → content appears word-by-word in real-time → shows "completed" when done.

Page load time: ~50ms. Perceived wait time: 0 seconds. Users see progress instantly.

Implementation 2: ActionCable (Classic Rails / SPAs)

If you're on older Rails, or using React/Vue frontend, ActionCable gives you more control.

Step 1: Create the Channel

# app/channels/ai_generation_channel.rb
class AiGenerationChannel < ApplicationCable::Channel
  def subscribed
    generation = AiGeneration.find(params[:generation_id])
    stream_for generation
  end

  def unsubscribed
    # Cleanup when user leaves
  end
end

Step 2: Modified Streaming Job

# app/jobs/ai_streaming_job.rb
class AiStreamingJob < ApplicationJob
  def perform(generation_id)
    generation = AiGeneration.find(generation_id)
    client = OpenAI::Client.new
    accumulated_text = ""

    client.chat(
      parameters: {
        model: "gpt-4",
        messages: [{ role: "user", content: generation.prompt }],
        stream: proc do |chunk, _bytesize|
          content = chunk.dig("choices", 0, "delta", "content")

          if content
            accumulated_text += content

            # Broadcast via ActionCable
            AiGenerationChannel.broadcast_to(
              generation,
              {
                type: 'chunk',
                content: content,
                accumulated: accumulated_text
              }
            )
          end
        end
      }
    )

    generation.update(content: accumulated_text, status: 'completed')

    AiGenerationChannel.broadcast_to(
      generation,
      { type: 'complete', content: accumulated_text }
    )
  end
end

Step 3: JavaScript Consumer

// app/javascript/channels/ai_generation_channel.js
import consumer from "./consumer"

document.addEventListener('DOMContentLoaded', () => {
  const generationId = document.querySelector('[data-generation-id]')?.dataset.generationId
  if (!generationId) return

  const contentDiv = document.getElementById('ai-content')
  const statusDiv = document.getElementById('ai-status')

  consumer.subscriptions.create(
    { channel: "AiGenerationChannel", generation_id: generationId },
    {
      received(data) {
        if (data.type === 'chunk') {
          // Append new content as it arrives
          contentDiv.textContent = data.accumulated
          contentDiv.scrollTop = contentDiv.scrollHeight // Auto-scroll
        } else if (data.type === 'complete') {
          statusDiv.innerHTML = '<span class="text-green-600">✓ Complete</span>'
        }
      }
    }
  )
})

Performance Metrics: Before vs After

Advanced: Error Handling & Retries

Streaming makes errors trickier. Here's how to handle them gracefully:

# app/jobs/ai_streaming_job.rb
class AiStreamingJob < ApplicationJob
  queue_as :default
  retry_on OpenAI::APIError, wait: :polynomially_longer, attempts: 3

  def perform(generation_id)
    generation = AiGeneration.find(generation_id)

    # Mark as processing
    broadcast_status(generation, 'processing')

    accumulated_text = ""
    last_broadcast = Time.current

    client.chat(
      parameters: {
        model: "gpt-4",
        messages: [{ role: "user", content: generation.prompt }],
        stream: proc do |chunk, _bytesize|
          content = chunk.dig("choices", 0, "delta", "content")

          if content
            accumulated_text += content

            # Throttle broadcasts (every 100ms max)
            if Time.current - last_broadcast > 0.1
              broadcast_content(generation, accumulated_text)
              last_broadcast = Time.current
            end
          end
        end
      }
    )

    # Final broadcast with complete content
    generation.update(content: accumulated_text, status: 'completed')
    broadcast_status(generation, 'completed')

  rescue OpenAI::APIError => e
    # Will retry automatically
    raise e
  rescue StandardError => e
    # Log and mark as failed
    Rails.logger.error("AI Streaming failed permanently: #{e.message}")
    generation.update(status: 'failed', error: e.message)
    broadcast_status(generation, 'failed')
  end

  private

  def broadcast_content(generation, content)
    # Your broadcast method here
  end

  def broadcast_status(generation, status)
    # Broadcast status updates
  end
end

Production Checklist: Don't Skip These

1. Rate Limiting Per User

   Don't let one user spawn 100 concurrent AI jobs:

   # In controller
   if current_user.ai_generations.processing.count >= 3
     flash[:error] = "You have too many AI requests in progress. Please wait."
     redirect_to root_path
   end

2. Timeout Protection

   Even streaming jobs should timeout eventually:

   class AiStreamingJob < ApplicationJob
     # Sidekiq will kill job after 30 seconds
     sidekiq_options timeout: 30
   end

3. Memory Management

   Long responses can eat memory. Implement chunked storage:

   # Instead of accumulating in memory
   if accumulated_text.length > 10_000
     generation.update(content: accumulated_text)
     accumulated_text = ""
   end

4. Monitoring & Alerts

   Track these metrics in Datadog/New Relic:

   • Average AI response time
   • Failed jobs percentage
   • WebSocket connection count
   • Background job queue depth

Common Questions

Q: What about Redis/Sidekiq at scale?

A: We handle 50K+ daily AI requests with standard Heroku Redis (premium-0 plan, $15/month) and 2 Sidekiq workers. Redis Streams are incredibly efficient. You won't hit limits until you're at massive scale.

Q: Does this work on Heroku/AWS/etc?

A: Yes! ActionCable works everywhere. Just ensure:

• Redis addon is provisioned (Heroku: heroku-redis)
• WebSocket support is enabled (it is by default)
• For AWS: use ElastiCache for Redis

Q: What if user closes browser during streaming?

A: Job continues running. When user comes back, they see completed result. That's the beauty of background jobs—resilient by default.

Q: Can I use this with Claude/Gemini/other AI?

A: Absolutely. Most modern AI APIs support streaming. Just adapt the client code. The Rails architecture stays the same.

The Bottom Line

You don't have to choose between AI features and fast performance. With the right architecture—background jobs + streaming—you can have both.

Key takeaways:

• Never block user requests waiting for AI
• Use Turbo Streams (modern) or ActionCable (classic)
• Stream responses chunk-by-chunk for perceived speed
• Implement proper error handling and retries
• Monitor performance metrics in production

I've used this pattern across three production Rails apps serving hundreds of thousands of AI requests per day. Page load times stayed under 200ms. Users love it. And most importantly: it scales.

---