PricingBlog

Streaming for OpenRouter

  • giantguf-1228356835367714978

    GiantGuf

    1 year ago

    Hi,
    How can I implement streaming for a chat using OpenRouter?
    Here's the documentation for OpenRouter:
    https://openrouter.ai/docs
  • Tod-1228356838203064331

    Tod

    1 year ago

    Great energy @GiantGuf! Your continuous contribution to the toddle Community just made you advance to Community Level 5!
  • yoelfdz-1228359272082182225

    yoelfdz

    1 year ago

    Hey @GiantGuf yes you can, but you need to implement it through a custom action. I've done it for Flowise
  • giantguf-1228359803957673994

    GiantGuf

    1 year ago

    @ I tried to go through previous conversations but couldn't figure out how to do it. 😅
  • yoelfdz-1228360462249492542

    yoelfdz

    1 year ago

    Depends on the api you are calling, Flowise uses socketio, you'll have to follow their docs
  • giantguf-1228362102188544031

    GiantGuf

    1 year ago

    Yeah, I have read their docs and as I said, I cannot figure it out so I need help. 😛
  • giantguf-1230633044533055599

    GiantGuf

    1 year ago

    Team toddle, do you have any suggestions?
  • erikbeus-1230774508512874527

    Erik Beuschau

    1 year ago

    Hi @GiantGuf
    From what I can tell, their API uses a standard POST request (with no streaming). toddle supports that out of the box, and you would just need to add the relevant headers + pass an object with model + messages for the body of the request.
    However, since toddle's APIs are currently not supporting any server side secrets, your API key would be exposed on the front-end.

    Therefore, I suggest calling openrouter's API from a back-end of your choice (buildship.com for instance) and make sure that your back-end is protected with some kind of authentication.
    1230774508315611237-Screenshot_2024-04-19_at_08.54.18.png
  • giantguf-1230961117917085758

    GiantGuf

    1 year ago

    @Erik Beuschau According to the docs streaming is supported. I have been able to do a standard request like you showed, but it would be much cooler if it could be streamed. 😛

    Ok, thanks for the heads up about the API key!
    1230961117694525541-image.png
  • yoelfdz-1231003140636151818

    yoelfdz

    1 year ago

    Hey @GiantGuf is there a specific reson why you want to use OpenRouter?
  • giantguf-1231517903946383400

    GiantGuf

    1 year ago

    @ Just that it had a lot of LLM options and reasonable prices.
  • yoelfdz-1231560784644145255

    yoelfdz

    1 year ago

    Sent you a dm
  • erikbeus-1231869824037027930

    Erik Beuschau

    1 year ago

    Currently, to support streaming you will need to create a custom action. This is something we aim to address soon though 🤞
  • giantguf-1231875898660093994

    GiantGuf

    1 year ago

    Can't see any dm 🤔
  • yoelfdz-1231878923583750206

    yoelfdz

    1 year ago

    Your account does not accept dms or friend requests @GiantGuf
  • giantguf-1231879534085800028

    GiantGuf

    1 year ago

    Right... 😅 Sent you a friend request.