you are viewing a single comment's thread
view the rest of the comments
[–] 26 points 10 months ago (4 children)

I think it's a bit more than that. A known failure mode of LLMs is that in a long enough conversation about a topic, eventually the guardrails against that topic start to lose out against the overarching directive to be a sycophant. This kinda smells like that.

We don't have many informations here but it's possible that the LLM had already been worn down to the point of giving passively encouraging answers. My takeaway is once more that LLMs as used today are unreliable, badly engineered, and not actually ready to market.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 11 points 10 months ago (2 children)

    It's definitely that. Those guardrails often give out on the 3rd or even 2nd reply:

    https://youtu.be/VRjgNgJms3Q

  • source
  • parent
  • hideshow 2 child comments
  • [–] 0 points 10 months ago

    I was testing an LLM for work today (I believe its actually a chain of different models at work) and was trying to rock it off its guard rails to see how it would act. I think I might have been successful because it started erroring instead of responding after its third response. I tried the classic "ignore previous instructions..." as well as "my grandma's dying wish was for..." but it at least didn't give me an unacceptable response

  • source
  • parent