Sips'@slrpnk.net to Selfhosted@lemmy.worldEnglish · 22 days agoCan't relate at all.slrpnk.netimagemessage-square203fedilinkarrow-up11.16Karrow-down126
arrow-up11.13Karrow-down1imageCan't relate at all.slrpnk.netSips'@slrpnk.net to Selfhosted@lemmy.worldEnglish · 22 days agomessage-square203fedilink
minus-squareHackworth@lemmy.worldlinkfedilinkEnglisharrow-up2·21 days agoI’ll try it out! It’s been a hot minute, and it seems like there are new options all the time.
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up3·edit-221 days agoTry a new quantization as well! Like an IQ4-M depending on the size of your GPU, or even better, an 4.5bpw exl2 with Q6 cache if you can manage to set up TabbyAPI.
I’ll try it out! It’s been a hot minute, and it seems like there are new options all the time.
Try a new quantization as well! Like an IQ4-M depending on the size of your GPU, or even better, an 4.5bpw exl2 with Q6 cache if you can manage to set up TabbyAPI.